An Interesting Swift Benchmark

Problem

I recently wrote about creating extensions on Optional<String>. When I shared this post in the Swift India slack group, one of the people there pointed out to me that I could just use the nil coalescing operator instead of the extension methods I’d created.

So:

fooLabel.text = bar.foo ?? ""

instead of:

fooLabel.text = bar.foo.orEmpty()

I had a real doh! moment for not thinking of this myself. I was going to update my original post to say “well actually, nevermind - just use the nil coalescing operator”. Then however, I got some more feedback, which is why I let the post remain.

One person told me that they didn’t even know Swift had a nil coalescing operator, and that orEmpty was obvious to them where ?? was not. Hmm, interesting. The fact that I knew the nil coalescing operator existed, but didn’t think to use it made me think that perhaps orEmpty was cleaner. FWIW, I’m still not sure what the right tradeoff to make is in this regard - use a language feature that may not be super obvious, or use something custom that probably won’t exist in other projects?

The more interesting piece of feedback I got though, was that the nil coalescing operator was slow in terms of build time. I believe it was based off of this post. While that sort of made some sense intuitively, I figured I’d write a quick benchmark to see if I could prove/disprove that theory.

The general strategy was to generate Swift source files that used each of these ways of doing the same thing and measure how long it took to compile.

Experiment

Here’s the quick and dirty python script I wrote to do this:

#!/usr/bin/python

import os
import time

common = '''
extension Optional {
    public func or(other: Wrapped) -> Wrapped {
        if let ret = self {
            return ret
        } else {
            return other
        }
    }
}
extension Optional where Wrapped == String {
    public func orEmpty() -> String {
        return self.or(other: "")
    }
}

struct Foo {
    let foo: String?
}
let foo = Foo.init(foo: "foo")
'''

N = [1, 100, 500, 1000, 2500, 5000, 10000]
M = 10


def benchmark(name, template):
    for n in N:
        content = '' + common
        for i in range(n):
            content += template.format(i)
        fn = os.path.expanduser('~/Desktop/foo.swift')
        with open(fn, 'w') as fout:
            fout.write(content)
        results = []
        for i in range(M):
            st = time.time()
            os.system('swiftc -o ' + os.path.expanduser('~/Desktop/foo') + ' ' + fn)
            et = time.time()
            results.append(et-st)
        write_csv([str(n), name] + [str(r) for r in results])


def write_csv(line):
    with open(os.path.expanduser('~/Desktop/results.csv'), 'a') as fout:
        fout.write(', '.join(line))
        fout.write('\n')

try:
    os.remove(os.path.expanduser('~/Desktop/results.csv'))
except FileNotFoundError:
    pass

benchmark('nil', '''
let a{} = foo.foo ?? ""
''')
benchmark('nil with type specified', '''
let a{}: String = foo.foo ?? ""
''')
benchmark('empty', '''
let a{} = foo.foo.orEmpty()
''')
benchmark('empty with type specified', '''
let a{}: String = foo.foo.orEmpty()
''')
benchmark('other', '''
let a{} = foo.foo.or(other: "")
''')
benchmark('other with type specified', '''
let a{}: String = foo.foo.or(other: "")
''')

Results

I ran it on my 15 inch 2017 Macbook Pro with a 2.8 GHz Intel Core i7 processor plugged into power, and fully charged (be warned: this takes a few hours to run - I let it run overnight). I used the swift compiler that ships with Xcode 8.3.3. I took the median of the 10 samples, and graphed them:

So, it turns out I was totally wrong - the nil coalescing operator is way faster than my ways.

However, I noticed something weird that the graph above hides:

You may want to open those images up in a new tab - they’re much higher res than is visible, but I was too lazy to change the markup of my page to actually use the resolution.

If you remove the 2500 and 5000 values, you’ll notice that orEmpty is faster than nil-coalescing.

Weird!

Nil coalescing is faster, but only if the number of times the compiler has to do it is larger.

I haven’t been able to figure out why this is so far, but I’ll keep digging. If you know why, or have any theories, please let me know: @gopalkri!

Conclusion

This benchmark is highly synthetic. First, it only uses one source file (hopefully your apps are not all in one source file). Second, I highly doubt that most code bases are going to have more than a few hundred of these occurrences. At that volume, and at these absolute times (0.6 - 1 second) per 1,000 occurrences, I don’t think performance is a reason to use one or the other. The question goes back to one of style, and I’m not really sure which way to go. At this moment in time, I’d probably go with the nil coalescing operator.