Apple has open sourced Swift’s benchmarking suite, a key piece in tracking Swift performance and catching performance regressions when adding new features to the language.
Swift’s benchmarking suite is a collection of Swift source files that implement test suites and benchmarking helper functions, plus a number of Python scripts that implement a test harness and facilities for metrics comparison.
Apple suggests running the benchmarking suite before committing any changes that may impact performance, and before submitting a pull request, which is done by passing the --benchmark flag to build-script, Swift’s build automation script is in charge of basic tasks such as building a Swift compatible LLDB, building for a given platform, running tests after build etc. This should help catch potential performance regression. In order to make sense of the benchmark results, it can be useful to first take a baseline benchmark on Swift’s master branch, then re-run the benchmark on the development branch, whose results will be automatically compared to the previous run’s.
By default, only Swift benchmarks for OS X are compiled, but additional platforms can be specified on the command line like this:
$ swift/utils/build-script --ios --watchos --tvos
When using the benchmark driver, developers can control the number of loop iterations in each test sample and the number of samples to take for each test, e.g.:
$ ./Benchmark_O --num-iters=1 --num-samples=1
As Apple engineer Luke Larson wrote, at the moment, Swift’s benchmarking suite includes 75 benchmarks, but Apple is encouraging contributions for new benchmarks, to helper libraries, and for general improvements. An important requirement for contributions, Larson says, is that they be covered under the same license as Swift. So ports of benchmarks covered under other licenses cannot be accepted.
Swift’s benchmarking suite is not currently integrated with the recently announced Swift continuous integration system, which builds and runs tests for OS X, iOS, and x86_64 Ubuntu Linux. Based on Jenkins, Swift CI system aims to make it feasible to support more platforms and architectures coming from the Swift development community. Swift CI system enables the execution of tests within a pull request, i.e., before committing to master, and notifies failures to the responsible developer. According to Larson, though, Swift’s benchmarking suite will be integrated within Swift CI system in the future.