In a recent post, Spotify engineer Patrick Balestra explained how they managed to improve Xcode build time by 50% using dependency manager Carthage to avoid rebuilding dependencies over and over across their teams.
When using Xcode, you have different options to handle external dependencies. You can do it manually or you can use a dependency manager such as CocoaPods, Carthage, or the Swift Package Manager. With the exception of Carthage, Xcode dependency managers usually rely on building all external dependencies from source, while Carthage supports downloading an already built binary from GitHub or getting it from a known local location.
Building dependencies from source has huge implications on productivity, says Balestra, since all developers will have to build external dependencies on their machines as well as each time they clean your project artifacts, e.g., when starting a new continuous integration build. From the analysis of their build logs, Spotify engineers came to a striking conclusion.
Comparing the time spent building the Swift source code of the Spotify for Artists app with its dependencies in our continuous integration environment, we quickly realized that around 60-70% of the time was spent compiling the same dependencies over and over again before we could actually build our own code.
This state of things could likely change now that Swift has got support for framework binary stability and the Swift Package Manager could at some point introduce support for binary distribution of frameworks too. But for the time being, the only available option to manage binary dependencies in Xcode is either manually or through Carthage.
The advantage of linking pre-built binary modules is even greater if they are shared across the team and the CI pipeline, which requires using some kind of distributed storage. In such case, team members and the CI pipeline can directly link the shared binary modules. Since Carthage only supports local storage of binary modules, developers usually use a third-party tool like Rome to cache Carthage artifacts to an S3 location where they are available to the whole team.
Rome speeds up future builds by removing the need to rebuild the same dependencies. This is achieved by caching the contents of the Carthage/Build directory to an external local folder (e.g: ~/Library/Caches) or a remote server (e.g: Amazon S3).
Since at Spotify engineers do not use S3 and rely on Artifactory, they had to extend Rome so it supported additional remote storage systems. Thanks to their work, you can now use a script written in your preferred scripting language to enable Rome to store remotely your Carthage artifacts wherever you want. This brought a significant improvement.
After merging support for Rome, it didn’t take long to notice an improvement. We saw an immediate reduction of around 50% in the time taken in our CI configurations that build the Spotify for Artists app: from around 20 minutes to under 10 minutes!
One major issue Spotify engineers discovered is the inability of LLDB to debug into cached binary modules, since the compiler stores absolute paths for the symbols found in the binary. This behaviour should have changed in the latest Xcode version supporting Swift 5.1 and binary module distribution, but InfoQ has been unable to confirm with Spotify which Xcode version they were using. We will update this post as soon as we get an update from Spotify.