Facebook has open sourced Yarn, a proxy package manager for JavaScript modules stored on npm or Bower registries.
Facebook has successfully used the npm client for many years, according to a post written by three of their engineers. It went well until their teams and codebases grew to a point where some issues with “consistency, security and performance” started to surface:
Many of our projects at Facebook, like React, depend on code in the npm registry. However, as we scaled internally, we faced problems with consistency when installing dependencies across different machines and users, the amount of time it took to pull dependencies in, and had some security concerns with the way the npm client executes code from some of those dependencies automatically.
Facebook mentioned having problems with npm install when run by a CI tool because their environment is cut off from the internet for security reasons. The immediate solution was to download separately and include all the required modules in the source code of their projects. But updating some of the modules had a major impact:
For example, updating a minor version of babel generated an 800,000-line commit that was difficult to land and triggered lint rules for invalid utf8 byte sequences, windows line endings, non png-crushed images, and more. Merging changes to
node_moduleswould often take engineers an entire day.
The third attempt was to remove the modules from the source code and keep them on an internal CDN. But that meant offering internet connection to development and CI build machines which was not acceptable. Eventually, they decided to build their own package manager called Yarn, one that Facebook considers to be fast, reliable and secure. And a web page was set up to highlight the differences in speed between npm and Yarn. In most cases Yarn outperforms npm by 3 to 30 times, but there are a couple of cases when npm does better. Bugsnag has also compared Yarn against npm and they found that Yarn is 11 times faster with a warm cache.
Yarn comes with a number of features:
- Offline Mode: If you've installed a package before, you can install it again without any internet connection.
- Deterministic: The same dependencies will be installed the same exact way across every machine regardless of install order.
- Network Performance: Yarn efficiently queues up requests and avoids request waterfalls in order to maximize network utilization.
- Network Resilience: A single request failing won't cause an install to fail. Requests are retried upon failure.
- Flat Mode: Resolve mismatching versions of dependencies to a single version to avoid creating duplicates.
Another notable feature is that Yarn works with both npm and Bower registries.
npm, Inc., the company running the npm registry, welcomed Yarn as yet another addition to the existing Node.js managers, noting that although Yarn pulls packages from registry.yarnpkg.com, this repository is a proxy of the official npm registry. Without specifically mentioned by Facebook, Yarn solves another purpose: having a secure backup of all Node modules in case the npm registry goes down, as it happened last spring when npm was broken for 2.5 hours affecting thousands of developers worldwide with failed build. If Yarn is not really needed unless one has the scale and development needs of Facebook, getting the packages from a proxy provides a layer of resiliency in case the original registry goes down.
Facebook made known that Yarn was the result of collaboration with Exponent, Google, and Tilde. The code has been made available on GitHub under a BSD license.
