Google has announced that they are retiring support for their Octane JavaScript benchmark, citing lack of real-world benefits; there are also signs that a big change is in the works for Chrome.
In a blog post, Google says that the benchmark was useful early in its life, but that it has become outdated since the benchmark doesn't reflect how web pages are built today.
Octane helped engine developers deliver optimizations that allowed computationally-heavy applications to reach speeds that made JavaScript a viable alternative to C++ or Java.
By 2015, however, most JavaScript implementations had implemented the compiler optimizations needed to achieve high scores on Octane. In addition, we began to notice that JavaScript optimizations which eked out higher Octane scores often had a detrimental effect on real-world scenarios.
The cynic's view of this news is that they were getting beat by competitors on their own benchmark. On a marketing page, Microsoft claims Edge is faster than Chrome on the Octane 2.0 score, explicitly saying "Google's own performance benchmark shows that Microsoft Edge is faster than Chrome and Firefox."
For their part, however, Microsoft seems to agree that the synthetic benchmark doesn't reflect the real world:
We are still frequently asked about JavaScript benchmark performance, and while it doesn’t always correspond directly to real-world performance, it can be useful at a high level and to illustrate improvement over time.
Brian Terlson, program manager for Microsoft's Chakra JavaScript engine, wrote in a comment on Hacker News that "Octane was good as far as benchmarks go but you have to balance with a much larger set of benchmarks and always ground your work in real-world scenarios (something we focused heavily on from day one, e.g. when we realized having an interpreter really helps startups on lots of real-world apps but doesn't do much for Octane score)".
It's clear that the browser vendors feel that they've tackled most of the big items on the JavaScript performance checklist. With features such as WebAssembly, performance improvements may come to us differently than they have for the past few years. What's more important now is improving what the user is actually experiencing, so a new class of benchmarking tools is required.
Multiple signs point to Speedometer as a good choice for now. Jeff Atwood has endorsed it, and a blog post from the Chromium team said the same thing. Speedometer uses the popular TodoMVC sample to exercise the browser and measure its performance. Browser Bench also has a browser graphics test (Motion Mark) and a JavaScript benchmark (JetStream).
The Chromium post also says that they plan on improving JavaScript performance based on modern JavaScript patterns and teased that something big is in the works:
Stay tuned for updates about our new engine architecture, designed for real-world performance.
Octane is still available on GitHub and developers can continue using it.
