Google has solved the concurrency limitations introduced by the Global Interpreter Lock existent in CPython by creating a new Python runtime in Go.
Google uses Python for YouTube’s front-end and APIs running on CPython 2.7, the reference implementation of the language’s interpreter. The code has grown to millions of LoC over the years, and generally served them well after some performance tweaks of the runtime. But they have come against one problem that kept showing up: “it's very difficult to make concurrent workloads perform well on CPython,” according to Google.
After investigating a number of other possible solutions, Google has decided to try and run Python code on Go’s runtime due to its strong support for concurrency. The Grumpy project was born. With Grumpy one can trans-compile Python code into Go source code which is then compiled into native code based on the target platform. The intermediary Go code is a series of operations applied to data structures representing Python objects.
Because it uses compilation, Grumpy takes some of the development freedom and power of an interpreted language. But Google claims these losses are compensated for by better optimization through static analysis and the ability to use Go packages via simple imports.
Another effect of static compilation is that Grumpy does not support and will “probably never support” some dynamic features available in CPython: exec, eval and compile. Supporting them would require “bundling Grumpy programs with the compilation toolchain which would be unwieldy and impractically slow.” While theoretically possible, the Grumpy team said they also do not plan to support C extension modules because “Grumpy has a different API and object layout than CPython”.
Google attempts to make Grumpy a replacement for CPython 2.7. Some would like to see Grumpy supporting Python 3.x in the future. Answering to a question on supporting Python 3, Dylan Trotter, Staff Software Engineer at YouTube, said “We have a large Python 2.7 codebase so that's what we've been focused on. I definitely would like to support Python 3.” He also said in another thread, “There are no concrete plans for Python 3 in the near future but long term I definitely want to support it.”
Google has open sourced Grumpy on GitHub under Apache 2.0, inviting developers to comment and contribute. A forum was also created for this purpose.
