Google AI has recently published results on ML-enhanced code compilation and how it improved developers’ productivity considering different metrics.
Code compilation is an essential tool in any integrated development environment (IDE) to assist developers in writing codes. These systems are mainly rule-based or semantic-based in which it has access to the full repository and structure of the programming languages. The rapid advances in natural language processing (NLP) opened a new direction to use deep learning models in providing smarter suggestions for developers. As one of the recent examples, we can call Copilot a new AI service for code compilation by GitHub.
This new architecture has leveraged both a system transformer and a semantic engine as a hybrid solution. For the first part, a transformer is trained based on the code repository. As they explained in the blog post:
We train a single model on eight languages (C++, Java, Python, Go, Typescript, Proto, Kotlin, and Dart) and observe improved or equal performance across all languages, removing the need for dedicated models. Moreover, we find that a model size of ~0.5B parameters gives a good tradeoff for high prediction accuracy with low latency and resource cost. The model strongly benefits from the quality of the monorepo, which is enforced by guidelines and reviews. For multi-line suggestions, we iteratively apply the single-line model with learned thresholds for deciding whether to start predicting completions for the following line.
This architecture is shown in the below diagram:
Transformer models are used to predict the remainder of the line or lines of code
Semantic engine checks if the suggested phrase is feasible considering the language grammar and rules. Semantic checks could be reference resolution, method invocation correctness, and type check to name a few. Abstract syntax trees are used to enable a full structural understanding of the written codes.
The following architectural diagram shows the final solution.
Final ML-Enhanced code compilation architecture
This system is used by 10k+ Google developers. It showed an acceptance rate of 25-34%, with more than 3% code completion by this system. It improved iteration time for Googlers by 6%.
One of the Google developers tweeted about this tool:
don't know what any of this means but the completions have been really helpful
Alongside this news, GitHub recently published the results of Copilot in a research paper for more visibility around the Copilot performance for developers.
