In a recent post, Uber shared how the development team manage the continuous deployment of the microservices, tackling the challenges of working with large monorepos.
Uber's microservices architecture, consisting of thousands of services, requires a reliable and efficient system for deploying updates, security patches, and new features. To ensure this process is safe and timely, Uber embraced continuous deployment (CD), automating deployments to production. This has been essential for maintaining code quality and minimizing delays in delivering changes. However, as Uber’s codebase and service demands grew, the existing deployment system faced challenges related to scale, standardization, and safety, particularly within large monorepos that housed multiple microservices.
Uber recognized the need for a more streamlined and automated continuous deployment system. In 2022, the company undertook a project to redesign its CD process, aiming to create a more uniform and automated solution for deploying microservices. At the time, Uber was managing around 4,500 microservices across three monorepos, processing 5,600 commits and 7,000 deployments weekly. Yet, only 7% of these services were automatically deployed using CD, with many deployments requiring manual intervention. This decentralized and highly flexible system led to inefficiencies and safety risks, particularly as there were over 100 unique deployment templates with no company-wide standards.
The newly designed system, named Up CD, was designed to improve automation and safety. It is tightly integrated with Uber's internal cloud platform and observability tools, ensuring that deployments follow a standardized and repeatable process by default. The new system prioritized simplicity and transparency, especially when managing monorepos. One key improvement was optimizing deployments by looking at which services were affected by each commit rather than deploying every service with every code change. This reduced unnecessary builds and gave engineers more clarity over the changes impacting their services.
Up introduced a unified commit flow for all services, ensuring that each service progressed through a series of deployment stages, each with its own safety checks. These conditions included time delays, deployment windows, and service alerts, ensuring deployments were triggered only when safe. Each stage operated independently, allowing flexibility in customizing deployment flows while maintaining safety. This new approach reduced manual errors and provided a more structured deployment experience.
A critical part of Up CD’s success was its user-friendly interface, which gave engineers an intuitive view of their service’s commit history and deployment state. The interface collapsed non-essential commits, making tracking changes and identifying potential issues easier. Engineers could also see real-time updates on the status of deployments, allowing them to quickly identify which commits had been deployed and to which environments.-1725634186219.png)
Since launching Up CD, Uber has seen a dramatic increase in the automation of service deployments. The percentage of services deployed automatically rose from under 10% to nearly 70% within a year. Despite the increased frequency of deployments, the rate of production incidents dropped by over 50% per 1,000 code changes, highlighting the system’s ability to safely manage more frequent deployments. However, the increased speed of deploying changes, especially those affecting shared code in monorepos, introduced new risks. Uber addressed these by staggering the deployment of such changes across service tiers, halting deployments if issues arose in the lower-priority services.
Working with monorepos is challenging, also other tech companies developed their own internal tools to improve the development process, like TikTok that releases Sparo a tool to optimize the performances of git operations for large monorepos.
In conclusion, Uber's revamped continuous deployment system has successfully automated and standardized deployments at scale, significantly increasing the safety, efficiency, and transparency of deploying microservices. By integrating tightly with existing tools and prioritizing automation, Up CD has enabled Uber to handle rapid growth while minimizing production incidents.