Monzo products are supported by an extensive microservice-based platform of over 2800 services. The company relies on planning and heavy automation to drive code migrations at scale and leverages config service to support gradual roll forwards and quick rollbacks in case of issues. Migrations are managed by a central team rather than service owner teams to avoid delays and inconsistencies.
Monzo’s vast portfolio of microservices poses unique challenges when rolling out sweeping changes, such as updating library versions for consistency and freshness. The company opted to centralize code migrations in most cases to speed up migrations and improve consistency across services.
The team responsible for running migrations laid down principles for applying changes at scale, including making migrations transparent to service owners, avoiding downtime, gradual rollouts to reduce blast radius, and applying the 80/20 rule for automation to avoid diminishing returns for tackling unusual use cases. The company heavily standardized its development stack, choosing Go as the programming language and adopting monorepo for all its source code.
Using a Library Wrapper to Switch Between Libraries (Source: Monzo Engineering Blog)
Will Sewell, platform engineer at Monzo, describes a typical process for rolling out a new library, using the migration from OpenTracing to OpenTelemetry as an example. For any extensive change, the migration team would present a proposal in Slack or during the architecture review meeting to raise awareness and solicit feedback from the engineering organization before starting work.
The team uses a combination of automatic code updates using language-specific code replacement tools and manual updates for any unusual use cases. To switch between external libraries, the team ensures the wrapper is deployed to all services first. Then, it uses a configuration service to switch implementations gradually for a particular set of users or a percentage of requests.