In a few years, the number of microservices has grown from 100 to 1600 at Monzo, and it continues to grow. Microservices are split when responsibilities grow, or merged when services are tightly coupled. Engineers can generate, deploy, scale and monitor their own services using code generation.
Matt Heath and Suhail Patel, engineers at Monzo, spoke about how Monzo builds, operates, observes and maintains their banking infrastructure at QCon London 2020.
Patel explained how engineers seek feedback when developing a new microservice:
Our company culture is very feedback-oriented. Engineers promote written proposals outlining problems we are trying to solve along with proposed technical solutions. These proposals are visible to the entire company and are an opportunity to provide feedback on a proposed set of interfaces and a chance to appreciate the broader context. This has a bonus of serving as excellent documentation for future engineers who join the company.
The biggest benefit Monzo has gained from microservices is organisational flexibility, Patel explained:
We build services which are granular enough to be easily understood. Ownership of services is well-defined but can be fluid based on the goals of the company. Monzo has grown significantly over the past 24 months; the structure of existing teams and emergence of new teams has changed significantly as a result.
InfoQ interviewed Suhail Patel, backend engineer at Monzo, about why Monzo uses microservices, how microservices look, managing a system with many microservices, and the benefits that microservices have brought.
InfoQ: What made Monzo decide to go for microservices?
Suhail Patel: Our mission continues to build the best current and business bank accounts in the world. From the very beginning, our technology choices needed to facilitate that. We chose microservices right from the very start after prior experience scaling large systems elsewhere.
When the Monzo Beta was launched, there were nearly 100 microservices supporting the pre-paid account. We got our banking license in 2017 and built a full current account offering. Since then, we’ve built many user facing features and integrated with many different payment networks (a lot of these integrations are built in-house). The number of services has grown to over 1,600 to provide this rich feature set.
InfoQ: How do microservices look at Monzo?
Patel: Microservices have a thin layer of business logic backed by a set of core libraries and abstractions which each service shares. Ninety-nine percent of our services are written in Golang and run on our core Kubernetes Platform.
Our tooling leans heavily on standardization. Our goal is to minimise how different services are engineered, relying on a set of battle-hardened top level libraries. We have tooling for generating everything an engineer will need to write a service and doing static analysis to check for potential sources of bugs at compile time. By hooking into shared libraries, our Platform Teams can make improvements or migrate an underlying implementation without changing a single line of service specific code in many cases.
InfoQ: How do you ensure that microservices have a single responsibility and have well-defined interfaces?
Patel: As a platform, we provide guidance and tooling to make it seamless for microservice building blocks to be attached together. Ultimately, the decision on interface boundaries is made within the team working on the implementation and maintaining the health of the service.
Our product is changing all the time; we don’t get the interfaces correct 100% of the time. It’s common to propose a split of a microservice which is growing in its responsibilities or to propose a merging of services that are tightly coupled and co-dependent. Making these changes can be done with little technical friction.
InfoQ: How do you manage a system that consists of many microservices?
Patel: On the whole, the goal is to make engineers autonomous as much as possible for organising their domain into the structure of the microservices they write and support. As a Platform Team, we provide knowledge and documentation and tooling to support that.
Each microservice has an associated owning team and they are responsible for the health of their services. When a service moves owners, other responsibilities like alerts and code review also move over automatically.
Our platform leans heavily on self service. Engineers can generate, deploy, scale and monitor their services without needing to interface with the team that runs the Platform. We do this by sitting on top of tried and tested infrastructure like Kubernetes and Prometheus which we operate with a sprinkle of our own tooling.
InfoQ: How do you use code generation, and what benefits does that bring?
Patel: Code generation starts from the very beginning of a service. An engineer will use a generator to create the skeleton structure of their service. This will generate all the required folder structure as well as write boilerplate code so things like the RPC server are well configured and have appropriate metrics.
Engineers can then define aspects like their RPC interface and use a code generator to generate implementation stubs of their RPC calls. Small reductions in cognitive overhead for engineers allows them to cumulatively focus on business choices and reduces the paradox of choice.
We do find cases where engineers need to deviate. That’s absolutely okay; our goal is not to prescribe this structure for every single service. We allow engineers to make the choice, with the knowledge that deviations need appropriate documentation/justification and knowledge transfer.
InfoQ: What do you standardize across teams, and how do you do it?
Patel: No team is forced to use any particular tooling, but we provide a very compelling, batteries-included service structure and associated libraries and tooling. What this translates to is a set of top level libraries used across all services. These libraries encapsulate the RPC transport layer, interaction with queuing systems, rich metrics, distributed locking, centralized logging, and much more.
Documentation also plays a large part in encouraging the use of standard tooling. If you are developing a new system, being able to leverage design patterns and code which power our most important services provides more incentive to use proven patterns and tooling. We’ve spent a lot of effort in explicit documentation such as knowledgebase tutorials and an internal Stack Overflow wiki.
InfoQ: What benefits are microservices bringing you and what have you learned?
Patel: The large majority of services across Monzo use the same code structure and tooling. This reduces cognitive overhead on making changes. This also has the benefit of increasing mobility, as engineers don’t need to learn a completely different stack when joining a new team internally. Engineers within teams can freely propose changes and are fully in control of development, deployment, and scaling of their services.
There are situations where services need to deviate from the paved road. A particular example is some of our machine learning services. These make heavy use of Python libraries for machine learning and often the cost of re-implementing them in Go is not justifiable. We can still define appropriate interface boundaries, use the same RPC layer and leverage the same benefits of the platform, without having to re-implement large parts of the platform stack.
Breaking down the complexity into small bite-size chunks means that each service is simpler and easy to understand. Iterative changes allow us to increase our velocity whilst reducing risk.