GitHub needed to fundamentally rethink how they did software development due to all of the different cultures, norms, and technology stacks that their teams brought to the table. They are migrating toward a microservices architecture that enables different teams and systems and technologies to work harmoniously together.
Sha Ma, former VP of software engineering at GitHub, spoke about how GitHub is introducing a microservices architecture to evolve their current Ruby on Rails monolith to accommodate future scaling at QCon Plus 2020.
GitHub started their implementation of the microservices architecture with data separation - grouping existing data into functional groups, and breaking up existing queries that crossed functional domains. Next, they identified core services as well as gravitational pulls of the monolith that should be extracted out first, to serve as the foundation for other services to build upon.
Ma mentioned that in parallel, they examined how they should change things operationally to support numerous, small, independently running services to work harmoniously together. They also made adopting microservices simpler by creating reusable workflow templates that everyone can use without having to reinvent the wheel every single time.
Shifting from monolith to microservices is more than just an architectural decision, Ma mentioned. She stated that it literally impacts every single developer in the organization, and fundamentally changes how they work and how they communicate (Conway’s Law):
Teams have to pay more attention to contracts, boundaries and communicating asynchronously. Teams also have more decisions to make as they gain more autonomy around tech stacks.
"It’s a long and arduous journey to transition from a large monolith towards a more distributed microservices architecture", Ma said. She mentioned that it’s not going to be an overnight shift, and it will most likely be a lot larger in scope and take a lot longer than people originally anticipated. "It will require a cultural shift; it will require training on best practices; it will require a change to how you work", she said.
InfoQ interviewed Sha Ma about why GitHub started their migration towards microservices, what advantages the monolith was bringing and how they intended to keep them, the challenges they faced, how the migration impacted the organization, and what they have learned.
InfoQ: What made you decide to start the migration from a monolith to microservices at Github?
Sha Ma: At GitHub, the primary reason for this shift in thinking is because of how we’ve grown in the last 18 months. We went from about 400 people to 1000 people in the engineering organization.
We also rapidly introduced a wide range of tech stacks through acquisitions such as Semmel, NPM, Dependabot and Pull Panda. In order to enable everyone to continue working productively without having to spend a huge amount of time learning Ruby and ramping up on the monolith, we decided that we needed to start doing things differently, and create pathways to enable development at scale outside our current Rails App.
InfoQ: What were the advantages that the monolith solution was bringing you? How did you intend to keep them?
Sha Ma: The monolith created a lot of simplicities for us; simplicities in getting up and running faster, simplicities in how we write code, and simplicities through shared knowledge. We have also built a lot of shared tooling and accumulated a lot of deep knowledge in scaling certain systems (e.g., Feature Flags, Database, CI/CD workflow).
For example, with high parallelism and pre-processing, we can already get the entire monolith to build in under 10 minutes. Similarly, we already support canary deploys in our deployment process.
We want to leverage this knowledge that has been built up over time as much as possible.
InfoQ: What challenges did you face and how did you deal with them?
Sha Ma: At GitHub, we started with first answering the "Why" and clearly articulating our goals for this shift. For example, for us it was less about the replacement of the existing monolith than the enablement for half of our development team to work outside of the monolith. We established some ground rules like the need to still maintain and improve the current codebase, as well as having a plan to migrate 100% of traffic to new services, so we could shut down old code paths that are no longer in use.
We also started with some global architectural decisions around data separation, extraction of core services, and shared workflow, etc. to make it easier for individual teams to get started down the microservices journey.
InfoQ: How has the migration to microservices impacted the organization at Github?
Sha Ma: Organizationally, this made us think more intentionally about teams and ownership areas. One of the ways it manifested itself is in how we are rethinking on-call responsibilities.
Previously, all dotcom developers (which is almost the entire engineering organization) rotated on-call responsibilities for the entire monolith. This created several problems as the engineering organization got bigger and bigger. People weren’t on-call frequently enough to really build that muscle, and it was difficult to pinpoint problems once you were in a production incident because of how large and complicated the codebase has gotten.
So, a small group of passionate engineers combed through our existing service catalogue and worked with the leadership team to assign out explicit ownership to each section of the code. Even when no active development was happening in a certain area, we made sure that the code was owned and traceable to a single team.
InfoQ: What have you learned?
Sha Ma: It’s important to put together a plan, so that new feature development isn’t paused for an extended amount of time in order to make this transition. Be intentional about the "How" and start the prep work (within the monolith) well ahead of actually creating your first microservice. Make this an iterative process, and re-evaluate initial assumptions over the course of this transition.