Transcript
Shibata: Who's squarely working with monoliths? Keep your hands up if you consider these monoliths as your legacy services.
Stewart: Who's migrated from monoliths to microservices? A few. Who's using Kubernetes for those, in production?
Shibata: Who's moving back to monolith?
Stewart: The people who started with monoliths and don't consider them legacy, that was a choice? Anyone want to raise their hand and say I chose to do a monolith? A couple.
Shibata: This talk is all about, where are we now? Microservices became mainstream. We are moving more towards a centralized architecture. Some of us are considering moving to Kubernetes already in there, like we've seen earlier. We as an industry have been talking about storage growth. We learned some lessons along the way. We're going to talk more about the challenges we faced transitioning in comparethemarket and how we solved those challenges.
Stewart: We're going to take you on a journey through several years of evolution at comparethemarket, discussing why we built our own platforms, and how microservice has facilitated growth at comparethemarket. Importantly, how we save our customer's money.
Background
Shibata: This is Adam Stewart. He's our Principal Solutions Architect. He's one of the key persons involved in building Comparison as a Service.
Stewart: This is Kenichi. Kenichi is a Cloud Solution Architect. He's responsible for our Kubernetes platform and all the great things around that.
Who Are Comparethemarket?
Kenichi, who are comparethemarket?
Shibata: Comparethemarket allows customers to compare prices on a number of insurance products, including car, home, van, pet, life, travel, and over 50s insurance. It has also expanded into comparison of items that can be switched, such as energy, utilities, broadband, and digital TV, as well as a large range of financial products such as loans, credit cards, and ISAs. Recently, I've been trying to get some mortgage on it. We are a price comparison website. We're aggregating quotes for all sorts of products from insurance to finance.
Stewart: Comparethemarket started life in 2006, just as the comparison and aggregation market was really starting to take off. During that time, I think everybody just felt comparison was the same. There were other competitors. We all looked the same. We were delivering the same message. It was all about saving money on car insurance. Really struggled to differentiate in the marketplace from those competitors. Then, in 2009, all of that changed, really, with the launch of the Meerkats. We've continued to grow consistently ever since then, really.
How many people in the room have used comparethemarket for comparison services? Quite a few of you, so you know who we are. For the rest of you, why not?
Takeaways
What do we want you to take away from our talk? Firstly, evolution over revolution? What do we mean by that? It's about continuing to deliver value to customers whilst we advance the architecture. Be brave. Things can and do go wrong. You need to be prepared for those things going wrong. We shouldn't be afraid to make those changes. It takes bravery. Use all the information you have.
Shibata: Lastly, platforms can help scale microservices. Having the ability to reuse multiple components while keeping them backwards compatible. Also, collaborating with each other is key. We will touch a little bit about healthy competition. How that drives ideas forward, while having consistent approach by doing collaboration.
Monolith First
Of course, we're going to start this journey with monoliths.
Stewart: Like most companies, we started with a monolith. We had a centralized IT department. We had specialized teams. We had silos of QAs, and Dev resource, and service delivery who took all these nice features. Had no understanding of how they were built, and how to support them, and bug fix them. It was all on-premise data centers. It got us to market. It delivered value. It saved our customers' money. I think that's one of the messages that I'd like you to take away as well is, there's a lot of technology out there. There's a lot of things that you can use. The most important thing is delivering value to our customers and making sure that we do that as quickly as possible, obviously, as safely as possible. We're done. We've delivered this. We're delivering value to customers. It's all working. What else do we need to do?
1st Evolutionary Branch
Shibata: It worked, until?
Stewart: Comparethemarket started growing. We hit 2009. The Meerkats came in and we saw a massive level of growth, probably not quite the same as some of the other people you've heard. It was significant. It's continued on that trajectory. It became apparent, really, the infrequent releases. We were releasing infrequently. We had a single monolithic application. We're trying to serve all of these different products, whether it's home, car, life out of this monolithic application. It just wasn't sustainable.
Shibata: The problem with the infrequent releases. We started, really, with our evolution of the systems by working with people first. Because we identified that monolithic application isn't really built in isolation. You build it with people, and collaborating with people. By working with people first, by bringing business closer to tech, and forming product teams, we're trying to solve some of the problems. Once you have these teams, what is the aim? If you have multiple product teams, what are the aims of those product teams?
Stewart: I think it is quite a common practice now. How many people work in product teams? Lots of people. I think the days of the siloed departments are mostly behind us. The trend is towards product teams and colocation, although moving more into remote based working as well. I think it's important just to reflect a little bit on why we did them and why that worked for us at the time. It's really about that rate of change. It's about the innovation. We're releasing new products to market. We're working through changes to those products. It was quite a new experience doing comparison. There was a lot of change in those early days. It was about self-service. It was about delivering stability. It was about being close to the product owners. All the things that the Agile manifesto tells you are good things, and as you go through all of that. I think it's important to reflect on that. However, some products, we're still sharing the original monolith. As we evolved, we had a core platform that was the monolith. We started building UI monoliths off the top of that. We were basically building frontends in a monolithic style on top of a monolith.
Shibata: Monoliths after monoliths. What did that look like for us? It's Stonehenge, we're in England, so monoliths. We have product teams building. We are building, running, and owning vertical slices representing each product. Then we were able to quickly add new products by just spinning up new teams, which is really good because you get a new team you get a new product on the market. At least that's what they say. This worked for comparethemarket. We were able to save even more money for the customers, which is the ultimate goal of comparethemarket as a company, to save customers some money. Product teams had freedom to innovate and make changes as they see fit. We now have even more monoliths. A lot more monoliths.
Stewart: That's ok though. We're meeting the customer's expectations.
Shibata: Not really. We're saving them money. We've struggled to deliver more features across all products. This is the key reason why our architecture started assembling Stonehenge, essentially.
2nd Evolutionary Branch
We started thinking about how we solve some of the problems that multiple monoliths brought. Having a single backend with multiple frontends seemed like a good idea at that time, because we had a single UI and a single backend. That's it, just like a lot of products. Of course, having the ability to write multiple frontend products, journeys, within a single backend did also create some problems, because when you do something in our space, it creates trade-offs. Because everything is straight up in our industry. The question is, what were the advantages or disadvantages of this new approach?
Stewart: One of the advantages we had with having moved to a product team organizational structure, was it really gave us the ability to actually start experimenting with our architecture. We could start trying different things out on different products. Teams had complete control over their product. They could streamline their processes and reduce the technical overhead of running multiple tiers. We started seeing the collapsing of some of that. To start with, we were very much anti-application. We started collapsing some of those tiers. We didn't need them. We could talk from the web frontend straight to the database. Why not? It's much simpler. It's much easier. Why do we need service layers in between? At that point we're talking about a world where .NET MVC was fresh and new. Everybody is like, this is the technology choice.
Shibata: MVC was really popular during that time.
Stewart: I don't know how many of you are .NET versus Java versus any other language. We were very much a Microsoft shop at that point in time. We'd transitioned from VB.NET, through to C# .NET. MVC was the natural choice. We started to see challenges around how we thought about and acted on our data. We've now got these monolithic applications. They're talking to single databases. What we were essentially doing was we were just putting all that data into our data warehouse. There was obviously, ETLs, there was cubes, all of that stuff. We were going straight from the application space into our other requirements for that data. That meant that whilst the teams had loads of control, about how they could change and evolve their product, any changes to the schema became costly. They became expensive to understand, to reason about, and to make a change in our cubes and other warehousing infrastructure. It created a coupling that was difficult to understand and fix problems with.
3rd Evolutionary Branch
Shibata: This data challenge led us to the next approach that we tried. We saw that distributing systems while keeping data centralized is inherently incompatible. You're trying to fit a round parcel into a square box, essentially.
Stewart: As we continued to grow, we knew that scalability would be a constraint. We could only scale vertically. That can be very costly when you run your own data center, the cost of bare metal, all of those things that go with it.
Shibata: At this point, the technology landscape was exploding. Whilst the ideas were not queued, the technologies available had made this more accessible. We're talking about cloud, decentralized architectures, and the implementation of that. There's a lot more technology. We started to experiment with this new technology such as NoSQL datastores, CQRS, and event sourcing. Some speakers touched on Command Query Responsibility Segregation. Initially, within our data centers, we'd done all of this. We proved the ideas will allow us to scale horizontally and solve the data problem.
Stewart: Just to set the scene as well a little bit in terms of where we are, timescale-wise, this is now the start of the cloud platforms. We're starting to see AWS starting to experiment with some stuff. We as a company weren't ready to go down that route. We're still on the data centers. We know that we can't continue to scale in the way we are. The initial experiments were largely successful. It did come at a high cost. The more we try and scale, the more kit we need. We need VMs. We started virtualizing all of that infrastructure. There's still a limit on how much you can actually do because you're constrained by the physical hardware.
As the various cloud offerings developed, this obviously presented an ideal opportunity. We can let somebody else worry about that hardware scaling. We can just start using it as a commodity. We use those learnings on what we'd done in that space and how we embrace the more distributed architecture to move that into the cloud. That was really our first foray into that environment. However, we still had the issue with data. We moved some of it. We still essentially had all of this data that could change rapidly. We were still trying to fit it into an old SQL data warehouse. It just didn't quite fit. As we tried to evolve, you essentially make it backwards compatible by dropping some of the data. You're missing out on all the value then of doing them.
Shibata: That was addition by subtraction.
Stewart: Absolutely. We needed a way to capture events through the system, to share beyond integration at the database level. This led to us implementing Kafka. Who's using Kafka? We were very early on to Kafka. We started exploring that very early on in Kafka's evolution. I think it was still in beta at the time. We could see how that could really help us to deal with this problem. We started delivering business events through Kafka. We've done CQRS. We got too granular with our events. They weren't useful from a business context. It worked in the application space. It didn't work beyond that. This now was an attempt to rectify that issue. We started generating more business events to send them through our Kafka stream. That was very tightly controlled to start with. At the time the concept of microservices were still being formed, but we could start already to see how this paralleled with our own architecture. It was a natural step then.
Shibata: The timestamp then would be a couple of years back.
Stewart: 2014, roughly that time. Martin Fowler and James Lewis did write a paper, which does reference this, in that time period talking about microservices. If you're interested, go and have a look at Martin Fowler's site, and you'll be able to read about that.
Microservices and Rewards
Shibata: At that time, we were preparing to build a new customer proposition to reward customers for their loyalty. At this point, we have lots of customers. Our previous loyalty program was toys. We had lots of customers that relied heavily on just toys, and obviously, having lots of toys that relied heavily on the database integration. Then we had to start to move away from this model and to build a newer system or a newer loyalty proposition that allowed us to keep more customers and attracting newer customers. Which is essentially the point of what we're doing. This presented us with an opportunity to make conscious decisions to implement a microservice architecture, using learnings from previous evolutions of how many monoliths we built. We became masters of monoliths. Then it doesn't work for us anymore because of this problem. Let's start thinking about microservices.
Stewart: This gave us a chance to really evaluate what tools, technologies, and processes we wanted to adopt. Taking advantage of containerization technologies, Docker. We targeted UNIX and moved development to Node.js. We've been largely a Microsoft house, really. We were using Windows. We were using the Microsoft languages. Having primarily developed in that, it was a great opportunity to try and explore that a bit further and use the right tools for the job. Having already moved to at least daily releases, we further streamlined the release process to remove blockers and move even closer to the ultimate aim of continuous delivery. At that point, we started adopting a DevOps culture. Lots of people talk about the difference between DevOps culture and DevOps functions. Essentially, the model was we had people within the team who were responsible for the DevOps aspects. Ideally, the whole team would have some element of knowledge around that.
Shibata: One big change this reintroduced was the use of services provided by other teams. We started touching about that earlier in the monolith world. When you have lots of small services, you need to do it better than when we did it before. As we implemented our rewards proposition, we identified other services that will be built and owned by other teams, requiring a change in working practices to support this collaboration. At this time, also, we tried to find other approaches to ways of working like mobbing and moving an entire team to a war room. As you would have expected, some things worked better than others. We found war rooms to be very good for the team initially, but it created a sense of isolation and hindered collaboration.
Natural Selection
Stewart: Following the process of natural selection, the things that worked coalesced into what you might now recognize as a standard pattern for a microservice architecture.
Shibata: This diagram really is our first attempt to do a microservices architecture, or what came out after that first attempt. Another first attempt at event sourcing that was too granular was leading had to change. It's really hard to change. We modified this approach to simplify how events arrays will go with the architecture.
Stewart: Conway's Law definitely applied here. Continuing with the product team approach allowed us to create the bounded context for the microservices, utilizing cloud enabled. It enabled an increase in speed to market and the flexibility to scale horizontally.
Shibata: We were able to remove database integration that gave the teams freedom to innovate, creating a proxy to handle events, simplify the architecture, which is the HTTP proxy there that sends data to Kafka. This removes overhead for the team.
Competition vs. Collaboration
Stewart: We started thinking more about how we shape our culture for success based on the evolution to a microservice architecture. The two things go hand in hand. You can't just go straight to services and that it'll work. There is a definite cultural aspect to that. Product teams were great for creating a sense of competition. Each team wanted to outdo the other. They wanted to be better than the other team. Each team wanted to be the first to take advantage of scalability which comes from a cloud-based microservice architecture.
Shibata: Very early on we realized that competition drives internal movements. The analysis by paralysis can easily be driven away by competition of different teams and different ideas to the marketplace of ideas, and iteration of new architectures and flow of new ideas. Too much competition can lead to silos and lack of sharing, which is the same problem as we had when we did that war room. We needed to find a way to allow for healthy competition, whilst enabling collaboration, so all teams benefit from advances in collective knowledge through the use of forums. We had some forums and sharing sessions like brown bags. We continued to evolve our culture. The knowledge is getting kept within the company and then being shared around using these avenues or mediums to foster a spirit of collaboration. At the same time that teams were competing, they were also sharing knowledge and collaborating on different aspects of the company.
Building on this, we gain consensus. We can easily do a temp check where your engineering teams are going, depending on what's being talked about in this forum. This allows us to move quicker when adopting cross-cutting architectures. This means massive changes that would have taken ages to be implemented, if you have these forums, and you have this consensus, and this marketplace of ideas, and all these brown bag sessions, that allows you to get that knowledge out really quickly. By building the community of practices, we were able to bind these decisions and then use this to solve really complex problems in a very simple way. Complex problems usually came about by microservice architectures. This is our way of dealing with those complexities. We don't deal with complexity by trying to simplify stuff. We deal with complexity by trying to understand it more deeply across the entire organization. We don't look at it using one lens. We use all lenses within the organization to look at it.
Stewart: As we started to collaborate more, it became apparent that we needed to focus on solving problems once rather than each individual team solving it. It seems obvious when you put it like that. At the time it was very much a product team into that base. Which leads on to Comparison as a Service.
Evolution to CaaS (Comparison as a Service)
Having decided that microservices would be the right approach, we needed to work out how to evolve from the rest of the monolith. We've done some trials. We've tried some stuff out. All seems to work really well. We started identifying the key responsibilities across all of those monoliths. We started looking at what commonality is there between them. We set about identifying a pattern that could be shared by all teams. We'd go into this with, is it a pattern? Is it a set of services that everybody shares? In the course of doing this, we realized that a number of the bounded context were rarely changed. When they did, they had unclear ownership. This allowed us to form a team around those core capabilities and create a set of microservices for consumption by those product teams. Enabled by the reuse of standard services, the product teams could focus on the aspect that they really cared about, building journeys and experimenting with improvements for customer experience. They facilitated a consistent experience for developers and the data teams.
In order to allow the product teams to continue to evolve and innovate, it was vital that we offered other extensibility points, which is achieved through various handoff to services owned by the teams themselves. That's what this diagram is depicting really is that we've got a set of core capabilities over here, which are owned by the share team. Then we've got some other capabilities, which are essentially owned by the product teams.
Shibata: That's everything we'd done. We didn't always get it right the first time. It took a couple of evolutions and iterations, but each provided value in building better experiences for customers and developers. This approach is already providing dividends for us. A good example of this is a recent regulatory change for IDD, Insurance Distribution Directive, which required functionality on many products. We delivered this in a quarter of the time on Comparison as a Service versus doing the same change on all of those product teams, still on the previous monolithic architectures. We have increased speed to market significantly with this new approach for new features, and reduced the effort required to create new products.
Why Move To A PaaS Model?
Stewart: Kenichi, how does PaaS fit into all of this?
Shibata: PaaS or Platform as a Service. We do not want our product domain experts to build logging, monitoring, infrastructure, security. At the same time, we wanted to balance this with the ability to innovate. Building a semi-opinionated PaaS model is our way to go. We want our product teams and domain experts to focus on building individual customer values that adds customer propositions.
Stewart: We've been experimenting with running a PaaS for a number of years as a way to achieve this desired state. Even before Kubernetes, we started talking about, how do we build a PaaS? How do we take away this pain? What really changed? How did we end up where we are today?
Shibata: That's true. There were a number of systems that tried to deliver PaaS, some very successful. For comparethemarket, the combination of containerization, microservices, and the emergence of orchestration technologies such as Kubernetes, Mesos, and then Docker Swarm. This all came together at exactly the same time when we needed them. This is not just all about technology either. It requires a change in how the business thinks about software, and the culture and working practices in place. Finally, we wanted to balance out the cost versus the rewards. By standardizing most of the underlying setup like CI/CD, security, logging, monitoring, and infrastructure, we were able to standardize the cost. Standardized cost can be studied. They could be decreased eventually. This is also done to maximize developer productivity, to increase overall customer reward or customer proposition.
Why Kubernetes and Containers?
Stewart: Why did we choose containers and specifically Kubernetes?
Shibata: Initially, we chose containerization, because it afforded us with the flexibility that comes with doing both .NET framework, which some of our stuff are still living in, and JavaScript in a single platform. Other languages were also available, but it allowed us to use the right tools for the right job. We chose self-managed Kubernetes way back in 2016. We had started to move to AWS, and most of our fully managed services were still in their infancy. In fact, GKE was the first available, fully managed service at this point in time. However, moving to Google Cloud was not an option for us at that point.
Stewart: We evaluated the choices available at the time, and those competing proof of concepts for Docker Swarm, Kubernetes, and Mesos. It was quickly apparent that Kubernetes fitted closer to what we wanted as a future state for the PaaS service.
Shibata: One of the differentiating factor for us was the lifespan of the Kubernetes project, at least what we thought was the lifespan of a Kubernetes project. Our opinion was this would continue to be developed and supported based on its origin in Borg in Google. It was initially built by ex-Googlers. At this point in time, Google was adding more and more features to GKE. We thought this would be the safest option.
Stewart: Of course, if it's good enough for Google, it must be good enough for us.
PaaS v1
Shibata: We started to build Platform as a Service, and what better way to start with Puppet. It had all the functionality needed. However, the developer experience was lacking and innovation was stifled. This first version or first attempt at building PaaS was built in collaboration with other people. During that time, Puppet was one of the most famous technology at that time. We spent a lot of time and effort building deployable Platform as a Service with some great features for the team running PaaS, but not so great for the people who were actually using the PaaS.
Stewart: This resulted in the exact opposite of what we set out to achieve. We'd actually made life harder for our developers. They spent more time on infrastructure issues than before on the PaaS. We started to think about how we solve that problem.
PaaS v2
After the Puppet show failed, we started working on version 2, which was kops based. Kops is one of the most adopted Kubernetes installers in the market. It worked well to solve both differentiated customer problems, as well as developer experience. However, it still stifled innovation as it had a single cluster per environment, and we had locked down the controls for governance reasons. Currently, we're exploring options to solve these problems and make the experience even better.
Shibata: Following the feedback from version 1, we knew that observability would be a key area for us. How it provided the insight, where we got the drop-off points for customers, for example, how we built a culture of observability that's adopted by product teams using mostly interfaces and standards. How we built those interfaces and built standardized libraries that produces standardized metrics. We started adopting open-source tech stack, for example, Prometheus and Grafana. Fortunately, it paid dividends because of the ease of integration with Kubernetes. It means that we were actually doing Grafana and Prometheus even before we went to Kubernetes.
Stewart: Is it worth managing your own cluster?
Shibata: Definitely not, unless you're running a fully managed service, you're selling it. Unless you're Google, or Amazon, or Azure. Or, in our case, there was no available product in the market during that time. Or, if there is no product that satisfies your requirement.
Stewart: Basically, it depends. Go into it with your eyes open. We learned a lot through doing this. The key thing about building a platform like this is think about what your core capability is that you're trying to offer to market. Is it a Kubernetes platform? If it's not, that's probably not where you want to invest your time and resources in building that out yourself. There are other options on the market now.
Shibata: One of the main reason why it's really hard, shooting from the hip here, is, managing upgrades and lifecycle of Kubernetes is hard. What I mean is etcd, the database where Kubernetes relies on, has a lot of incompatibilities with regards to version of Kubernetes binary itself. It makes upgrade last quite a long time. However, there's always a new version of Kubernetes, just right around the corner. You need to keep up because there's new vulnerability fixes, security fixes, and stuff like that. Upgrading is quite hard when you're doing it self-managed. Fully managed, it's so much easier.
Stewart: For a PaaS to be successful, it needs to provide a good developer experience. We learned that the hard way.
(Cloud) Platform Ops
Shibata: We can't build a platform in one go. We decided to evolve the platform based on a clear strategy. A very high level strategy, but still quite a clear strategy for us. This can change over time. Having a line in the sand allows us to focus on the most important aspect, or more urgent areas. This is key in prioritizing to build a successful delivery of a Platform as a Service.
Stewart: Essentially, build your platform in stages, work collaboratively, and take the developers on the journey with you. Apply everything that you've learned in building systems to how you build platforms. The same rules apply.
Shibata: We also don't want the evolution to be restricted or constrained by knowing the end state of the platform. However, we also do not want to lose sight of what we're trying to achieve. Simply, what does a complete platform look like for us in comparethemarket?
How Will We Evolve Platforms?
Stewart: Finally, we want our developers to use our platforms as components and only focus on building differentiating systems that our customers will directly benefit from. We want to engage with the teams early and often.
Shibata: This ultimately means treating your platforms as your internal products, treating your developer teams as your customers. What we have here in this diagram, really, is we have building blocks for the developers to play around with so they use the right technology or the right tool for the right job. The only thing that your product teams need to be concerned about is, what is the outcome? What does good look like for the end user, for the customers?
Evolution not Revolution
Stewart: Innovation and playfulness creates evolution. Revolution cuts to the roots. We're not trying to say you can't rewrite things, you can't change things. Be aware of what you're trying to do in that regard. Innovation and playfulness, and that will evolve your systems.
Shibata: We need to acquire new information all the time. We need to try to get our currently held beliefs or traditional beliefs in order to cause dialog. You need to have and explore these ideas with your teams and your communities, or communities of practices. Then you need to develop these ideas further. For example, in the words of F. Scott Fitzgerald, "Intelligence is the ability to hold two opposing ideas at the same time." Finally, when facts change, change your mind. It is this intelligence that gave us this ability to experiment and pivot our hypothesis based on new information provided. You won't get sucked into, that's how it's always been done. At the same time, trying to balance that out with what is the need of your customers or your business.
What Have We Learned?
Stewart: Throughout this journey, what have we learned? Evolution. Grass isn't always greener. There is a bit of a trend to throw things away and start from scratch. It can leave you exposed when new requirements come along. Regulatory change, for example. Evolution, it is better to incrementally improve rather than get stuck making big changes, and frequently leads to better outcomes. We're not talking at the level of a service where rewriting that service probably makes sense, but it's about evolving architectures, evolving cultures.
Shibata: The second one is be brave. People think hypothesis based on new information is important. Do not be afraid to change course, especially if you know there is a better direction based on your information. Lastly, platforms. Your developers are also your customers, use them to understand how to evolve your internal systems as well as customer-facing systems.
Evolutionary Architecture
Stewart: Hopefully, you're getting the trends for evolution here. Moving from legacy systems can be difficult. In fact, it can feel like it's never ending. Each problem solved, you might find out more. This can lead us towards actually starting from scratch might be the better approach. There is another way and that is to embrace that evolution. By choosing the appropriate tools and technologies to complement your existing architecture, you can create your own evolutionary branch.
Shibata: Finally, we would like to leave you with a phrase attributed to Charles Darwin. "It is not the strongest species that survives, nor the most intelligent. It is the one that is most adaptable to change".
See more presentations with transcripts
