Sam Newman on Information Hiding, Ubiquitous Language, UI Decomposition and Building Microservices

Charles Humble: Hello and welcome to The InfoQ Podcast. I'm Charles Humble one of the co-hosts of the show and editor in chief at cloud-native consultancy firm Container Solutions. Today I'm joined by Sam Newman, Sam is an independent consultant and is the author of a number of books including two on microservices, Monolith to Microservices which has been the subject of a previous InfoQ Podcast and Building Microservices. The second edition of the latter is just out and having just finished reading it I was keen to talk to Sam to find out more about what has changed between the new edition and the first version which was published in 2015. Microservices will also be one of the tracks at QCon Plus the software developer conference which will be back this November the 1st to the 12th online.

QCon Plus features 16 tracks curated by domain experts to focus on the topics that matter right now in software as well as the microservice patterns and anti-patterns track of the tracks covered this November include the cloud operating model, from remote to hybrid teams, and architectures you've always wondered about. At QCon Plus you can find out what should be on your radar from the world's most innovative software leaders driving change and innovation. Each will share with you their solutions, best practices and actionable insights to reduce uncertainty on which technologies should be part of your roadmap. We hope to see you online this November. You can visit qcon.plus for more information. And with that, Sam, welcome back to the InfoQ Podcast.

Sam Newman: Thank you very much for having me.

Can you talk about information hiding, and why you feel it's so important? [01:50]

Charles Humble: And congratulations on getting the second edition of the microservices book out there, I think it's a fantastic book. I've just finished reading the early access version and I believe as we're recording this, the second edition is going to print and will be out in the next month or so. I was interested by some of the things that have changed and in particular you talk a lot more about information hiding in the second edition drawing on David Parnas's work. Can you talk about that, why you feel it's so important.

Sam Newman: When I wrote the first edition I think we were just dealing with what people were doing, we were looking at this architecture, looking at that architecture, and just seeing the different style of SOA that microservices was becoming and it was a process of just almost recording what people were doing rather than trying at that stage, you don't really have the time or the energy to be able to distill down the essence of what unified these approaches, bit of a hand-wavey way of saying it right there. But basically there was too much stuff going on and there was no time to step back and think about what were the core ideas at play here, we could see there was something different but not what, a lot of the early focus was other things. And after time and energy looking at it and throughout writing the first edition, I kind of zeroed in on this idea of this independent deployability as being the big hook for microservices.

But then in the intervening time after writing that first edition, I spent time working with people who were struggling to deliver on that. And I realized talking about independent deployability was great but how do you achieve it. And so I'd been put on the trail of information hiding, during the first edition Martin Fowler who helped review the first book had mentioned it, and I do cover it in passing I think in the first edition. But I came back to that concept as well, is this the hook for the hook in a way? If you get information hiding right does it therefore make independent deployability much easier? And I realized that actually it does. And if you go back to the original work by Parnas when he's talking about information hiding as a way of building good modular architectures, the concepts of what we want from independent deployability are right there in the stuff he wrote back in the early '70s.

And so at the beginning of last year I'd kind of come full circle almost just thinking of microservices really fundamentally as being a modular architecture, albeit a modular architecture with the complexity of a distributed system. And so therefore taking Parnas's advice about information hiding made even more sense in that context, so that's kind of where it came from. So the first edition, the world is chaotic and I have no time to work out what the hell's going on, I'm just capturing the good advice and then in the second edition, I've had a bit of time to think there's some core hooks, I think, that are important to emphasize more in the second edition to hopefully help people deliver better on the promises of microservices.

Can you give a definition of exactly what you mean by information hiding? [04:41]

Charles Humble: Can you give a definition of exactly what you mean by information hiding?

Sam Newman: Yeah. I mean, there's some specific definitions that Parnasmkj uses in his work but the kind of one I go to is the default, if you think about a microservice boundary your default position is almost you don't expose anything. So any information you expose, so if you expose a method, a data structure, a piece of functionality to something outside of your microservice boundary, well, then an external consumer can make use of that. And so once they make use of that they then effectively have a contract between you and them, it's an interface they're using. Once they have an expectation about how that function works about the presence of that piece of data, that then makes it harder for you to change your microservice without breaking them. If you want independent deployability, you need backwards compatibility. By hiding things inside your service boundary the things inside your service boundary can change without fear of breaking your external consumers.

So it's almost like partly saying, things that are hidden can change easily and things that are shared are part of your contract and you have to be very careful about how they change if you want proper independent deployability. The more you hide, the more freedom you have as a developer working on a microservice to change things safely. I talk about information hiding really is just about clarity, being really clear to a developer as to what things can be changed fairly safely and are not going to have a nasty ripple effects in production. Developers don't go to their day job wanting to break computer systems, they want to do a job. And with information hiding, right? You're saying, this stuff is hidden no one else can see it, that's yours, anything you change in there is going to be okay.

Now, anything in this part of your code base, this is where you're shared with other parties, the outside consumers, you need to be really careful in that world. So this is where you're exposing your REST endpoint or gRPC endpoints and whatever kind of external contracts that you might have. And so when you change anything in that world you need to have safeguards in place to make sure that you don't break people accidentally, this is why I'm a big fan of explicit contracts for service interfaces, things that are contributive in contracts. But information hiding allows you to really focus that energy and work on the small parts of your microservices code base which actually are used by other parties.

Charles Humble: And then for those bits that you are exposing to other parties, how do you then control that? Do you have some sort of schema? Do you have some sort of contract which defines what is and isn't allowed there?

Sam Newman: For me, again, this is something I've definitely come full circle on. I got burned using SOAP, many users did, and, well, we reacted against SOAP. We threw SOAP away because it was for a lot of reasons and none of them are good reasons, but one of the things we threw away there was the concept of the schema, WSDL, not a great schema format in my opinion, but we got rid of that schema format, and then most people now are doing service interactions with implicit schemas rather than explicit schemas. I agree with Martin Fowler, is you always do have a schema between how a client and service communicates. If you don't have an explicit schema the client still has expectations on what it thinks your end point should do, there is a schema it's just now entirely implicit, so having an explicit schema allows you to be really clear.

I would definitely prefer now having explicit schemas for my service end points and that clarity gives you clarity for consumers of your microservices as to the structure of your service interfaces but it also gives you clarity as the person implementing that microservice as to what you signed up for. Now, schemas can only help with the structure but that's still incredibly valuable, especially when you consider there's good tooling out there to compare two different versions of an explicit schema and tell you if one is backwards compatible or not which is something you can then put into a simple build process. So even before you deploy your microservice, you might be able to pick up breaking changes in service compatibility which is just amazingly great and so much simpler than say having to have lots of large test cycles and things.

Why do you think ubiquitous language is important and why do you think people often don't pay attention to it? [08:48]

Charles Humble: Right. Yes. Then you've talked a little bit about service boundaries already here and obviously Eric Evans Domain-driven design has become firmly established as one of the kind of key texts for people thinking about building microservices. You particularly mentioned bounded contexts and aggregates in the book but you also talk about ubiquitous language, this idea of having a shared language between the business and the language used in the code itself. And although it seems kind of obvious, I think it's one of those things that kind of weirdly gets missed a lot. Could you talk about that? Why you think it's important and maybe also why you think people often don't pay attention to it.

Sam Newman: The ubiquitous language in many ways is the key stone of domain-driven design and it's amazing how many people skip it, and it's foundational. I think a lot of the reason that people skip ubiquitous language is because to understand what terms and terminology are used by the business side of your organization by the use of your software, it involves having to talk to people. It still stuns me how many enterprise architects have come up with a domain model by themselves without ever having spoken to anybody outside of IT. So this fundamentally, the ubiquitous language starts with having conversations. This is why I like event storming as a domain-driven design technique because it places primacy on having that kind of collective brainstorming activity where you get sort of maybe your non-developer, your non-technical stakeholders in the room and listen to what they're talking about and you're picking up their terms, their terminology, and you're trying to put those terms into your code.

This is ultimately what brings the biggest value from domain-driven design, a developer joining that code base where the terms and terminology used are the same that are used in the business, the same ideas in people's heads inside the organization. A developer working on that code base is going to become a better domain expert and will be much more able to have conversations with people outside their development team and use the right terms. This is partly about aiding communication, it is also I think, a big part about building empathy, without that I think it's very hard to have these great collective relationships. And so I think it's something you can't skip, I think if you do skip I think it invalidates so much of what comes. I mean, ultimately the language is how we name the concepts, if you decide to skip the whole step of understanding what that common language is, the concepts you come up with are similarly devalued.

Do you have a preferred definition for an aggregate? [11:07]

Charles Humble: We should perhaps touch on what an aggregate is as well, because there are quite a few different definitions out there, do you have a preferred one that you use?

Sam Newman: I'm pretty sure the one I use is just a bit of an amalgamation of how Eric describes them and how Martin describes them which is, it's a collection of objects which you always want to manage together from a point of view of state management, which is a very dry definition, they're typically things that have state and identity and often they represent real world concepts. Almost the canonical example of this might be an order aggregate, right? So you can think of an order aggregate as something that you place, that the order might have stages it goes through. But you could think about it from a data point of view that an order might have say an order table that maybe each row represents a state of one order, this might be the customer ID, maybe the shipping location it's going to, but that we'll then also have associated with it many order line items, so each item you've purchased, maybe a quantity, what the item is and how much the item costs.

Collectively that represents the concept of an order in that domain but from an implementation detail point of view, we might have to manage that state in two separate tables, assuming we're using kind of standard relational model, but you'd always want to manage those order line items and the top line order record together as a single unit. So it's also this core concept around thinking about aggregate, so not only should they really represent real world concepts but also they're from a transactional management point of view, we'd always want to keep the management of an aggregate within a single transactional boundary.

And so even when you're breaking say monolithic systems apart into microservice architectures, if you're using a domain driven design approach, as you identify those aggregates you still want to kind of keep the ownership of an aggregate within side a single microservice, because you can always then implement the management of those aggregates like a state machine and have a single microservice managing that state machine. And so really it's kind of where at the high level domain-driven design is all about talking and interaction with non-technical stakeholders but at the more detailed level around aggregates, it does move much more into how we think about state management of things as well.

Can one microservice handle the lifecycle and data storage of more than one aggregate? [13:15]

Charles Humble: And can one microservice handle the lifecycle and data storage of more than one aggregate?

Sam Newman: Yes, and that's quite common. So you could have a microservice, might encapsulate, say, five, or it might manage 2, 3, 4, 5, 10, 15, 20 different aggregates inside it. So the idea is that you say each aggregate you pre-order is associated with one micro-service but one microservice might manage many aggregates. We could have a domain model for how microservices handled domain models effectively if you want to take from that point of view. And often, this is where kind of lots of people get obsessed about the granularity of a microservice, actually it doesn't matter too much because there's lots of different factors that go into size and that's probably a separate conversation. Some aggregates are incredibly simple to manage, other aggregates the logic around them and the algorithmic complexity around those aggregates might be much more developed. So we can't say that's got 25 aggregates in it therefore it's doing too much because those aggregates might be incredibly simple things or there might be incredibly complicated things that are highly critical to your organization. So yes, it's not immediately cause for concern if that's happening

 Can aggregates have relationships to other aggregates? [14:23]

Charles Humble: And can aggregates have relationships to other aggregates?

Sam Newman: Yeah. And then the example I gave before with sort of the order as being one of the canonical examples of an aggregate, an order is placed by a customer. So, you would imagine in your relational data model you'd have the order line in your order table and that would have an ID for the customer that placed that order and that is effectively how from an implementation point of view is showing the relationships between aggregates. And now obviously with a microservice architecture where this gets interesting is that the aggregate, the one you might relate to, so maybe when my order relates to a customer that customer aggregate might actually be managed by a separate microservice somewhere else. So you still have kind of almost a logical concept of something like a foreign key relationship but that foreign key relationship is effectively now pointing you not to another table inside the boundary of that microservice but it's now pointing you to an aggregate managed by a separate microservice that you would have to resolve that relationship probably via some kind of service call.

Can you also give a definition of bounded contexts? [15:22]

Charles Humble: Great. Thank you. Another thing that has a lot of definitions is actually bounded contexts. So could you maybe give us a definition for them as well?

Sam Newman: Have you ever heard the phrase when someone says, monads are like burritos, it's a bit like this in bounded contexts there's so many different definitions out there. Again, I like Eric's original definition of it which talks really about a bounded context like a cell in biology. But the way to think about it is, think about your organization, your organization has lots different bits to do lots of different things. You've got the finance department, they do all sorts of things, they manage your accounts receivable, they manage your payroll, you've got the warehouse team who manage the stock that you have and shipping products. From the outside you kind of have a high-level idea about what the warehouse does, you understand that the warehouse is responsible for the stock that it carries, receiving goods, shipping products out, but you don't need to know the detail of how the warehouse does it, you don't need to care about how many shelves there are in your warehouse, you don't need to care from the outside which forklift trucks that they've got, you just need to know they do a job.

Sam Newman: A bounded context is effectively a way of us modeling these kinds of groups of functionality within a domain model so from the outside the bounded contexts allows you to kind of access some sort of higher level concepts and ideas like I can check the stock level with the warehouse. Inside the bounded context we have all the detail and all the aggregates, as we talked about before, that might be required to implement the functionality of that given warehouse bounded context. So really bounded contexts are all about information hiding so from the outside you see an explicit boundary with explicit responsibilities that it provides to the wider system.

Inside that boundary is all the details but that's hidden from you, you need to engage with that detail if you're working inside that bounded context. So bounded contexts really are how you implement the concept of information hiding in domain-driven design. I'm not sure if that was originally the intention of what Eric was talking about, it's been a while since I've read the full blue book, I'm not sure if information hiding is explicitly referenced in Domain-driven design. But for my money this is it, right? This is the connection. In the same way that encapsulation is an example of information hiding in object oriented programming, bounded contexts are an example of information hiding in domain-driven design.

Aside from DDD are there other techniques that can be used when thinking about decomposition? [17:43]

Charles Humble: And we've said that microservices are basically a form of modular architecture and we've been talking about domain-driven design in that context, thinking about decomposition. Presumably domain-driven design isn't the only game in town, what other approaches or techniques are there that might be useful or helpful if indeed that there are any?

Yes. I find that domain-driven design is the best general purpose approach to decomposition, but whenever I'm sort of working with a team trying to find boundaries I'm not going to look at just the domain model in isolation, I'm going to overlay other concepts on top and they might help you tweak. Because if you think about it, there's often a degree of subjectivity in terms of where you draw some of these boundaries, especially when you get into the bounded context there can be some fluffiness. So the other things that I like to bring into sort of scope in terms of helping find those boundaries are going to be the organizational structure. Now, typically people are adopting microservices to speed up delivery, is not always the case but for a large amount of people that's what they're trying to do. So looking at where your boundaries of your architecture without looking at the boundaries of your organization would be a bit crazy. I mean, obviously you would expect your domain model and the bounded contexts, especially would normally follow your organizational structures anyway, but not always.

So overlaying that organizational structure is worthwhile often because you realize the organizational structures have got a problem. Other overlays that can be useful is looking at things like volatility, so are there parts of your code base that are changing more frequently than other parts of your code base? Now, again for me, this is an overlay not a place to start with, so if you're decomposing an existing monolithic structure it might be good to know which parts of your code base change more frequently. There are some sort of general purpose decompositional techniques I don't like, I mean, the classic example would be things like the three-tiered architectures because I don't think they make much sense as a way of actually speeding up delivery, really, they're based on old fashioned concepts around manufacturing theory which don't really work and we know they haven't worked for 40 years. So things like the three-tiered architectures they don't tend to be a good approach if you're trying to speed up delivery, although often fixing three-tiered architectures often means also changing org structures which is a different thing.

Again, this is also why I disliked things like pace layered architectures or bi-modal IT which are just often concepts infrom insert name of big analyst firm here to allow enterprise organizations to not actually have to change anything they do and sort of to codify the current very slow pace of change, so in general I avoid layered architectures. We can sort of simplistically say that what we're trying to do with microservices is talk about maybe more end to end vertical slices and move away from a previous world of predominantly being horizontal layered slices. When you want to deliver a piece of functionality with a layered architecture you often have to change multiple different layers and if those layers are owned by different teams, that results in lots of handoffs which slows you down. If we can move to a world where we're predominantly talking about end to end vertical slices then we're hoping that the changes that we want to roll out for our customers are more likely to sit within those vertical slices and we can speed up delivery and that's simplistically kind of the reason I'm trying to move people away from layered architectures.

Why do you think having a dedicated front end UI team is a mistake? [20:54]

Charles Humble: Now, one of the things that's covered much more extensively in the second edition of the book than the first is around decomposition in UI. And one of the things you talk about is that you prefer to see non dedicated front-end teams, you prefer to see the front-end people as part of the team responsible for a given microservice and you argue that it's a mistake to have a dedicated front-end team. It's always seemed to me that there are quite persuasive reasons for going the other way on that, obviously UI is a specialist skill, good front-end engineers are hard to find, a single microservice team might not be able to keep a front-end person sufficiently busy, and in theory, at least having a single front-end team gives you a more consistent UI. So I was interested in your view on that, why do you think having a centralized team here is a mistake?

Sam Newman: I think all of those arguments are good arguments, I think you could also have made all those arguments around a centralized DBA function, a centralized security function, a test function, an infrastructure function, an ops function. And those are good arguments and in many cases make a lot of sense, but we've also recognized that we want to bring some of those functions into our team and that's made us better, now, we've traded some things off around that absolutely. So for me I think it's another example of a problem that we've been dealing with over the last sort of 20 years or so which is we've had centralized specialist functions which allow you to have team structures which are fundamentally around your core competency, right? You have the UI developers and the UI development team and they're specialists, you have the DBA and the specialist team.

The challenge is those handoffs I mentioned before, when you had a separate dev team, a separate support team, a separate test team, a separate infrastructure and ops team, a separate security team to deliver software you've got all these handoffs that happen and that just slows us down, right? So we've been putting those functions in, and so for me the UI is just a continuation of that idea. I think there are some specific challenges in making those UI teams work though, so we thought if I now own delivery of an entrance slice of customer functionality inside my team then I can go a lot faster. Because we can come up with an idea, myself and the UI, and remember, you can still have front-end specialists inside a team, we can work together to deliver that change and put it out.

But also I remember this thing that happened with DBA functionality when we started moving DBA work more into delivery teams, it used to be that the DBA has do all the DBA work and some of the DBA work was really hard and a lot of the DBA work was really simple like adding a column. And so what happened was when we started pulling DBA work more into development teams is it freed up your DBA specialists to do the hard stuff and you could have developers do the easy low hanging fruit. So for me, I think if you pull your front-end functions into your teams, there are simple front-end things that a service side developer absolutely can do in the same way that there are some simple service side things that a front-end developer can do.

So you're giving people the opportunity to grow their skillset, you're kind of creating, I hate using the term full stack because I'm with Charity Majors on this, you're not a full stack developer unless you also build the chips, but more of a full stack team than a full stack developer then you still can have a role for UI specialists but they're focusing on the hard stuff. The thing you lose, of course, when you pull those front-end teams into your delivery team rather than having dedicated is that lack of consistency in the same way that you might argue about a lack of consistency in your data model. Now, in microservices that's not an issue because we break our databases apart but with the UI, this is your front of house, your customers, and you see some organizations that really seem to be okay with having a high degree of inconsistency to their users.

Amazon is a good example, it used to be the Amazon shopping site was quite poor from this point of view, lots of inconsistency of experiences, that's got better, but the AWS web console is a great example of where you see this inconsistency in a way that does, I think, harm the customer experience. The trade-off at Amazon is this allows us to go fast and, guess what? It seems to be working for them. But you can still do things around consistency of experience even if you have separate UI teams, examples of how you can do this is you could have enabling teams, this is a term from the Team Topologies book who provide things like living CSS style guides, they can be incredibly effective.

So you basically have some people that collaborate to come up with like, this is the style book we're going to use, so everyone just inherits from that style book, that's a simple change that you could do that already helps unify things. So I think you have to recognize that you have lost some things by going from a dedicated front-end team to breaking them apart but you also might recognize that there may be ways to deal with those downsides. And even if those downsides still exist, the upside may still be worth it and in most experiences, especially when you're dealing with larger projects of work, I definitely feel that avoiding that separate second team needed to coordinate with work is vital.

Charles Humble: It's interesting you mentioned enabling teams there in your answer as well, this idea from Matthew Skelton and Manuel Pais' Team Topologies book because I think that can also help with the underutilization problem that you might get.

Sam Newman: Yeah, absolutely. And their work came from looking at dev ops organizational structures. And in there when we were looking at embracing the concept of dev ops, we'd often have this idea that you would now have specialists that didn't have enough work in one team just be dedicated to one team and so you'd have this kind of roving concept. So as an ops person I might spend half my time embedded with team A and half my time embedded with team B and literally I used to play those roles, I would go and sit in with those teams and I'd moved to the other team later in the week, and we recognized that was a good way to deal with the lack of utilization of a specialist within a single team. And what they've done in the Team Topologies book is maybe kind, almost call that out more in the concept of having the enabling team structure. The mindset behind Team Topologies book is your primary unit of delivery is what they call the stream aligned team, right?

So you have a team dedicated onto delivering a stream of valuable work, right? So we focus on this part of our functionality and we deliver this and we own that responsibility for delivering and this is also a shift of responsibility as well. And then you have these enabling teams and their job is not to control or not to say what does or doesn't happen, their job is to enable so when you're in an enabling team, my job is to help the stream aligned teams do their job and this again though is an important shift. Often those people in their traditionally siloed roles were placed in a situation where not only did they have to do the work but they were responsible for doing the work and they were accountable for any problems in that area as well. Now you're kind of trying to invert that, you're saying, no, you're responsible for the UI, I am responsible for helping you own your UI and work out how we do that, and so that's a tricky switch.

Now, I think the biggest switch is not, we don't have a central UI team anymore, I think the biggest switch is understanding how much responsibility can move and what teams are ready to take on board that responsibility and also the mindset on both parts, right? So if you're used to being a person in charge of this piece of functionality or this responsibility, be it as an ops person, a security person, a UI person, and now you're being asked to sort of seed some of that responsibility to other teams, that can be difficult to do and it's almost like a consultancy mindset in a way that you have to adopt which is can be really, really tricky. But yeah, if you want these sort of end to end fast delivering stream aligned teams to work, enabling teams are essential to make them because you will need help. That's why I really love the terminology they use, enabling team I think is the right term for that.

For web apps you lean towards page based decomposition, can you talk about why that is? [28:35]

Charles Humble: In terms of UI decomposition you offer a couple of different approaches in the book, page based and widget based, and for web apps you lean towards page based, can you talk about why that is?

Sam Newman: I'm a bit old school. I started off as a web developer back in 1995, 1996, right? So I was at university and I made side money doing websites, a lot of them involve little animated GIFs of workmen saying, this site is under construction.

Charles Humble: I remember them well, I did a few of those as well.

Sam Newman: So there's a fundamental belief I have that single page app experiences can be incredibly useful for delivering very rich styles of graphical user interfaces on the web that were never envisaged has been the things we built in the web, right? So single-page apps make sense in certain occasions, but this sort of wide array of types of web based user interfaces which makes sense as websites. I read the Sydney Morning Herald a lot because I used live in Australia and it's probably the best way to get news about the NRL, and so they have built their news-based website as a single page app and the experience is horrendous partly, as a result of this, it would be much simpler for them to build it as a series of webpages. Now, if you have a series of webpages each page can be served up from different microservices, it's a very easy decompositional look because then you're effectively using URL routing as a way of dispatching requests to different microservices.

What seems to have happened is the assumption now is that every experience that we're going to deliver over web must be a single page app and therefore we have to pick one of these single page app frameworks to build it. And there're some situations where it's justified but a lot where there isn't, the key thing is no one ever thinks anymore, should it be a webpage or should it be a single page app? That switches don't seem to be engaged anymore, and so as a result we ended up building websites which are a lot more complicated than they often need to be. I did, I can't remember, well, I think I did a check recently, I think the Apple homepage at one point had a page load size bigger than the Alpine Docker image, right? So I've got full fledged Linux operating systems that are smaller than some web pages now.

And yeah, sure, it's cash and everything else but it does seem that we've got a bit to arrive there. And actually the problem is that this adoption of single page apps as the way to do things has proved problematic because single page apps as they were originally envisaged did not make modularization possible, they weren't built with the concept of decomposition on the user pane in mind, again, because a lot of the people that built those things did not have a lot of experience of building user interfaces. And I say this, I used GWT for a while, right? So Google Web Toolkit, I use this while I was working at Google and that was built basically in such a way to try and ignore the fact of the web even existed and you can't do that.

You look at Angular which has been very, very successful, but the clue is single page app, I am the app, I own the browser, it is mine, there's no concept of breaking that apart. So it took us a good five years before we started seeing approaches to allow us to sort of decompose those single page apps in ways that allowed for different parts of that UI to be updated at different frequencies. And it's astonishing to me that we had solutions for that on the web 10 years ago, 20 years ago, and we've waited years until single page apps have something similar. So often the reason I've seen dedicated front-end teams has been because of a decision to adopt single page app frameworks that cannot be broken apart easily or historically couldn't be.

And so for me it's like, just ask the question, could it be a webpage? Because if so, your life from a decompositional viewpoint can be significantly easier because now you've got the option of decomposing your UI around pages and even within an individual page you could then further decompose it in terms of widgets. Now, if your app has to be a single page app because that's the kind of experience you want to create and that's absolutely fine, but then you're going to have to look into solutions around micro front-ends effectively in terms of having a container app and then having separate apps you swap in and then being able to communicate sort of between those components within the browser pane which adds complexity as that stuff has historically been an afterthought with the single page app frameworks.

 Do micro-front-ends  have any particular problems that you're aware of? [32:52]

Charles Humble: Right. Yes. We've talked about micro front-ends before actually, Wes did an interview with Luca Mezzalira who is VP of engineering at DAZN and they talked about micro front-ends on that episode of the podcast, so I'll link to that in the show notes. I've not used them in anger but instinctively it's an approach I quite like, at least if you're working with a single page app. Do they have any particular problems that you're aware of?

Sam Newman: There's a couple of particular issues you need to watch for with a single page app. Now, I like the approach, assuming you really do need a single page app I think is a good thing to do so I think if you can justify using single page app frameworks, using some of the micro front-end frameworks make a lot of sense. There are still some caveats around it, the first thing I would caveat all of this by saying, I do not spend my life doing front-end development anymore so you need to take some as a pinch of salt although I have spoken to experts in this space. The couple of the issues really you need to think about are, well, the page load size can get kind of interesting. So you can have say two apps that you spliced into the same browser pane using a micro front-end framework, if they're using two different, say, point releases or even minor point releases, of the same single page app framework, their list of transitive dependencies can be quite different.

So although they might all depend even though app A and app B might all depend on the same library, they might all depend on slightly different versions of the same libraries just because of how the transit dependency chains work and so you're going to have to bring those in and sandbox them all inside that browser pane. Now imagine you've got 4 or 5, 10, 15 different single-page apps that are just there to develop simple widgets, you can actually see a much larger page load size occur as a result so that can be problematic. Now, of course, one way to deal with that is say everyone is got to use the same version and then of course you undermine the whole ability to be able to roll out these single page apps so that's one of the big issues I see.

I think the other one is kind of just how you handle the communication between them. The thing is, by definition there'll be solutions in the Vue community and in the Angular community and in the React community about how you solve these types of problems. But also kind of ideally what you want with micro front-end stuff is almost to be language agnostic in a way as well because there are organizations that have wanted to move, say, from Angular to Vue and if their application was a single monolithic piece, that's a huge migration but if you use micro front-ends, you could then migrate a piece at a time over to Vue and that I think is a great thing about micro front-ends, but then you've got to work out how you do the columns basically between those components.

In the book you argue strongly against end-to-end testing.  Why is that? [35:15]

Charles Humble: Yeah. We're coming to the end of our time. There's one other thing I want us to quickly pick up which is that in the second edition of the book you are very assertive about not doing end to end tasks much more so I think than in the first version of the book and I was curious about that, why is that?

Sam Newman: It's in the first book but I'll be equivocating a lot more. I remember getting the feedback at the time which is, don't equivocate in the first edition, if you believe it just say it, and it's like, well, but I'm not sure. And in the second one, I'm like, yeah, no, I really don't think we should be doing it. And I want to be really clear about this, what I say is, as you move from say a single process monolithic system to a distributed microservice architecture end-to-end tests become a lot more expensive to write and they become a lot less valuable that you'll get a lot more false negatives, right? They will fail for reasons that aren't down to your software, there'll be more and more expensive to create them in the first place because the test scope becomes broader. When they fail it's harder to find out what caused them to fail and they're not going to necessarily give you the same degree of confidence around system issues as you may have had with a simpler non distributed system.

So over time you'll find that you're spending more and more time and energy on your end-to-end tests and getting less and less value. So I kind of almost see it as a sliding scale is as you become more distributed you want to put less energy into your end-to end-tests. You're going to bring that effort and energy down and put that effort and energy, it's not a zero sum game, right? You're going to need to shift some of that energy into looking at, firstly, different types of testing and I would argue that contributing and contract testing can be really effective in terms of picking up some of the things that you would have picked up with end-to-end tests, I think schema compatibility checking can also pick up some of the issues the end-to-end tests would have picked up.

Sam Newman: And then you're also putting some of that effort and energy into doing more work in production so you're doing things like parallel runs, you're doing things like automated remediation and canary releases and things like that. I mean, fundamentally I think we have a problem that we've always seen in the world is there's testing and then there's production and they're two different worlds. And really is, well, the activities we do are actually quite conceptually when you're testing, you're trying to validate, is this software good enough? And in production we have all this monitoring stuff, you can't see the air quotes, I'm doing air quotes around monitoring, which is saying, is our software good enough? We're asking the same question using totally different tools.

Sam Newman: But I do think there's a lot of that verification of our software that could be done in a production environment and should be done in a production environment that will be better and in some cases looks a lot like old end-to-end testing. So for me, as systems become more distributed, your end-to-end tests are going to become more expensive, less valuable. So divert some energy into other types of verification, look at schema compatibility checking which I think is analogous to sort of typing in statically typed languages, contributive and contract checking for picking up semantic breakages, canaries maybe, parallel runs certainly for some types of functionality and all those things can help you pick up issues before a customer ever sees them.

Charles Humble: That's great. Sam, thank you so much indeed for doing this and lovely to talk to you and thank you for joining us this week on The InfoQ Podcast.

Sam Newman: You're welcome. Thanks for having me.

Mentioned

• Building Microservices, 2nd Edition by Sam Newman
• Information Distribution Aspects of Design Methodology by David Parnas (PDF)
• Domain Driven Design InfoQ Resources
• Domain-Driven Design: Tackling Complexity in the Heart of Software by Eric Evans 
• Luca Mezzalira on Micro Frontends at DAZN (InfoQ Podcast)
• Team Topologies by Matthew Skelton and Manuel Pais