Security Considerations and the State of Microservices with Sam Newman

Show Notes

What are you up to these days?

• 1:40 - I was working with Atomist in Australia, and I was thinking about moving back to the UK.
• 1:50 - I had no idea what to do next - so I decided to answer a few questions from people, and before long I discovered I had created a consultancy.
• 2:05 - I advise startups and training and consultancy on microservices, cloud and continuous integration/deployment.

You wrote "Building Microservices" in 2015. What's the state today?

• 2:40 - The hype has been replaced with people trying to deploy microservices. 
• 2:50 - People are asking the (sensible) question "Why are we trying to use microservices?" - which is the right question to ask.
• 2:55 - The majority of the people I talk to have existing IT and are attracted to microservices in order to go faster.
• 3:25 - Some of the companies are doing it because they are smart and savvy about it; some are doing it because everyone else is doing it.
• 3:30 - There's a clear distinction between those that have a good idea of why they are doing it and those that do not.
• 3:50 - There's still a bit of fuzzy thinking - they are the golden hammer for the silver bullet.
• 4:00 - The reality is that they are an approach to architecture that has pros and cons.
• 4:10 - In the wider ecosystem we are getting better tools to handle them, such as linkerd and service mesh - the deployment platforms are getting better.
• 4:25 - With Kubernetes and CloudFoundry on one hand, and the service meshes like linkerd on the other hand, things are getting better - the kind of things that Amazon suggested were the things you didn't want to have to do but still needed.
• 4:40 - Some of that gets handled by those platforms - but fundamentally, microservices are a distributed systems development architecture.
• 4:50 - You can push as much smarts into the platform or service mesh as you want, but fundamentally the challenges with distributed systems remain.
• 5:10 - I don't expect everyone working with microservices to have a detailed mathematical understanding of the properties of distributed systems, or to know about the CAP theorem, or to understand the issues with distributed transactions, or to have heard of Leslie Lamport - those aren't the important points, but I have been working with people who are building distributed systems and don't know that time doesn't synchronise between machines.
• 5:35 - Those are fundamentals of distribute systems, which I'm still seeing a poor amount of penetration of people understanding.

What are the first principles of microservices?

• 5:55 - Don't do it? I mean, have a good reason for doing it.
• 6:05 - When you have an illness and go to your doctor, they will think about medication to give you - with both benefits and side effects.
• 6:20 - Hopefully the doctor understands medicine and your body well enough and your physiology in your particular condition  - is the cure worse than the disease?
• 6:30 - Microservices are no different - they have an amazing number of benefits if they are done well, but if you do them poorly then there can be serious down sides.
• 6:45 - So the starting point is: do you know why you are doing microservices and what is the outcome you want to achieve?
• 6:55 - The thing I try and get from my clients is: can you explain to me what benefit your customers are going to get from you adopting microservices? Can you explain it in terms your customer will understand?
• 7:10 - Can you work out whether you are achieving the goal you expect?
• 7:15 - Finally, having done all that work, can you think of any other ways that you can achieve those same results?
• 7:25 - For example, if the goal is to ship software quicker, then measure your cycle time; then look at where the constraints are in your cycle time.
• 7:35 - Are the issues with the architecture, or are there improvements to the automated deployment process that can deliver these results?
• 7:40 - So getting to the basics is important, as once decided what they are, start really small and simple.
• 7:50 - For example, extract out some small functionality or write a new service that achieves a measurable goal, one piece at a time.
• 8:05 - The key is to get it into production. A large amount of learning comes from managing the services in a production system.

How mature are the service meshes today?

• 8:20 - A lot of them are being used by startups who are tech savvy.
• 8:35 - The team at Monzo (used to be called Mondo), who are a FinTech banking startup, really love linkerd.
• 8:45 - Their team is super sharp and technical and has always been happy with embracing the cutting edge.
• 8:50 - When you think about about how mature the technology is, it's also about your acceptance of the cutting edge.
• 9:00 - For example, when going into a big bank, they aren't likely to be able to use linkerd because that presupposes using Kubernetes.
• 9:10 - I love Kubernetes, but it can be complicated if you only want to run a single app.
• 9:20 - So yes, it's being used in production.
• 9:25 - But it's as much about whether you want to play in that space; what's your appetite for running a new piece of technology on top of Kubernetes?
• 9:40 - There's a lot going on and there's a lot of change happening; some organisations have an appetite for the opportunities created by such change, and they can absorb that, but other organisations need stability and baby steps.
• 10:00 - They may also not have the same problems that some of this is trying to solve.

So how do you define a microservice?

• 10:45 - It's varied a bit over the years. What I currently define it as: independently deployable services modelled around a business need.
• 11:10 - I used to use the word small, but the micro in microservices suggests size already.
• 11:20 - James Lewis spotted people deploying SOA who were writing small scoped services that could be written and thrown away.
• 11:50 - They also happened to have a small number of lines as well, and were initially called micro apps.
• 12:20 - It was the independent deployable aspect that meant they were easy to replace.
• 12:30 - The size isn't important; it's the independent nature that is important.
• 12:40 - Both Google and Netflix self-identified as being companies that were deploying SOA, now microservices, and their services aren't 50 lines of code, they are several thousand.
• 13:05 - So instead of focusing on lines of code, having more services can be more flexible. So size does help to influence the number of services you have.
• 13:20 - James is fond of saying "Microservices buy you options."
• 13:30 - The flip side is there are loads more problems and things that need to be managed.
• 13:45 - So over time I realised the key part wasn't size, it was making them independently deployable.
• 13:50 - I focus much more on how many services you have rather than the size of the individual pieces.

What does defence in depth mean to microservices?

• 14:30 - When you take a monolithic application and deconstruct it to multiple microservices running over a network, you effectively create multiple potential boundaries at which your application can be defended.
• 14:55 - One advantage is that you don't have all of your data in one single database any more as it can be split up.
• 15:00 - The main example I use is an on-line CD selling shop.
• 15:10 - My customer personal identification information (PII) is in one system, my payments are in another system, and the catalogue is in yet another system.
• 15:20 - Those service boundaries can be defended independently of one another.
• 15:25 - If an attacker breaks into one of those systems, they haven't necessarily penetrated another.
• 15:40 - This allows more attention to be placed in the areas that are more important.
• 15:50 - If all the data is in one place, then I am as worried about my least sensitive information as my most sensitive information.
• 16:00 - For example, if attackers break into my catalogue, then they can gain some information about whatever is being sold - but in most cases that information is on the website anyway.
• 16:15 - I don't want them to manipulate that information, but I don't particular care if they get it.
• 16:20 - On the other hand, the customer PII is very sensitive - I want to place a large degree of care and attention to defend that dTa.
• 16:30 - Microservices allow the defence in depth to focus the attention to those parts or for gaining certification by restricting the systems that are covered by it.

But the data is being transferred over the network now?

• 17:00 - Microservices give and they take away. Data that used to be in one process on one machine is now flowing over the network between nodes and multiple processes.
• 17:15 - That dramatically increases the surface area of where that information can be read or manipulated, and you also open up some interesting problems about being able to masquerade as clients or services.
• 17:30 - Early on, people were running Microservices over their internal corporate network, without any particular care of who could access them.
• 17:40 - So it took a while to get the point across that anyone who could gain access to the network could gain access to that information, and in some cases, manipulate it.
• 17:50 - The nastiest attack could be to join a Kubernetes cluster as a node, and offer to start running jobs, and suddenly people are running workloads on a compromised machine.
• 18:15 - So you have the potential to create systems that are more secure; but with a naive stance around security you could create systems that are less secure.

Does this mean using HTTPS everywhere, or are there other considerations?

• 18:30 - Well, it depends on the protocols that you are using but if you are using HTTP then you should.
• 18:45 - But it's important to realise what HTTPS gives you - it's great on the public internet as it gives you trust of who you are talking to, but on your corporate internet it's giving you the same thing.
• 19:10 - That's only one-way trust (assuming you have done the right thing with certificates of course). You have the knowledge that data isn't changed in transit.
• 19:25 - However, you have no client-side guarantees. You can't stop a random person connecting to the service.
• 19:55 - Unless you are also validating that the client is allowed to connect, then even if you are running over HTTPS you don't have a secure system.
• 20:05 - You have to look at client side authentication, such as client-side certificates, HMAC.
• 20:20 - When you have service to service communication - as a producer and a consumer - then you have to know that the producer is someone who you trust and that the consumer is authorised to use the producer. 

What do you recommend to manage certificates in this way?

• 20:55 - Certificate management is always painful.
• 21:00 - For server side certificates, Amazon has a number of interesting systems.
• 21:10 - Services like LetsEncrypt have a protocol which allows you to request non-wildcarded certificates for particular use cases.
• 21:20 - Client side certificate management has fewer tools for managing it; Netflix's Lima was announced in 2015 but I've not seen many people use it outside of Netflix.
• 21:35 - I've seen a few people jerry-rig systems like that using puppet and chef to push certificates out to hosts.
• 21:45 - I've not looked into what the service meshes are doing here.
• 21:50 - Both linkerd and isdio have excellent support for people doing synchronous point to point communication, so its architecture has the potential to make it significantly easier, but I've not looked into what it currently does.

How can you use protection, detection, response and recovery with a micro service?

• 22:35 - You need to do some kind of basic threat modelling first.
• 22:40 - We often the distracted by the technology of a solution without really understanding the risks.
• 22:55 - In the example of the on-line CD shop earlier, the things that I'm drawn towards are the customer PII or payment information such as credit cards.
• 23:05 - What I would do is to define the threat goals, such as an attacker breaking in and getting that information.
• 23:10 - Based on that, what approach do we need to take to protect that information?
• 23:30 - It's often fairly easy to identify the Crown Jewels.
• 23:50 - If you're integrating with a third party payment provider using a browser based handoff to a provider, then you may be able to ring fence it easier.
• 24:00 - If the credit cards are flowing across your network and you're integrating with a back end payment provider, then you have to think about the vulnerable data flowing over your network, and immediately becomes something you should care about.
• 24:20 - If you have that data flowing over your network you have additional PCI requirements as well as a duty of care.
• 24:40 - If you're using HTTPS then you can prevent the information being sniffed over the wire, or at airport lounges, but their are bunch of other issues you have to deal with.
• 24:50 - For example: what happens when that data stops flowing around? If it's in memory on a machine can an attacker gain access to it?
• 25:00 - I was doing some research the other day - apparently 44% of data breaches are caused by unpatched machines.
• 25:05 - So these vulnerabilities were caused by things that were already known about and a fix was available but it just hadn't been deployed.
• 25:15 - If the data is being breached because the machine hadn't been updated, then that's a really basic thing to solve.
• 25:25 - So now if the data is stored into a database, are you worried about the data at rest? Is it encrypted?
• 25:35 - In Kubernetes before 1.7 you could store secrets - but they were stored in plain text in etcd - so if you could gain access to etcd then you could gain access to all the secrets.
• 25:40 - You have to think where the data is going to sit.
• 25:50 - These are really basic things - keep your machines patched, think about data movement, think about data at rest - those low hanging basic prevention mechanisms.
• 26:00 - You then move on to the more subtle problems that may be easier to miss but need a more sophisticated attacker.
• 26:10 - One such example is the confused deputy problem.
• 26:15 - The way to think about it is like the certificate problem from before; the user of a service really thinks that it is the service it should be.
• 27:30 - If you have a user service and a website service, the data needs to be authorised as well. It's not sufficient for the website service to know that a logged in user is requesting personal details; it can't simply substitute another user's information.
• 28:00 - So it's important that services don't just proxy requests blindly; either the user service needs to be hardened so that the website service can't let the information, or the website service needs to know that the content of the logged in user isn't entitled to see that information. 
• 28:25 - There's an important difference between authentication and authorisation. 
• 28:30 - If you can trick the website service for asking the user service for information that it shouldn't be able to access, that's the confused deputy problem.
• 29:20 - Typically I would use some kind of interceptor pattern at the perimeter.
• 29:35 - The most common pattern for microservices is using a token based mechanism, which can be independently validated and encode information user.
• 30:00 - This token can then flow with the request down to nested services which can then validate the authenticity as well as what authorisation rights flow with that token.
• 30:10 - When you start having centralised access control lists (ACls) it flies in the face of what microservices are trying to do.
• 30:25 - If you put the information for the user service in the user service it seems silly to go out to a third party system.
• 30:40 - Some systems I've seen need round trips with third party systems at this point: rather than the user service making its own determination, it has to go out and check things again.
• 30:55 - This becomes hugely burdensome beyond the increased surface area for failure with synchronous calls, and it's slow.
• 31:15 - If the token can be validated locally without needing to go to a third party system, and encode additional information in the token as well, so it can take decisions locally.

Does mean the panacea is serverless architecture?

• 31:45 - Potentially - but it's not the whole story.
• 32:00 - For the server itself, the patching is being handled for you, but your application is still running on that architecture along with its run time and libraries.
• 32:10 - If you're using Python or Java or another library, those need to be patched as wel as well to avoid security vulnerabilities.
• 32:20 - For the application world there are a lot of good tools; for example, snyk.
• 32:30 - They look at the maven dependencies of your application and check the versions of the libraries and when they are out of date or if they have CVE vulnerabilities listed. They can even send you a GitHub pr to update the creation automatically.
• 32:50 - These services are out there,but you have to monitor your applications dependencies and patches, not just the server ones.
• 33:30 - For security in general, with function as a service it's mostly a good story.

What are the kind of new security issues that you might not be familiar with for serverless architectures?

• 33:50 - A lot of the incident based services use IP addresses for identification, which doesn't apply in a serverless architecture.
• 34:05 - A function is something that will potentially run in the future rather than right now.
• 34:20 - You're looking instead of how it is running now to the configuration of how it might run in the future.
• 34:30 - You're moving from an IP based security to an API based security. Configuration such as how you have set up your job in Amazon is more important.
• 34:40 - Mostly the security story is good. Amazon in particular has done a good approach on this - secure by default is a good idea.

Mentioned

• CloudFoundry 
• Istio 
• Kubernetes 
• Linker-d 
• Service mesh 
• Book "Builduing Microservices"