Gareth Rushgrove on Kubernetes as a Platform, Applications, and Security

Show Notes

Can you introduce yourself?

• 01:00 I'm director of product at Snyk; we do security tooling for developers covering open-source components and containers.

Are there core capabilities of a platform based on Kubernetes?

• 01:45 One of the things with Kubernetes that surprises a lot of people is that it is both a low-level abstraction and a higher-level abstraction.
• 02:00 If you are thinking of Kubernetes as your future application platform, and it turns out that Kubernetes has got low-level building blocks, then you have to build something on top of it.
• 02:10 On the other hand, if you are coming at it from a migration path of a bunch of VMs, the Kubernetes API is incredibly high level.
• 02:30 It depends on where you are coming from.
• 02:35 If you buy into the idea that you're building a platform on top of it, what is missing in the Kubernetes API is the application.
• 02:45 You've got Services, you've got Deployments (which adds an abstraction layer over replication sets and pods) - but how do you organise teams, source code, CI around?
• 03:10 Kubernetes doesn't have that built in, but there has been moves in that direction for a while.

What's the state of Kubernetes tooling?

• 03:30 There's a lot of opportunity; that was what attracted me to Kubernetes 4-5 years ago.
• 03:35 The first talk I did at KubeCon was called "Kubernetes and the potential for higher level tools" - and I still fundamentally believe that will happen.
• 03:45 We haven't moved towards it as quickly as I thought we might, and to a certain degree that's also down to wide-scale adoption.
• 04:00 It's not just the high level high context people who were interested in Kubernetes because of its API surface; lots of people have been using Kubernetes to get work done.
• 04:10 Because that potential was there, you do see some of those tools; some have come and gone, some were too early.
• 04:20 Most people haven't needed the high level tools yet, and that's led to a certain degree of fragmentation.
• 04:30 I think what's happened was a lot of people jumped on to Kubernetes too early, before the tooling existed, and built their own tools.
• 04:40 Kubernetes was API-first, and lots of people built their own tooling, which was relatively straightforward to do; you'll see some of that consolidating over time.
• 04:50 KubeCon is a great proxy for that; in Barcelona, something like 80% of attendees were new to the event, and in San Diego, something like 70% were new to the platform.
• 05:05 It's easy to forget that most people are new to Kubernetes.
• 05:10 In that context, if you come to the platform and find the array of tools bewildering, that can be a problem.
• 05:15 I created a spreadsheet a while back, partly as a narrative of what tools were built on top.
• 05:30 I don't think too many tools have been created; the ones that see large-scale adoption will win, rather than the ones with the most interesting functionality.

What's your thoughts between pure open source and CNCF as well as vendors?

• 06:00 I think they can be quite individual and come from a different point of view.
• 06:05 Some of it comes from what you think of open source, and whether you think open source is important.
• 06:10 I came back from FOSDEM a few weeks ago, and there were lots of conversations about what open source is.
• 06:20 When I got into open source, it was about the freedoms for me; we've had conversations about licences and maybe open source has become synonymous with free, but we're coming back to licenses again.
• 06:40 I think you'll have people coming from different points of view.
• 06:45 I believe open source is important.
• 06:50 I don't like the idea of a world where all the open source software that becomes popular and sticky is spat out of large corporate organisations.
• 07:00 It adds value to the ecosystem; but I like it where there's a mix.
• 07:05 I've learnt that building big software takes a lot of people, and gathering that amount of people together for free take a long time and is hard.
• 07:15 Doing it commercially is easier; it doesn't mean it's easy.
• 07:20 I like the mix that you can go out on your own and create software, or with a small group of people.
• 07:25 You can see open source projects coming from start-ups, which might succeed or fail, and from end users.
• 07:40 The interplay is interesting, and I think that relationship is all about the end-user first.
• 07:50 If you don't focus on the end user first, you're not going to succeed.
• 07:55 If you build something for yourself, you are the end user - it's really easy.
• 08:00 If you're building for someone else, whether it's for a startup or a giant corporation, if you're not putting the end user first then over time it won't work out.
• 08:10 If you have a lot of money, you can get somewhere quite quickly, and you can squash things, but on balance it might not work out.
• 08:15 Foundations have a different role to play, and work differently - CNCF is central to quite a lot of things in the cloud native space.
• 08:30 People like the idea that the foundation is making lots of decisions, but actually what's happening is it's lots of people in lots of projects that are making decisions.
• 08:35 The CNCF provides a place where lots of the technology people can talk, even though they might be from competing companies.
• 08:45 That's the benefit of the foundation.
• 08:50 The other aspect is it has driven awareness of cloud native applications and infrastructure, and that shift across lots of companies (including small ones) is a benefit to everyone.
• 09:10 Large companies, which might not listen to individual users, can be associated with a group to move towards automation and devops - all of those things, we can agree on under one banner.
• 09:45 When we get down to the specifics of how to offer a specific thing, we can offer competing solutions.
• 09:50 That's the key benefit of a foundation; people may look down on marketing, but actually it turns out it's really powerful when done well.
• 10:00 The foundation is also a place where people who are employed by fiercely competing companies can talk about things together, even though they work for different companies.
• 10:10 Technically it just needs a common project, but the foundation provides things and opportunities that wouldn't exist otherwise.

What is the role of standards in this?

• 10:30 Standards get a bad reputation, and people run away - I don't think that's going to change.
• 10:40 I would like to see more work done around standardisation.
• 10:45 What we're seeing is people trying to build de-facto standards.
• 10:50 Where that works, it works really well; the projects around the cloud native space are generally to do with containers; securing them, managing them, building them, using them as an abstraction.
• 11:15 That fundamentally happened down to a widely adopted implementation in Docker, but the smart move in standardising the low-level bits with the OCI, the runtime image formats, etc.
• 11:40 The number of companies with container registries expands every few months, and the reason why it's not a problem for users is because they're all fundamentally based on the distribution spec.
• 11:50 It turns out that they aren't 100% compatible; there's always things that people will do either on purpose or not on purpose, but for the most part they are transferrable between registries and runtimes.
• 12:10 Google cloud run enables you to run containers here or there, hypo was doing that before perhaps ahead of its time; and all of these were based on standards.
• 12:15 You don't want too many standards, and you don't want to create standards first and then hit people over the head with them.
• 12:25 I do think specification standards we can all agree on common undifferentiated things is a good space.
• 12:30 The serverless working group around Kubernetes around the cloud event spec have been doing things there.
• 12:45 They've been finding areas in which they can agree, and writing those down as lightweight standardisation processes.
• 12:55 I was recently involved in the Cloud Native Application Bundles (CNAB) spec at the beginning, with Microsoft, Docker, Pivotal etc. about the higher level packaging.
• 13:05 A lot of that doesn't have to be end-user facing, either; there's a lot of people who are using Docker without knowing about OCI - but they get the benefits of that.
• 13:20 There are a lot of people wondering how they might use it, but instead, should be wondering which products to use that might use it.
• 13:25 Most people don't need or want to care how it does it; they care about the products solving the end user problem, which is the way it should be.

What are your thoughts on the good and bad of buildpacks?

• 14:05 Software packaging and container builds are important.
• 14:10 One of the things Docker brought it at the right time and right place was to move software packaging from a specialised part of the IT department to self-service.
• 14:40 Docker packaging shifted really rapidly to developer teams doing packaging all the time.
• 14:50 There are tradeoffs about how that happened; ultimately with docker build as the tool and the OCI specification as the result.
• 14:55 One of the things you can't argue about is how widely adopted and simple it was to make that transition.
• 15:05 IF you talk to the Debian packaging people, they'll list off a whole set of things that are wrong with the Docker files, and they have a point.
• 15:15 There are several million Docker files on GitHub - it really helped to democratise packaging.
• 15:30 There are tradeoffs; so the question is, can we have other types of build tools?
• 15:40 Jib is a specialised tool from Google for the Java ecosystem.
• 15:50 If you can remove a lot of the general purpose nature abstraction; as long as you fit into this gap, and the tool will take care of the rest of the complexity for you.
• 16:10 I think buildpacks have ultimately been of interest for those that have already been using CloudFoundry.
• 16:15 As a direct end-user tool, I think it has some downsides; where I see much more interest in buildpacks is being built in to something they are already using.
• 16:25 Buildpacks aren't the end user interface; they're an implementation detail - so I really like the recent Spring integration.
• 16:40 If it gets built into the latest Spring projects, it will automatically build your Spring project into a container using buildpacks.
• 16:50 You don't need lots of things being able to do that; you just need something that has encapsulated all of those problems.
• 17:00 There's other build work going on as well; the evolution of 'docker build' into a library called buildkit; lots of software is now using buildkit to do builds without needing docker.
• 17:15 That's then plugged back into docker as the next generation of the build process.
• 17:20 There's still some integration required for Windows containers, but there's already a lot of improvements about syntax and extending the Dockerfile syntax.
• 17:30 For me, the Dockerfile is a great blunt tool for getting to the 80% goal very quickly with a low barrier to entry.
• 17:40 I think we'll see tools that take a different approach; single tools around specific ecosystems like buildpacks which generate an OCI image.

What do you think a best practice CI/CD pipeline looks like for Kubernetes?

• 18:10 I think the reality is that it is based on your organisation context.
• 18:15 There are multiple projects in the CNCF and the CDF from projects and end-users to large vendor-backed projects.
• 18:30 What it means is that there's a richness to the tooling, but everything is moving really quickly
• 18:35 It's probably also why I would wager that Jenkins is the tool that everyone's using for doing their builds.
• 18:45 Everyone's using it, and can make it do what they want.
• 18:50 In a greenfield environment with a lot of Kubernetes experts, the answer is going to be different from that from one in which there are SaaS solutions.
• 19:05 I'm interested in Tekton [https://github.com/tektoncd/pipeline] (a project stewarded by the CDF), built by Google, RedHat, and other folks.
• 19:25 What it's doing is providing the elements for software build on top of Kubernetes.
• 19:30 What it adds to Kubernetes is pipelines, tasks, as constructs in the same way you'd have deployments, pods and services.
• 19:40 For me, that's serving an interesting problem; at that point, you could have interoperable tools on top which are using those details.
• 19:55 I think that might be how we get to genuinely better tooling.
• 20:00 There are other tools available; Argo and ArgoFlux is taking a slightly different approach.
• 20:10 I would love to see other things built on top of Tekton, because the interoperable bit appeals to my nature.
• 20:15 The Kubernetes-native objects is a useful API abstraction.
• 20:20 It's a powerful abstraction, but there's a high barrier to entry.
• 20:35 Viewing these as as end-user tools is probably the wrong level.
• 20:40 Viewing the tools that use these things is probably the correct way of doing it.
• 20:45 If people feel that Kubernetes is complex - but there are a lot of people who are using Kubernetes without attending KubeCon - they're not using them and they don't find it complicated.
• 21:05 The moment you add custom resource definitions and operators, the complexity goes through the roof.
• 21:10 I feel that they're an implementation detail of what can be done, but not an end user visible thing.

What do you think are the biggest security challenges with Kubernetes?

• 21:40 Security problems are always socio-technological; they're always a mixture of both people and technology.
• 21:45 It's nearly impossible to say which one is more important, because it will depend on the actual problem at the time.
• 21:55 We talked about previously, about the shift of software packaging to something that developers own (and associated problems).
• 22:10 We are seeing it move from a smaller number of people who bear all the responsibility and assume all the load and act as a gate to good or bad is breaking down.
• 22:20 Security is something that is shifting towards being a developer problem.
• 22:25 That isn't to say - in the same way as software packages - that there aren't expert people who know how to do the high-end bits.
• 22:30 Security definitely needs security experts - with the shift to devops, the sys admins didn't disappear, they became SREs.
• 22:50 You have seen the value and appreciation of SREs grow over time - security is going to go through the same existential crisis.
• 23:00 The security people are going to become the new security experts, and instead of playing whack-a-mole with thousands of things and focus on the high-risk, high-security problems.
• 23:15 Kubernetes is happening in the space where there's a lot of things shifting; packaging, operations, configuration, security etc. is being pushed down to developers.
• 23:25 So what are the security tools and problems that are fundamentally application security problems, but they spill over into historically what would have been IT and infrastructure space.
• 23:40 The line of IT owning databases and load balancers and the devs owning the applications is breaking down; the developers are bringing their own databases and owning more of the data.
• 24:00 It's in that context that all of this is happening; developers need help to understand what is happening, but security needs new tools and to understand the evolving space.
• 24:15 The fundamentals, like ephemeral API driven infrastructure, is the new normal, and it's not sufficient to (say) to get a list of all of the IP addresses?
• 24:30 The problem is that the spreadsheet will be out of data before it's even used, so is it useful?
• 24:35 So it moves the discussion to security fundamentals - one of the main things is to go back to basics, and do a thread modelling exercise.
• 24:55 You'll then discover the problems as you go through that process.
• 25:10 There's a CNCF SIG about security, and there were a security days at KubeCon which was about a cross-section of end users.
• 25:30 The work that's going through that community between all stakeholders and security people is important.

If I'm new to the security space, are there any quick wins?

• 26:10 One is understanding that Kubernetes' defaults aren't that secure.
• 26:15 That's something the Kubernetes project acknowledges, and it isn't as much a mistake as much as hindsight of massive success.
• 26:20 If you understand the defaults, like things are running by default as privileged or with writable file systems; if you can avoid those, you can remove a lot of attack vectors.
• 26:40 Another is that a pod can use as much as it wants to on a node, and the other things get thrown off; you end up with risks around resource starvation.
• 27:00 Adding limits can mitigate some of those problems.
• 27:05 You need to address the secure defaults on a per-application basis.
• 27:15 The biggest problem aside from people deploying Kubernetes control planes to the internet (don't do that!) is the security of the application itself.
• 27:30 People might not attack Kubernetes, but if the applications have SQL injection problems or cross-site scripting problems, the most secure Kubernetes platform won't help you.
• 27:55 You don't win by protecting the data or compute if you don't also protect the application you put onto the platform.
• 28:00 Ultimately thinking about the security of the applications and vulnerabilities they may have are the best places to start.
• 28:20 It doesn't require you to buy into a lot of tooling to make it happen.
• 28:30 If you are in a large organisation, and have intrusion detection systems or business intelligence solutions you can get your data in to, you should use those.
• 28:35 You still need a level of expertise to use the data and find the signals.
• 28:45 There are things on the developer side that you can scale out faster to cut down the signal in order to help.
• 28:55 Bang for buck, secure your applications and understand that Kubernetes isn't secure by default are the two main ones.
• 29:00 Ian Coldwater did a talk on "Think like an attacker" 
• 29:15 Ian also did a talk from Duffy Cooley from VMWare on the problems of a security point of view of Kubernetes defaults 

What is the best way for people to follow you on-line?

• 29:35 I'm garethr on most social media Twitter, GitHub, Speaker Deck 
• 29:45 At Snyk you can see things we're building on this space.
• 30:00 I also send out a weekly newsletter.