Helm 3: A Mariner's Delight

Transcript

Evenson: We're talking all about Helm 3 today, and we've titled this talk "A Mariner's Delight," because as I think of everybody sailing on the high seas of Kubernetes, hopefully, Helm is the tool that you know and love and could get used to. We're going to focus very heavily on Helm 3 and what that brings to the table and also go back and learn a little bit of history about Helm 2 and teach you what Helm does and how it can be useful to applying packages to Kubernetes. With that, let's get into it. We got a packed agenda I'm going to go through soup to nuts. There's going to be demos and everything. If you can hold your questions to the end, I promise I will answer them and we'll get to them.

First of all, a little history on Helm. We're going to go through why is Helm important and why does it exist in the Kubernetes ecosystem. We're going to look at Helm 3, which is the next iteration, next major version of the Helm release, which everybody is eagerly awaiting. I'm going to teach you about Helm 3, how it overcomes a lot of things, problems in the ecosystem that Helm 2 identified. Hopefully, everyone will give me a standing ovation at the end. That's the hope. We're going to go through breaking changes, new features, increasing reliability. As you release and use Helm to release your applications on Kubernetes, we're going to talk about how you can use it in a CI/CD context and actually get increased reliability using some open-source tools. Then we're going to go on to what's next, what does this mean from you and where do you go from here. I will be around at the conference, happy to talk to everybody after this. If you have very specific questions, I am more than happy to answer them.

First, who is this guy up on stage, talking to you? My name is Lachlan Evenson. I look after open-source program management in the cloud-native compute foundation ecosystem. You may have seen me from such communities, such as the Helm upstream charts maintainer. I do that. I make sure and I saw on the Octoverse report yesterday that helm/charts repository is number six in changes to open-source computers over the last year out of all the repositories on GitHub. It is high churn, Helm is a high-use tool. It's very popular in the ecosystem. Also, the Kubernetes 1.16 release lead, so if you've got any feedback for that or you want to learn how to be a release lead yourself, happy to teach you on that. It's certainly a labor of love, but you can thank me for all those wonderful things that went into Helm Kubernetes 1.16, or not. Happy to take the brunt either way. I'm also a CNCF ambassador, and what that means is you can ask me about any CNCF projects. I'm happy to talk to you about any of them and what they do in the ecosystem. Feel free to ask me about those things.

Why Helm?

First, let's get into what Helm is and how it fits into the Kubernetes ecosystem. Simply put, Helm is a package manager for Kubernetes. What does that mean? The best way I like to think about Helm is, if you're a Mac user, everybody knows Homebrew, everybody's used to apt, yum, de facto packaging for applications on different platforms. Think about Helm as a package manager for applications on Kubernetes, specifically. Helm, the project, was donated back to the cloud-native compute foundation in June of 2018. It's used by 68% of the people, according to the last CNCF survey, in the ecosystem to deploy applications to Kubernetes. It is the third most popular ecosystem tool in the cloud-native compute foundation tool belt, only third to Kubernetes and Prometheus. It's a fairly well-known tool. If you're not familiar with it, take a look at it. It's used in a lot of different places and a lot of different companies publish this software using Helm charts to be able to have a one-click install for their packages.

What does that this really mean under the hood, where the rubber meets the road as you're practicing using Helm? It allows you to define your applications and template them, because as we know, in Kubernetes, it doesn't have the notion of an application. It has notions of individual building blocks of applications, services, deployments. Helm allows you to package them all up and say, "This is my WordPress application as I know it," which consists of services, deployments, discs, so on and so forth, and allows you to install, upgrade, rollback, and have all those common lifecycle verbs at your fingertips that you know and love from other package managers. It also allows you to build chart repositories. The packaging format for Helm is called charts, and you can build a repository of charts that people can then leverage. You can distribute your software with charts internally and externally.

As I mentioned before, I am the maintainer of the upstream charts repo, which is a set of curated charts in the ecosystem that allows you to have one-click install of any popular software on top of Kubernetes. You want a Jenkins, you can Helm install Jenkins. You want a Redis Cluster on Kubernetes, you can Helm install Redis. Charts allow you to package up and share your applications with other people or different organizations. It's reusable, repeatable, and packaged up.

V3 Overview

I'm going to go straight into Helm 3, because we've got a loaded deck on Helm 3 specifically, and I think that's the most exciting thing that most people are here for today. What have we learned? Helm 2 has been around for several years. I think circa 2016, Helm 2 was released prior to Helm 1. Helm 1 was completely client-side, Helm 2 went client-server, and Helm 3 is going back to client-side. More on that in a moment. What we've taken as a community is all the best practices feeding back from the Kubernetes community to how they want to deploy apps and rolled it into one piece of software, which is Helm 3. We used Helm 2 as a sounding board for how people want to use applications on Kubernetes, how to package them, and rolled all that feedback into one binary. Let's see if we get everybody excited about that. We've also made it really simple, much more simple than Helm 2, so there's a lot less moving parts, and I'll show you what that means. We've focused on security as a priority. A lot of feedback is, "I want to use Helm, but I cannot use it in my production environment because the architecture is insecure in Helm 2." We will show you how we've addressed that in Helm 3 so that you can use Helm in production and know that it is secure.

In order to get to Helm 3, we needed a major refactor of Helm. Who was around in the Kubernetes 1.1, 1.2 days? If you know that there was no RBAC, there weren't even deployments, there wasn't even ingresses. A lot of the resources didn't exist that you have today in Kubernetes. Helm comes from that legacy of very early boilerplate Kubernetes, where it was designed where there weren't many security controls. As the Kubernetes ecosystem has grown up and added security controls, it has exposed the inherent insecurity of the architecture that Helm employed prior to having all these things. We needed to go back and look at this. We also didn't have custom resources. Kubernetes just shipped an inbuilt set of APIs, and you could not extend them. The first version of extendability to bring your own APIs was back Kubernetes 1.4, and they were called third party resources. If you go back, they were horrible. Thank goodness, we have CRDs today and we also have RBAC. We're able to use the tools under the hood that Kubernetes provides and roll them in as native features to Helm. Again, focusing on simpler, more secure, and production-ready. One of the other things we wanted to do was make Helm more Kubernetes-native. Now, people have grown up with Kubernetes, and they're used to a specific vernacular and way to interact with Kubernetes. Helm was kind of a little off on its own world. What we've done with Helm 3 is brought it back in the line of what people know and expect with Kubernetes. When you're using kubectl, you expect a very precise experience, and we've modeled that experience in Helm as well, so that you can be comfortable coming from the Kubernetes-native tooling into the Helm ecosystem. We've inherited security controls from kubeconfig, which allows your identity to access the cluster, which we did not have in Helm 2. Using RBAC and replacing custom APIs for charts and deployments with secrets. We've removed a lot of custom code and rolled that in the Kubernetes-native resources, meaning we don't have to carry a lot of that code in Helm itself, which brings me to Tiller.

Tiller in Helm 2 was the server-side component, which exposed a lot of security risk. Basically, it sat in cluster, and I like to call Tiller the first Kubernetes operator. It basically interacts with the Kubernetes API, but it was doing so in a way that was insecure. It allowed unsolicited, unauthenticated commands to come into it, and then it would execute directly against the Kubernetes API, often as a root user. That made everybody in the Kubernetes ecosystem fairly uncomfortable as RBAC became a thing and as many other tools that we know and love in the security aspect of Kubernetes. We've decided with Helm 3 to actually collapse Tiller and remove the server-side component, and that took out a big security risk in the architecture, which was we were using our way to basically pseudoroot into your cluster. We've gotten rid of that, and Helm now operates client-side and directly accesses the Kubernetes APIs.

The released objects are also rendered client-side as well and stored in a secret. I'll show you how that all works. Really, not having this server-side component means people can get something out of Helm very quickly. They can download the binary, run an install, and not be reliant on a server-side component already being installed in the cluster. Cluster administrators can actually determine what you can install where, lowering the barrier of entry there.

As I mentioned, we made Helm 3 a lot more Kubernetes-native. With this, we actually employed a whole new set of verbs. Worry not, we aliased all the old verbs to the new verbs. What we wanted to do was make sure that most things operated, but there are some breaking changes, which I will show soon. Basically, helm delete is now helm uninstall, inspect is show, fetch is pull, search is search repo, and purge is now the default. Purge, when you're deleting, is now the default to remove all the released artifacts from the server-side where they're stored. The idea is that people can come in and drop in the Helm 3 binary in place of the Helm 2 binary and they have an experience that works and allows them to move over to the verbs in Helm 3.

Breaking Changes

Now, we're going to get into where the rubber meets the road. I'm going to share the breaking changes. There is a link here that is a doc that details all the changes between Helm 2 and Helm 3, so you don't have to heed on to my every slide. These are all detailed on the Helm website as well, where you can find all this information. I'm going to go into particular depth about all these things, but I want everybody not to panic. The idea here to move to Helm 3 was not to segregate the community and have a Helm 2 ecosystem and a Helm 3 ecosystem. It was to bring everybody over to Helm 3. The intent with the move and the community work was to bring all your Helm 2 artifacts, bring everything you know and love about Helm 2 over to Helm 3, and then you can keep that experience going. The most often question I get asked is, "I've been using Helm 2 for years. Can I move to Helm 3? Do I have to rewrite everything?" The answer is no, you should be able to drop in the Helm 3 binary in place, and I urge you to do that if you're using Helm today, if you take into consideration the following things I'm going to state. Don't worry, they're fairly low.

Let's go into namespace changes. A release is when you install an application, a packaged application on Kubernetes. That constitutes a release, and you may have many releases as you upgrade, rollback other specific application but that metadata. If I install an app and Wes installs an app, that metadata about who's installed it where and what the release information is stored server-side in Kubernetes as a secret. That is different and it is stored in the namespace that you release that application to. This is a behavioral difference from Helm 2 to Helm 3. In Helm 2, that release metadata was stored where Tiller was actually resided as a ConfigMap, usually in the kube-system namespace.

Templated resources. Helm packages are templated Kubernetes resources. If you'd specify the --namespace flag, it will annotate those resources with the namespace. You can say, "I want to deploy WordPress in the Lachy namespace," by saying helm install --namespace Lachy, and it will go and do that. Caveat to that is it doesn't actually create the namespace on your behalf. We did that in Helm 2 as an ease-of-use, but that is anti-Kubernetes. You should not create any resources people don't ask you to do. I'm going to model the behavior in Helm 3 in a demo shortly, but you need somebody to create, whether it's yourself, that namespace. Again, we're detaching use cases and different personas. There's a developer who wants you to install an app and there's a cluster administrator that says, "Here's a namespace for you to just deploy that application." We assume somebody is issuing namespaces. If you wear both those hats, you need to create the namespace. As I said, don't be scared of that. I'll show you how it works. Again, we're modeling kubectl create where it won't create a namespace on your behalf.

Now, people ask me, "Why do you do that?" That's really a pain in the butt. If you want to do that with Kubernetes, you can use an admission controller called the NamespaceLifecycle controller, which will actually, if you create a resource and a namespace that doesn't exist, it will create it on your behalf. Again, it's not really best practices. If you're used to that kind of experience, you can recreate it using Kubernetes, but if you're not, we hope that somebody will come and create that namespace on your behalf.

Chart dependency management. Like all package managers, if you have dependencies on other packages, like I have WordPress and I need a MariaDB or a database, you can say, "I take a dependency on the chart for MariaDB or MySQL," or whatever it is that supports the data storage for WordPress. Now, what I want people to think is dependencies were added to Helm 2 very late in the game so that we're bolt-on. They weren't actually part of the chart definition. We bolted them on. What we've done with Helm 3 is we moved them into the chart definition itself. If you're using dependencies in charts, the toolchain should still work, but some of the subcommands will break under Helm dependency. That command is actually changed to Helm package, but most of the workflows that I've seen continue to operate if you're taking a chart from the old style to Helm 3, the old style being a Helm 2 chart to Helm 3. Just something to note, but if you are to build a chart today with Helm 3, the dependencies would be reflected in the chart metadata itself, not in a requirements.yaml and a requirements.lock, as they previously used to.

CRD installation, so custom resources. Many charts chip with custom resources in-built. For example, the Prometheus operator, where I can configure Prometheus from a custom resource, will ship with a set of custom resources. In Helm 2, we model those custom resources as CRD install hooks. We say, "Here are some CRDs that you need to install prior to installing the operator, which looks at those custom resources." This is about order of operation. In Helm 2, they were all predefined as these hooks, which were essentially annotations on the resource that said, "Please install these before you install the application or the operator," as it were.

In Helm 3, you'll get a lovely warning message to say, "You've got CRDs, but I'm not installing them." What Helm 3 expects is that you put these in a directory at the root of the chart called CRDs, and it only has custom resource definitions in there. You'll get a lovely warning message. Don't worry, but you need to move those CRDs. This is a big thing for people using Helm 2 charts that'll eventually be updated, but as you can see over on the right, you can see the CRDs directory with crontab.yaml that would contain the custom resource definition for crontabs. Everything else is the same, but if you find that you're installing a chart and you get a nice warning message, "By the way, I didn't do your CRDs," this is exactly why. You have a Helm 2 chart with a Helm 3 binary.

Release metadata – I briefly touched on this. It's no longer stored in the same namespace as Tiller, because there is no Tiller. It's stored in the namespace that you're deploying the release to. Also, it's stored as a Kubernetes secret, and it is double base64 encoded. I mention this because most people will go and pop the hood on this afterward and say, "What is that thing?" It's double base64 encoded because we have different storage back ends for Helm. Kubernetes secrets happens to be one, but we have MySQL that wants a base64 encoded blob put in. It doesn't do the translation. We encoded on the in-code before we store it in there. If you go and grab and pop the hood on that blob, you'll see you need to decrypt it twice, and there'll be a big JSON blob with all your release metadata, so who installed it, when it was installed, what exactly happened. If you want to go and pop the hood, you're going to have to double decode it.

Finally, it is not backwards compatible with Helm 2. The Helm ecosystem has published a Helm 2 to 3 plugin. If you have a cluster and you've installed packages on it with Helm 2, you run this plugin, it will migrate that release metadata to the Helm 3 format so you can pick up the Helm 3 binary and actually introspect those applications as they run. There's been a lot of work in the community to make sure that people can actually bring their released artifacts from an operating Kubernetes cluster without having to say, "You got to greenfield it and start again." Check out that tool. It's written by members of the Helm community that we're very thankful for. It basically converts from the old release metadata to the new release metadata.

Deprecated functions. I felt compelled to actually mention one deprecated function here, Release.Time. People embed functions in their templates to generate different aspects of their templates. Release.Time wasn't very configurable, but now, you can just call them "now" function and dictate how you want your time to be sliced down. No big deal, but if you have Release.Time there, you will get an error message and say, "Please go and use now."

Generate-name. If anybody's done a Helm install without giving it a name, it generates these quirky little animal names, like crazy emu or something like that. We now have changed that not to be the default behavior. Again, the paradigm in Kubernetes is to do exactly what's asked. We were generating things we didn't feel that was in line with the Kubernetes ecosystem. We've gotten rid of that. Now, you actually need to say, "I give it this name." If you don't want to give it a name, you have to explicitly say, "Generate a name." If you have Helm in pipelines or running off code, it's already there, and you pop this in, and why isn't it working? Most probably, you're relying on the generated name. You're going to need to pop that flag at the end.

Helm 3 End-to-End

I'm going to do an end-to-end demo. I actually recorded this so I could talk, because I can't talk and type at the same time very well. I wanted to do you justice with the demo. I think what you'll find is a Helm 3 is very much similar to a Helm 2 demo. I'm going to go through everything here to show you the end-to-end experience to install an app and what it looks like in Helm 3 as opposed to what people are familiar with in Helm 2.

I'm going to show you how a Kubernetes cluster running 1.16.2, and I have Helm version 3; 0.0-rc.3. What we're going to do here is I'm going to show you a list of repositories. This is the upstream stable repo. I'm going to run an install of WordPress. I'm going to call it wp and take the stable upstream WordPress chart and run an install using Helm. I'm installing WordPress on Kubernetes with a single command. We can see that that's actually been deployed, and then I will actually go and show you the resources using kubectl. Here are the pods where you see we have a MariaDB and a WordPress pod there. We can see we have a release in the default namespace of the WordPress chart. I will show you the release metadata here. You can see the secret named release.v1.wp.v1. For posterity, let's go ahead and pop the hood on that. You will see the data payload is a big double encrypted JSON blob with base64. Not encrypted, encoded. Don't want to make that mistake.

I'm going to show you doing an install of the NGINX Ingress, which is another popular chart in the Helm ecosystem, into a namespace called nginx-ingress, which does not currently exist. Just to hop on there. Namespace not found. I haven't done anything, so what I need to do is, either an admin or if my user allows, create that namespace, and then I should be able to go back and install the NGINX Ingress into that specific namespace. We can see that that now has deployed. I will show you the deployment status. We've installed that. I'll go and show you the pods using kubectl get pods in the namespace nginx-ingress, and we should see that we actually have the Ingress controller installed into that namespace. Namespace releases, so I did it without a namespace. I only stored default. I did it with all namespaces, so you can see both the NGINX Ingress and the WordPress. Releases are now namespace-specific. Finally, I show you just with the namespace specified. I'm going to do a delete without the namespace. It doesn't know about it. All the releases are namespaced now, not just one global namespace like they were in Helm 2. Finally, I'm going to go and clean up the WordPress release as well in the default namespace. That's been uninstalled and share with you the secret. The release metadata is gone, so it's purged by default, and you can see that the WordPress application is now being deleted and terminated.

Just to reiterate, we went through the whole lifecycle using Helm 3 install/uninstall. We did a list, a non-namespaced and a namespaced, because everything is namespaced now in Helm 3. I also showed you the release metadata where you can have a look at that yourself. That's all there is to it with Helm 3. It's that simple to install apps. The other thing that I was showing is they're all Helm 2 charts from the upstream chart repository that's published under the CNCF. They have not been migrated to Helm 3 specifically, but the tool will just install them as long as they don't have any of those constraints that I mentioned. You can see that many of the upstream charts do not have those constraints. I've gone on breaking changes, now the exciting stuff. Let's talk about what's actually new in Helm 3 and what you can get excited about.

New Features

Chart repository API. In Helm 2, charts were stored in a chart repository. A chart repository was generated from the Helm tool itself, which was basically an index YAML file that needed to be served by a web server. It only supported basic authentication in the client, but in Helm 3, we're moving to charts being stored in container repositories. For those following along at home, there has been updates in the OCI distribution, which is the governing body that takes care of container registries and how they are actually operated. In the distribution spec v2, they have allowed storage of any arbitrary metadata. Now, you can store Helm charts in a container repository, which means you can leverage all the authentication and all the tooling in the container ecosystem to store your charts. Who wants to build another web server to support charts anyway? Nobody. I certainly didn't. With the chart repository API, we ask people to try this out, some public providers support distribution v2, which you need to supply charts and have them stored in the container repositories. You need to figure out who they are, and then you can test this.

By default, this is turned off in Helm 3. You need to enable an experimental flag that says, "Pull your charts from a chart repository that's stored in a container repository, an OCI standard repository." This is great as people managing infrastructure. You now don't need to have another repository just to store your Helm charts. You can store them where you store your containers, and typically, the container registries that are out there are rock solid, whether you're running them on-prem or public cloud. We know that it's a known entity, they're stored, they're backed up. You don't have to manage that on your own. Put all your stuff in the same place. I dare say, you'll see a lot more packaging taking advantage of distribution v2 for storage. Just a word that that's coming. Everything will be stored in a container registry soon. Surprise, surprise.

Chart API version v2. We've bumped the API version to v2. Now, this is a little bit, as I like to say, handwavy. It's mainly for tools and third party tools to say, "This is a Helm 3 chart or a Helm 2 chart." You can install different logic if you're building tooling. As I mentioned, there are dependency changes. They're stored in the chart.yaml. We also have library charts. Let me talk about library charts because I think this is exciting for most people who are using maybe customized and other tools in the ecosystem. In Helm 2, when you define the chart, it needed to install resources. It had to intrinsically have things that are installed against the Kubernetes API. With library charts, you can bump all pieces of your templating or pieces of your helper functions to a library chart that's imported. What this means is, as a chart author, you can import from one place. Let's say your company has a standard set of labels that it wants enforced on all its Kubernetes resources. Go put them in a library chart. They say, "Please, can you add one more label?" You add it to the library chart, and all the charts that import that library chart get that function. Now, we're lowering having to copy and paste things around. Then, if you have hundreds of charts, then you make one change and copying and pasting to 100 charts. Library charts, we call them library charts and application charts. Library charts store functions that you could import and reuse, and application charts actually deploy resources to Kubernetes cluster.

This one, I don't know why I'm really excited about it, but I am really excited about it. You can bundle up validation to your values.yaml. values.yaml in Helm is the parameter you want to be able to tweak, so if you want to change a container image, you want to change how many replicas. What we can do using JSON Schema, is you can actually provide typed values and have them packaged up. If you say, "I have a parameter called replicas and it must be an integer," and I go ahead and put a string in there, it's going to say, "You need to put an integer in there." Give you early failure detection and error reporting. As a chart maintainer, this is optional, you can provide a JSON Schema so that you can actually have your values typed.

The other thing I've seen people do is there are fancy web UIs people build to install Helm charts. You can already prepopulate that from the schema that you provide here in your web app. I have a dropdown list. Here are accepted values, I can generate them on the fly, rather than having them untyped in Helm 2. Let me just pop that up. As you can see here, we have a property called image. We give it a description. Under there, we have a repo and type as a string. We can strictly type everything under here and package it up with a chart. Everybody excited about that, or is it just me? Maybe just me, that's ok.

Three-way merge for upgrade. This is super awesome. Again, we're falling into the Kubernetes. The way kubectl apply works is it takes your new config, looks at your old config, and also takes a look at the config of what's actually running, and merges the three of them. Helm 2 only looked at the new and the old and diffed them. If somebody had gone in and edited a deployment artifact on the cluster live, which nobody ever does, added something manually, that wouldn't have been taken into account. You blow that away in Helm 2. In Helm 3, it will take that into consideration in the diff and present that to you as a user. Again, it's more in line with what kubectl apply does, but we're adding the current state, which was a big blocker to many people that had tools that would change things after Helm had touched them.

Increasing Reliability

I'm going to move into a period of looking at how we can actually make releases more reliable. One of the things with getting rid of Tiller – Tiller used to operate as the cluster admin in most cases, which meant, whatever you threw at Tiller, it was able to do. Now, with the removal of Tiller, people don't know if they have access to do things. What we end up seeing is a lot of people have failed releases. "I want to install WordPress, I install a resource that I don't actually have permissions to install." When I say pre-release checks, what I'm thinking about here is other tools we can use to make sure that when I land a release, it's actually going to be successful. I'm going to take you through three or four different tools from the ecosystem that are all plugged into the Helm ecosystem where you can actually look and do pre-release checks. Either you as a user on your workstation installing to a Kubernetes cluster can use them, or if you want to bolt them into your CI/CD process to do some very early linting, can this user do this, is this possible, is there policy violations, you can take these tools and plug them in as well.

The top three ways I see Helm releases fail today are as follows. Invalid Kubernetes resources. You put in values that aren't valid in Kubernetes. We've talked about JSON Schema. That's one way to do it. I'm going to share another way. Denied by policy. For those people who are using OPA, the Open Policy Agent, and admission controllers which say, "You can't define that container registry. You can't have a resource without limits," they're late binding server-side errors. What you'll end up with is a failed release. Role-based access control, RBAC. "I want to create a service." You can't create a service, Lachy. What that'll end up is in a failed release. What we want to do is make sure preempting the failure using standard tooling that's reproducible. I'm going to give you some examples here.

Here is an invalid Kubernetes resource. This is supposed to be ugly, so don't worry that it's ugly, but what I'm doing is installing the nginx-ingress, and obviously, I've set the controller.replicaCount to a string word of two. If anybody can decipher that error there, that's the error you're going to get. Basically, what that says is you can't have a string where there's supposed to be an integer. If you can figure out where that is actually stated there as a user, and this is your day one on Kubernetes, you're a better person than me. If you can't, come up, I'll give you a handful of stickers. I'm going to show you an easy way to do that.

The other one is this one. If anybody, I'm partly to blame for this Kubernetes 1.16, we deprecated a whole bunch of APIs. What you'll see is, "By the way, deployment in v1beta1 no longer exists in Kubernetes 1.16." It was there in 1.15. Again, not a nice error message there. "What? I just worked yesterday. I don't know what happened." I put the link there for posterity. If you're moving to 1.16, please take a look at this. We deprecated a lot of workload APIs, and I imagine a lot of people are having headaches with this.

We're going to talk about a plugin called kubeval. Kubeval is a Helm plugin, so you do helm plugin install. It is written by a gentleman called Gareth. Gareth works for a company called Snyk, but he does this on his spare time. You can go grab it. It's a tool that plugs natively into Helm, and we'll show you what it looks like. This time I've replaced the second verb, helm install with helm kubeval, and passed it the same parameters. This time, you can see in bold, "spec.replicas isn't a valid type. I expected an integer and you gave me a string." Just using one very lightweight command, I can actually throw an error that's actually readable to the chart author or the person trying to run the install that's usable.

Again, let's go back to that first use case, stable/nginx-ingress. What I'm doing here is saying, "Is this going to install against Kubernetes 1.15 APIs as defined?" This tool is going to go pull the schemas from the upstream Kubernetes repositories and say, "These are all valid resources." If I went and did that against 1.16, it would say, "By the way, deployments is not valid in 1.16." Again, one tool tied straight to Helm, you can run it and pass it the same parameters. Super simple, your releases will already be way more reliable.

Another example is policy. What I'm going to do here is use a tool called conftest. Again, we're using a Helm plugin called conftest. What conftest does is use OPA policy. If you're using Open Policy Agent policy server-side in Kubernetes, you can take that policy and bring it client-side and have it validated before you push it to Kubernetes. It's portable through Open Policy Agent, but here, I'm using the Kubernetes best practices, conftest publish best practices for all your resources, and it says, "By the way, you don't have a memory limit set or a CPU limit set for any of these things. I'm going to exit with an error." Here's a way you can define policy, and if somebody installs anything on your cluster, you can actually go and make sure that it's valid, client-side, early binding before you have a failed release. Let's have a look at the success case of this. I'm going to supply those limits. I'm running the same command, but I'm feeding in those limits, and the command exits without error. Finally, we're going to take a look at the RBAC case here. Without Tiller, you lose your cluster root admin, which obviously we're focused on security here. This is a good thing. I wrote a really ugly script just to demonstrate it, but there's actually a command called kubectl can-i, and there's an API that exists in Kubernetes that says, "As this user, can I perform this operation on this resource?" There's a nice big bit of bash at the top here, but what that bash does is it says, "For any of the resources that this Helm chart spits out, can you pass them and tell me whether I can create them against the Kubernetes API?" You can see in bold, on the third line there, kubectl auth can-i create, and I've actually passed the resource to say, "I can't create a ClusterRole, I can't create a RoleBinding, but I can create a Deployment, Role, RoleBinding, Service, and ServiceAccount." That is a very common thing that many charts package cluster admin level resources that you, as a user, won't have access to. Again, you can use this can-i API, and there are a set of tools here you can go. I've given you a link to a doc and I'll publish these slides, but here's the upstream documentation of how you can build tools around the Kubernetes API, because it ships with that can-i API. There's also a who-can plugin for kubectl that Aqua Security. It says, "Who can do this," rather than "Can I?" There are some ways you can plug this into your toolchain and get the most out of it.

What's Next?

What's next? Favorite one, security audit. Helm just had their security audit. They published the findings I think a week or two ago. It went great. Apart from Linkerd, it was the second highest rate of security audit in the CNCF. No major vulnerabilities found. There was one symlinking very minor vulnerability found. That's already been patched, but just full disclosure there. Why you saw this as we're preparing Helm for graduation? Graduation means it's the highest level in the cloud-native compute foundation as a top-level project. An independent entity has done a security report on the code quality, threat vectors, everything. It's all there for some late bedtime reading. If you can't get to sleep, it's certainly a good read, it'll put you to sleep, but it's a really good document. I mean that in all seriousness. Again, they security-audited only Helm 3, because we want to deliver the security posture of Helm 3, because the ecosystem was so heavy on that being one of the postures we want to make for the Helm 3.

You noticed that I had C3 up. Helm 3 release date, let's just drop it now. It's real soon now. You got the scoop here, please don't tweet it, but it should happen any moment.

Finally, getting into support. After Helm 3 is released, for those people using Helm 2, as I said, we don't want an ecosystem of two different parts for you, but we have Helm 2 and Helm 3 running. What we're going to do is once we release Helm 3, we're going to allow bug fixes and security fixes for six months, and then own these security fixes for the following six. Twelve months out, Helm 2 will be completely end-of-life. What we're asking people to do is try the RC. If you had been, then it's not going to blow things up. Take it with a grain of salt, but you should be able to drop it in. I've given you enough information here to be able to do that. Read the FAQ to make sure that your chart or your tooling is able to use this. Obviously, we want to make sure that everybody's feedback is incorporated. Anybody who's using Helm out there, and they have a workflow that doesn't work when they migrate to Helm 3, please let us know. It's our goal to make all these workflows as many as we could support work so that we don't get people upset about moving to Helm 3. We want it to be nice and smooth. Fantastic, easy, actionable.

Questions and Answers

Participant 1: I have two quick questions. First of all, the Helm update is still necessary when we're using dependencies?

Evenson: It depends if you want to package your dependent charts and snapshot them and have them in the chart repository when you zip it up or whether you want to pull them at runtime from an upstream source. You still need to use package update to pull them down into the actual chart before you package it.

Participant 1: Also, for local charts, right?

Evenson: Yes.

Participant 1: Ok. The second one is, the CRDs, when I do helm uninstall, of course, it doesn't install the CRDs, right?

Evenson: Helm uninstall does not uninstall the CRDs, which was a main pain point with Helm 2. It would process the hooks in reverse. When you uninstall an app that had CRDs, it would not uninstall the CRDs.

Participant 2: You did mention the OCI. I just wanted to check if you can publish the Helm chart into Docker registry.

Evenson: Docker Hub does not yet support distribution v2 yet, which is the specification that lays out the metadata required to store Helm charts in a container registry. They are part of the OCI, so I would assume. I don't know if there's any Docker folks here, but I would assume they would update to make that allowable in future. It's not there yet. I think Harbor has it. ChartMuseum has it. Azure Container Registry has it. They're the only three I know, but there may be more.

Participant 2: Ok, thank you. I have another question. Can we use Helm update for scaling?

Evenson: Yes. You can use Helm update to scale or change the parameter in a template, which could trigger a scaling event on a given resource. If you have a stateful application that's either using, typically there, a stateful set or a custom resource, if you change the template so that it becomes an applied that changes the replicas, then that will trigger the behavior. Helm doesn't do anything that changed their resource, and then it relies on Kubernetes to perform.

Participant 2: So it will not redeploy the entire thing.

Evenson: No, only the delta using the three-way merge. If you say, "I need this one field changed, which said three to six," it'll change that field, and then Kubernetes will pick that up and roll out change your stateful application.

Participant 3: Once I deploy my release through Helm, are you saying we should not use kubectl to make any configuration changes, ConfigMaps or scaling up, all of them?

Evenson: I can't give you a prescriptive advice. I think using Helm to perform all the operations might be easier, but if you have intrinsic knowledge of how Helm works and what it does, I know some people write tooling around not using Helm for day two. They'll lay down, use Helm, and then they use another set of tooling to actually deal with resource changes over time. The goal of Helm is to provide rollback, upgrade, install, delete, so all the lifecycle hooks that you would want, but how that plays out is really up to the chart maintainer to determine.

Participant 3: If I say that I'm not using kubectl and using all the Helm to upgrade my pieces, you call it as best practice?

Evenson: I would say the best practices use declarative. As long as you have it some way that's stored and that you're operating off resources that you had stored in some kind of versioning system, then that's probably best practice. Imperative changes, like editing replica counts live on a cluster, that's going to get lost.

Participant 3: Ok, thank you. Now, the release name is stored within the namespace, now we can have the same release name...

Evenson: Across namespaces, yes. If you and I are working on the same cluster, assuming we're not, that was something that was limiting in Helm 2. People wanted to call it WordPress and WordPress and we're in different namespaces, why can't we do that? Yes, it's now namespace-specific.

Participant 3: Ok. One last question about kubeval plugin or pre-release check. Are these equivalent to testing your chart?

Evenson: Yes.

Participant 3: If yes, then can we do it in the earlier environment rather than the production environment?

Evenson: Totally. Yes, they are pre-checks. Have you seen the chart testing repository in the Helm? There is a community maintained chart testing, and that's the tooling we use to test the upstream charts. We have linters and installations against a cluster. We make sure that everything works. You can use that as well and pull that down into your own deployment environment.

Participant 3: In my environment, the policies are not going to be the same. Is there a better way to install various policies within this chart?

Evenson: If there are environmental differences in policies, you'd need to store them in the chart and have them configurable depending on the environment. If you're using something like conftest, it'll pull policy from any repo. It pulls them from an upstream git-repo or any git-repo. You could have policy that's environment-specific and test against that environment's policy if you wanted to.

Participant 4: Bringing the CRDs upfront is really cool, but is there more general lifecycle support? I'm thinking namespaces obviously, even anti-pattern maybe, service accounts, that kind of stuff that have to be made. I'm thinking the Istio chart literally has a separate init chart writer.

Evenson: Right, just to build everything. Not at this point. The lines are blurry, I would say, between operators and Helm charts. Some people expect the operator to take care of that, and other people expect the resources to be pre-created. It's going to depend on how you want to use it and what the split of your personas are internally. Do I have an Ops person create the service accounts? We're willing to listen if that's a better practice for us to do as part of the Helm chart community.

See more presentations with transcripts