Q&A with Matt Fisher of Microsoft about Helm 3.0 Release for Kubernetes

At the recently concluded KubeCon+CloudNativeCon 2019 conference in San Diego, attended by over 12000 attendees, Helm and in particular the release of Helm 3.0.0 stole the headlines.

Helm, which is a package manager for Kubernetes akin to the OS package managers such as yum, apt, Homebrew, etc., makes it easier to install and manage packages and thier dependencies. Helm's history precedes the first KubeCon as outlined in the Q&A with co-founder Matt Butcher. As a result, it had acquired some technical debt, including the much controversial server side component tiller.

InfoQ caught up with Matt Fisher, software engineer at Microsoft and one of the maintainers of Helm and Draft, regarding the Helm 3.0.0 announcement.

Fisher talks about the features of Helm 3.0.0, which is a major release, including why and how they overcame the technical debt, primarily related to tiller. He also talks about Helm’s commitment to backwards compatibility, which has always been important to the project. In this Q&A, he outlines a few key changes between Helm 2 and Helm 3.

InfoQ: Helm 3.0.0 is a significant release, correct? Before we get into tiller, can you provide a high-level overview of the features in the new release?

Matt Fisher: The most apparent change is the removal of tiller.

Helm 3 uses a three-way merge patch strategy. Helm considers the previous release's state, the "live" state, and the proposed state when generating a patch. In Helm 2, it used a two-way merge patch strategy.

The release ledger for a release is now stored in the same namespace as the release itself. This means helm install wordpress stable/wordpress can be managed per-namespace.

Another change was the introduction of JSON Schemas, which can be imposed upon chart values. This ensures that values provided by the user follow the schema laid out by the chart maintainer, allowing users to catch errors earlier in the release process, and ensures that values are being provided as the chart maintainer expects.

Some features have been deprecated or refactored in ways that make them incompatible with Helm 2. helm serve was a tool where it was useful when Helm 2 was first introduced, but other tools like chartmuseum have succeeded the project, so helm serve was removed from the project.

Some new experimental features have also been introduced, including OCI support.

There are way more features available in this release, though. The FAQ in the documentation goes into more detail on what changed and why.

InfoQ: Let’s talk about tiller now. Can you briefly talk about what it is, why you had to get rid of it, and how did you get rid of it?

Fisher: It's important to give a little background context here.

The first version of Helm (AKA Helm Classic) was introduced at the first KubeCon... Way back in November 2015. Kubernetes 1.1.1 was released earlier that month, and 1.0.0 shipped only four months prior to that in July 2015.

Back then, Kubernetes had no concept of a ConfigMap. ReplicationControllers were all the hype (remember those?). The Kubernetes API was changing rapidly.

When Helm 2 was being built, we needed an abstraction layer from the Kubernetes API to allow ourselves some room to guarantee backwards compatibility.

Tiller was created as that abstraction layer. It provided us with a layer where we could control the input (via gRPC), the output (also via gRPC), and provide some backwards compatibility guarantees to users.

We're pretty proud of the fact that Helm has maintained a strong commitment to backwards compatibility since 2.0.0.

Over time, Kubernetes' API layer has become more stable. Helm 3 is our opportunity to refactor out some of those protective layers we put in four years ago, tiller being one of them.

InfoQ: Containers security is front, left and center with respect to container adoption. What other security issues should developers and architects be aware of as it relates to Helm 3?

Fisher: A few weeks prior to Helm 3's release, the CNCF funded Cure53 to perform a security audit of Helm 3. Cure53 has performed audits of other CNCF projects including Prometheus, Envoy, Jaeger, Notary, and others.

The entire report is worth reading as no summary can do it justice. Cure53 provided a conclusion as part of their summary which reads:

To conclude, in light of the findings stemming from this CNCF-funded project, Cure53 can only state that the Helm project projects the impression of being highly mature. This verdict is driven by a number of different factors described above and essentially means that Helm can be recommended for public deployment, particularly when properly configured and secured in accordance to recommendations specified by the development team.

Security audits are one of the benefits of CNCF projects and we are grateful for them and the analysis performed by Cure53. This analysis has provided some concrete areas we can work to improve and given us confidence in what we have.

InfoQ: Despite tools like helm2to3, there are breaking changes that would prevent migration from Helm 2 to Helm 3, correct? Would you summarize these changes and how to obviate the difficulty in the migration process? Also, would you recommend a mixed-mode development that uses both Helm 2 and Helm 3?

Fisher: Helm 3 is a major release. After years of maintaining backward compatibility with each minor and patch release of Helm 2, we have taken the opportunity to rewrite substantial parts of Helm for this release, and there are some backward compatibility breakages. Most of these changes relate to the storage backend in Kubernetes, or how we deal with certain CLI flags. Most of the work was wrapping our heads around how Helm 3 would look with Tiller being removed from the architecture.

Once that was complete, we needed to ensure a smooth migration path forward, which introduced the helm-2to3 plugin.

One of the big design goals we had in mind was to avoid making any breaking changes to charts. We wanted to ensure the community that has built their charts will not have to maintain two separate versions of the same chart: one that works for Helm 2, and one that works for Helm 3.

We've heard from community members adopting a strangler pattern for their clusters, where releases that were maintained with Helm 2 continue to use Helm 2, but new installations use Helm 3 going forward.

We strongly discourage users attempting to use Helm 2 and Helm 3 to manage the same release, though. It won't work. The backend has seen some significant changes, and this is one of the places where the helm-2to3 migration tool comes in to assist with their migration efforts.

InfoQ: Can you briefly summarize the ecosystem that Helm plays in? In terms of tooling for Helm, what are the top tools that you recommend for developers and architects? Where are the major gaps?

Fisher: Helm is a package manager in the same line as apt, yum, homebrew, and others.

A package manager lets someone who has knowledge of how to run an application on a platform (e.g., postgresql on debian) package it up in a way that others who don't have this knowledge can run an application without needing to learn it.

That's why I can use apt install regularly without needing to know the business logic of running all those apps on my system.

Package managers let you compartmentalize and automate expertise.

In terms of tooling, you can't go wrong with chartmuseum or Harbor for hosting your own chart repository. chart-releaser is another take on the chart repository API, relying on automation using Github Releases and Github Pages for hosting chart packages.

The Kubernetes extension for Visual Studio code has some lovely visual tools for interacting with a Kubernetes cluster. I find myself using it more often for simple Kubernetes operations (like kubectl edit), or for when I'm trying a assemble a new Helm chart based on what's currently installed in my cluster.

InfoQ: Finally, What’s the roadmap beyond Helm 3? Any other tips for those who have not used Helm or for those who want to contribute to the Helm community?

Fisher: Right now, we're watching the dust settle and seeing where things land with the community. We are looking towards stability and enhancements to the existing feature set. Moving experimental features (like the OCI registry integration) towards general availability and gathering feedback from users trying out the Go client libraries. Security fixes and patch releases are being scheduled as needed, of course.

For those who want to contribute, chatting with other community members in the #helm-users Kubernetes Slack channel is always a good first step. It's also a good place to reach out if you're hitting issues. I am constantly humbled by the community members who share their stories with others, or help others with certain issues they've seen before.

The core maintainers are also looking for feedback from community members integrating Helm (the CLI or the Go SDK) into their own projects. The #helm-dev Kubernetes Slack channel is great for reaching out to core maintainers about high-level topics.

Additional details of the Helm 3.0.0 release, which introduced some new features and removed the cluster-side component tiller are available at helm.sh. More detailed documentation on the Helm is available in the docs. There were also a number of sessions at the conference that focused on Helm, including a Helm 3 Deep Dive.