Transcript
Kromhout: Welcome to our QCon Online 2020 Session, wherein we look at the past, present and future of the Helm project. I'm Bridget Kromhout. With me are Helm maintainers, who can give us insight into this open source software project, which today is graduated CNCF project used by upwards of 70% of all those using Kubernetes itself.
Helm's Origin Story
Let's look at how that got there. To set the stage for that we have Matt Butcher. Matt, would you like to give us Helm's superhero origin story?
Butcher: In 2015, I was working for a company called Deis. Deis was popular for work, making a PaaS solution called Deis Workflow. We had been looking at a number of different orchestration systems. A lot of the work we were doing was already container-oriented. We were pretty convinced that containerized workloads and schedulers were going to be the next big thing. After doing some R&D work, we decided that Kubernetes was really looking like the most promising solution, and so we decided to go all in on Kubernetes development. In order to get everybody on board with this, we had a two-day all-company meeting. This brought in everyone from marketing and communications through engineering and ops. We all got together in a room, discussed Kubernetes, discussed what we were going to do, how that was going to change our plans. Then we decided to do a hackathon project, and just break everybody into teams, and just try some coding. People did all kinds of different things, and had lots of different ideas.
The team that I got together with was Jack Francis and Rimas. The three of us sat down and said, "What would it look like if we built a package manager for Kubernetes?" For every nook and cranny of time we could get for the next couple of days, we kept working on this little thing. At the time, it was called K8S Place. We thought that was really cute. We built this little simple package manager for Kubernetes. At stake in this entire thing was a $75 gift card for Amazon. We really wanted that $75 Amazon gift card. We worked really hard. Got up there at the presentation thing at the end of this two-day all-hands and presented our hokey little package manager. It was cool. It was close, but we did end up -- and I know this is the most important part of the story -- winning the $75 gift card, which I consequently squandered on coffee.
The next day after the hackathon was over, the CTO and CEO of the company called me in. That's one of those things where you're like, "Uh-oh, what did I do?" They said, "Hey, we were thinking. That K8S Place thing was kind of cool. A package manager for Kubernetes might be a really interesting thing to build. The inaugural KubeCon is coming up in only a month. Let's do it, let's turn it into a big product, turn it into a product that we can share there." I said, "Okay." Then he said, "Just one thing. We're not really wild about the name K8S Place. Do you think you can come up with something else?" I went, "Yeah, sure." Jack Francis and I sat down with a nautical dictionary, which was very interesting experience for both of us because neither of us spent any period of our time on the ocean or on boats or anything.
We just kept reading through words out loud back and forth to each other until he said out loud, "Helm." I went, "Helm? That sounds like a good idea." Then we worked a little bit on packages and came up with Charts. That kind of really got us rolling. Then we spent the first couple of months developing that. Showed it off at the inaugural KubeCon. That KubeCon was very small, but it was a lot of fun. Everybody there hyper interested. Kubernetes 1.2 had just come out, and so everybody was really interested in what we could do with it.
Few months after that, Google called us and said, "Do you want to come up to Seattle, visit us, and we can sit down and talk about some new ways of pushing Helm forward?" We went up there, and sat down, and did a big design session. That's where Helm 2 came from. Out of that meeting, we came out with a plan and started working on that. Within eight months, we had dozens and dozens of companies, hundreds of engineers all contributing bits and pieces to the Chart repository, to Helm itself. That's where Helm 2 dropped.
Then, a couple of years ago, we started saying, "Helm hasn't quite kept up with the recent developments in Kubernetes. We need to revisit some of the assumptions that we had made back in the Kubernetes 1.2 days and see if they still hold true in Kubernetes 1.12." They didn't. Helm 3 was born out of this effort to try and figure out how to evolve the project, really meet the way the clusters are working today, and meet the way that the systems that DevOps teams are using today. That's how Helm got started.
Kromhout: That is a whole heap of an origin story. This is actually fascinating because you start unpacking that and you think, "There's the Helm project itself, but the way people use it, there's a lot more to it than that." The community is vast and contains multitudes. This is an interesting scaling problem.
Scaling Problem
We also have Matt Farina here to tell us a little bit about how that works.
Farina: I think scaling the community is the interesting part, because the community isn't just work on the core Helm client that everybody uses, but it's all these charts. It's all these packages that you can install. When I joined on to the Helm project, which was a little later, I joined on to help the Helm Charts. When I joined on, there were less than 100 charts in the stable and incubator repositories. These are chart repositories that have been hosted by the Helm project. It was growing.
One of the first things that I did when I came in, it was how do we figure out how to manage this growth? Because at the time, you had a handful of charts maintainers who managed all of the charts, they managed all of the applications every time a pull request, every time a change request came in. How did that work? It turns out, that's a lot of work when you look at all of this [inaudible 00:06:29] and doing it by hand. As you wanted to scale up without burning out your maintainers, especially in these charts, what do you do? You add automation. We spent a fair amount of time, in that first bit when I joined on as a charts maintainer, figuring out, how do you automate, and how do you scale the community? How do you scale managing all of these applications out of that central repository?
Most of that work was in the stable repository, because it's stable applications. How do you scale that? We added layers of automation to do those things that people were manually checking. That was the only way we could have scaled because, as Helm was growing, there's a lot more than 100 applications. The repository just grew. Both a number of applications and the amount of activity, each chart was getting. You just saw this scaling grow. This all happened in the Helm 2 era.
When Helm 2 came out, it was Helm 2, Alpha 2 added the stable repository, added by default to Helm 2. When Helm 2 came out, you just saw this massive growth in charts and applications. We had to figure out automation. We actually got to a point, and we went so far as we brought in the Kubernetes automation that has owners' files, we can have individual people owning individual charts, being able to merge things into those charts. We actually couldn't handle the scaling problem anymore, even with all this automation we added on.
A couple of years ago, Helm was designed with distributed charts in mind. You could have charts and all of these different repositories, add different repositories and install them. It's been like that for years. We weren't taking advantage of it because everybody was putting everything into the stable repository. We decided to make the intentional move to distributed charts, because there were pain points. People who wanted to maintain their charts, they all had to use one workflow that we came up with. They couldn't do their own workflows. They couldn't integrate it with their own application releases. They had to wait on charts maintainers to do things. Sometimes that was frustrating for both them and us to do the work.
We wanted to go do a distributed model. We took advantage of Helm's ability to do that. We did some things such as stand up the Helm Hub, which has now been overcome by the Artifact Hub, which will search for more than charts. It was a way to allow you to search for all those distributed charts across many repositories hosted by many people around the globe. We took that automation we used in the stable and incubator repositories to test everything. We made it available for anybody to use in their own CI pipelines. Since many people are using GitHub to host these repositories, and want to host these charts, we created GitHub Actions that you can use that tie together some of these other tools we have. That way, you can publish your own charts repositories right within your workflows.
We started working in tooling to enable that distributed effect. That's a lot of what we see today is more and more distributed charts from companies and organizations that you can find through the Artifact Hub.
Kromhout: We just jumped from 2015 to this year of our apocalypse, 2020 where all this exciting stuff is happening. You alluded to the Helm Hub moving to Artifact Hub. We jumped over even what Helm Hub was, which you've mentioned, helping people find charts, search for charts. There is a lot going on in this space right at this moment. We've blazed right past history into exactly what's happening right at this moment.
What Is Happening Now
As people are watching this talk on November 2020, what is happening now, Farina, in this space?
Farina: When Helm version 2 hit was announced with its end of life cycle, so Helm version 3 said, "We're coming out with version 3. Here's how long we're going to support version 2 for, because we can't support it with security fixes and bug fixes indefinitely." We gave ourselves a year-window of support, and we said, "At the end, there is going to be Helm 2." Helm 3 didn't add the stable repository by default anymore. There were many reasons for that. That opened up the opportunity for us to say, "What do we do next?" We had already been going towards that distributed model. The charts maintainers announced that the stable and incubator repository would hit its end of life and be archived when Helm 2 hit its end of life. It was announced about the same time, so everybody would know. Then we've been trying to move people off over that time. That end of life comes up here on November 13th. The stable and incubator repositories will be done accepting contributions.
One of the things that came out in April of this year was that we were looking at turning down the Helm repositories that you could get through Google repository buckets. That was first brought up by Google to say there's a cost to either hand it off to the CNCF -- due to financial reasons, it's actually expensive to host all the bandwidth from this, we weren't able to -- or turn it down. They hosted it until the end. A couple months ago, we started hearing people saying, "We need to keep this around in an archive form." There were many people who haven't moved their charts out yet. They still need to get access to them even after the fact because they haven't found that new home we had hoped for yet.
We've announced now that the stable and incubator repositories are moving to GitHub Pages. They're going to be hosted out of there on a Helm domain, charts.helm.sh/stable and charts.helm.sh/incubator. You can move to that, but that doesn't stop the end of life from accepting new contributions, which happens on November 13th. Now they'll be around for this longer form where you can get them in an archive deprecated format, but the end of life is still coming.
We are in an effort now to help everybody transition their charts to new locations to be found in a distributed manner, where you can self-manage in your own form and find those on the Artifact Hub, which is artifacthub.io.
Kromhout: When you say, "End of Helm stable and incubator repository is coming," I hear, "Winter is coming." As a Minnesotan, I think, "Exciting." Then I also think, "We're giving people news of exciting changes that they might have to take some action, they might have to read these blog posts and make sure they're ready for it." There are other changes in the Helm project that are also exciting, but maybe don't require as much immediate action, just informative.
Other Project Changes
Butcher, do you want to take us through what are some of the other project changes?
Butcher: Probably the most important one stems from our kind of critical learning that as a project increases in volume, and becomes more successful, and has more users and more people contributing, we can't use the same methods we used when we started. Back in those hackathon days, it was fine to open a PR, review it really quickly, merge it, go on to the next thing, have multiple people doing this at the same time. Around Helm 2's lifecycle we got more formal about issue queue management and things like that. We've really arrived at that point where we've realized that a lot of very mission-critical applications rely on Helm as one of those building blocks in the wall that holds everything together. We realized we needed to apply a little more rigor to the process of figuring out how we're going to change things over time.
Recently, we introduced a new way of managing the process of introducing new features. It's called the Helm Improvement Proposal, HIP for short. Insert various Shakira "Hips Don't Lie" jokes right here. The cool thing about HIPs is we based it on the way that Python does its feature development cycle. The idea is it gives the person with the vision for a new feature an opportunity to express according to a structured format, how that vision would play out. Then it gives a lot of, frankly, overworked, sometimes highly distracted Helm maintainers a well-understood format to read through it and say, "I understand what they're thinking. I have a couple of questions, because as it's articulated, I'm not sure about this." It gives us this whole process for opening a constructive dialogue.
Frankly, I'm so excited about it, because we had gotten into the pattern of just throwing up an issue, saying, "I want to do this." Then starting the development cycle, and then realizing three months into a mega PR that we had not thought about something. The way this whole structure works really provides the framework necessary for identifying potential problem spaces, and then dialoguing about how we can fix them up. I think we've gotten about six HIPs in process, a couple of them have gone through already. I'm really, really enjoying it. To me, I think this is what's going to help us bridge from where Helm is today to where Helm is going to be in the future.
Kromhout: I'm not positive if it's reasonable for us to say, "More process, delightful," but it is kind of. Farina, I think you've found that this is moving things along in a more useful way.
Farina: Yes, I think it is. In fact, if you look at HIP 2, the HIP 1 describes the process itself, and HIP 2 is the first one that's a real change. It actually takes Helm into a known release cycle. We had had a felt-out release cycle before. Quite often it dragged just behind Kubernetes releases. It was never documented, it was never understood, people didn't know when to expect things. That's the first HIP that's a change is one that brings it into a known release cycle with known dates and expectations on when that would happen. Because if you're a big shop who's using this as part of their critical infrastructure, as Matt said earlier and we see happening, then you want to know those things. This is one of those maturing things that we've been able to take on recently. The process helped us actually work through, and ask questions, and understand what are the needs of people.
Kromhout: I think that's a really good point about being a mature project means that there are a lot of people relying on you. If you're part of somebody's load-bearing infrastructure, you don't necessarily want to shoot from the hip the way you might have when you were a hackathon five years ago. You mentioned the regular cadence of releases. That's one of the things that I'm pretty excited about it just because if we're trying to plan things, whether it be around a conference, or around when we want to get a blog post out, or just around someone's vacation, and making sure that if they want to weigh in on something, they'll be there for it, it's nice to have a little bit more predictability. I think that if we're looking towards what's going on in the future, I would say that cadence is going to be really helpful. I'm happy about how we're doing more translation of docs. By we I mean the community is contributing translations of docs, so far into Korean, Japanese, and Chinese. Though, it'll be fantastic if we get your language of choice. Pull request accepted. Please come over to our docs repo. I think that getting community contributions of whatever sort is really valuable. I'm pretty stoked about that.
Future
What do you see coming in the coming months and years, Farina?
Farina: One of the things that I see coming, and maybe that's because it's one of my personal pets here, is security. I want more security around charts and just artifacts in general in the cloud native space. In context of Helm, how do you make Helm more secure, and passing around charts more secure? Currently, we have provenance files, which is a way of signing charts. You can copy a chart and its provenance file, if you have one, from one location to another. You can store it internally, do what you want, and always verify where it came from, and who it came from cryptographically using PGP.
It hasn't had a huge uptake, and so one of the things I'd like to see is more uptake and making that security process easier, and helping people feel more capable in doing it more easily. PGP, open PGP, GNU, GPG, none of that is necessarily easy to use. I never remember the command line switches. How can we make that easier? Because that security element, when we're doing so much in open source, and so much in cloud computing, and people are using these things with customer information and data, and in that critical infrastructure. How do you help up the security game on that? That's one of the things that I'd like to see.
Kromhout: Absolutely. Again, that dovetails really well with, this is a graduated CNCF project, it is essential to many people's infrastructure. Making sure that we can meet and exceed the needs that everyone is placing on Helm for security is pretty important.
Farina: It dovetails in with some of the other CNCF initiatives too. There's TUF and Notary. There's other projects in working like in-toto. Looking at how these security projects can play in, I think that's useful. With distributed charts, not just those coming from who you trust with Helm, this is one of those things in that trust relationship. If you're getting something from VMware Bitnami, did you really get it from them? If you're getting it from some other company, did you really get it from them and can you trust it? I think that's important in the distributed model of how you have that level of trust.
Kromhout: Farina, I get my charts from the internet.
Farina: You probably run a random Bash script too that you're told to, right?
Farina: Curl Bash for life, man. Farina wants us to be more secure, enterprise grade, etc. I like it. Butcher, where do you think we're going?
Butcher: I am all for more Curl Bash. I think if we can just Curl Bash everything in the Helm universe... No. When we started, you tossed out a number there that still surprises me, even though I know it, that 70% plus of production Kubernetes clusters use Helm. We've tossed out around a couple other numbers. Thousands of contributors to the charts repository, thousands of contributors to the Helm repository, hundreds and hundreds of charts, so many at this point that we can't keep track anymore of where they are, what they do. Artifact Hub is our friend in that case.
One of the things that has become evident as we've matured as a project is that it takes a lot of people to be able to keep something like this moving forward. The HIPs have been a great process change to help us be more careful and more cautious and more thoughtful about how we do this. Moving forward, I think one of the big challenges for the Helm organization as a whole, the charts part, the Helm client part, everything, is going to be finding enough people to help us get the work done, enough people who are willing to spend a couple of hours a week working on this, or champion their particular charts through reviews, or work on a particular feature, or even just jump into a Slack channel and answer a couple of people's questions as we bring more and more people on board with the Helm project.
I think probably 2020 is going to be an almost make-or-break year for us. It will be beginning of our sixth year as a project. I think if we can rally enough people behind us or alongside of us to kind of help move the community along and answer the questions as more and more people come in, we can end up with a strong, healthy community where we can continue to evolve and grow Helm as a long-standing mature tenure product. That's kind of my hope for the future.
Kromhout: I like it. I'm also realizing, and I don't know if you know this, but it's not March 2020 anymore. Horrifyingly, it's November 2020. When you say 2020 is going to be the year we're going to do a lot of things, I started looking at the calendar and saying, "Boy, there's a lot we're going to fit into the rest of 2020, or perhaps 2021."
Butcher: I was starting to say 2021. I don't know what year it is anymore. [crosstalk 00:24:09]
Kromhout: 2020 is a year of doom.
Farina: Maybe we just get a do-over. That's what this is. Next year is a do-over.
Kromhout: I like that plan. On that note, if you are interested in and excited about that, please join us on GitHub, github.com/helm/helm.
See more presentations with transcripts
