At the Golang UK Conference, Peter Bourgon introduced ‘Go kit’, an open source microservice toolkit that can be used to facilitate and standardise the creation of Go-based services within the modern enterprise application stack.
Bourgon, an engineer at Weaveworks, began the talk by stating that although Google's Go is fast becoming the ‘language of the server’, it has not yet reached critical mass in the modern enterprise. In order to make progress in solving this issue, Bourgon has created ‘Go kit’, a microservice toolkit, as a mechanism for easing (and standardising) the creation of Go-based microservices in larger technical organisations.
The remainder of the Go kit talk provided a high-level overview of the toolkit, its motivations for existence, and usage scenarios. Bourgon peppered the talk with a series of live coding demonstrations that highlighted key architectural choices made within the framework and the semantics of usage.
InfoQ sat down with Bourgon after the talk, and asked a series of questions about Go kit, microservices, and the use of Go within the enterprise.
InfoQ: Welcome Peter! Could you explain to us what 'Go kit' is please, and also explain your motivations for creating the framework?
Bourgon: Last year, I was working on the core infrastructure team at SoundCloud. We were early adopters of the so-called microservices architecture, and had built a Heroku-like internal platform called Bazooka to containerize and run our services, many of which were written in Go.
Go was great for product teams: it was fast to learn, easy to maintain, and simple to refactor. It was great for operations, too: Go services tended to use an order of magnitude fewer resources per business operation than the other languages at the time (Ruby, Scala, Clojure, Node).
But as the years progressed, and we grew more sophisticated as an engineering organization, it happened more and more that teams chose Scala. This was not due to the language, but rather in spite of it, as by any measure Scala is a beast of a thing. I'm a fan of Go, so I can admit some bias, but I think it's arguably the perfect language for microservices, especially in larger organizations. So, I was sad to see this sort of seismic shift.
I asked around, and what became apparent was that Scala, or rather the JVM, simply had better support for sort of "microservice plumbing". By that I mean the largely uninteresting, but hugely important, work of connection pooling, safety checking, recovering gracefully from errors, managing metrics and instrumentation, routing logs, and so on. SoundCloud settled on Twitter's Finagle stack; there are others. While all of this was possible in Go, it was comparatively verbose, and there weren't yet settled best practices. After a while, the inertia was on the JVM's side.
So I decided to do something about it! Go kit is my attempt to provide a set of standards, best practices, and usable components for doing microservices in Go. We ship reasonably opinionated components—circuit breakers, rate limiters, logging and instrumentation packages, and adapters to popular service discovery platforms—which can be used piecemeal or wholesale. If Go kit is successful, engineering organizations of any size will feel confident in choosing Go for their business logic.
InfoQ: How does Go kit make it easier to write microservices, in comparison with simply using the standard Go libraries?
Bourgon: I would like to answer this question in two parts.
First, the Go standard library is definitely a gold standard in a lot of ways, and Go kit doesn't attempt to supplant it. Rather, we build a bit of scaffolding around it, to meet the challenges that come with microservices in the large.
For example, if you want to write an e.g. user service, served over HTTP, it's entirely possible to do with plain net/http.Server. Certainly GitHub is replete with projects that do exactly that, perhaps making slightly different assumptions about environment, requirements, etc. And those services work well, in isolation!
But when you have a system composed of many such services, perhaps even written by developers or teams that don't sit near to each other, you'll inevitably start getting drift among some ancillary concerns: different ways to manage errors; subtly different log formats; different philosophies about dealing with malicious clients.
Past a certain scale, those differences become real problems. It turns out that it's really important to be able to systematically reason about this big, decoupled, distributed system you've managed to build. And Go kit gives you the components, the common foundation, you need to do that.
Second, Go kit contributors come from a diverse set of backgrounds and organizations, and have worked on, and in, microservice architectures for a long time. As a result, Go kit encodes a battle-tested set of beliefs & best practices about how to do microservices properly.
For individuals and small teams just getting started, or for larger organizations migrating to microservices for the first time, perhaps without the institutional knowledge that comes with operating microservices at scale for a long time, Go kit provides some guidance about how to do things. I've seen, and made, my share of mistakes with microservices, and I'm doing my best to make sure Go kit users don't have to repeat them.
InfoQ: What type of developer/organisation are you targeting with Go kit?
Bourgon: Go kit's stated goal is to make Go a viable option for business logic microservices in the modern enterprise. That's a term I kind of made up: I mean companies like SoundCloud, Spotify, Hailo, 37Signals, Bit.ly, Imgur, Secret, up to and including really big orgs like Netflix and Twitter. These are companies that are largely consumer-focused, tech-driven, and are incentivized for growth. Those incentives drive them to take technological risks, and when those risks pay off, they set the tone for the software industry. Many ideas we now take for granted—like immutable infrastructure, or dev/ops, or Functional Reactive programming—tend to start out as experiments in modern enterprises like these. I want to give Go the opportunity to be the same kind of success story.
That said, Go kit is designed to scale down just fine. If you're a hacker working on a project by yourself, Go kit gives you a great bootstrapping experience. If you're a small team with some established services already, Go kit can slide in alongside them without any major changes or disruptions.
InfoQ: Go is becoming an increasingly popular language in which to write microservice-enabling technologies, such as Docker, Kubernetes and the HashiCorp suite of infrastructure tooling. Why do you think the Go language is chosen here?
Bourgon: Indeed, Go very quickly became the language of cloud infrastructure. I think a lot of it has to do with the toolchain; it's difficult to overstate just how wonderful it is to have a compiler that can produce small, statically-linked, natively-compiled executables for every modern platform in a single go. Just that single property of Go makes it hugely attractive for a lot of infrastructure tooling.
Beyond that, I think Go was effectively the first language in which writing multithreaded (i.e. concurrent) network servers wasn't acutely painful. I was one of the few who came from C++, and it was precisely for this reason. The way Go approaches concurrency—the fact that it's built into the language proper, with a great deal of attention paid to keeping that feature clean and orthogonal to the rest of the language—was a revelation in 2009, and in my experience remains unmatched to this day.
InfoQ: What is the best way for interested InfoQ readers to get involved with or contribute to the project?
Bourgon: Go kit is still a nascent project, with a lot of work left to do, and a lot of interesting directions to grow in. The future is bright and broad, and anyone interested in contributing can still get in on the ground floor, so to speak. You can check out the GitHub repo at github.com/go-kit/kit to see the current state of the project, and look at the issues page to get a sense of our near-term roadmap.
We have a mailing list at Google Groups, which is a good place for broader discussions and design ideas. And we have an active community on gophers.slack.com in the #go-kit channel. That's the best place to reach me, or any of our contributors or users, if you just have some questions or suggestions.
InfoQ: Thanks for your time today Peter. Is there anything else you would like to share with the InfoQ readers?
Bourgon: If anyone reading this has built this kind of project at their organization, and has advice to share, or stories to tell, I'd love to hear from you. Please come by the Slack channel, drop me an email, or get in touch on Twitter, at @peterbourgon.
And—especially—if anyone reading this is interested in using Go kit at their organization, but needs some specific feature, or isn't sure about how an integration would work, definitely get in touch. I'm happy to work with you to build whatever piece you might need, to give Go a chance to show its strength.
Thanks for having me!
The video for Bourgon’s recent Go kit talk will soon be available at the Golang UK Conference website, and more information on Go kit can be found on the project’s web page and Github repository.