The New York Times development team have released ‘Gizmo’, an open source Golang-based microservices toolkit, which provides standardised configuration and logging, health check and metric endpoints with configurable strategies, and basic interfaces to define service expectations and vocabulary.
According to The New York Times ‘Open’ website, a blog about code and development written by The New York Times software development team, the organisation has been adopting the Go(lang) programming language over the last three years. Go is used by the NY Times development team for a variety of tasks, with the most common use being the building of JSON APIs. This focus on creating API-based Go services led to the creation of the ‘Gizmo’ microservices toolkit, which has been made available as open source software.
InfoQ sat down with JP Robinson, principal software engineer at The New York Times, and asked about the motivations for creating a microservice toolkit, where this offering sits within the growing landscape of Go-based microservice frameworks, and the future roadmap.
InfoQ: Welcome JP! Could you explain to us what 'Gizmo' is please, and also explain your motivations for creating the framework?
Robinson: Gizmo is a collection of composable tools, funcs and interfaces to help developers build Go services, specifically APIs and pubsub daemons. The toolkit was started by a small group of developers building the proprietary email platform at The New York Times.
After we built a handful of APIs and pubsub daemons and a few other teams became interested in using Go, we found we needed some common vocabulary and structure to help us easily move from project to project and to allow our developers to simply concentrate on the business problem instead of things like "How will I get metrics on my endpoints?" or "What do I need for a graceful shutdown?"
InfoQ: How does Gizmo make it easier to write microservices, in comparison with simply using the standard Go libraries or one of the many other emerging Go microservice frameworks?
Robinson: Gizmo gives developers the ‘feel' of the standard library but adds additional functionality that provides the bare minimum for successfully orchestrating microservices. For instance, Gizmo has a service middleware that wraps around http.Handlers, and its endpoints are http.HandlerFuncs. This allows developers to write code that looks like the standard library, but they get the added benefit of Gizmo automatically collecting metrics about status codes and request execution times.
Reusing the standard library also allows any Go developer to quickly glance at the code and know exactly what's going on. Another advantage is it opens Gizmo up to hook into the wide array of available existing open source tools that adhere to the standard library's interfaces. A good example is Gizmo’s use of `github.com/NYTimes/gziphandler` or Gorilla toolkit's `handlers.CombinedLoggingHandler`.
InfoQ: You mention in the recent "Introducing Gizmo" NYTimes blog post that Gizmo could compliment Peter Bourgon's Go Kit (which has been covered by InfoQ). Could you explain more about this please? Would it be beneficial for developers to become familiar with both frameworks?
Robinson: Once I saw Peter Bourgon’s Go Kit talk from FOSDEM 2015, I got excited and immediately started looking over the list of “related projects” on the project's README for something we could use internally. NYT has a lot of different services, most of which speak JSON over HTTP, so the RPC-centric service frameworks in the list weren't going to work for us out of the box.
I also felt several of the available frameworks were abstracting away a lot of the feel of the standard library, and we needed something very simple for high adoption. From there I decided to take on Go Kit's non-goals:
- Re-implementing functionality that can be provided by wrapping existing packages.
- Gizmo's SimpleServer is really just a server composed of Gorilla Toolkit's mux and context, github.com/Sirupsen/logrus, github.com/rcrowley/go-metrics and the standard library. RPCServer is composed of the same guts plus gRPC.
- Supporting messaging patterns other than RPC.
- Gizmo's pubsub package offers simple Publisher and Subscriber interfaces with a couple of implementations.
- Having opinions on process supervision.
- Gizmo servers expose a configurable and extensible health check endpoint. We expect service discovery systems (i.e. Consul) to interact with Gizmo through them.
- Having opinions on configuration passing.
- Gizmo server and pubsub packages are heavily coupled with the config package. It isn't opinionated on how the config structs are populated (environment variable, Consul-template backed JSON file, Consul key/value store, CLI flags). We found that making this package flexible sped up adoption across teams that orchestrate servers differently, but still gave us the same starting ground once the configurations were acquired.
From here, the idea is to build out a GoKitServer and/or GoKitService into Gizmo's server package. My hope is that the server would combine the packages of Go Kit endpoints and tooling with the existing Server and Service interfaces available in Gizmo.
InfoQ: Gizmo appears to not be overly opinionated about API design, in comparison with say, RightScale's Goa framework. What was the rationale behind this?
Robinson: I've found keeping the toolkit simple yet flexible sticks to the Go way of building software and also speeds up adoption by keeping the learning curve gentle. There are many ways you could structure a Gizmo application and use it in the wild, which I think could turn into a problem without guidance. I'm currently working on a series of blog posts for open.blogs.nytimes.com that will provide guidance on how I think Gizmo apps should be structured, built and tested.
InfoQ: What is the best way for interested InfoQ readers to get involved with or contribute to the Gizmo project?
Robinson: We're still heavily relying on the SimpleService and JSONServices here at The New York Times. I'd love for someone to take a whack at putting together a GoKitServer, a FastSimpleServer composed of github.com/julienschmidt/httprouter (instead of Gorilla's mux) or maybe even a WebServer composed of the templating toolset from Hugo (to compete with Node.js/express). Also, the Kafka pubsub implementations are still experimental and could use some testing.
If those tasks are a tad intimidating, we'll gladly accept PRs for improved testing on any level. We merged a PR recently that added a ContextService and ContextHandlers to the toolkit. I would like to encourage folks to try them out and create issues for any bugs the may find. Either way, I recommend users hop on the Gopher Slack group’s #gizmo channel if they want to discuss changes or improvements to the project.
InfoQ: Thanks for your time today JP. Is there anything else you would like to share with the InfoQ readers?
Robinson: Thanks for reaching out and giving Gizmo some exposure! We knew open sourcing the toolkit to the awesome Go community would greatly improve it and, hopefully, encourage other organizations to adopt the language. I'm looking forward to seeing Go become more highly adopted in the “Modern Enterprise” world and hoping Gizmo will be a part of it.
Additional information on the New York Times Gizmo microservices toolkit can be found within the ‘Introducing Gizmo’ article on the organisation’s ‘Open’ technology blog, or within the Gizmo GitHub repository.
