Daniel's full question: Hello. We are here at QCon London with Peter Bourgon, engineer at Weaveworks and creator of Go kit. My name is Daniel Bryant. I am one of the editors of InfoQ. Thank you for joining us today, Peter.
My pleasure to be here.
Daniel: I noticed you have two talks at QCon London, which is great. But it does mean I have lots of questions for you, if that is all right.
That is absolutely fine. Hopefully not too much overexposure for the attendees.
2. [...] Could you explain a bit more why you think Go would be beneficial for the enterprise?
Daniel: You were talking about Go and Weave Net. So if you do not mind, we will cover the Go side first. A lot of people are saying Go is super cool, very hipster, used in a lot of container technology, but not so much in the enterprise yet. I know you are championing that. Could you explain a bit more why you think Go would be beneficial for the enterprise?
I am trying to champion that. See, it is my perspective that Go is almost perfectly suited for a certain class of application, and I think that class of application lines up very nicely with a lot of other trends in the enterprise market at the moment, and that is chiefly microservices and the microservice design pattern. So, what I am doing with Go kit, sort of my side project, is that I am trying to level-up Go in certain areas that are important to the enterprise, that are important to the microservice space, and chiefly I am trying to do it in a way that is lining up with the Go’s philosophy and idioms. Because you are definitely looking at ecosystems like Java and you can say “Here is what our Java does and let’s just port these idioms and patterns over to the Go language” I fell in love with Go long before I had dreams of Go in the enterprise, so I want to keep the Go praxis and the Go philosophy alive, and so that is what is driving me there.
I think it is a good match as a language, primarily because the idea behind the microservices, this bounded context, this domain driven design, this thing where you keep all the code in your head, and you give it to one or a small number of developers to build and then maintain and then kind of quickly throw away at the end. This is the way I think Go wants to be written. You keep the abstractions to a minimum, you focus very closely on the problem domain and I think they are good match about reason. There is definitely stuff in Go that is missing and I am trying to fill in the gaps there, especially stuff that is important to large enterprises. In Java and Scala we have this thing called Finagle which is very popular. It came from Twitter, it is sort of an RPC framework for microservices and other things like that. So I am kind of in the footsteps of Finagle with a few changes, making Go a lot more attractive to people building software in that environment.
Daniel: Very interesting, Peter. On that note actually, I recently wrote an article for InfoQ covering Gizmo, which is the New York Times sort of extension, if you’d like, or integration with some of your work.
Gizmo is something that I was made aware of when they announced it. Apparently, the chief developers there at the New York Times saw one or a couple of my talks about Go kit and said “OK. This is probably a good idea. This lines up with what we want to do, but we have specific needs internally, in our organization, that Go kit is not addressing right away. So let’s take this idea and run with it.” That is awesome. I am really happy that Go kit served as inspiration. I mean, more than anything, that is what I hope it does. From what I have seen of Gizmo, I have not deployed it myself, I have only read the code, but it lines up very closely with what I want Go kit to be. So, certainly already there is some influence and inspiration, and I hope that as time goes on we will continue to collaborate. I am very excited to see what they are doing with it, but yes – this is one of the great consequences of putting Go kit out there is that it inspires people to solve these enterprise class problems in the Go way and this is really what I am interested in: keeping the Go spirit alive, even in this environment that is potentially going to exert influences away from that philosophy.
Daniel's full question: I think that is really interesting there, Peter. Something you have touched on a couple of times and I hear a lot of people in the Go community talking about this is the need to maintain the Go idioms and so forth, you mentioned. If you come from a language like Java, JavaScript, it is very tempting to force another language into that kind of development. Have you got any tips for developers like myself who can be tempted to fall back into what is comfortable?
Yes, absolutely. I mean the thing with Go, and a real advantage to it from where I sit, is that it is such a small language that it is actually not only possible, but highly recommended to read the spec from beginning to end, and to read the so called ‘Effective Go’ document, which is sort of like a language primer, to take the tour which is online. You can do each of these things in a long afternoon, and when you are done with it, you have this coherent vision of a programming language and a programming model that draws a lot from other ecosystems, but it is entirely its own thing. When you absorb its opinions and its idioms and its eccentricities, it comes together in way that, at least for me – I remember in the early days Andrew Gerrand who is the chief developer advocate for Go within Google, he said for him, Go made programming fun again coming from a Python background.
For me it is exactly the same, as I came from C++, but the kind of problems I was solving there were very tedious and Go allowed me solve these problems, but in way that fun, and I hope and I truly believe that, if you approach Go with that same attitude that, you can solve these hard problems, or these enterprise problems in a fun way, in a way that makes you feel good as a programmer and so that is my advice: start with the core documents, try not to worry too much about bringing over the patterns that you have in previous languages or the things that you are used to doing. Take a fresh start, take a few days, do not worry about dependency injection. Say “OK. I need to build this thing. Let’s build the thing and see what emerges”.
Daniel: That is great advice. It is very tempting. Dependency injection is one of the first things that coming from a Java background, that I stumbled into ….
Exactly.
Daniel's full question: You mentioned design patterns there, Peter. Have you got any favorites ones, or any ones that you would recommend if I am starting out with Go that I should look at first?
Absolutely. For me, the most compelling feature of the language - a lot of people say Go routines, the concurrency story, a lot of people say the HTTP interface and that sort of thing – but for me, the most important and interesting feature is the interface, and the interface is the primary means of abstraction in Go. You do not have inheritance, you do not have any of these object-oriented things per se, but you do have this sort of duck typing, and it is really quite powerful once you get in the right mindset. So, for me, the number one pattern is to embrace the interface in all the ways that it can manifest, and one really good example is the middleware decorator pattern, and this is where you take something, you define a behavior in terms of an interface.
So one thing I like is the HTTP “doer”. An HTTP “doer” is something that takes an HTTP request and does it and it returns an HTTP response. With this single method interface you can define a whole chain of middlewares or decorators that do different things with that request: you can log it, you can instrument it, you can actually send it over the network, you can proxy, you can do all sorts of things. This is a really a great pattern. I think my friend Thomas and I did a talk at dotGo last year or the year before, I think, which was called “Embrace the interface” which goes into great detail here. Definitely check that out. I think it is a great pattern.
5. [...] Can you recommend for avoidance of any classic anti-patterns? [...]
Daniel's full question: That is awesome. Can you recommend for avoidance of any classic anti-patterns? Do you see – and I am not going to pick on one language here – but do you see new developers to Go constantly falling into the same trap? Maybe error handling, I think, is a popular one?
Yes, there is a lot of them and I should plug my talk tomorrow because this is exactly what I am going to be talking about. “Go, a successful program design six years on”, has really a lot of anti-patterns that I have stumbled across, especially by new users but also by veterans of the language. Error handling is one. In Go there is not exceptions, there is not this sort of control flow that you might be used to. Instead errors are very much values and the same as an integer or a string. So you are expected to program with them, the same as you would any other value in the language. So you see a lot of blog articles, I see always ‘if error not equal to return’, and I just do this over and over again and feels like a boilerplate to me – how can I clean this up?
Well, the fact that you see that in the language, kind of indicates that you are not thinking about errors in precisely the way Go intends you to. Your errors are not something you get and then you just immediately return. There is something you can do things with: you can maybe retry, you can maybe abort your processing and do it in a slightly different way. It is just a value that you should program against in the same way that you might build a testing DSL or something like this. You do not need to build different abstractions then what the language already gives you. So, definitely, too naïve or too simple error handling is one anti-pattern.
I see a lot of anti-patterns related to the use of global state, and I think if you ask the core language designer, I am going to go out on a limb here and say they would agree. In the standard library, there are several instances of package global state that I think are mistakes, and I think if they could do it again, they would change. So I am thinking here the global Logger object, or the global HTTP default client or something like this. If you are writing your own code, I think you are probably tempted to do things like this, because it feels fast and easy and convenient. But in almost every instance I find that using global state and using things like the init method to do sophisticated initializations, start-up work, this is almost always a bad smell. So, I have a lot more in the talk. We could go on for hours about this actually.
6. [...] What is the future of Go kit? Any sort of roadmap?
Daniel's full question: I will definitely plug into the talk and hopefully that will soon be on InfoQ as well so you can catch that one. Just finally wrapping on the Go bit there Peter, what is the future of Go kit? Any sort of roadmap?
Yes, definitely. I mean Go kit is totally open source and all the work is done in the open. So you can check out gihub.com/go-kit/kit. We have all the issues there. The roadmap is pretty clear. The main things I am focusing on now are integration with existing software systems. So a deeper integration with a wider variety of tracing systems, like appdash or the Mondofoster system I am looking at. There is now Zipkin that is sort of expanding to this idea of open tracing – I really want to get into that a little bit more. Integration with different discovery systems, a deeper integration with those, publishing and endpoints in that sort of thing not only consuming. I have a big push now to work on code generation. There is a lot of what feels like boilerplate – I think we can automate away.
So I am interested to dive deeper into that. A lot of things on the roadmap there, but I guess I should say that we have done enough iteration. Now I am pretty confident in the core abstractions of Go kit and the core components of it. So, at this point it is really about just kind of doing the grunt work to fill in the gaps, to make it useful in a wider variety of enterprises and in a wider variety of use cases. My goal was always to make Go kit useful to large organizations like Twitters and SoundClouds and this sort of thing. But I see a lot of people using it as the basis of not only in a small technology company bases on a microservices architecture, but also people using it as the basis for building an elegant monolith which is kind of another design pattern, where everything is designed in a sort of componentized way, but it is still running in a single process and that is quite interesting to me. So it is seeing growth beyond what I had expected. In any case, the roadmap is very clear and it is all public.
Daniel: That is great, Peter. I saw you tweet the other day about this issues in Go kit, even newbies, for example, can jump on board and get started with it as well.
Exactly. Yes. We make a special effort to highlight those things that people who are interested in joining open-source in general or Go open source in particular, can get started with the stuff that I am very highly committed to helping out with. So if you are at all interested in learning more about Go or getting more involved in the Go community, I think that Go kit is a really great way to do that. Not only because I have those issues, but also because I take great pains to make Go kit very, very clean. I am probably very annoying to my contributors in that I do not allow anything that falls even a little bit askew of what I consider to be clean code. So it is a great clean code base to learn from, I think.
7. [...] Could you fill in the rough details or what Weave Net is?
Daniel's full question: Perfect. It is always good for newbies to learn that kind of thing. So moving on now to your other talk where you talk about Weave Net and Weave Mesh. Just to give our listeners a little bit of context, could you fill in the rough details or what Weave Net is?
For sure. I work for Weaveworks, which is a company that is best known for Weave Net, which is a software defined network for containers. What that means is that in the early days and still today, it solves the multi-host Docker communication problem. Since we started there is now this idea of the Docker networking plug-ins and the space has grown quite substantially to include Mesos and Kubernetes and all these things need, they sort of offload the networking story to plug-ins to other components. So Weave Net is something that gives you a nice and very unique solution to this problem based in a lot of different domains. I guess that one of the key selling points of Weave Net is both ease of use - we take great pains to be the easiest possible way to get started with multi-host networking - and also we do everything in a totally pier-to-pier, distributed, masterless architecture. So, with other networking solutions, especially in Docker and Kubernetes, like Universe, you are obliged to hook everything up to a central source of authority, like an etcd cluster or something like this. And Weave critically, and its key differentiator I think, is that we do not that. We do everything through a sort of a Gossiped pure network.
8. [...] Are looking into academia to pull out interesting ideas or how is that working?
Daniel's full question: Perfectly into my next question. I was actually looking through your talk abstract, and you mentioned about Gossip protocols, Raft and also CRDTs as well. They are kind of almost academic style leanings. Are looking into academia to pull out interesting ideas or how is that working?
Yes. Always. I mean in the field of distributed systems, we are definitely, as a union of both industry and academia, we are always at a bleeding edge because we are at a point now in our technological evolution, maybe even as a species, that maybe even we are exhausting the limits of physical computation in a way that requires really outside of the box thinking, to use a terrible phrase, to solve these kinds of problems. So this is where Weave Net certainly, but Weave Mesh, which is the thing I gave a talk on today, that is where it exists. It exists at this intersection point between the latest in distributed systems theory coming out of research at Berkeley and Carlos, Peter Alvaro, Peter Bayliss – all these people are really pushing the boundaries on distributed information theory, distributed systems theory and practice and so industry. So Weave Mesh is a union of communication theory, Gossip stuff, with information theory and CRDT stuff.
9. [...] Is it something you agree with?
Daniel's full question: Do you regularly read papers, Peter? I know that we have had a keynote here at QCon London yesterday by Adrian Colyer – The Morning Paper. He is famous for that. He talked about why more people should read morning papers. Is it something you agree with?
Totally. Adrian’s keynote was amazing and extremely inspiring to me. I reflected on this immediately afterwards. His dedication to learn something new every day is something I have always tried to do, and I am not nearly as successful as he is. A paper a day is incredible to imagine. But I try to read papers. I do not really have a bandwidth to do it at his pace, but I am familiar enough in the distributed systems world with the key papers – maybe one a month. And I certainly do not claim total understanding of them, but I try to keep abreast.
For me it all comes back to a paper by Google several years ago. I think it might have been 2009. I could be wrong on this. It is called “The Data Center as a Computer: Warehouse-scale Computing”. It outlines the ideas of cluster schedulers, of treating compute as sort of cattle rather than pets, so computer’s resources abstractly. It touches on all sort of things. In the paper, it actually spends quite a lot of time talking about power consumption in data centers or something like that. That is like half of it. But more interestingly to me, it is like the use of containerization to segment workloads, to couple long running services with batch jobs into a single scheduling system and that sort of thing. And this I think lays the ground work for containerized schedulers like Docker, Docker Swarm, like Kubernetes, like Mesos. From this, I think this is a necessary precondition to start talking coherently about microservices. So it all starts there for me. I am not aware of microservice papers at the moment, but I would love to find some.
Daniel's full question: Sounds good. Just wrapping up, the container networking space – where do you see the future of this going? Because obviously it is the hottest contended area at the moment with plenty of companies interested. What do you think Weave’s vision is and also, the general vision of the industry? Where do you think it is going?
Yes, it is a really hot space and I mean that in the sense that it is almost a Cambrian explosion at the moment. There is not only all these companies that are interested in casting their own vision of the future, but also a huge amount of interest in the consumption side, right? So people building platforms for resale, and people taking these core components and constructing something of their own design to implement their business vision. It is really hard to say where it is going. I can only say that I feel the pace is accelerating. Absolutely. And to me at least, it is not clear where it is going to end or what the good patterns are. What I can say for sure is that the winning entry in this space, or where we are all headed, is definitely going to be developer-focused, right?
The developers or the application designers, are the ones that, in the end, are creating the core business value for the organizations and so any solution is going to have to address them as the first order of concern, and I think this is really what is the main differentiator for Weave, that we always make a point to put the developer experience first and that is primarily ease of use, but it is also coherency, it is also being able to say “In order to deploy this SDN, you don’t need to be concerned with this other set of things that it ancillary, it is not part of the core story. You just need to be concerned with your application, its immediate needs, and really looking at things from an application perspective”. So I think that is part of the future. Beyond that, it is really hard to say. And I think that is really exciting to be in the space right now, because I do not think anybody really knows where it is going to go. It is a whole lot of energy and a whole lot of smart people working on it.
Daniel: I think that is the perfect place to end. Thank you for your time today, Peter. I really appreciate it.
Thank you so much.