I’m the CTO of Basho Technologies. At Basho we produce a lot of open-source software, including Webmachine, which I think we’ll talk about today, including also Riak which we sell a version of - that’s what our business is all about. In addition to the distributed systems and other obvious that I have there, I’m personally very interested in compilers, protocol design, that sort of things.
We actually released Webmachine about two and a half years ago and we’ve used it ourselves quite a bit as well as we are now seeing it used all over the place. You can think of it not much like a traditional web framework, but more like a toolkit for really easily building well-behaved HTTP applications and systems. So it’s much more about the web than something that lets you write some objects and somehow expose them to the web. A normal web framework that people usually think of is usually a mostly finished application that most of the way it looks and most of it talks to databases and all those things are 90% defined and you fill in whatever the logic and data model is for your app. This is great, because it does all this work for you and it’s very easy as long as the application you are building is one that looks mostly like the applications that the framework author built and that’s a huge timesaver for people building certain classes of applications.
But even though you can build those sorts of direct user-facing CRUD in the database applications and Webmachine and people have build full web applications that way, the best places for Webmachine tend to be middleware or systems that didn’t previously have and HTTP interface, but you found the value in exposing them on the web. Instead of it being you take your object and you somehow cram. The web, you write a little bit of code that describes the web-relevant parts of your system: how can you tell when something was last modified, how can you tell if a resource or a request is well-formed, how do you produce a BODY of a given type. You do that instead of saying "I have all these objects. How do they respond to ‘GET’? How do they respond to ‘PUT’?" and things like that.
3. What led you to choose such a RESTful architecture over how traditional frameworks work?
When we started making Webmachine that was really the purpose of us building it: we saw huge value in the architecture that’s made the web successful. We looked around for a framework or toolkit that would let us take the best advantage of that architecture in exposing some of our systems like Riak with good REST-friendly HTTP compliant interfaces and we didn’t really find anything that would fit our needs. So, those web architecture elements or REST elements can be seen in the ways that companies like Akamai are successful, the ways that people can build mashups out of other systems. All of those things, the loose coupling that you get in an environment like HTTP, the ability to work really well with intermediaries because you have a uniform interface, all of those elements I think are strengths anywhere, but especially when what you are building is a layer that you don’t know what else will be in front of it.
It’s certainly true that a lot of the systems, the more traditional object and class and "here are some components" web frameworks have a lot more tools and part of that is because the whole point of a lot of those frameworks is to make you not think any differently about your programs than you always did before. So there are lots of tools for writing a generic, big pile of objects Java programs and you get to reuse those tools if you haven’t changed the way you think at all. With Webmachine on the one hand you don’t have quite as much advantage that you get out of those tools, but on the other hand, you get to write very simple small pieces of code that tend to be purely functional or close to it, but in a very natural way you write a function over resources that says "Here is the end of the entity tag" and so on.
It’s very easy in an editor or whatever your favorite editor is to write small self-contained pieces of code that your existing whatever tools you prefer to program in do support very well. The fact that they don’t support the web aspect of this specifically almost doesn’t matter because the web-specific part of a lot of those other component frameworks is the one that they shouldn’t have. What I mean there is in a lot of other frameworks, you have some objects and then you have to define how it responds to a "GET" and how it responds to a "PUT" and you have methods for this, which is completely opposite to what the web is all about. HTTP has already defined what "GET" means and what "PUT" means and if you are writing that from scratch, for everything you do on the web. Writing a new method or a new function for what "GET" means you are going to not do it right. You are going to leave out some things that are correct behavior, you are going to do some things wrong.
So it’s those parts of some other frameworks that you need extra-tooling support for whereas Webmachine takes a very different approach and says "These things are at the uniform interface of HTTP. ‘GET’ always works the same, ‘PUT’ always works the same." and so on. So, all you define are a few facts about your resources or objects or whatever you like to call them and Webmachine manages figuring out what status code to respond with and all those sorts of things.
Yes. That diagram is often used to stand in for Webmachine and help people understand Webmachine, actually directly is reflected in the Webmachine code. That flowchart is what’s called "The Decision Core in Webmachine" and when a request starts on Webmachine, that flowchart is walked and the way it works is that almost every decision in that chart of deciding what status code to respond to or respond with and what other data is going to go along with it, every single one of those decisions has a sane default. You don’t have to fill in all these decisions at all. All you really have to do is say the very least "How does my resource produce a basic representation? What does the HTML or whatever look like of a basic response?" Webmachine will follow all those default choices through the framework and end up at say 200 OK and here is your response.
But then, all you have to do, if you want to take more advantage of HTTP, like if you want to have conditional requests or better cacheability or anything like that, all you have to do is write individual functions or methods that overrun the defaults for whichever of the decisions you’d like to do that for. Some of the classic examples that everyone knows about are things like ETags and last modified and all these sorts of things, but there are many you can look at. HTTP I think is a very deceptively simple protocol in that everybody has used it, everybody has written some code that uses it, it feels simple, but it’s actually really complicated when you look through all the details of different status codes can interact and what different headers can mean.
Webmachine, through that decision process, manages all of that and encapsulates all of that so you don’t have to write all of that "What do we do if we’ve received this conditional header?" and all these sorts of things. All you have to do is provide some predicates that say "This is a fact about these resources" or "If this thing is true about the request being made, then this other thing is what’s true about the resource". That’s the core model of Webmachine.
6. How is the testing and debugging experience on a RESTful framework like Webmachine?
It’s really great and there are two ways in which I think it’s exceptional: one of them is that same flowchart. It is also a visual debugger and if you turn debugging on a Webmachine it will actually show you the path that was taken through that flowchart for every request that you debugging on, for obviously you don’t want to do this normally for productions systems because that generates a lot of data. And it will not only show you the path, which is very often all you need to figure out what went wrong "This decision over here I went off in this direction I didn’t want to." But it even shows the state of your request before and after each of those decisions and since the only thing that changed that state are the functions that you’ve dropped in at one of those decision points, it’s very easy to inspect where some parameter got set to the wrong value or something like that.
A part of what makes that possible is that each of these decisions is a function and here I don’t just mean whatever way you have procedures and methods, but a function in the sense that they should give in the same input to provide the same outputs. That doesn’t only enable the visual debugger, it allows some other people to build some really cool tools. Webmachine has now been cloned in three-four other languages and platforms. One of them is actually on a platform that includes a proofing system, so you can have a Webmachine system, a whole running web application that you can prove some things that are true about it. You do a little bit of work and say "If all these factors are true about my system and requests, I can actually prove that these things will be true about the response."
That by itself it’s kind of neat, but a bigger deal is that then you can change something, you can add a new function and see if those things are still true and that’s really powerful. Even if you don’t end up actually using a relatively arcane thing like that, it reflects that you get this very easy composability and you can inspect small parts of a Webmachine application in isolation and you don’t have to trace through all kinds of recursive method calls and things like that. Each piece is very isolated and so debugging tends to be pretty straightforward.
Certainly. For instance, in one of the various functions that getting called, whether it is_authorized or it is how you describe whether or not to present an HTTP authorization header and things like that, you can have your security model implemented via that function and it doesn’t have to be threaded through where you define "GET" and where you define "PUT" and all these things. You can have, for instance, all of your resources call the same function for determining authorization, whether it’s there or some other part of your system, some other part of your application and because the Webmachine process is all a sequence of individually executed non-interlinked functions that can’t call each other.
It’s very easy to inspect these early functions that get called very early in the process and say "If this one returns false, I know I couldn’t possibly ever return to the user." So it makes it simpler. It doesn’t write your security model for you. That’s still up your application, but it makes it very easy to inspect whichever mechanism you choose to use there and be confident that if it works correctly, then you won’t be doing the wrong thing the rest of your application.
8. What are your future plans for Webmachine?
That’s a really interesting question because Webmanchine has sort of reached the point of maturity. It’s no longer as under heavy changes it was for the first year or so that we were writing it and then the next year or so after we first released it. Now it’s been out for a couple of years, it’s mostly mature and we’re still developing it and on those rare occasions when someone finds a new bug or a new corner of it, we still do things and there are a couple of relatively rarely used pieces of the HTTP protocol that aren’t yet in the decision core model that we’ll probably add at some point, especially since it’s more demand for some of them now. For instance, HTTP has had in it for quite some time, since 1.1 and slightly before the notion of the upgrade header which lets you switch to a different protocol and almost nobody ever used it until people started trying to do things like WebSockets and things like that.
WebSockets themselves don’t work much like HTTP. It’s just the way to hide the fact that what you are really doing is a plain old socket but you started talking HTTP. Webmachine isn’t going to do all the sorts of things that will turn it into a web socket server or some sort of other generic network server, but they can certainly do the pieces of HTTP that make it easier to build those, like adding an upgrade support and things like that.
Yes. There is an active effort right now in the standards community to improve HTTP standards and I don’t mean to change the content, but to recognize and fix the fact that the HTTP standards in HTTP 1.1 as it exists today is really difficult to decipher. You have to become an expert and you have to read many documents and remember everything because all cross-references are mixed up with each other. Much of the challenge in building Webmachine in the first place is that there are many pieces. For instance, everything about conditional requests and which conditions should be checked first and satisfied first are things that are mostly not very explicit in the protocol, but in many cases there is really only one right way to do it, but you have to figure it out by trial and error.
10. You mean one right way because...?
Because you have consequences that don’t work. If you choose the wrong condition to the test first, you couldn’t possibly satisfy some of the others. Also, we explored when we added all these features that people mostly don’t use in most web frameworks and noticed that because many of those features are so underused, when we let people start using them, we found out that many web clients, whether it’s browsers or otherwise have extremely poor support. For instance, many browsers caching support compared to their content negotiation support: caching in terms of holding onto a local piece of data so that I can show it to you again without going over the network, content negotiation meaning that in HTTP there is fairly complex (and this is actually another answer to your question "What’s hard?").
Content negotiation is tricky to get right in the first place and it’s this notion of a client being able to express preferences, being able to say "What I’d like most for the return value of this document is JSON. But if you don’t have JSON, what I’d like most is HTML. If you don’t have that, I’ll take whatever you give me." That’s sort of the thing.
11. Which is underutilized in almost all web frameworks.
It is. And it’s a very powerful mechanism when you use it well, but it turns out for instance that many browsers ignore some of the content negotiation information with regard to the cache. If you do content negotiation and caching at the same time on the same resources, there are some common browsers that will actually give you the wrong data that doesn’t match the content that is negotiated. We actually early on had to caution some of our users and so on and make it easier to not do things wrong in that way while still giving that power. It’s been a challenge in some cases to implement some pieces like content negotiation, like conditional rights, but also to find the places where even using those things correctly after we implemented them, we don’t always want to encourage in the cases where implementation on the other side, on the client side might not be able to work well.
Exactly - intermediaries. One of the big advantages of using HTTP is being able to have all these intermediaries, but some of them are better than others in different regards.
Not quite that old, but it’s getting there. Right now there is an effort that’s called HTTPbis that I’m very supportive of and I’m very glad it’s being done. The idea is that the next version of the HTTP protocol won’t be about changing and improving the protocol (there are some small things will have to happen), it’s really about fixing the standard’s form, of changing the standard without changing the protocol so that you can divide things up into five or six pieces of the protocol specification that are cross-referenced only when they have to be and they are all mixed up with each other. I think the biggest thing that ought to change about the HTTP specification is the one that’s being worked on right now, which isn’t really any of the semantics of the specification, but the form of it, just to make it easier for other implementers to understand.
In addition to the recognition that in a lot of cases the REST architecture and the basic web architecture notions are very valuable, I think it’s also the case that some frameworks in your very mainstream common languages are starting to either adopt in the case of existing frameworks or be built around in the cases of new ones, ideas that have historically been more on the fringe - notions of concurrency or notions that may have originated somewhere in functional programming or techniques outside the mainstream of programming. The past couple of years have been popping up more and more in the context of new web frameworks that are not fringe-frameworks, that are in fact the popular ones.
I think one of the greatest things we’re seeing now is not so much that there is one set of features that’s being added in the some direction, but that there is a bit more crosspollination and the people from different worlds of programming are actually affecting each other in a beneficial way.