Meteor is a platform for writing applications in JavaScript and the reason we’ve built this is because there is a shift in how we are seeing applications and by apps I mean interactive applications that you would use to communicate with other people or to look and store data. Those apps are written more and more to run inside a web browser, or on a mobile device, rather than have the software run inside a data center. So there is a shift in how the architecture of those applications looks. It’s not the first time that applications have moved from one end of the network to the other in the nineties of course we wrote apps to run on Windows on the desktop, and the web replaced that model because it was such a better way to deliver an app, and you could type in an URL instead of install a CD. But with the rise of JavaScript as a powerful language, and the investments that Apple and Microsoft and Google and others have made in JavaScript and in the browser, with the rise of mobile devices and the ubiquity of mobile networks, it’s now possible to run software on the client again, and all of the best interactive web experiences are built that way. So the photo browser in Facebook for example is a case where the smooth being able to swipe from photo to photo, seeing comments in real time, it’s a better experience because that software is running inside the browser. And of course in the case of Facebook they can throw enormous engineering resources to solve a problem and build the interface they like.
Google Plus is another example of software written this way it’s actually an app running in your browser it’s not a website. So, for Google and Facebook you can build these things by hand, it turns out for most developers, developing software in this style is incredibly difficult, it takes a lot of custom engineering work, you end up solving many of the low level problems, and we can talk about some of these, each time you build an application. And it leaves people in a tough spot, they want to build rich, engaging experiences but they also don’t have months of spare engineering resources to build them with. So Meteor is the solution, Meteor is an off the shelf platform that lets more developers build rich real-time applications that run on the client.
We look at the application as a whole, and I think it’s important to have a solution to the full technical challenge of building a rich client application. So there is code on the client but there is also code on the server that we can’t forget about because the server is still in this architecture the privileged environment where your security code runs, where your batch processes run, where your emails that get sent out come from the server, we can’t ignore that half. And the challenge as I’ve mentioned of writing an app that has these characteristics is that there are so many moving parts now. It used to be easy to write a web app, right? You’d use Rails and you’d use MySQL and the book was about an inch thick and then you were done.
But today you need all of the sorts of things you mentioned. So we need something on the server, maybe that’s Node.js, if we’re at the leading edge or something like Rails or Django, we need something on the client, and you mentioned a number of options that solve overlapping parts of the problem, we also need a protocol for those two pieces to communicate, and in the web era we use REST of course, but in a real time application that’s not sufficient because we need a way for the server to be able to tell the client about new data, the basic problem is that the clients don’t reload pages, they don’t refresh themselves automatically as you click around so we need a way to deliver new information into the client so that you are not always looking at stale data. And most of the libraries that you mentioned don’t attack that problem, so you are still left as the developer designing a protocol so that the server can send data to the client, and so that the client can ask the server to make changes, as users fill out forms and modify information.
Full question: Up until recently if I was trying to build a client server application, if I was trying to build a rich client application with the server side backend and I wanted real time updates my tech stack might look something like say a Backbone on the frontend, for the client MVC, I might drop in a SocketIO and maybe build something using one of the Node frameworks so that has JavaScript on both sides and SocketIO in the middle, and I would kind of stitch those pieces together and deal with various incompatibilities and issues. So are you saying that Meteor basically provides this as a single framework for covering that range of requirements?
That’s right and what we typically see is that stitching those pieces together gives you part of the solution, but there are limits to what is possible when building that by hand. So one of the things we like to show is in the original Meteor screencast, a single line of code lets you define a Mongo-backed collection of data, and in Meteor that data is automatically available in the client, so that inside client JavaScript you can do a Mongo database query which doesn’t go over the wire it actually is fast because it uses a local cache, and that cache is kept up to date as each of the clients makes changes to the data. So the programming style becomes much easier you can write in just a few lines of code, using familiar APIs like Mongo DB and the HTTP APIs you can build a full application. And the challenge is, when you write an application like this, particularly at scale, you are writing a massively distributed application, let’s not pretend otherwise and again it’s possible to build and to stitch together some of the low level components of that but the best systems are designed with that design point in mind, and so Meteor gives you a principled approach to solving these problems.
So for example you can put together a client and a server with a custom protocol running over SocketIO but what you lose in that case is tooling to inspect the protocol when it goes over the wire. You don’t have the equivalent of Curl, you don’t have a proxy server, you don’t have a way for a third party app to integrate with what you’ve built, because they would have to agree on your private SocketIO encoding. In Meteor we try to solve this is a standard way, so all apps use a protocol we call DDP, apps can connect to each other so you can subscribe to data from a third application, we are starting to build tools and people in the community and we should also talk about that, are also building tools that allow you to inspect what’s happening to be able to integrate with other components at the same time. So it’s both a faster development experience when you have a platform as a whole, but it’s also a broader experience in the sense that you have more capabilities than if you built these components by hand.
That’s a great question. Our goal is to make all applications, again interactive applications easier to write for more people. We are starting with a couple of particular focuses, so I would encourage you to have a new application as opposed to an existing legacy codebase, we’ll certainly find ways to integrate for example a backend that’s been written in Java with a Meteor frontend, but the focus right now it’s a green field application, a new app, we selected Mongo as the first default database because the JavaScript JSON object model is a great fit for developing in JavaScript on the client side. I would encourage you to use that as you backing store, and there are applications in which that’s a particularly good fit. We’ve found two use cases that people in particular have started focusing on: one of them is the example of a small team or a startup, where development velocity is the most important thing. They’ve got to build a great app, they can’t compromise on the quality of the application, but it’s super important to be able to write that application quickly, and to be able to iterate quickly. So in Meteor for example you can define your interface, and you can create the user experience first, before you go back and think about what are my REST endpoints, how am I securing them, what does the scaling picture look like? We are not painting you in a corner, but you can defer some of those decisions because as we know when we build apps, often times the first and second and third ideas aren’t what we end up productizing. And I’d like to defer as much of that task as possible. The other case we’ve seen a lot of interest in is real time analytics, so people who have a Mongo database with data in it and what they want to be able to do is plug that into an interactive client possibly backed with D3 or one of the other graphing tools. Meteor gives you a very easy structure for moving that data from the server into your client in real time, keeping it in sync and then driving automatic screen updates either as HTML or SVG.
Peter: If I am looking to build an application using this, something that immediately comes to mind is the testing story. And I think there is two parts to that: there is how do you typically write the unit tests and then do you have a solution for what has historically been quite difficult which is providing acceptance level test functionality for applications that have a lot of JavaScript and that can sometimes trip up some of the traditional acceptance test tools.
It is a challenge and it’s even harder as you build these distributed applications because one of the things you want to test is the interaction between the client and the server. And setting up that environment and doing it in twenty browsers at the same time, so that you can qualify mobile Safari and you can qualify an older version of IE, and the newest versions of Chrome, that becomes a testing challenge both, as you said, the low level unit tests but also as a whole the application. That’s a great example of something that we have seen tremendous interest from the community about and I should say Meteor is open source, and we have thousands of developers world wide building applications on Meteor, the most exciting thing about that is that parts of the core that we haven’t yet built as the core development team in San Francisco are parts that other have jumped into and so we have seen a number of really interesting efforts around integrating some of the popular unit testing frameworks with Meteor and that’s been I would say probably one of the most popular topics on the mailing lists. But also more forward thinking questions about what does a complete acceptance level testing framework for an application of this style look like. We have a public roadmap for the project, at roadmap.meteor.com, you’ll find testing is one of the things we are going to be building soon. And a lot of what’s there and the specifics of how we’ll do it, comes out the conversations we have at events like QCon and on lists like Meteor Talk where people can flesh out ideas and play with concepts about what might be the best way to approach a problem like that.
Our own website is built in Meteor it’s important to eat your own dog food early, it’s an early tech preview, it’s what we call it and so you'll want to be comfortable with quickly changing technology stacks. In particular some of the APIs in Meteor are still in flux because we want to explore in the context of these real production apps that people are building, what the best approach is. So as one example of that we have one library called Spark today in Meteor, which is how we automatically manage the DOM. So redrawing the screen in as efficient a way as possible, as the underlying data in the local database or other parts of the applications change. We are in the middle of rewrite of that library and the reason is that we have discovered as we have worked on larger and more complex apps some in-house and some that members of the community have written that there is a cleaner way we think to define the programmatic API around modern DOM manipulation. And so there is a tension because on the one hand we do have, as I said, thousands of developers that want to be able to know that these applications that they write today will be against the correct APIs on the other hand and we hear this very clearly from the community they want those APIs to be as excellent as possible and so we are moving forward but we are doing it in a way that allows us to step back and examine each of these design choices as we go and evaluate “Is this the best thing we can do for the developer experience? Is this the easiest possible way to write a modern app that doesn’t compromise on technical quality, that doesn’t compromise on user experience”. If it’s not, then our inclination is to take a second or third look before we bake it in.
Absolutely. So certainly everything is based on HTML5 and CSS, my goal is to make that be one of the few technologies you need to learn as a developer to be able to write an application and I think those two will be with us for a long time so that’s a very good bet. JavaScript of course is the third. Meteor does play nicely with he ecosystem so Meteor is a collection of packages each package has a specific task we include as packages with Meteor for example jQuery so you can absolutely mix and match whatever programming styles you like. Now a lot of what people have done with jQuery historically which I think of as imperative DOM manipulation, a short program that changes what is on the screen, in the Meteor model that is typically done using the reactive templates that we provide. So in Meteor you would define a template, that template depends on certain data sources that come from the database, and as that data changes, the template automatically withdraws, that is Spark's job that I have mentioned before. That gets you most of what you need and you don’t have to write individual event handlers, you don’t have to write particular items that grovel through the DOM for an element, and then replace it and so on and so forth. But there are places for that, there is of course a huge ecosystem of date pickers and other UI elements that have been written in jQuery we want to support those we want to make it easy for you to drop those sorts of things into your application.
Yes, Meteor works well with PhoneGap and we have a number of people in the ecosystem that have built applications like that and have explained how they’ve done it and so we are very grateful for anyone who takes the time to write a blog post, send out technical information on the list, that’s how the ecosystem grows, because the core team is focused on those key APIs that get us to a stable 1.0 and it’s the energy that comes from someone who says “You know what? I want to see if I can get this working in PhoneGap and when I’m done I am going to explain how I did it so that others can follow”. It’s really exciting and gratifying to see that.
We’ve seen people take a number of different approaches Meteor and any real-time application with web sockets is a little challenging to host on some of the legacy hosting platforms that don’t support the real-time stack. And so one of the challenges for deployment which we’ll have to face in the next year or two, is moving the web infrastructure that we’ve built for request/response-based HTTP apps to a technology platform that supports this real-time web socket based application. So we provide a free hosting service called Meteor Deploy that’s designed that way that anyone can use, really as a way to share applications quickly, certainly we have people building on Heroku, we have people building on EC2 as you said. One of our growing interests is software that will help developers host applications very quickly and effectively on whatever environment they want.
So I think the trend is very much toward hosting in the cloud but hosting on infrastructure that you have some control over and so we would like to make it very possible for an organization that is building a Meteor application wants to host it alongside their existing web-based applications at Amazon, Rackspace, wherever it may be and solve most of the challenging problems around, how do I plug my web socket down to my stack, how do I run multiple copies of Meteor as I need to scale, how do I manage like what is the equivalent of a proxy server for the web sockets and how do I scale to millions of clients and so on. These are all things that properly belong in the open source framework that we are building so we will work on that as well.
We try to support the browsers that are in the field I believe IE8 and above is supported. I think it’s important for the framework itself to be fairly permissive about what is supported. I don’t know that every application developer will take advantage of that of course but there are some who need that and so it’s important for us to do that. The other half of your question is about mobile browsers and of course there is this huge explosion of mobile browsers so we think about challenges like for example there are some mobile networks where web sockets aren’t well supported and so we want to bake into the framework as a default behavior under what circumstances do we use a web socket, under what circumstances do we fall back to some sort of long polling strategy, each of those strategies have a different implication for battery life on your phone, a different implication for whether or not you see a spinny when you are using the app, and so those are things we focus on so that application developers can just build the application and know that it will work on the widest possible variety of platforms.
Well I think that any platform is as much about the ecosystem and the developers as it is about the code. And we take the development half of that picture very seriously so I think of what do I look for, in a technology platform, when I select it? I want to make sure that I believe in the technology, but I also look at how many people are using it, does it have the energy behind it that gives me confidence that it will be available as a long term platform? And I think setting those stages in Meteor has been very helpful so as I said we have thousands of developers around the world, I was just last night at a meetup in New York city and we got a bunch of Meteor developers together for that, I think that is happening in fifty or sixty cities around the world now each month. So it’s important I think as a developer to think how am I going to move toward JavaScript, how am I going to move to the client? Let’s make sure that their first experience with Meteor which is typically a mailing list question or an event that they go to, is an excellent one and anything we can do to help show off the energy and the excitement, the creativity in the ecosystem I think it’s very valuable. Because we are at a point now where developers make technology choices, no one is going to parachute it at the top of the organization with a sales team and force you to use a particular database, or a particular development framework. It comes from the developers. And I think that building an ecosystem around that and understanding that can have a dramatic impact on everyone’s success in the community.
Peter: Matt, thank you very much for taking the time to be with us here, I am Peter Bell for QCon.
Thanks.