Node.js is a server side framework based on top of Google’s V8 JavaScript Engine, that aims to assist developers in building highly scalable network programs, by using evented, non-blocking I/O. InfoQ had a virtual panel with the creators of some of the most popular 3rd party libraries and frameworks that utilize Node.js.
Node.js author Ryan Dahl has given a very good brief explanation of what Node.js is during his JSConf 2010 presentation[PDF]:
- Server-side Javascript
- Built on Google’s V8
- Evented, non-blocking I/O. Similar to EventMachine or Twisted.
- CommonJS module system.
- 8000 lines of C/C++, 2000 lines of Javascript, 14 contributors.
Node.js has been gaining lots of attention lately and Peter Griess Principal Engineer at Yahoo! Mail has mentioned that his company is looking into using it in production.
Currently there may be about 300 projects related to Node.js on GitHub and InfoQ has contacted the creators of some of the most popular ones for this panel.
The participants were:
- TJ Holowaychuk from Express, a Sinatra inspired Node.js web development framework,
- Guillermo Rauch from Socket.IO, an simple HTTP Socket interface implementation and server,
- Matthew Eernisse from Geddy, a web development framework for Node.js. similar to frameworks like Merb, Rails, Pylons, or Django,
- Astro from node-xmpp, an idiomatic XMPP library for Node.js
- Peteris Krumins and James Halliday from StackVM, a startup that tries to make virtual machines more accessible over the web using Node.js,
InfoQ: Would you like to tell us a little bit about your project? What problem does it try to solve and how does it go about it?
TJ (Express): As you already know Express is heavily inspired by Sinatra, initially this was mainly to aid in making developers feel comfortable on the new platform with new idioms. That being said it has evolved quite a bit in 1.x, it now utilizes the popular middleware framework Connect (which I co-authored), and removes some previous dependencies that were not a good fit for the project.
Some frameworks for node are interesting in their own right, however most attempt to be the "be-all end-all" library. My vision of Express is about options, not limitations.
Guillermo (Socket.IO): Socket.IO tries to make building realtime JavaScript applications possible by providing a simple API that abstracts all the differences between different browser implementations of HTTP transports.
Matthew (Geddy): Geddy is an MVC web framework in full-stack JavaScript. Geddy tries to make it easy for developers to write complex web apps in JavaScript, sharing code between client and server.
Geddy uses patterns common in existing MVC frameworks such as Rails or Django: routes with controllers/actions, models, templating. But Geddy does all this with JavaScript, so things like models, validations, and templating can also be used in the browser.
Geddy has simple, resource-based routing, and proper content-negotation, so it's super-simple to use it for lightweight Web services that serve structured data.
Astro (node-xmpp): My goal was creating an XMPP library that can be naturally used in a Node.js environment. One previous library was based on Strophe.js for the browser and just didn't match the conventions of Node.js. It didn't use EventEmitter, didn't make use of the SRV-capable DNS resolver, and most importantly, didn't support both XMPP client and component connections.
Peteris & James (StackVM): Our project is called StackVM and it brings virtual machines to the browser with HTML and JavaScript. StackVM shows what VMs can do when they’re not bound to the desktop. For example, users will be able to embed VMs in their websites, share their work with collaborators, create online product demos for applications, and create new mashups that weren’t possible or practical to build before.
InfoQ: What are the main components of your project, how do they interact with each other and with Node.js?
TJ (Express): The Connect middleware framework plays an important role in Express 1.x, components previously known as "Plugins" are now Connect "middleware". This abstraction means the community can form powerful frameworks with all the commonalities in a much shorter period of time. The core components and features of Express as of 1.0beta are:
- Robust routing
- View system including partials and collection support
- Environment based configuration
- Session based persistent "flash" notifications
- Content negotiation
- Response utilities
- Redirection utilities
- express(1) for generating skeleton apps
Guillermo (Socket.IO): Socket.IO is composed of a JavaScript client that runs on the browser, and a JavaScript component that runs on the server (Socket.IO-node).
Matthew (Geddy): Geddy is an MVC framework. You define routes that map URLs to actions on a *controller.* You can create *models* to define the objects you want to use in your controllers. Finally, you respond to requests by rendering a *view,* which can be some sort of template, or just formatted data.
Geddy uses Node.js for its HTTP server, and for server-side execution of all the routing, controller actions, and template-rendering. With full-stack JavaScript, a lot of this same code also runs in the browser.
Astro (node-xmpp): Because Node.js is all about scalable server software performance was very important to me. That's why I first set out to evaluate the available XML SAX parser libararies and then started creating node-expat myself. That is just a C++ binding to libexpat, one particular fast parser library.
node-xmpp itself was modelled a bit like xmpp4r, the Ruby XMPP library, because I was used to use that a lot. Client & Component both derive from Connection which does the XML stream processing. Connection is based off a vanilla TCP client stream so a user still has access to those features: raw data management, send queue management et cetera.
Peteris & James (StackVM): Our main component is the StackVM itself (http://github.com/pkrumins/stackvm/) which uses several other Node.js modules that we have written for Node.js:
- dnode (http://github.com/substack/dnode) DNode simplifies communication among StackVM processes and the web browser by providing a simple, object-oriented interface for remote function calls with transparently wrapped callbacks to pass data around.
- node-bufferlist (http://github.com/substack/node-bufferlist) With bufferlist, we can build binary stream parsers in an asynchronous way.
- node-rfb (http://github.com/substack/node-rfb) This module uses node-bufferlist to talk the client side of the RFB protocol used by VNC.
- node-png (http://github.com/pkrumins/node-png) We use this module to render PNG images for the screen updates captured by node-rfb.
- node-jpeg (http://github.com/pkrumins/node-jpeg) This module takes output of node-rfb and produces JPEG screen updates.
- node-video (http://github.com/pkrumins/node-video) With this module, we can record screencasts of the screen updates captured by node-rfb.
- node-base64 (http://github.com/pkrumins/node-base64) This module takes output of node-png, node-jpeg, base64 encodes it and sends it to browser (because there is no easy way to stream binary data).
Then we also use Socket.IO (http://github.com/LearnBoost/Socket.IO) and Socket.IO-node (http://github.com/LearnBoost/Socket.IO-node) modules written by others to communicate between the browser and StackVM.
So here is how it all works: StackVM uses node-bufferlist and node-rfb to open a connection to a virtual machine’s VNC port. As screen updates come in, node-rfb forwards them to node-png or node-jpeg, which generate images. Then they are forwarded to the browser via Socket.IO. DNode abstracts over the Socket.IO transport to route messages between the server and browser in a more asynchronous way.
InfoQ: What were your main reasons for choosing to work with Node.js? What are the key benefits for your day-to-day work?
TJ (Express): My interest in node began roughly a year ago, when I started Express. I wanted a familiar and easy way to define a web app using a familiar language. I downloaded node's source shortly after I had heard about the project and began poking around in the internals, and I liked the direction it was heading. After having worked with Ruby for the past few years, there was a simple truth about JavaScript that always stuck, the more I used JavaScript the more I liked it.
Guillermo (Socket.IO): Node makes writing realtime applications really easy. The main reason is that in Node, you're in the driver seat because it allows you to create your own HTTP server in a few lines of code. That kind of power cannot be achieved in typical stacks like the common Apache+PHP one, because PHP is only executed when the request is completed, and is expected to run for a short period of time, produce a response and exit.
On the other hand, in Node requests can extend for long periods of time without sacrificing performance or scalability, as it's a fully asynchronous, single-threaded, single-process framework.
Matthew (Geddy): I chose Node.js because it's the very first credible implementation of server-side JavaScript that was not based on the JVM. I code a lot of JavaScript in my day-to-day work, and not having to switch back and forth between two languages for client and server is an enormous benefit.
Astro (node-xmpp): I want to master concurrency in IO-intensive applications. The canonical choice would be programming with threads, but they're expensive. Erlang and GHC tackle this by mapping light-weight green threads onto one native thread per CPU core. Despite of this feature developers still have to bear with synchronization and avoiding race conditions.
In asynchronous programming you're not developing serial "threads" of code, and thus it doesn't feel that natural. On the other hand being relieved of all the synchronization mess really felt more easy to me. Once you've got that concept it's just getting used to the "staircasing", that is nesting functions in nested functions.
Peteris & James (StackVM): Performance is really exceptional in Node.js without having to drop down and write code in C. Node.js doesn’t have any legacy blocking code to worry about so everything is asynchronous from the bottom-up. There are also lots of libraries for doing new experimental stuff like websockets and the community is very active. Using the same language on the browser and the server makes it easier to reuse code and JavaScript is an easy language to whip up quick prototypes in.
InfoQ: Do you think Node.js is mature enough for mainstream adoption? What do you think about the surrounding ecosystem?
TJ (Express): That is a tough question, compared to many other communities it is still quite small. Personally I feel the project (and the community) has great potential, and chances are it's Django, Drupal, or Ruby on Rails equiv will eventually come along and pump even more traffic into contributions.
Guillermo (Socket.IO): Node is getting really mature really quickly. Today, it has almost half as many followers on github (http://github.com/ry/node) as the Rails framework (http://github.com/rails/rails), which is hands down one of the most well known frameworks for developing web applications. And this was achieved in a very short period of time.
Matthew (Geddy): It's mature enough for some purposes, but it's still way to early to do the kind of general-purpose development that people today use something like Ruby or Python for. Give it a year or two.
Astro (node-xmpp): Mainstream adoption can be very easy one day because JavaScript is quite a popular language.
Node.js is in motion and you should often test your code against the newest version. I think it can get problematic if you're going to release code that should work for everybody regardless of whether their Node.js version is fresh or a few months old already.
Peteris & James (StackVM): If it isn’t mature enough in one regard or another, we’ll help to make it more mature. It’s open source, so if we find a bug or we need some new feature we can hack it together ourselves if necessary. The community-driven ecosystem is great: open source, plenty of ideas, many engaged developers who know their stuff, tons of documentation, blog articles, and new Node.js libraries on GitHub every day.
The API does sometimes change between versions slightly but Ryan Dahl has promised a new era of stability in the upcoming version 0.2.
InfoQ: Are you using any other Node.js libraries/frameworks in your project or everyday work? Is there any other Node.js library/framework [besides your own] that you think is especially useful and that other developers should look into?
TJ (Express): On a day to day basis I use a number of my own libraries, as well as the following works which I have great respect for the developers and their projects:
- Socket-IO http://github.com/LearnBoost/Socket.IO (Guillermo Rauch)
- Formidable http://github.com/felixge/node-formidable (Felixge)
- Less.js http://github.com/cloudhead/less.js (Cloudhead)
- Redis client http://github.com/fictorial/redis-node-client (Fictorial)
Guillermo (Socket.IO): We're using node on a daily basis. Aside from Socket.IO, I maintain Mongoose (http://github.com/learnboost/mongoose), an ORM for the NoSQL document-oriented database MongoDB, and other smaller components.
Matthew (Geddy): Node-Jake is a JavaScript build tool similar to Make/Rake:
http://github.com/mde/node-jake
Logan is a small x-env (cross-environment) test runner for testing code that has to run in Node.js, in the browser (and also TheRubyRacer, which embeds V8 in a Ruby process):
Astro (node-xmpp): There are a few to mention. First, the package manager npm by Isaac Schlueter. You don't want to miss the convenience of a package manager anywhere.
ext.js by TJ Holowaychuk has been quite useful: it augments basic JavaScript types with a lot of convenience methods everybody should find useful.
Not to miss Senchalab's Connect web framework. The Node.js built-in HTTP support is its killer feature, and Connect is going to be what Rack is for the Ruby world. Of course, that won't look as straight-forward because it's all asynchronous.
Peteris & James (StackVM): The node package manager, npm (http://npmjs.org/) is a great way to use and discover libraries written by other developers. We use Socket.IO, which abstracts over websockets and provides fallbacks to support websocket-style communication for browsers that don’t have websockets yet. We’ve been experimenting with node-compress to speed up data transfer to the browser on top of Socket.IO too.
InfoQ: If a team wants to start working with Node.js, what are the main pitfalls that they should look out for? Any tips or “best practises”?
TJ (Express): Generally the same as with an environment. GIT is a must in my opinion, I have never been satisfied (in fact repulsed) but other SCMs. Setting up a continuous integration server, even a simple one like CIJoe http://github.com/defunkt/cijoe is a must, and tests tests tests! I cannot stress testing enough, for all my node projects I use the "Expresso" project http://github.com/visionmedia/expresso which supports code coverage reporting, letting you know exactly what code you missed, these simple steps should keep a project rolling.
Guillermo (Socket.IO): My recommendations are:
- A very strong, almost fanatic, familiarity with the JavaScript language.
- Solid understanding of all the different layers that make a website work. That means understanding the low level aspects of it (solid knowledge of HTTP, and protocols and networking in general would help tremendously)
- Try to "forget" about some different ways of achieving the same things in other languages, and learning the "Node" way of doing them (for example, problems that you would solve using threads in Java, you solve differently -and easily- in Node)
Matthew (Geddy): It is still pretty early with Node.js, so there are still API changes that break code. Be prepared to make changes to your code -- or use a framework like Geddy or Express, where the framework author can keep a consistent API through Node's changes.
Astro (node-xmpp): While JavaScript's syntax is really trivial and can be understood by anybody who has seen a curly-braced language, the prototype-based inheritance system can be confusing at first. Be sure to educate your mates on the implications.
If you still feel easier with conventional class-based inheritance you might want to look out for one of the libraries that emulate inheritance the Ruby way in JavaScript.
Peteris & James (StackVM): Blocking I/O is probably the biggest pitfall to be aware of, since the entire program will wait for that call to finish before doing anything else. Use blocking calls sparingly and only for initialization, especially when writing network services where performance matters. Asynchronous I/O and event-driven programming requires a different mindset from most programming. Embrace it.
InfoQ: From your experience, how does a JavaScript codebase scale as a project gets bigger and bigger? What tools are you using and how good do you think tool support is for real-world Node.js projects?
TJ (Express): I think this one is completely up to the developer. JavaScript has a history of being a "nasty looking" language, but it does not have to be. I think inline comments are quite important, and aid in creating visual whitespace as well which prevents the code from appearing cluttered and messy. As far as organizing the project as whole it never hurts to break out logical chunks into separate files which should be common practice for any language.
Node's adoption of the CommonJS module pattern is actually my favorite organizational utility I have used in a language before. Typically most environments consist of a flat namespace, so in reality you still run into namespace collisions all the time, the CommonJS module system is great at preventing this.
Guillermo (Socket.IO): Having JavaScript run on the server and the client is certainly something that makes people excited. I think it's specially important for companies as recruiting becomes easier, and entire teams can work on many different aspects of the same application. Node also has many tools for developing in a professional environment:
- A very simple yet powerful module system (based on CommonJS)
- Lots of testing frameworks (Expresso, Vows, etc)
Matthew (Geddy): On the server side, JavaScript now has pretty much the same tools as Ruby or Python -- CommonJS modules provide a mechanism to keep code modular. Build tools like Node-Jake allow you to automate build and packaging so they happen in a predictable way.
Client-side JavaScript requires a little more work. But JavaScript objects provide a natural way of encapsulating functionality, and there's a long history of best practices for JavaScript "programming in the large" on the client. Large companies like Facebook create their own frameworks, but more sophisticated toolkits like Dojo or YUI have lots of affordances like this.
Astro (node-xmpp): I belong to the fraction that would recommend static typing for larger projects.
I'm also in favour of Test-Driven Development; not sure if the Node.js community has chosen a de-facto standard library for testing.
Peteris & James (StackVM): Node.js has a module system which helps a ton with project scalability, but large software projects tend to be far buggier and harder to test than small ones in any language. With StackVM, we’re trying to avoid the trap of a large codebase for as long as possible by factoring out functionality into reusable, open-source modules. So far, this approach has kept the main StackVM codebase more flexible and the supporting libraries are far easier to test in isolation. People using our modules in the community have even already identified some bugs for us. Probably the most useful tool for Node.js right now is npm, the node package manager. Using npm, we can easily make use of libraries that other programmers have written and we can share our own modules more easily amongst ourselves and the Node.js community.
InfoQ: Are there things that you think are missing and should be added to Node.js? How do you think it should evolve?
TJ (Express): I think it should continue as-is, being a low level foundation for frameworks to grow on. It should be considered a "blank canvas" to work from, with very little imposed upon the developer,
Guillermo (Socket.IO): SSL support in the HTTP server is incomplete and buggy. Besides that, it just comes down to get people using it, realizing the best patterns from looking at the common usage, and fixing bugs.
Matthew (Geddy): People are working on a package manager (NPM), so hopefully once that solidifies we'll have an easy way to install libraries and specify dependencies for projects we're building.
There is a wide array of embryonic projects attempting to deal with datastores and persistence, but no obvious best solution. We still need a solid ORM that works as much as it can to provide a common API for both relational and non-relational stores.
There is a lot of pent-up demand for server-side JavaScript, so I think the ecosystem will evolve very naturally to fill in all the gaps we currently have.
Astro (node-xmpp): In particular I would like to drop the node-base64 dependency from node-xmpp. A patch exposing Base64 encoding/decoding had been posted, but didn't get merged.
I saw CommonJS mentioned here and there. It's a project that aims to standardize libraries across multiple JavaScript interpreters/frameworks. Their module system has been implemented and npm uses their package.json spec. Other than that I fear there won't be too many shared patterns because contrary to many other implementations everything has to be asynchronous in Node.js.
It's also interesting to many people trying to get web frameworks right.
Asynchronous programming can sometimes be a hassle and I understand if developers sometimes omit the callback references just to keep the control flow more straight-forward.
Peteris & James (StackVM): Whenever we think up some potential new library that we’d like to see in Node.js, we hack it up ourselves and release it so other programmers don’t need to waste time reinventing the wheel. Plenty of other programmers do the same thing, and the supporting ecosystem of libraries evolves on its own naturally. The Node.js core is intended to be small with the bulk of the functionality provided by supporting libraries, which is a good way for a community-driven project to distribute the workload and encourage participation.
InfoQ: What is the future roadmap for your project?
TJ (Express): At this point and time just keeping Express lean and mean. I am always accepting new ideas and patches for focused features that fit well with the project. Developers could even now consider Express a "frameworks framework", as itself could easily become a building block for larger more opinionated frameworks. Lastly I would like to thank Ciaran and Aaron for their great contributions, and my employer Sencha for allowing myself and Tim to give back to the community.
Guillermo (Socket.IO): Right now I'm hard at work at making sure it can be as fast, lean and well-tested as possible.
Matthew (Geddy): There is still a lot of work to be done on Geddy:
1. ORM query API
The persistence piece of the current model code is very much just a stake in the ground. ORMs are very complicated, and anything that tries to interface with both relational and non-relational stores is going to be even harder to build.
2. A plugin API
Geddy needs an easy way to allow snap-in functionality built by third parties.
3. Authentication
Authentication is the perfect example of what should be implemented as a plugin.
4. Better modularity
Geddy was written to provide an integrated, "out of the box" experience that lets developers start writing real applications with a minimum of fuss. However, it should be easier to use components from Geddy "a la carte," or replace one specific piece with something else.
5. Support for other template libraries than simple EJS
People like to use crazy stuff like Haml or Moustache. Geddy should let them go crazy with that stuff.
6. Test coverage
Geddy has extensive tests for its router, and a solid suite of tests for models and validations, but the rest of the code isn't covered. Geddy's code needs to be refactored to be more testable so we can have more complete coverage.
Astro (node-xmpp): I was pretty happy about all the Github watchers I got in no time after I published the project. Other than that I'm still waiting for actual contributions.
My very own goals are met because I just wanted the XML parsing and authentication done, not to implement any particular XMPP XEPs on top of the library. That's what I leave to the application programmer for now.
Peteris & James (StackVM): In the shorter term, we’re focused on adding more features and tuning performance. Further out, we’ll probably be more focused on hardware and scaling.
You can find more information on Performance and Scalability, or JavaScript, right here on InfoQ!