Multi-Process Node.js: Motivations, Challenges and Solutions

Although Node.js doesn't have traditional threads or bother directly with issues like multiple-processor concurrency, these issues do arise in production environments. InfoQ has conducted a virtual panel with the creators of Node.js projects that deal with these concerns.

The participants were:

• TJ Holowaychuk, creator of Cluster,
• Kris Zyp, creator of Multi-node,
• Vladimir Dronnikov, creator of Stereo,
• Rob Tweed, creator of Q-Oper8 and
• Pedro Teixeira, creator of Fugue.

InfoQ: Would you like to tell us a little bit about yourself and how you got started working with Node.js?

TJ: I'm TJ Holowaychuk, I'm 24 and I live in Victoria BC, Canada, and I'm working with the awesome team at LearnBoost, building revolutionary education software. I started off as a graphic designer roughly 4 or 5 years ago, later getting involved with software and realized that in many respects software can be just as creative, if not more so. I'm 100% self-taught in both fields, however I made the usual mistake of learning PHP before anything else.

After a year or two of writing Drupal modules & web applications I started dabbling with other languages, and soon realized that PHP is terrible. I went on learning a few more languages and ended up settling with Ruby & the Sinatra web framework, which I used to write corporate data-mining software for roughly a year and a half.

Related Sponsored Content

My first year of Ruby opened my eyes to some new concepts, although the more experienced I became the more I disliked the magic the language provided, which was often misused and caused nothing but ambiguity and headaches. I started getting interested in parsing technologies like Ragel and Lemon which lead me to Ryan Dahl's "ebb" server for Ruby, which I believe was the early inspiration for Node.

Kris: I am Engineer working at SitePen (http://www.sitepen.com/), Dojo committer, an active participant (representing Dojo) in TC39 (EcmaScript committee). I have been involved with working on bringing JavaScript to the server for a number of years, before Node.js existed, with the Persevere project (http://www.persvr.org). With the advent of Node.js, unparalled JavaScript performance performance and scalability on the server was made possible, and Persevere has since become very Node.js focused, leveraging this powerful platform.

Vladimir: I'm a freelancer from Russia, Tver. Got interested in server-side JavaScript circa 3 years ago. I had been playing with http://dev.helma.org/ and http://ejscript.org/, looking for a good SSJS solution to a RESTful framework, when some day google gave me reference to Node.js. I recall it was at version 0.1.18 or such. I played a bit with example but asynchronous programming was way new to me, so I bookmarked the page and kept looking. I found @kriszyp's Persevere 1.0 which is written in Java, then came to his Node.js rewrite called Persevere 2.0 which was at early alpha stage that time. I started using it, then digging into its code, then contributing. This process gave me much to understand Node.js coding paradigm and enter async programming. So I can say I see Node.js evolving from the very early stages.

Rob: I'm co-founder of a UK company called M/Gateway Developments.  Since 1996 we've specialised in creating web application development tools, technologies and frameworks, in particular for the healthcare and financial services sectors.  My particular interests are in browser-based business applications (particularly for mobile devices) and a class of NoSQL database that uses so-called global storage. GT.M, CachŽ and the recently-launched Globals are the three databases in which I specialise.  CachŽ (and, to a lesser extent, GT.M) pretty much dominates the healthcare marketplace, and our products underpin a great many web-based healthcare applications around the world and particularly in the US. I've been working with Javascript since the late 1990s, and have always believed that the integration of Javascript and global storage was a potentially very powerful combination.  It was after watching the video of Douglas Crockford's talk called "Loopage" over a year ago that I realised that Node.js was going to be the way to realise that integration.  I've been working with Node.js in combination with GT.M and CachŽ ever since, and it is, indeed, proving to be as powerful as I expected.

Pedro: I'm a programmer, freelance worker, doing mainly Ruby and Node.js. I started doing Node after a friend of mine told me about it back in early 2010. I got very curious about it and really enjoyed Ryan Dahl's 2009 JSConf.eu presentation. I also liked the no-frills approach of the three-liner "Hello World" http server script, so I decided to dig in. I then started experimenting with it, mainly using the http server API and trying out some web frameworks that were out there. Discussing Node with friends, some questions were raised about it's internal workings, so I started looking into the source code, understanding the internals. Soon after that, since all my freelance work was only Ruby, and as a way to force myself into deepening my knowledge about this platform, I decided to propose a talk about Node to the Codebits.eu conference, which was accepted. So my efforts studying Node internals had to me enforced, but the lack of online resources was something that struck me as an obvious need that was still unfulfilled, so I decided to start a series of screencasts, now Nodetuts.com.

InfoQ: Would you like to give us an architectural overview of your project and how it actually works?

TJ: Cluster implements a fairly typical prefork server, in which by default one child process (worker) is spawned per CPU, however you can specify this number to anything you like. Cluster provides crucial functionality like zero-downtime restarts and worker resuscitation.

Cluster is a must-have even for small deployments, suppose you have a single core the redundancy provided by spawning a few workers to retain availability is usually still necessary. Cluster is also very performant, as it is not a reverse proxy. It works by creating a socket and binding in master, then passing the fd to the children, which can then accept(2). While this does not provide the flexibility of a reverse proxy, it's very simple to integrate with any Node server (not just HTTP!), a simple example looks like this:

    cluster(server).listen();

Cluster also has a strong notion of plugins, bundled with a few for exposing a command-line for managing your cluster, logging, automated reloading (great for development), a REPL for managing the cluster to view stats, spawning additional workers etc. The community has come up with quite a few useful plugins as well, logging to remote services, email notifications, live monitoring to name a few. Below is an example of cluster with some settings and plugins in use:

cluster(app)
  .set('workers', 8)
  .use(cluster.logger('/path/to/log/dir'))
  .use(cluster.debug())
  .use(cluster.repl(8888))
  .listen();

One of these experimental plugins is named "cluster-live", a small node application built with Socket.IO and Express, providing real-time statistics for your cluster. As shown in the screenshot below, Cluster can may relay connection and request-level data, which can be extremely powerful. 

(Click on the image to enlarge it)

Cluster can also function as a generic process manager, for example it's trivial to run a cluster of Kue workers to process job queues. Cluster running without a server looks something like this:

var proc = cluster()
  .set('workers', 4)
  .use(cluster.debug())
  .start();
if (proc.isWorker) {
  var id = process.env.CLUSTER_WORKER;
  console.log(' Êworker #%d started', id);
  setInterval(function(){
    console.log(' Êprocessing job from worker #%d', id);
  }, 3000);
}

Kris: Multi-node is designed to make it very easy to run a NodeJS web server on multiple processes. Starting multi-node with an HTTP server object will cause Multi-node to spawn child processes and each of these child processes is given an environmental variable so that the multi-node module in these child will recognize that is child to another master (and not spawn more children). Multi-node then uses Node's socket descriptor passing capability to deliver the socket to each child for shared listening. Multi-node also provides inter-process sockets for communication between the processes, and a simple framing protocol to simplify sending messages to other process. It also listens for dying children to (optionally) respawn children for greater robustness.

Vladimir: I started Stereo as my rewrite of @kriszyp's, being inspired by his excellent article. I had been charmed by the beautyness of CoffeeScript, thus *.coffee sources. The difference is mainly in intercommunication network between workers. I always wanted any particular worker to be as dumb as possible in sense of minimizing 'knowledge' it has about other workers. In particular, this means worker process should not contact another worker directly, but instead should contact master which in turn should broadcast sent message, and initial worker should then process the message. This requires an additional roundtrip for local messages, but allows to scale well. In order to reduce increased dependence on master node (which starts to present a single point of failure), master should again be as dumb as possible and do nothing but relaying messages. I also implemented a simple REPL interface to control the number and status of workers. This is a helpful option both to debug and control the production solution.

Rob: First a bit of background:.  InterSystems (the vendor of the CachŽ database) recently launched a free database called Globals, which is essentially the core global-storage engine that powers CachŽ.  With encouragement from me, they have added a native Node.js interface which, largely as a result of it running in-process with Node.js, is proving to be capable of truly stunning performance. Q-Oper8 was initially developed as a solution to a couple of key issues that the Globals database presented for Node.js:

- by running in-process, some people in the community have raised concerns about it having such an intimate relationship with the Node.js server process

- InterSystems have implemented two versions of their in-process APIs: synchronous and asynchronous.  The experience coming from the technical developers at InterSystems is suggesting that the synchronous APIs perform significantly better than the asynchronous ones.

Having dealt with asynchronous access to the GT.M and Cache databases in my work to date, I believe that it will be a real problem for serious uptake of Node.js as a technology for developing large-scale business/enterprise database applications.  Asynchronous logic is just too weird for database application developers used to standard programming practice, too time-consuming to figure out and write, and it presents a seriously high maintenance cost.  However, Globals, with its extremely high-performance and its synchronous APIs looks like it could completely change the accepted practice for database access from Node.js.  But, of course, synchronous database access is an anathema to the entire Node.js ethos.

So, Q-Oper8 was primarily designed to create an environment in which Globals' synchronous APIs could be safely used without breaking the golden rule of never blocking the Node.js server process.  As it turns out, it has also provided an interesting generic multi-process solution for Node.js that has wider applicability.

Q-Oper8 consists of four main parts:

- a master Node server process that is 100% asynchronous and non-blocking

- a pool of persistent child Node processes that, crucially, only ever handle a single action/request at a time

- a queue of pending actions/requests with an API for adding requests to that queue

- a queue processor that attempts to allocate requests/actions on the queue to available child Node processes

The really important feature of the Q-Oper8 architecture is that you pre-determine the size of your child Node process pool and these are started before anything else happens. Subsequently, your child Node processes persist throughout the lifetime of the master Node server process, so there is no setup/teardown overhead or latency when handling requests/actions: the child processes that are flagged as being in the available process pool are instantaneously available for use by the master process.

It's the fact that each Child Node Processes only ever handle a single request at a time that makes it possible for them to support synchronous coding, since they don't need to worry about blocking anyone else.  Once started, everything is then completely event-driven, so there are no polling overheads or delays. Processing of the queue is automatically triggered by two main events:

- adding a request to the queue

- a child Node process becoming available

So, in summary, if you have to do anything that will require synchronous database access, you simply add a request to the queue. That request gets passed to a child process (if one is free), and the synchronous activity happens in that child process, blocking nobody else.  The main server process is unaffected, and it can continue to carry out other asynchronous activity at the same time as it processes the Q-Oper8 queue and results returned from the child processes.

Q-Oper8 appears to work very well, and delivers other benefits.  It:

- separates the close in-process relationship of the Globals database from the main Node.js server process

- also protects the main Node server process from errors or problems that might occur when handling particular requests/actions.

- further isolates the main server process from processing that requires significant amounts of computation that would otherwise slow down the performance of the main server process for all concurrent users

- allows load to be distributed across multiple Node processes, allowing Node to exploit multiple-core CPUs.

Pedro: I'm a big fan of the Unicorn HTTP server, which is a Ruby HTTP server that allows code hot-swapping, meaning that you have no application downtime when you are deploying. I think that works on Unicorn by forking the worker processes from the new master.

Fugue also allows for no-downtime application deployment, but works in a slightly different way: Node is not fork-friendly, but allows you to pass file descriptors between processes by using an old off-band "trick" present in UNIX Domain sockets. Node makes this really easy, with a really intuitive API. So, when the master boots, it spawns off new worker processes and, once they're up, they ask the master for the server file descriptor using a UNIX Domain socket, and then they all go listen on that file descriptor. So they're all listening on the same server socket, all of them individually accepting connections. Then, when a new request comes in, the OS selects one of those worker proceses. The beauty of this is that the master is not at all envolved in accepting and using connections. The master can even die and the server will still accept connections, and the OS will load-balance between the worker processes.

When you want to redeploy your app, you push your code and signal the master process, which in it's turn launches a new master. This new master then launches it's workers, and if and when all of them are up, it signals the old master, and the old master kills off the old workers and finally kills itself. This is a somewhat complex ballet, but it means that all these worker processes are all listening to the same server socket, which then means your application has no downtime.

Also, Fugue allows for fail-over of the worker processes. Say, for instance, that a worker process dies. It will be automatically re-spawned by the master.

Besides the HTTP server, Fugue can also be used on TCP servers or any object that inherits from it (like the Node http.Server or the Express.js HTTP Server), so, unlike Unicorn, you can use it for load-balancing and fail-over any TCP-based server.

InfoQ: Are there any other alternatives you might to a team that wants to take Node.js multi-core?

 TJ: Things like node-http-proxy don't necessarily replace Cluster, nor does Cluster replace it, but Nodejitsu's http-proxy module is a light-weight reverse proxy fantastic for setting up "virtual hosting", processing incoming requests, so they compliment each other quite well if you need that sort of thing. Depending on deployment needs one might run http-proxy with Cluster for redundancy, forwarding to several machines or processes. Alternatively you may have a Cluster of proxies forwarding to more Cluster servers, each serving up a different application, there are lots of possibilities but it is a great tool for utilizing a machine, or simply for availability.  

More information you can find here.  

Vladimir: I believe https://github.com/LearnBoost/cluster is the way to go.

Rob: Before developing Q-Oper8 I looked at the available multi-process solutions for Node.js, such as Cluster, Multi-Node and hook.io.  None appeared to include the criterion that I saw as critical to safely handling synchronous database coding: namely the limitation of child processes to handle a single request at a time.  If you're OK sticking with asyncronous database logic, and you don't have computationally heavy processing, the other multi-core alternatives will probably be more than adequate. However, with Q-Oper8 and Globals I reckon we now have the best of all worlds: all the awesomeness and speed of Node.js combined with standard synchronous database programming against an equally super-fast and extermely adaptable database.  I'm really excited to see how this combination can now be exploited within the healthcare and financial services sectors that we specialise in, but there's no reason to believe it needs to be limited to those sectors.

Pedro: I started building Fugue on September 2010, and the Node world has evolved immensely in the meantime. On my side, I got entertained by freelance work and by new open-source projects and other adventures, so the time to make it evolve has been scarce. Also, the TJ Holowaychuk's Cluster project uses these techniques and extends on that, and I think it's solid, so nowadays I would prefer using Cluster instead of Fugue.

There are other alternatives if all you want to do is load-balance an HTTP Server inside the same box like Spark, Spark2 and probably others.

But if you want to go multi-core you could use entirely different techniques depending on your application. Node already has a great API for managing processes (spawning, killing and communicating with the standard pipes), and Node 0.5 (soon to be stable 0.6) introduced a Node fork API, which establishes a message channel between the parent and the child process. I could easily see architectures where you pre-spawn worker processes and then order work to be performed by using your own protocol using this channel.

If you want to do off-line work there is the classic queue approach, which can also work cross-machine. A project that I use for this is Kue, which uses Redis for message persistence and delivery.

I'm also excited by this new project named hook.io, which allows you to spawn a bunch of processes and have them all talk to each other using this Event Emitter bus. You can use this architecture for spawning different Node processes (on different machines if you like) and have them all send events to each other to coordinate work. If you do this right you can have a fault-tolerant architecture that moves away from the client-server architecture and into something like a mesh of cooperating nodes, which I think will be the future for complex applications written in Node.

InfoQ: What is the roadmap for your project? How would you like to see it evolving?

TJ: Mainly maintenance, I would like to continue with making the master process more robust, perhaps an auxiliary process that would manage statistics etc to keep apps like cluster-live out of the master process.

Kris: This is a pretty stable project and is focused on filling one need and doing it well. However, there are some additional features that may be added at some point. We could improve the support for multi-process logging and add module monitoring for automatic restarting when modules change.

Vladimir: Actually I don't think now that multi-core solutions scale well horizontally, which is the primary need of every modern project if it starts to succeed. Multi-core solutions mostly use short-range communication, say unix domain socket, so _the whole cluster is limited to sit on one machine_, which is neither extensible, nor robust because it presents a classical single point of failure. Moreover, I believe passing file descriptors (to share listening 'master' socket) via `net.Socket#write` in pure Unix feature (didn't explore the current state of things). So I drifted to using a balancer ( this is my choice) to several equal Node.js worker processes. That's why I consider multi-core kinda interesting sugar Node.js provides for free, and no more.

Rob: The original version of Q-Oper8 used the new child_process.fork() method that became available in the new unstable 0.5.x version of Node.js.  I'm now porting it back to the stable 0.4.x version by using the standard Node.js child_process APIs. Otherwise I don't forsee many further major changes.  As a solution to the requirements of the Globals database, it seems to do what's needed. However, I would love to see its architecture being subsumed into a generic set of capabilities within Node.js itself, or re-implemented at a lower level - primarily to obtain even better performance. Q-Oper8 is no slouch: I've been seeing queue processing throughput of over 18,000 requests/sec on a very basic 4-core AMD Opteron server, but I can't help thinking that it could be further enhanced by architecting it at a lower level.

About the Panelists

TJ Holowaychuk is a open-source engineer who has backed Node since its infancy. He has also authored many popular Node.js modules, including the Express web framework, Cluster, Stylus, and Jade, among many others.

 Kris Zyp is an Engineer working at SitePen, Dojo committer, an active participant (representing Dojo) in TC39 (EcmaScript committee).

Vladimir Dronnikov, got his Ph.D. in Physics and Mathematics from the Tver State University. He is a freelance system administration, and before web development, he was contributing to http://busybox.net

Rob Tweed is a Director and Consultant at M/Gateway Developments Ltd in the UK.  Following a number of years at Touch Ross Management Consultants in London, since 1996 he has specialised in web, Ajax and (more recently) mobile web application development, in particular for the healthcare and financial services sectors and provides support, training and consultancy to a worldwide base of customers.  An avid supporter of Open Source development, his areas of expertise include Javascript, Node.js, ExtJS and Sencha Touch, and he is a long-term advocate of global-storage based NoSQL databases such as CachŽ, GT.M and Globals.

Pedro Teixeira is a Ruby, Rails and Node.js Software Engineer. He's the author of the Node Tuts screencasts and the Hands-on Node.js book. He also wrote some Node modules, including Fugue, Alfred.js in-process key-value store and others.