Flocker is a volume and container management system for Docker based on ZFS. It allows for stateful containers, such as databases, to be moved between virtual or physical hosts. This provides a capability that is analogous to the live migration features of some virtual machine hypervisors. Version 0.1 has been released by ClusterHQ as an Apache 2.0 open source project.
The two key components of Flocker are a network proxy at the front end and ZFS based volumes at the back end. The front end proxy can route requests for services to whichever host is running the active container. When a container is migrated between hosts the ZFS data volume is first synchronised and a handoff takes place from the active container to the replica so that the replica can become active. There is a short pause in availability as the handoff takes place, and a stated aim of the project is to allow no-downtime migrations between containers.
The Flocker architecture inherits from HybridCluster, which also made use of a front end proxy and ZFS storage fabric. Besides the focus on Docker the other main shift is to ZFS on Linux (rather than FreeBSD). Due to intellectual property concerns stemming from its origins at Sun, and subsequent acquisition by Oracle, ZFS has had a troubled journey to Linux. ClusterHQ founder and CEO Luke Marsden commented:
ZFS is an extremely mature, stable and ground-breaking filesystem. ZFS on Linux (ZoL) is now a native kernel module (not FUSE) and is nearing production readiness both in terms of stability and performance. It is already managing many petabytes of data at institutions like Lawrence Livermore National Laboratory and under our own stress testing has held up well for months without kernel deadlocks or panics. The licensing question is a non-issue and Oracle has no hold over the OpenZFS project (see the ZoL FAQ). ClusterHQ has recently hired Richard Yao, the second most active contributor on the ZoL project, to help accelerate the final fixes required to make ZoL incredibly rock solid. Richard will be writing up the state of ZoL on the ClusterHQ blog in great detail in the coming weeks.
Version 0.1 doesn’t support container linking, and Docker’s native linking system isn’t aware of the Flocker proxy and thus can’t take advantage of it. This can be worked around by passing environment variables with linking parameters pointing to the proxy rather than a specific container. Flocker already makes use of some parts of of Red Hat’s geard, and the ClusterHQ team plans to implement geard style linking in a subsequent release. On the topic of proxies Marsden said:
In order to be able to make containers portable, we believe that containers should be addressed via a proxy layer with some knowledge of the type of protocol they are serving. It's highly desirable to be able to access the same container via any public IP address in a cluster of nodes, since this allows effective IP based failover and portability between data centers by simply publishing multiple A records. Therefore, a container's address should be e.g. (https, "clusterhq.com") if it is an HTTPS web server. Or it could be (tcp, 1234) for a generic TCP service which should be made available on all the IP addresses a running cluster.
For the time being Flocker doesn’t play nicely with other parts of the Docker ecosystem such as Fig, Deis or Panamax. Providing robust state management to containers that would otherwise only be used for more superficial stateless systems does however provide a great deal of utility, so Flocker integration could become an attractive target for other projects. Flocker also only works for planned migrations such as those associated with capacity management and known faults that aren’t immediately terminal. Given that the ClusterHQ team has previously provided fault tolerant systems infrastructure for hosting providers it’s reasonable to expect that similar functionality will find its way to Flocker (or commercial offerings built on the Flocker base), and indeed ‘automatically configured continuous replication and failover’ is on the roadmap.
Developers wishing to try out Flocker can follow the getting started guide, which uses Vagrant and VirtualBox to run VMs with Flocker demos using MongoDB and PostgreSQL. Marsden’s recent presentation to the Edinburgh Docker Meetup on ‘Data focused Docker clustering’ is also available on YouTube.