The open source release of Docker in March 2013 triggered a major shift in the way in which the software development industry is aspiring to package and deploy modern applications. The creation of many competing, complimentary and supporting container technologies has followed in the wake of Docker, and this has lead to much hype, and some disillusion, around this space. This article series aims to cut through some of this confusion, and explains how containers are actually being used within the enterprise.
This articles series begins with a look into the core technology behind containers and how this is currently being used by developers, and then examines core challenges with deploying containers in the enterprise, such as integrating containerisation into continuous integration and continuous delivery pipelines, and enhancing monitoring to support a changing workload and potential transience. The series concludes with a look to the future of containerisation, and discusses the role unikernels are currently playing within leading-edge organisations.
This InfoQ article is part of the series "Containers in the Real World - Stepping Off the Hype Curve". You can subscribe to receive notifications via RSS.
InfoQ recently sat down with Adrian Mouat, author of “Using Docker”, and explored the motivations for writing the book, the reasons why Docker has captured the IT industry’s attention in such as short time span, and how containers can be best utilised within a typical software development lifecycle.
The “Using Docker” book is presented in three parts; background and basics, the software development lifecycle with Docker, and tools and techniques. Part 1 covers the historical context and development of Docker, provides installation instructions for a range of platforms, and examines the fundamental principles behind containers. An example web application, written in Python, is introduced and deployed with Docker. The sample code can be downloaded from an associated ‘using-docker’ GitHub repository, which allows the reader to follow the steps taken when ‘Dockerising’ an application.
Part 2 focuses on how Docker can be used within a typical software development lifecycle. The creation, packaging and execution of a simple web app is described in depth, and container-specific concepts like image distribution, volume mounting and Docker networking are discussed. Mouat takes care to relate container concepts and usage to a traditional continuous integration process, and examples utilise familiar tooling like Jenkins. Part 2 of the book concludes with a detailed discussion of container monitoring and logging with the ElasticSearch/Logstash/Kibana (ELK) stack.
The final section of the book, Part 3, examines tools and techniques from an intermediate to advanced level of knowledge (building upon the learnings from the previous two parts of the book). Networking and service discovery are covered in detail, as are the topics of orchestration and clustering, using platforms such as Docker Swarm, Mesos and Kubernetes. The final chapter of the book presents a comprehensive discussion of security in the context of containers, and provides information on executing containers with ‘least privilege’, image provenance and limiting access to the underlying host operating system resources.
InfoQ sat down with Mouat, and asked questions about Docker, the impact containers are having on the software development lifecycle, and what he believes is the future for Docker.
InfoQ: Welcome to InfoQ Adrian! Could you briefly introduce yourself, and also explain a little about why you chose to write 'Using Docker'?
Adrian Mouat: My name is Adrian Mouat and I'm currently setting up Container Solutions Labs, a sister company of Container Solutions that will focus on products and services in the container and microservices space. My background is in software development, where I've worked on a wide range of projects from large data processing frameworks to small webapps. I was introduced to Docker at a company I previously worked for where we used it to power a Continuous Deployment (CD) workflow for a small python webapp. I was impressed with the elegance and flexibility of the solution, which led to my writing a blog post about it. O'Reilly saw the post and asked if I would be interested in putting together a proposal for a Docker book. Thankfully, they liked the proposal and Docker became my life for the next year-and-a-half!
InfoQ: What interests you about containers, and why do you think they have captured so much attention?
Mouat: I believe containers will become a standard part of the normal software workflow. They enable new possibilities in computing, such as immutable infrastructure and portability between environments. Things that were hard to accomplish with VMs become much easier with containers, largely due to their speed and size, which enables the fast feedback loop craved by developers. Organisations have to pay attention, because if containers can enable a workflow that is just a few percent quicker, that has a knock-on effect that allows them to evolve faster than the competition. Even so, I've been amazed by just how quickly they've caught the attention of the industry.
InfoQ: If an organisation is looking to embrace Docker, where would be the best place to introduce containers within the software development lifecycle?
Mouat: That's a good question. Docker can help to solve a lot of problems in various areas. A good introduction can be to use Docker to instantiate Continuous Integration (CI) or testing environments - done properly that can significantly speed up the development pipeline and doesn't risk breaking production. However, you'll gain the most benefit from using containers throughout the lifecycle - having developers build images, pushing those images through CI and then on to operations for deployment. Done properly you will minimize the differences in environment between development and production which cuts out a bunch of potential issues - it kills the "but it worked on my machine" problems if you like. Of course, most organizations aren't going to want to change everything at once. Developers - rather than operations - are most likely to adopt Docker to begin with; they are naturally attracted to shiny new tools whereas operations focus on keeping things stable and running. I think this will change overtime as Docker proves itself as a trustworthy technology and operations start to reap the benefits that containers can have on deployment.
InfoQ: We saw a comment you made on Twitter that the rapid evolution of Docker means books often struggle to keep up-to-date. Do you think Docker is evolving too fast, and how do you plan to keep up with the rate of change?
Mouat: There is a fast pace of change. The first version of the book is up-to-date with Docker 1.8, which is still ahead of the packages in some Linux distributions. The orchestration frameworks and various tools also struggle with the pace, sometimes only supporting older versions of Docker. However, I believe we're already seeing a slow-down. The basic APIs and formats are starting to solidify and standardise, which will help a lot - one of the biggest worries is breaking changes, which causes existing tools and code examples to fail. Also, as the Docker codebase grows and the number of users increases, it will naturally become more difficult to make large changes without a lot of thought and consultation.
InfoQ: How secure are containers currently, say, in comparison with VMs?
Mouat: That's a common question. The first point to make is that containers vs VMs is a false dichotomy; there's no reason you can't use both (and you probably will).
That said, a lot of people do make the comparison and it's an interesting discussion to have.
Most people argue VMs are more secure as:
- they are a comparatively old and trusted technology and;
- each VM has its own kernel which is separated from the host by the hypervisor, unlike Docker where containers share the host's kernel.
However, containers also bring a lot of security features to the table, such as making it easy to create tiny containers with a correspondingly small attack surface or run containers with read-only file-systems and limited sets of privileges. Docker are committed to on-going improvements in this area and notable recent additions include support for user namespaces and seccomp.
InfoQ: What advice would you give our readers for running containers in a secure fashion?
Mouat: Remember the principle of least-privilege. Applied to containers, this means a container should only be able to access data and resources that are essential to performing its function. There are several ways this can be enforced, including:
- setting file-systems to read-only;
- limiting the amount of memory a container can access;
- isolating network access to other containers and;
- reducing the kernel calls a container can make.
If you want to learn more about least-privilege, check out the talk "Least-privilege Microservices" by Nathan McCauley and Diogo Mónica. Also, the above methods and more are fully covered in the free mini-book "Docker Security" (registration required), which is based on materials from Using Docker.
InfoQ: Over the past year a lot has changed in the networking and storage within Docker. Do you think these topics are approaching stability (we're thinking about the discussion around libnetwork in particular)?
Mouat: I wouldn't say there is stability yet. The discussion you allude to is where the Kubernetes project claim they can't work with the Docker networking model. It's also notable that Weave still recommends users run the stand-alone binary which inserts a proxy between the Docker client and daemon; they find the proxy provides a cleaner and simpler interface that has given them less issues. However, this is likely to change in time and I would stress that it is possible to run Weave as a straight plug-in without the proxy. I've seen less disagreement in the storage space; possibly because there is less diversity in the hooks that are needed to support various implementations, but this is only speculation on my behalf.
InfoQ: There is a lot of development in the associated cluster management and orchestration space e.g. Mesos, Kubernetes, Docker Cloud. Could you share any of your thoughts on the use of these technologies?
Mouat: It's a real battleground. Docker Cloud (formerly known as Tutum) is Docker Inc's SaaS offering and will presumably integrate Swarm support for clustering and orchestration. Mesos is being used by several large companies and has the longest track record. Google are seriously pushing Kubernetes and have developed a very powerful framework based on their knowledge from using containers internally over the last decade. It's hard to say how this will all shake out. There is also Nomad from Hashicorp who are hoping to take a slice of the pie.
InfoQ: Do you believe that companies are ready for the operational challenges of working with containers, such as logging and monitoring?
Mouat: The monitoring problem becomes a lot more acute when using containers, especially if you adopt a microservice approach - you can easily end up with thousands of short-lived containers that traditional monitoring tools will struggle to cope with. Lukasz Guminski, one of my colleagues at Container Solutions, wrote an in depth article on this, and Adrian Cockcroft has also written and talked extensively on the subject, including this InfoQ article. The good news is that there are a lot of new companies and products in this space, such as Ruxit, Prometheus, and Weave Scope. It's certainly possible to get a working solution at the moment, but I would expect to see a lot of innovation and growth in this area. For companies moving towards containers, my advice is simply to be aware of this issue and look into the various tooling that can help you.
InfoQ: Can you offer any comments about your predicted development of the container space within 2016?
Mouat: I'm mainly curious about the orchestration space. If Kubernetes ends up with the majority of the orchestration and clustering space, they could potentially switch container runtime (for example from Docker to rkt) which would be a blow for Docker. I expect to see a lot of action and discussion in this space.
We'll also hear more about different implementations leveraging the Docker API. We already have the excellent Triton from Joyent which provides bare-metal hosting of Docker containers using a bespoke Docker engine built on top of SmartOS and LX-branded zones. There has also been a lot of work on around lightweight VMs as containers, including projects such as Bonneville (now called vSphere Integrated Containers) from VMWare and Clear Container from Intel. Finally we will definitely hear a lot more about unikernels; where the kernel and OS is stripped down to the bare essentials for running a single application. The recent acquisition of Unikernel Systems by Docker Inc likely means that we will soon see unikernels being supported with the standard API tooling. Unikernels have already got a lot of press and some controversy, but look like being a great fit for IoT and other areas with where low resource usage and minimal attack surface is important.
Adrian Mouat’s book “Using Docker” is available via O’Reilly Media.
About the Interviewee
Adrian Mouat is Chief Scientist at Container Solutions, a pan-European services company that specialises in Docker and Mesos. He is the author of the O’Reilly book Using Docker: Developing and Deploying Software with Containers. In the past, he has worked on a wide range of software projects, from small webapps to large-scale data-analysis software.
The open source release of Docker in March 2013 triggered a major shift in the way in which the software development industry is aspiring to package and deploy modern applications. The creation of many competing, complimentary and supporting container technologies has followed in the wake of Docker, and this has lead to much hype, and some disillusion, around this space. This article series aims to cut through some of this confusion, and explains how containers are actually being used within the enterprise.
This articles series begins with a look into the core technology behind containers and how this is currently being used by developers, and then examines core challenges with deploying containers in the enterprise, such as integrating containerisation into continuous integration and continuous delivery pipelines, and enhancing monitoring to support a changing workload and potential transience. The series concludes with a look to the future of containerisation, and discusses the role unikernels are currently playing within leading-edge organisations.
This InfoQ article is part of the series "Containers in the Real World - Stepping Off the Hype Curve". You can subscribe to receive notifications via RSS.