At DockerCon EU 2015 Brian Christner presented an overview of ‘Docker Monitoring’ and shared best practices, a guide to the Docker stats API, and a comparison between three popular monitoring options: cAdvisor, ‘cAdvisor + InfluxDB + Grafana’, and Prometheus.
Christner suggested that best practices included labelling container and setting resource limits, and stated that although Google’s cAdvisor container monitoring tool is easy to use, the combination of cAdvisor, InfluxDB and Grafana offers improved application integration and scalability, and the Prometheus monitoring platform offers client libraries and alerting ‘out of the box’.
Christner, cloud advocate at Swisscom AG, began the talk by introducing the usage of Docker at Swisscom, which included a persistent Database-as-a-Service (DBaaS) offering on Docker and Cloud Foundry (built in partnership with ClusterHQ), and several internal Docker products that support various appllications ‘from application cloud to TV’. These projects utilise Docker due to the possibility of higher density of applications per server (and associated infrastructure cost reduction), decreased time to market for developers, and a ‘one size fits all’ approach to deployment artefacts. A core challenge with using Docker was the required change in the approach to monitoring, from which Christner was keen to share his learnings.
Best practices for monitoring Docker include labelling containers with descriptive key/value pairs e.g. ‘--label environment=”production”’, setting resource limits, and limiting the amount of alerts generated to improve the signal to noise (“Don’t overlert yourself!”). Christner presented an overview of the ‘docker stats’ command, and stated that this tool is great for troubleshooting, both locally and remotely. The Docker stats API was discussed, and Christner commented that this is typically the basis for all other Docker monitoring tools, and can also be used to feed container resource information into an organisation’s existing monitoring solutions.
Google's cAdvisor (Container Advisor) provides "container users an understanding of the resource usage and performance characteristics of their running containers". cAdvisor's container abstraction is based on Google’s lmctfy container stack, and has native support for Docker containers and should support other container types “out of the box”. cAdvisor is deployed as a running daemon that collects, aggregates, processes, and exports information about running containers. The information can include container-level resource isolation parameters, historical resource usage, histograms of complete historical resource usage and network statistics.
cAdvisor can be combined with InfluxDB, a time series database, and Grafana, a metrics dashboard, to stores and display information. Christner has created a “How to setup Docker monitoring” blog post and associated Docker Compose configuration file, which can be used to create a monitoring environment that utilises cAdvisor, InfluxDB and Grafana with a single ‘docker-compose up’ command.
Prometheus is a systems and service monitoring system, which was born out of SoundCloud’s work on an improved monitoring system over StatsD and Graphite. Prometheus collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true. The Prometheus GitHub repository README.md states that the main distinguishing features as compared to other monitoring systems are a multi-dimensional data model, a flexible query language to leverage this dimensionality, multiple modes of graphing and dashboarding support, and support for hierarchical and horizontal federation. A Docker Compose configuration file that can create a fully-functional Prometheus monitoring environment can be found in Christner’s GitHub account.
Christner concluded the talk by comparing the three container monitoring approaches presented above: cAdvisor, ‘cAdvisor + InfluxDB + Grafana’, and Prometheus. Although cAdvisor was stated as the easiest to use, it has limits with scaling and alerting. The combination of ‘cAdvisor + InfluxDB + Grafana’ provides good scalability and also offers client libraries, but does not support alerting out of the box. Prometheus may not scale easily, but does support alerting in addition to providing client libraries for many languages.
Figure 1. Docker Monitoring Comparison Roundup
The slides for Christner’s “Docker Monitoring” presentation can be found on SlideShare, and additional information can be found in a series of blog posts at brianchristner.io.