Docker announced several software and cloud platforms as its Ecosystem Technology Partners (ETP) for managing logs generated by applications running inside Docker containers. Being an ETP logging partner is an endorsement that the software or platform works reliably with Docker to provide logging information via the logging driver mechanism.
Docker already had support for various external logging systems like syslog (the Linux standard central logging system), GELF (Graylog extended log format supported by both Graylog and Logstash), Fluentd and AWS CloudWatch logs in its version 1.9.0. Graylog, Logstash and Fluentd are open source log management tools. The ETP program is an attempt to formalize the various logging providers under a common framework. According to Nick Stinemates, VP Business Development and Technical Alliances at Docker:
The Docker “Ecosystem Technology Partner (ETP) Program” is designed to highlight partners in the Docker ecosystem that have demonstrated quality integrations with Docker engine. We developed the program to address the requests we’re receiving from users seeking recommended solutions for various aspects of the Docker ecosystem ranging from technology areas such as monitoring and logging to networking.
Docker’s logging works by specifying a logging driver while starting the container so that the container’s stdout and stderr streams can be redirected elsewhere. This can be either a standard logging mechanism like syslog or a centralized logging system like Sumo Logic or LogEntries. In addition to the standard output and error streams, applications running inside the container may also write their own log files to either syslog or to a custom log file inside /var/log. In the latter cases, the log files won’t be handled by the logging driver directly.
The idea of logging drivers was born in a docker-dev mailing list thread and later formalized as a Github issue. Docker has added increasing support for different logging drivers over the course of its releases with the latest being support for AWS CloudWatch in Docker 1.9.0.
In real-world deployments, container performance and lifecycle metrics will also need to be monitored. Towards this, Stinemates says:
The Docker Engine API exposes performance metrics, lifecycle events, and log data. This is the common base for vendors to implement against. Logging drivers extend the Docker Engine API to enable custom log handling behavior, where previously only the Docker log streaming endpoints were an option.
Today’s applications often generate huge amounts of logging data. Will container performance suffer as a result? Stinemates says that, “performance is not an issue because performance is tied to the logging driver itself. The logging drivers are modular so users can select the driver that works for their environment and performance requirements."
On being asked about how a new tool/vendor can join the ETP program, Stinemates said that:
To be included in an ETP initiative a partner needs to be enrolled in the Technology Partnership program. The partner needs to have made some significant impact to the Docker ecosystem - like integrating existing services to our APIs, building a product based on or servicing our platform. The Partner implementation should not compromise Docker portability, should maintain and promote Docker as a keystone in the infrastructure, and surface Docker without wrapping or disintermediating.
The complete list of ETP log management partners includes Amazon Web Services CloudWatch, Elastic/Logstash, Graylog, Rapid7/Logentries, Loggly, Papertrail, Sematext Logsene, Sumo Logic and Treasure Data/Fluentd. Some of these tools have additional features on top of plain log collection.