At the recently concluded KubeCon, Google announced the beta release of Stackdriver monitoring for Kubernetes. The key features include central visibility of Kubernetes-orchestrated container metrics and logs along with other metrics in the existing Stackdriver dashboard, and better Prometheus support.
Stackdriver, Google Cloud Platform’s monitoring product, can collect metrics and logs from servers and applications. Some metrics are inbuilt into the platform, whereas others need an agent to be installed on the servers being monitored. Agents can also be written to push custom metrics via the API. Stackdriver also has ability to build custom dashboards, choosing the metrics one wants to view, filtered by labels. As long as the metric exists in the Stackdriver system, it can be viewed. These same abstractions are being used to provide Kubernetes monitoring.
Stackdriver monitoring for Kubernetes consists of log collection, metric collection and events. Logs from individual containers, logs from system components like docker and kublelet, and system events like pods being scheduled are pushed to Stackdriver by an agent that runs on each Kubernetes node. This logging is enabled by default on Kubernetes clusters running on Google Cloud. The same is true for monitoring also. The monitoring data collected is available for comparative viewing along side metrics from other parts of the system. Stackdriver metrics are grouped by resource types, such as VM instances and load balancers, and the new feature introduces new resource types such as Kubernetes nodes and pods. This monitoring can take advantage of the existing Stackdriver components like alerts and incidents.
Some features were present before the announcement, like container stdout and stderr being sent to Stackdriver logs, and OS metrics being pushed to Stackdriver monitoring. However, there were also issues - e.g. with the default log parsing pattern which could not detect multi-line events like stack traces. Kubernetes clusters that do not run on Google Cloud have multiple options for monitoring like Heapster and Prometheus.
Commenting on the Prometheus integration, J D Velásquez, Google Cloud product manager, said,
We will expand our current integration with Prometheus to make sure all the hooks we need for our sidecar exporter are available upstream by the time Stackdriver Kubernetes Monitoring becomes generally available.
A sidecar container is a helper container that runs along side another container in the same pod. The documentation for setting up Prometheus with Google Kubernetes Engine (GKE) mentions a separate service with a single Kubernetes pod, but the code for running the sidecar container is available on Github, and it lays down the restriction that the cluster should run on either Google Compute Engine (GCE) or GKE.
Velásquez added that "Stackdriver Kubernetes Monitoring Beta is available for testing in Kubernetes Engine alpha clusters today, and will be available in production clusters as soon as Kubernetes 1.10 rolls out to Kubernetes Engine." Stackdriver data can be further exported to BigQuery, Google Cloud Storage and Google PubSub for subsequent analysis.