Google recently announced they are adding features to its cloud platform that will enable integration between its cloud-based log service and its cloud monitoring service. The intent of providing these capabilities is to allow customers to efficiently move between the Cloud Logging and Cloud Monitoring services when discovering the health of their own services.
Both Google Cloud Logging and Cloud Monitoring are currently available for developers to use. The intent of the Cloud Logging service is to log events from its cloud platform. The Cloud Monitoring service exists to allow developers, or administrators, to monitor the health of their services and send out notifications when thresholds have been exceeded. The current gap with these tools are they run independently of each other. Deepak Tiwari, product manager, Google Cloud Platform has recognized these challenges and Google is attempting to improve the experience. “You often want to be alerted when particular events occur, or occur at an elevated rate, or exceed a specified count. Existing monitoring tools do not allow you to easily connect your logs to your monitoring console. With this launch, Google Cloud Monitoring adds to its current out-of-the-box metrics the ability to use granular metrics from your logs.”
The new integrated features are in beta at the moment, but will provide the following capabilities when released:
- Logs based Metrics allow developers to create a metric by using filter expressions, such as all log entries where HTTP status code is 500. With this metric in place, it can now be added to a Cloud Monitoring dashboard. Developers can also enable threshold-based alerts off of these monitors in order to notify interested stakeholders.
In a recent blog post, Google has provided details on how this works. The following image illustrates the process of taking a metric and turning it into a monitoring dashboard.
Image Source: http://googlecloudplatform.blogspot.com/2015/10/create-metrics-alerts-and-dashboards-based-on-your-Google-Cloud-logs.html
- Advanced Logs Filters support more complex filtering based upon Boolean expressions which can target a specific time range, sampling percentage and filtering on metadata or custom user-defined fields.
- Charts to Logs allow users to drill down from a monitoring dashboard into a specific log for issue resolution. In this scenario, a developer may be looking over the health trend of its service. When they discover an anomaly, such as a system resource peak, they can drill down into the specific timeframe within their logs for further diagnosis.
One of Google’s goals for these new features is to reduce the amount of time customers spend context switching between logs and monitoring tools. This is an area customers have been struggling with, but an early adopter of the new features is already seeing the benefit as explained by Arjen van der Ende, engineer at Q42. ”We were sitting on a lot of valuable log data but had been missing insights and there was no good way to gather metrics from them. We experimented with some open source solutions, but it's a lot of work to manage it for the amount of logs we produce…We can now add new metrics on a running production system which has helped us diagnose issues and monitor new signals."
