Recently, Amazon announced the general availability of the Anomaly Detection feature in Amazon CloudWatch, a monitoring and management service providing customers with data and insights from AWS, hybrid, and on-premises applications and infrastructure resources. With the new anomaly feature in CloudWatch, customers will be able to isolate and troubleshoot unexpected changes in metric behavior of their resources.
The tech giant announced a preview of the anomaly detection feature for CloudWatch earlier this year, in July. This new feature for CloudWatch applies machine-learning algorithms to continuously analyze system and application metrics, determine a healthy baseline, and surface anomalies with minimal user intervention. Moreover, the foundation of anomaly detection lies in an accumulation of over 12,000 internal machine learning models. However, according to a tweet by Corey Quinn, cloud economist at the Duckbill Group:
For the impatient, this feature is basically using Exponential Smoothing (https://en.wikipedia.org/wiki/Exponential_smoothing) buried under the umbrella of Machine Learning to develop confidence intervals around CloudWatch metrics.
CloudWatch users can enable the anomaly detection feature through the AWS Command Line Interface, AWS SDKs, or AWS CloudFormation templates. When activating the feature for a standard or custom CloudWatch metric, it will start analyzing historical values of the chosen metric using statistical and machine learning algorithms. These algorithms will determine predictable patterns that repeat hourly, daily, or weekly for the metric. Subsequently, anomaly detection will create a best-fit model to help users differentiate normal and problematic behavior for the metric, which they also can adjust and fine-tune, by for instance excluding specific time ranges from the data that is used to train the model.
Source: https://aws.amazon.com/blogs/aws/new-amazon-cloudwatch-anomaly-detection/
Finally, on top of the generated models, users can create alarms based on when a metric is outside of its set anomaly thresholds. Or the user can in the CloudWatch console go to Alarms to generate an alarm based on Anomaly Detection. Furthermore, users can specify the type of action when the alarm executes, for example, sending a notification to an SNS Topic, which will send an email to its subscribers.
Source (screenshot): https://www.youtube.com/watch?v=8umIX-pUy3k
Users who currently leverage third-party tools such as Datadog or Splunk for anomaly detection could consider using the CloudWatch Anomaly feature instead. A respondent on a reddit thread about the new feature stated:
This was one of the few reasons I used Datadog years ago. I was surprised it took them this long to add it.
Anomaly Detection is available in all of Amazon’s commercial regions, and more details of the feature are available on its documentation website. Furthermore, the use of this feature is free except for the use of CloudWatch alarms - pricing of CloudWatch itself is available on the pricing page.