In a recent blog post, Amazon announced CloudFormation Drift Detection which organizations can leverage to automate configuration consistency across AWS cloud resources. The CloudFormation Drift Detection feature allows organizations who have templated their configurations and deployments, known as stacks, to detect when configuration drift occurs from out-of-band changes. These out-of-band changes have been directly applied to cloud assets, instead of leveraging a templated deployment approach.
To avoid configuration drift, Amazon is suggesting the customers use a CloudFormation Change Set to apply changes. This way your deployment template is kept up to date and can be used to provision AWS services in a consistent manner. Jeff Barr, chief evangelist at AWS, explains a common problem that customers encounter as they try to manage their cloud assets:
In reality, many organizations are still working to fully implement Infrastructure as Code (IaC). They are educating their staff and adjusting their processes, both of which take some time. During this transition period, they sometimes end up making direct changes to the AWS resources (and their properties) without updating the template.
An example of how this drift may occur is an administrator who initially provisions cloud services using a template, but then makes changes outside of the template and applies these "out-of-band" via an API call or manual invocation, such as an EC2 instance type change or altering an Auto Scaling parameter. The problem with out-of-band changes is that any future deployments using the template that has not been updated with all changes will lead to inconsistent configurations, which can create operational challenges.
To address drift challenges, Amazon has introduced a drift detection feature called CloudFormation Drift Detection. After a CloudFormation stack has been created that contains a list of resources defined within a template, drift can be detected from the AWS Management Console, CLI or from your own code. Drift can be detected within a few minutes from the out-of-band changes being applied so that administrators can quickly address this. Differences in configuration are detected by comparing the current stack configuration with the one specified in the template and identifying divergence. In addition, detailed information for every difference is provided.
For many organizations, they recognize the value in using an IaC methodology. But, organizations may lack the discipline to maintain their configuration management which creates future issues. Eric Shanks, a cloud infrastructure consultant, suggests using CloudFormation Drift Detection which can be used to enforce greater discipline:
As a consultant I’m often trying to drive the point home that this tool [CloudFormation ] is great and some sort of IaC should be used to manage the environment. That's usually easy at the start, but inevitably something will happen or an emergency will happen where a manual change gets made to the environment and the code doesn’t get updated. Maybe that isn’t a big deal, but what happens when someone then updates the code for another change and applies a change set. That’s right, that manual change that was made in the middle probably got wiped out and whatever issue happened before is probably back.
The current list of AWS resources that drift can be detected for includes: API Gateway, Auto Scaling, CloudTrail, CloudWatch Events, CloudWatch Logs, DynamoDB, Amazon EC2, Elastic Load Balancing, IAM, AWS IoT, Lambda, Amazon RDS, Route 53, Amazon S3, Amazon SNS and Amazon SQS. Support for additional resources is expected in the future.
Detecting drift can be accomplished from the AWS Management Console by selecting a stack and then calling the Detect drift operation. The detection will run immediately, but may take several minutes to run based upon the number of resources included in the stack.
Image source: https://aws.amazon.com/blogs/aws/new-cloudformation-drift-detection/
After the drift detection job has completed, a report will be available that includes a list of resources that are in sync and modified.
Image source: https://aws.amazon.com/blogs/aws/new-cloudformation-drift-detection/
For resources that have been modified, there is detailed report available that will highlight the differences between the desired configuration and the actual configuration. This level of detail provides administrators with prescriptive guidance on what has changed.
Image source: https://aws.amazon.com/blogs/aws/new-cloudformation-drift-detection/
The CloudFormation Drift Detection feature is now available and can be found in the following regions: US East (N. Virginia), US East (Ohio), US West (N. California), US West (Oregon), Canada (Central), Asia Pacific (Mumbai), Asia Pacific (Seoul), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Europe (Frankfurt), Europe (Ireland), Europe (London), Europe (Paris), and South America (São Paulo) Regions.