At QCon SF, Yevgeniy Brikman presented "Automated Testing for Terraform, Docker, Packer, Kubernetes, and More". Key takeaways from the talk included the recommendation to use an appropriate mix of all testing techniques, such as static analysis, unit tests, integration tests, and end-to-end tests. As demonstrated through the use of a "testing pyramid", he suggested creating many unit tests and static analysis tests, fewer integration tests, and only a handful of end-to-end tests. Ultimately, testing infrastructure code will establish confidence and get rid of the fear of deployment.
Brikman, the co-founder of Gruntwork, a company that provides "DevOps as a Service", began the presentation by talking about fear-inducing issues within the DevOps world, such as outages, security breaches, data loss and change. Because of this, Brikman suggested that DevOps teams deploy less, leading to deployment only working 60% of the time. A better way of dealing with fear is to do automated tests, which raises confidence. As Brikman puts it, everyone knows how to write automated tests for application code, but how to test Terraform code that deploys infrastructure that works or Kubernetes code that ultimately deploys services is different.
An overview of several techniques to test infrastructure code was provided:
- Static Analysis
- Unit tests
- Integration tests
- End-to-end test
The first technique Brikman discussed was the static analysis of infrastructure code without running it or having it deploy resources. The code goes through a compiler, parser or interpreter to check the syntax and structure to determine whether the code is valid or not. Brikman provided examples for Terraform using terraform validate, and demonstrated the validation with a Terraform demo. Next, he pointed out other tools for detecting common errors like a security group allowing all inbound traffic. And finally, he talked about a dry run, by executing the code but not deploying anything. Again, there are tools available for this like Terraform plan, and using Kubernetes, there is a command:
kubectl apply -f <file> --server-dry-run
Next, Brikman discussed unit tests for testing code in isolation. For cloud infrastructure, this means a single module containing one or several depending resources together. Furthermore, the only way to test it is to deploy the infrastructure in that module in a real environment. As Brikman states in his talk, there is no pure unit testing for infrastructure code.
Moreover, the strategy is to test the code by deploying the defined resources, validate if they work, and undeploy them. Also, there are tools available to support this strategy like Terratest. Brikman then provided a demo performing Terraform unit tests, and he ran through the related code, which is available on GitHub. Finally, he did a similar demo for Docker/Kubernetes.
After covering Unit tests, Brikman dived into integration tests, a technique to test all the unit-tests together. The basic approach for an integration test is similar to unit tests. He did a demo with the same code of his previous Terraform unit test demo, but now deploying resources from two units. Subsequently, after the demo, he shared how to speed up integration tests by running tests in parallel, and also noted that when running in parallel, one could run into conflict issues - when for instance resources names are not unique. Additionally, he explained the concept of test stages and retries.
Finally, Brikman discussed end-to-end testing to test the deployment of an entire infrastructure. The approach could be the same as for unit test: deploy all the infrastructure, validate if it all works, and tear it down. However, Brikman pointed out that that’s an unusual way of doing that. Therefore, he introduced the test pyramid in his talk, showing static analysis at the bottom, followed by unit tests, integration test and at the top end-to-end tests.
He stated:
The thing about this pyramid is that as you go up, the cost to write the test, how brittle the test is, and how long it takes to run goes up.
Brikman said that the more resources your test deploys, the great the chance of failure when running end-to-end tests. The unit and integration tests could be retried; however, end-to-end tests are too brittle and slow to be useful. Hence, Brikman indicates it's better to do them incrementally:
- Deploy a persistent test environment and leave it running.
- Each time you update a module, deploy & validate just that module
In conclusion, Brikman recommends using all of the testing techniques available; however, the test pyramid needs to be applied accordingly.
Additional information about Yevgeniy Brikman’s QCon San-Francisco talk can be found on the conference website. The slides are available, and the video of the talk will be released on InfoQ in the coming months.