The 2nd edition of the book “Chef Infrastructure Automation Cookbook” brings updated recipes that work with Chef 12. Usage of ChefDK is also introduced, making it easier to install, run and test Chef in developer's workstations.
InfoQ reached out to the author, Matthias Marshall, to get a better insight on what's new and his view on the evolution of the configuration management tool space.
InfoQ: What’s your background?
Matthias Marshall: I studied computer science in Munich, Germany and worked as a software developer for a couple of years implementing web applications. Later I moved into a mixed developer - system administrators role. We practiced what is called DevOps today since 2003 and I started blogging about it in 2008.
InfoQ: How did you get started with configuration management?
Matthias: Prior to rolling out Chef to automate our whole infrastructure I wrote my own configuration management tool based on Capistrano. It turned out that, while Capistrano is a great and stable tool for deploying code, its inner structures are not very supportive for more advanced requirements like configuration management. That’s why I started looking. By that time Chef and Puppet were maturing and I settled for Chef as it appealed more to my developer soul.
InfoQ: Can you detail a bit those advanced requirements?
Matthias: In Capistrano, you have the concepts of roles and servers. You can tell Capistrano to deploy your code to either servers having certain roles or to individual servers. So far, so good. But the code internally was not implemented in a way that would have made it easy to extend it. It was mostly plain arrays and IDs instead of objects, which I could have extended. That’s why I gave up extending Capistrano and moved on.
InfoQ: How would you compare today the strengths and weaknesses of the main configuration tools available (Puppet, Chef, Ansible, Salt)? Are there specific use cases where you'd recommend one tool over another?
Matthias: Right now Puppet seems to be the most widely used followed by Chef. Ansible and Salt have matured a lot in the recent years and Ansible now being part of RedHat only emphasizes on its maturity and future. Choosing the right one for your infrastructure depends on a lot of factors like experience and the types of systems you manage. It’s hard to give any concrete advice here.
InfoQ: Why did you choose the cookbook style to write about Chef?
Matthias: When automating my own infrastructure I was regularly searching for existing best practices and cookbooks. For me it was difficult to distill best practices from what I read about Chef. And a lot of cookbooks don’t work out of the box or it’s hard to understand how they work and what exactly they do. The style of providing individual small how-to sections helps readers to learn more about Chef in easily digestible chunks.
InfoQ: What is the target audience for your book?
Matthias: While the book is not a beginners guide it starts by covering basic tools and approaches and then moves on to more complex topics like setting up cloud servers or installing full application stacks like Ruby on Rails or Wordpress.
It helps if the reader already has some experience with Chef. The book will then help to learn about how to attack concrete challenges. It will provide building blocks for the reader to build his own solutions.
InfoQ: What was the feedback on the first edition of your book?
Matthias: I got feedback that it really helped to see the official Chef documentation in context with other tools in the Chef universe like Berkshelf, Vagrant, etc. Readers told me that they liked showing some best practices and boiling them down into simple explainable steps.
InfoQ: Why a second edition two years later?
Matthias: The first edition sold over 4.000 copies. But after nearly two years a lot of the included recipes were not working any more due to a new major version of Chef (12) and various updates to cookbooks. Additionally the company name changed from Opscode to Chef and the ChefDK got introduced - all that made the book very hard to read because its contents didn’t match reality anymore.
But reviews were great and the demand high. That’s why Packt and I decided to update every single recipe and publish a second edition.
InfoQ: What's new in the second edition?
Matthias: Shortly after I finished the first edition, Chef 12 came out with a lot of new stuff. While most content in the book was still valid, a lot of details had changed. That’s why I really touched every recipe in the book and updated it to the new version of Chef and more current versions of used tools and cookbooks.
Berkshelf and Test Kitchen are now integrated into the new ChefDK - a much cleaner way to install Chef on your workstation. Chef 12 incorporated chef-zero for its local mode now. These are all major changes in the Chef ecosystem which I had to address in the second edition.
I added a how-to on using Chef provisioning, which is now a part of Chef. With Chef provisioning (formerly known as Chef Metal) you can spin up virtual machines and bootstrap them - all from within Chef.
Last but not least the company developing Chef changed its name from Opscode to Chef Inc and renamed its products. The second edition incorporates the new names to make it easier for the readers to map what they read to what other people talk about.
InfoQ: Can you briefly describe what exactly ChefDK brings to the table and how that affects the development workflow?
Matthias: ChefDK bundles a lot of the most popular tools in the Chef universe like Berkshelf to manage cookbook dependencies and Test-Kitchen to run integration tests on multiple VMs. It puts all those tools under one directory to make managing it easier.
InfoQ: What about chef-zero? In which use cases does it apply?
Matthias: Chef-zero is a replacement for Chef solo. It provides a fully featured in-memory Chef server. Chef solo was limited because it did not provide you with some of the more advance features like search. With chef-zero, these limitations are gone.
InfoQ: Do you still encounter many organizations running production in bare metal servers? For those migrating or adopting a mixed cloud plus in-house approach, can Chef or other tools help them make their transition smoother? In which way?
Matthias: Yes, a lot of companies are still using bare metal servers. Transitioning individual services or the whole infrastructure really needs a decent configuration management approach either with Chef or any of the other tools available. You want to move from individual, well known and maintained servers (a.k.a “pets”) to arbitrary cloud instances, which you simply terminate if something goes wrong (a.k.a “cattle”). This is only possible using configuration management tools.
InfoQ: When you launched the first edition you wished for a tighter integration between the Chef Server and source code repositories. Has your wish been addressed by changes in Chef since then? Or did you solve that problem in another way?
Matthias: No, you still need a source code repository for your Chef recipes and you need to upload them to the Chef server. But Chef is working on a feature called policy files, which “allows you to specify in a single document the cookbook revisions and recipes that should be applied by the chef-client“. Policy files are already available in Chef 12 but are still experimental.
InfoQ: Since Chef's ecosystem is always evolving, how do you keep abreast of the changes that take place? In other words, what's the easiest way for someone picking up your book in 1 year from now to know what is still applicable and what has changed?
Matthias: The basic concepts don’t change that fast. While a lot of cookbooks I present in the book will have newer versions available, most of the inner workings will be similar. Maybe you won’t need some of the workarounds I show to make a cookbook work for you in 1 year from now. I would propose to take my book, try it out, look at the “How it works” section to understand what’s going on and take it from there to see what has changed since the release of the book. It never harms to look at the history of the source code of a cookbook at GitHub, too.
InfoQ: Where do you see Chef and the whole configuration management tools space in 2-3 years?
Matthias: Chef as a tool is already very stable and capable. We’ll see more best practices and usage patterns spread. Chef will move more into the enterprise space supporting network devices as well as the complete deployment processes.
Newer tools like Salt, Ansible will mature and enrich the configuration management tools space but the most fundamental shift will come from things like Amazon Lambda or Otto by HashiCorp. These approaches are abstracting away the whole configuration management part and focus on deploying functions or apps transparently.
InfoQ: Can that transparency also bring about new bottlenecks, i.e. having dedicated people controlling the configuration management part “under the hoods” while others are isolated from it? Will that not take away part of the shared understanding between dev and ops promoted by infrastructure-as-code?
Matthias: Yes, configuration management will move “under the hoods” and it will become a more specialized skillset. When the tools and providers take over the largest part of your configuration management, a lot of teams and organizations will not need to worry about it anymore. This is like with power supply. Nobody (besides the power companies) cares too much anymore about how to produce and distribute electrical power. For a lot of companies it will be enough to use computing services as they’ll be provided by service providers or tools.
InfoQ: There have been several initiatives in moving Chef to the enterprise space. In your opinion what are the main enterprise-specific requirements (if any) that Chef and other tools failed to support before? Or is it just a matter of lowering the barrier of adoption with a smoother transition path?
Matthias: Chef has added a lot of features in the auditing and security space required by enterprises with Chef 12. The work on policies is another step into that direction. But taking a complex, sometimes decades old, infrastructure and automate it isn’t an easy task. I guess that finding better ways to deal with existing infrastructure and how to move it under configuration management are needed.
InfoQ: During the last Chef Summit, Dominica DeGrandis humorously tweeted that early adopters didn't need documentation but now, because enterprises started adopting Chef, there is a need for it. Do you agree?
Matthias: Like with any piece of code you can create a big mess with your infrastructure code. As the tools are pretty new in comparison to programming languages and a lot of people are writing configuration management code, which haven’t been properly trained in things like object orientation you see a lot of messy infrastructure code. Another reason for this is the fast evolution of the tools and best practices. You easily end up with messy code, if you’re new to the subject or if the tools change too fast. So, yes, I agree. But as the tools and the practices on how to use them mature, things will get better than before.
About the Book Author
Matthias Marschall is a Software Engineer "Made in Germany". His four children make sure that he feels comfortable in lively environments, and he retains in control in chaotic situations. For years he's focused on helping developers, teams and entire organizations to quickly and continuously release software that provides users with real added value. Together with Dan Ackerson he runs www.agileweboperations.com where they share their early successes with DevOps. Twitter: @mmarschall
