Configuration management is the foundation that makes modern infrastructure possible. Tools that enable configuration management are required in the toolbox of any operations team, and many development teams as well. Although all the tools aim to solve the same basic set of problems, they adhere to different visions and exhibit different characteristics. The issue is how to choose the tool that best fits each organization's scenarios.
This InfoQ article is part of a series that aims to introduce some of the configuration tools on the market, the principles behind each one and what makes them stand out from each other. You can subscribe to notifications about new articles in the series here.
Taste Test, by Matt Jaynes, is a short, but instructive book that uses a simple scenario to compare Ansible, SaltStack, Chef and Puppet. On the recently released 2nd edition it adds new chapters on Docker, the communities around the tools, and how they fare on security. InfoQ took this opportunity to talk with Matt to know more about his thoughts on the tools and his approach to configuration management when consulting.
The book does not intend to be a full, deep introduction of the different tools. As Matt puts it on the book:
This isn't a deep exploration of these tools. Instead, I aim to give you a great head start by saving you the weeks of research you might have spent trying out the tools in order to choose one.
It has a full hands-on approach: it uses a simple configuration scenario that the reader can follow by executing the written instructions on the book. At the end of the book, you'll have two configured web nodes with all the four tools, Docker and even with shell script.
The book has several strong points: a like for like comparison of the different tools; it is short and to the point; it has a good noise-to-signal ratio. Given that each tool refers to the same concepts with different names, the author chose a common, neutral terminology throughout the book. This decision makes it easy to compare each tool's key concepts without getting confused by the sometimes... colorful names.
On the other hand, if you want to deepen your knowledge in this space you will have to look elsewhere. There is no mention on CFEngine, which would cover all the five major tools. The price is a bit steep, although the book can save a lot of time on basic research. Kindly, Matt offers a 9$ discount for the first 30 InfoQ's readers here.
Taste Test uses a very simple two-nodes scenario to demonstrate the various alternatives. Each node contains one static web site serving specific content. Matt adopts this background to demonstrate key features of any configuration management tool:
- master/children node setup
- installing packages
- user/group setup
- deploying a static file
- deploying a templated file
- running a service
Matt ranks Ansible as the best option, while considering that Chef and Puppet have problems with usability. In Matt's eyes, usability is paramount to success so Chef and Puppet have an important handicap that must be addressed:
In writing the book, Ansible took the least time to set up the project (~2 hours). Salt has a higher learning curve and took a bit longer (~5 hours). Puppet had a few rough patches and took ~9 hours. Chef was the toughest and took ~12 hours.
Ultimately it was outdated documentation, confusing flows, and inconsistencies that hindered both Puppet and Chef.
The three new chapters discuss Docker, the communities around the tools, and the tool's security track record.
The Docker chapter contains a basic introduction to the tool and shows how to configure the two-nodes scenario. The most interesting part though, is the list of misconceptions that the book cautions against. As with every other tool, Docker is not a silver-bullet.
The communities' chapter tries to make a comparison between the communities around each tool, based on metrics like GitHub stars, Twitter followers, event (conferences, sessions) mentions and job trends. Unsurprisingly, Puppet and Chef have many more followers, job postings and event mentions. Maybe surprisingly, Ansible and SaltStack have a lot more GitHub stars.
The security chapter ranks the tool on three subjects: reporting transparency; attack surface and security record. According to Matt, Puppet comes out first on reporting transparency due to its dedicated page, while Ansible comes on top both on attack surface and security record.
InfoQ took the opportunity to talk with Matt Jaynes.
InfoQ: How do you support your claim that all tools support 10000+ nodes? I'm asking this because there are common arguments on the scalability of the various tools and you seem to think that this is not a differentiating topic.
Matt: Remember that most configuration management tools use a standard client/server setup similar to a typical web app. So, there's nothing really special about scaling up CM tools as opposed to scaling up any client/server setup. Just as a web browser requests an html page from a web server, a Puppet-managed node requests configuration files from the Puppet master server.
So, to scale up a tool like Puppet, you would use similar techniques as what you'd use for a web app:
- use a higher powered web server (nginx, etc) that supports more requests
- use a beefier server (more CPU, etc) for the master
- use multiple masters to balance the load
The main difference between scaling a web app and scaling a CM tool is that you can control the request frequency and timing of the clients. A web app has fluctuations in traffic based on how many humans decide to visit the app. A CM tool's clients can be configured to make their requests at the frequency and time of your choosing, so you control how to spread the traffic out.
Note: Ansible has a different approach than the other tools (it pushes the configuration out via SSH), but it also provides the capability to use a standard client/server setup via its "ansible-pull" mode.
Each tool is different, so you'll scale them up differently, so that is a differentiating factor at larger scales. However, my book is for those just getting started with a CM tool, so the audience is primarily those at low scale. I mention that all of these tools can scale in order to focus beginners on the attributes that really matter for them: usability and security.
Finally, here's some examples of these tools being used at 10000+ node scale:
InfoQ: Your example is based on a small scenario. Would any of your conclusions change if we were looking at a larger scale?
Matt: The small scenario is designed for speed in evaluating the tools. The things that matter most are usability and security. Since the usability and security factors don't change much at larger scales, I don't think it would change the conclusions.
InfoQ: Do all the tools scale smoothly along with the infrastructure they're managing, from a usability point of view?
Matt: Puppet and Chef are challenging to get started with and don't get any easier when you try to scale them. Ansible and SaltStack are more usable and that usability continues when you get to scaling them. The more usable a tool is, the easier it is to understand. Understanding is the key to being able to secure and scale your systems.
InfoQ: You state that the main competition to the various tools is all those organizations that don't use any tool at all. Why do you think there are still so many organizations in this situation? Is it just that all the buzz is relatively recent and things take time to become mainstream or do you find other explanations?
Matt: If you look at how a startup begins, you see there's usually a technical founder who is a developer. Then as the company grows it adds a few more developers. It isn't till later on that they finally add a systems engineer. So, who built the systems along the way? It was often developers with only limited systems experience.
Also, a new company is under intense pressure to survive and grow, so the systems are a low-priority and just get built and maintained manually by the developers.
I think there's a bit of a renaissance happening in systems right now because the learning curve has come down so dramatically. With Puppet and Chef, there's a daunting process that will take weeks or months. But now with Ansible, you can be up and running in a few days. The bar has been dramatically lowered for being able to automate your systems now, so more developers and systems engineers are embracing that.
Also, there's a huge competitive advantage for your company if you have lean, automated systems and more companies are realizing that. If you can get the exact same output from your systems for $2K/month instead of $200K/month, you can then focus that extra $198K on building your business. If your competitors can't figure that out fast enough, they'll fail and you'll win.
Businesses are starting to realize how much they can save when it takes 2 minutes to build a new server instead of an intensive 2 week manual process. The gains to be had are huge and now that the cost is so low to make this happen, more companies are embracing it.
InfoQ: What approach do you take when you consult with an organization that doesn't do configuration management? How do you ensure that they keep at it once you leave?
Matt: My approach is:
- Remove risks that could cripple or kill their business
- Accelerate growth with low-cost, scalable systems
The first priority is protecting their business by removing risks (security breaches, outages, data-loss, etc). Often companies don't have basic security, backups, or monitoring in place, so fixing that first is paramount. We recently saw Code Spaces fail as a company because of something that could have been prevented with less than an hour of work.
The second priority is optimizing their systems for business growth (simplifying, automating manual processes, etc). This is where Ansible really shines. Not only can we use it for the orchestration tasks like provisioning and deploys, but we also use it to configure and maintain the systems. It's easy to learn and so it's a quick win that the client will continue to use.
In the past, one of the hardest parts of my contract work was convincing the engineers at the client company to continue using the CM tools instead of falling back into manually managing the systems. It was because the tools at the time were painful and difficult to use. That's a big reason why I value usability so highly and why I ultimately recommend Ansible to my clients.
InfoQ: Do you use different configuration management tools depending on the organization's context? If yes, what criteria do you use?
Matt: Usability and security are my top priorities for the tool. For Unix/Linux systems, Ansible is the best choice.
I don't work with MS Windows systems generally, but for those, the choice would probably be Puppet or Chef since they are the most mature on that platform.
InfoQ: Do you find any area where the tools are lacking and should improve in the future?
Matt: Usability is paramount. If it's not usable, it will seldom get used. If you don't understand it, you can't properly secure or scale it. Ansible is doing the best job on this right now, but it also has plenty of room for improvement.
Most users try to use a tool by consulting its documentation. Just having usable documentation is the first step, but many of the tools' documentation is broken, out of date, confusing, etc.
The best way to fix the usability for a tool (and its docs) is user testing.
If these tools want to gain traction and new users, then I'd ask them to please please please do actual user testing. There are a lot of vendors that think they are doing this, but they're not. Having your docs in the wild and being open-source is not the same as actual user testing.
User testing is this:
- choose a project a first time user would do
- find a user that has never used your tool
- have the user attempt the project
- observe the user and see where they hit rough spots
- remove the rough spots
- repeat
I've been advocating this for awhile, but most companies apparently would rather spend their money shouting about how awesome they are than to fix basic usability problems. All the conference sponsorships, ads, swag, etc are meaningless if a new user has a painful experience when they try a tool.
I'm not really sure where the cognitive disconnect is here. I had one company contact me 5 separate times over a 6 month period to ask me to promote them and each time I gave them this user-testing advice. Last time I checked they were still spending tons on marketing themselves at every event, yet the tool and docs are still unusable by anyone but the most persistent and masochistic engineers.
I'll get off my soap box now, haha. Fortunately, the free market will take care of these problems over time :)
InfoQ: If you were to write a 3rd edition, where would it focus on?
Matt: The focus would be primarily on updating any significant changes to the tools themselves. I added a chapter on Docker in the latest edition and I'd probably expand that a good bit in the next edition.
About the Book Author
Matt Jaynes is a systems engineer who's been consulting on development and operations projects since 1999. Past clients include: Sony, CBS, Yahoo, Intuit, Scribd, Whole Foods Market, Showtime, Bravo, Mensa, University of Maryland, American Cancer Society, Brooks Brothers, and many smaller companies.
Configuration management is the foundation that makes modern infrastructure possible. Tools that enable configuration management are required in the toolbox of any operations team, and many development teams as well. Although all the tools aim to solve the same basic set of problems, they adhere to different visions and exhibit different characteristics. The issue is how to choose the tool that best fits each organization's scenarios.
This InfoQ article is part of a series that aims to introduce some of the configuration tools on the market, the principles behind each one and what makes them stand out from each other. You can subscribe to notifications about new articles in the series here.