Spencer Krum and Elizabeth K. Joseph shared their experience both using and providing the public infrastructure used by OpenStack at the configuration management developer room at FOSDEM.
Consuming Open Source Infrastructure
Spencer Krum, cloud engineer at HP, works on the HP distribution of OpenStack that uses the upstream Puppet based infrastructure created by the OpenStack.org team. HP applies their own patches to OpenStack and then test them internally, to build their own OpenStack distribution.
We benefit from consuming the infrastructure because we have confidence that the architecture is viable. Any departures you take from upstream is technical debt.
In HP case, the upstream OpenStack.org team used two OpenStack clouds, while downstream at HP they used one OpenStack cloud across two datacenters, something that caused networking issues amongst other problems. For downstream the goal becomes deciding what to replicate, while for upstream it is decoupling components from each other so they can be easily consumed downstream.
At HP they contributed changes upstream, ensuring the contributions were small changes or conditionals to consider specific cases that could be customized downstream, both for practical reasons, because big patches are harder to process and makes life harder for the upstream committers, and for social reasons, creating a trust relationship between committers and contributors.
They had to face issues with different versions of Puppet between the upstream and downstream repositories, usage of old modules, like the Puppet Apache one, and a monolithic Puppet repository. There was an effort to migrate from Puppet 2.7 to 3.7, splitting the data into separate git repositories and the monolithic git repository into repositories per module, using Puppet Blacksmith to release modules to the Puppet Forge, and running integration testing against proposed changes.
The current plans for the upstream OpenStack repository includes using semantic versioning for modules, and using git clone to fetch module dependencies to avoid transitive dependency issues. In the downstream repository at HP they plan to run everything in OpenStack to test their own environment, update to Puppet 3 and deploy more testing.
Fully Public Puppet
Elizabeth K. Joseph, automation and tools engineer at HP, works on the OpenStack.org CI team, and talked about the project public infrastructure.
She outlined the benefits of having public infrastructure:
- Provides examples.
- Encourages better practices.
- Allows others in your organization to suggest changes.
- Sharing with the community is nice.
OpenStack has a policy to use only open source tools, to avoid conflicts between commercial companies. There are other projects that have open sourced their infrastructure configuration and provide a public view, Debian, Mozilla or Jenkins for instance.
In the specific case of Puppet, she recommends to:
- Leverage existing modules, or write new ones with the intent of sharing.
- Split existing system configuration and Puppet configuration.
- Split out non-sensitive custom configurations into a different module.
- Use Hiera for sensitive data.
- Document modules, linking to the source, explaining how to use the configuration and how to contribute with commits and bug reports.
- Add a license to configuration files. This is important for reusing them.
OpenStack uses pic, a web interface to PuppetDB aiming to replace the reporting functionality of Puppet Dashboard. It allows them to give visibility to developers on whether their patches are applied in servers, or getting the status of the server, without having to ask the infra team. It empowers developers to have the visibility over the infrastructure.
On the question about using Puppet and not something else, se answered:
We use Puppet because we are a small team and the person trying Puppet and Chef got Puppet running first. If we were to start from scratch we would probably use Ansible to avoid having agents everywhere.
