Avishai Ish-Shalom, CTO of Fewbytes and co-organizer of DevOps Days Tel Aviv pointed out the difficulties posed today by inconsistent configuration definitions and updates to system administration. He offers five design principles that help solve these problems.
Multiple formats requiring larger skillsets, lack of validation and feedback, and especially custom formats that are hard to auto-generate are some of the problems. According to Ish-Shalom, the popularisation of modern configuration management (CM) tools emphasized some of these problems as they break the automation and standardization of systems.
Ish-Shalom mentions Linux's popular conf.d pattern as a bad example where a configuration management tool might automatically apply a certain configuration in a machine but manually adding a new file (or even just changing their order) can alter the final configuration. CM tools lack sufficient granularity and context to detect such deviations.
Furthermore, Ish-Shalom adds, removing those files does not even guarantee that the actual system configuration is updated. The heterogeneous ways in which a system's configuration is applied (e.g. reload vs restart) makes it hard to ensure that even trivial configuration changes can be automatically and consistently applied. Even immutable infrastructure is not a solution for all cases, as Ish-Shalom told InfoQ:
Even with immutable infra and disposable instances configuration is still a big problem. To some degree, it's even more problematic. Configuration resembles the notion of "state" in an application and while some of it can be changed in VCS, integrated in build time, tested in CD pipeline and deployed to your immutable infra just like code (which is awesome and definitely the way to go if possible) - many other configurations cannot be treated so; think of environment configurations like the location of the database - it is a dynamic value that exists only in the production environment and may change at any time due to failure. Think of feature flags, at any moment you may want to switch them on or off and the whole point of them is to avoid another deployment to activate a feature. Finally, while some organizations have progressed to immutable infra and disposable components, most of the world is still using legacy infra - and in fact more companies are building legacy infra right now then immutable infra.
The five design principles proposed are based on two core ideas: creating a REST-based configuration API and separating configuration files by the type of system update they require.
The API should support read and write of system configuration via standard GET and POST methods. This allows cross-checking that the current system configuration matches the definition in the CM tool via GET, as well as applying a new configuration via POST. Both of these principles support a third one which is to give CM tools full responsibility for managing configuration updates, thus avoiding the conf.d pattern.
Together with the separation of configuration files by type of system update (a fourth principle), the CM tool can then, for example, merge multiple configuration (files) that require the same update type and apply the change with a POST to the configuration API, later polling with a READ to check it was successful applied and not modified.
Finally, the ideal configuration format should be standard and serialized (for example JSON or YAML) to allow auto-generation by CM tools. If not possible, treating the configuration as a plugin and extracting variables to an external configuration file is the best alternative.
InfoQ asked Ish-Shalom whether such a configuration API could add potential security vulnerabilities:
The configuration API is no different then any other control API. You can encrypt it and require high grade authentication (e.g. SSL client cert), expose it on a different port to trusted networks only or simply use the same authentication method you are using for the rest of your API. Public cloud services (PAAS and SAAS) are a great example, we configure them via API daily and it feels natural and comfortable - and security is built in from the start.s
Ish-Shalom will talk about the DevOps definition of work at the next DevOps Days Ljubljana.