"Config" is a new SaaS offering for managing configuration files. Created by Bien David in 2017, the company looks to simplify how teams store and access configurations used by systems, apps, modules, environments, and server instances. InfoQ spoke to the team behind Config to learn more about how these problems are solved.
The Config platform runs in both public and private environments, and supports properties stored within INI, XML, JSON, YAML, and TOML configuration files. It includes capabilities such as version control, change review workflows, search, and support for continuous integration and deployment.
Config is currently in private beta, and includes multiple pricing plans. The price increases as the customer adds more users, or wants features like webhook notification, on-premises support, or more supported apps per system.
What are the challenges with managing configurations today? How does this work with existing solutions that teams have in place already? InfoQ reached out to Bien David to answer these questions.
InfoQ: What's the primary problem faced by developers and sysadmins when it comes to config file management?
Config: I would say the primary problem is how to easily manage config files across all the environments. I'm not talking about system config files that are the same across servers, as those can be part of the server image or rsync'd. I'm talking about config files that change per environment, like applications that go through a Dev to Prod migration.
My answer doesn't show you the complete picture though, as the problem depends on your situation and job function. So let me talk a little about the problems faced by three groups: programmers, sysadmins and managers.
Programmers. When you have an application that needs to be deployed on multiple environments (local, Dev, QA, Staging, Prod) and multiple instances (cluster, region, etc.), the problem is how to make sure everything is in sync. How do you add a new configuration entry that is common on all servers? How about adding a new environment specific configuration? Related to this, is the question of where configuration files are stored.
When programmers start a new application, especially on a framework we are unfamiliar with, configuration goes through a certain life cycle, partly because configuration is not a high priority. Things start off with values being hard coded, then they turn into constants, then move on to a configuration file, then to an external configuration file or made into system environment variables. Configuration files eventually mature with environment overrides, or get transformed into centralized configuration. Even at this mature state, there is no easy or standard way of managing configuration that differs across environments. There's still the manual effort of synching configuration. There's still coding involved specific to your application framework or configuration library. And the problem only grows as the application grows, as more servers are used, and as the team becomes bigger and more distributed.
Next is the question of where to store these configuration files. You can store the configuration files on the servers themselves, or store them as part of the source code repository, or store them in a separate configuration repository, or store them on a centralized configuration server. Regardless of where the configuration is stored, you still have to making sure common and environment specific values are correctly applied in the master copies, and deployed to the right locations. And let's not forget the fact that some companies lock down Staging and Prod environments, making the process of modifying configuration take longer.
Sysadmins. Sysadmins view the configuration file problem differently from programmers. First, they have to manage OS and system configuration. They need to easily provision new servers and desktops, with all the standard software that comes with them, and with the necessary environment or server specific configuration. Second, they need to worry about application configuration files in locked down environments. For custom applications, this usually means configuration files with less documentation and with no help available online. In the best case scenario, the application developer is still part of the company and is available to guide them.
Managers. The third group, the group seldom discussed, are technical managers who worry about making sure the Prod server is running smoothly. They need to fix Prod issues fast, or better yet, prevent them from happening. Since the application and servers are identical, ideally at least, configuration then is what varies across environments. So this is one of the things usually checked when something works in one environment but not the other.
Have you ever experienced something like this? The technical manager is scrambling because something is broken in Prod. They want to review the configuration and the logs but don't have access. This is an old application and the original developers are gone, so they hunt down the sysadmin. The sysadmin is not familiar with the application configuration, so they work together and locate the config file directory, compare with other configuration files, and use available documentation to make sense of it all. There should be a better way.
InfoQ: Is there a problem that those users don't even realize they have right now?
Config: Many programmers, myself included, believe in YAGNI. Without a standardized configuration file management solution, we will do what is easy and straightforward for the application platform or framework we are using. Configuration files are not usually high priority, so we end up with various approaches. This works well during development, but becomes a problem when projects mature, and we have to manage many applications on multiple servers, all with their own little quirks.
For sysadmins, configuration management is one of their primary job functions, and the popularity of IaC is a testament to this. I think sysadmins know the problem, but some are hindered by company adoption.
The last thing I want to mention is the move to disposable applications and containers. If you can effortlessly bring up a new application running on a new container, then the only thing you need to worry about is configuration. This makes configuration file management much more important.
InfoQ: Where did you see a gap in the available solution space that inspired you to create Config?
Config: Great question. I have a configuration file problem which I haven't found a straightforward solution for. Old questions in Stack Overflow without an easy to implement answer. 2017 blog posts complaining about the same problem. We are in the age of AI and my relatively simple problem still exists.
So how do you manage all your application configuration files across all your environments? Here are the solutions right now, with Config filling the gap between storing configuration files in source control, and accessing them from a centralized configuration server. Think of Config as hosted source control with an easy-to-use web interface designed just for configuration files. A deployment time dependency so no runtime overhead, but with the benefits of centralized and runtime solutions, because of its on-demand configuration generation.
Manual <-> Source Control <-> Config <-> Microservice <-> Centralized Server <-> IaC
I won't go into the pros and cons of each. I will however compare existing solutions with Config.
Manual vs Config - If you have a small app that will remain small, running on one or two servers, doing things manually works. The only reason I can think of for using Config is if you have a lot of these small applications. You can use Config as an application inventory, so you have an overall view of all your applications, and their corresponding configuration locations, values, and documentation. If you outgrow your small setup, manually doing things won't scale.
Source Control vs Config - For example, this will be storing your configuration files in Git. I recommend a separate repository for configuration files and their environment overrides. Other common approaches include: 1) a configuration template as part of the source code repository that you need to customize, 2) prod environment only configuration in source code repository, 3) all configuration files, including environment overrides, in the source code repository. Config is built with environment variables at the core. If you use Git, configuration files are just plain text files, and you still have to manually manage the environment differences, and implement code (or use a library) to handle environment overrides. Config lets you manage a configuration file, where you can add a new common key that applies to all environments, or tab through the environments and make environment specific changes. Like Git, Config has versioning and you do a pull to get the configuration you need. Config however, only pulls the configuration file for the environment you asked for, generating the correct configuration on-demand. When compared to hosted Git, like GitHub, Config is better since your configuration files are always private. You can't accidentally make it a public repository, or easily make a public clone. Other Config features not found in Git (or GitHub) includes review or approval workflows, at rest encryption, push deployment, and type safe editing.
Microservice vs Config - An example of this is Spring Cloud Config. Works really well if you are using Spring Boot. Not really Spring specific but you need to write your own client. Config is better in the following situations. You prefer a deployment time solution, and not manage another server and introduce another runtime dependency. You don't want to make code changes, or looking for a solution that will work with third party applications with configuration files. You are looking for a user-friendly interface to manage your configuration.
Centralized Server vs Config - This will be something like using ZooKeeper. Half the battle is getting consensus and company adoption. If your company already uses a central configuration server and you are given access, this is the way to go. Config is better if you want to support applications that use files for configuration, and you don't want to write intermediary scripts to convert configuration between files and their KV counterparts. Part of our roadmap is the ability to push configuration files from Config to Git or a central configuration server. This is an example where Config complements existing solutions.
IaC vs Config - This is Configuration Management like Ansible. Here we move more on to the realm of sysadmins. Ideal for system provisioning and configuration. Config is better if you wish to manage configuration files the same way regardless if it's your local developer machine or Prod. This is really where Config complements an existing solution and not replace it. Ansible can do all of its magic with system provisioning and configuration, and when it's time for application configuration, let Ansible pull the configuration file from Config. This is a similar concept with Ansible pulling configuration from Git.
For companies who are not yet ready to go to the cloud, Config's main con is the fact that it's a SaaS offering. We alleviate this problem by offering an on-premises plan. We will also work on making it available on cloud marketplaces, like AWS Marketplace, so you can quickly set it up on your own VPC.
Config's tagline is "The easiest way to manage your config files across all your servers and environments". The key word here is easiest. Just import your existing configuration file, and choose between push or pull deployment. Tab between environments, compare environment variables, and search across applications. It takes minutes to get things up and running on our free forever plan.
InfoQ: How do configs stay fresh on the client when source values change? Is that up to the client to figure out (e.g. cache invalidation), or for the server to notify of? What's a good practice?
Config: There are two pieces to this. Deploying configuration and reloading configuration.
Deploying Configuration. We wanted Config to work with as many applications as possible. This meant we had to be a deployment time dependency. In simple terms, instead of manually editing a configuration file using vi or notepad, we will update it for you. So how do we update the configuration file? We have a couple of different options, depending on your process or infrastructure. The easiest is push deployment, which you can trigger from the web interface. We SCP the configuration file to you server, so we require SSH access. It supports batch push, to handle multiple servers at once. Another option is manual pull, similar to how you pull files from Git. Another option is auto pull, which is simply pull under cron or task scheduler. The last option is manual export/download.
Reloading Configuration. Once the configuration file is on the target machine, how does the application know to reload the configuration file? We have a few options, and they all require support from the application itself. During push, we allow a custom script to run, which you can use to send a SIGHUP command, at least for applications that support it. Another option is for the application to detect the change in the file timestamp, and reload it accordingly. Our last option is, users can embed a Config timestamp variable in their configuration file. On every deployment, we will update the timestamp within the configuration file. The application can then regularly check if the timestamp property has changed. Very similar to timestamp detection, but this time a configuration value is checked.
I recommend supporting SIGHUP, but it won't work for all applications, Java for example, and won't work on all operating systems, Windows for example. In lieu of SIGHUP, implement your application where you can give it a command that will tell it to reload its configuration, on-demand.
InfoQ: With Config, how does it work for someone who has an existing investment in configuration file management? Is it a drop-in replacement? Does it work alongside those existing solutions?
Config: If your configuration file management is simply using Git to store all your configuration files and all its environment overrides, Config will be a better solution. If you have automated most of your Git operations, you can have Config push to Git and keep your existing process in place. This will help you quickly get started, and allow for a phased transition.
If you have a microservice solution, your microservice still needs to get the configuration from somewhere. These configuration can come from Config, instead of the file system or source control.
If you have a centralized configuration server that you are happy with, stick with it. If you are using version control alongside your configuration server, Config can be that version control. Config can further complement your configuration server by giving you an easy way to support applications that only works with files, or complex configurations that don't fit nicely in a KV store.
Config complements existing Configuration Management solutions. Part of your IaC scripts (manifests, playbooks, etc.) will be pulling the configuration files from Config, instead of the file system or source control.
InfoQ: Trusting an outside party with sensitive configurations requires trust. How did you go about building an as-a-service offering that offers the necessary protection? Can you describe your architecture?
Config: Thanks for asking, as this is a common question. The best practice is to encrypt sensitive information when data is at rest. This means regardless whether the configuration file lives in a server in your internal network, on a cloud server, in your company laptop, in GitHub, or in Config, nothing sensitive is exposed. However, it is common for sensitive configuration values to be unencrypted, and some configuration files are for third party applications that you have no control over, so it's not a solution that works in all cases.
To help you with the ideal scenario of encrypting sensitive values in the config file, we will provide resources/tools to easily encrypt and decrypt values. In case this is not an option, Config allows you to specify which configuration values are sensitive, and we automatically encrypt the data at rest. This does mean these values won't come up when using our search feature. For companies who are not yet ready for the cloud, we offer an on-premises plan. Run your own Config instance in your internal network or VPC.
The best solution to the sensitive configuration question is to not store them in the configuration file at all. This is where a secrets management solution like Vault comes into play. Part of the Config roadmap is to add support for secrets management servers. Config will pull the sensitive information from Vault and place them in the generated configuration file. This makes it easy for existing applications to utilize Vault.
In terms of infrastructure, we are using Docker on Linux. We use minimal CentOS servers, updated daily. We use NGINX Docker as load balancer and web server. We use Java Docker for running our back-end REST API. We use Postgres Docker for our database. We only open ports as needed for the app to work and for containers to talk to each other, and the database is not accessible from the internet. Offline backups are encrypted, security monitoring is in place, and security testing is done on a regular basis.
InfoQ: We frequently hear stories about credential leakage as a result of configs being improperly uploaded to GitHub. In general, what do you consider good configuration file hygiene?
Config: I touched on this topic in my answer regarding security and trust. Let me summarize and also give a few tips. If you need to store sensitive values in the configuration file, encrypt them in the file and decrypt in the application. Using a library, like Jasypt for Java, can help. If you wish to use a hosted Git repository, make sure you are using a private repository. This is where Bitbucket is better than GitHub. Config is always private, so you are sure your configuration files won't be exposed to the public by accident. The best solution will be to move sensitive configuration out of configuration files into a secrets management server.
Additional information on Config can be found at configapp.com.
