At the Velocity conference in Amsterdam last October, Alex Schoof, principal engineer with a focus on security at Fugue, explained how to manage secrets in a distributed and scalable environment like the cloud. Schoof proposes decomposing a secret management system into multiple components, each with its own particular function, and isolating them to reduce the surface of exposure to the strictly necessary.
According to Schoof, this design is the result of taking into account five principles for secret management at scale. A secret management system should rely on authentication, authorization and access control policies to guarantee that each client only has access to the subset of secrets strictly required.
Schoof stresses that the system should be easy to use to allow for quick access and update of secrets that expire. This is important to avoid having users working around the system by hardcoding or storing secrets in code or plain text files as that would defeat the purpose of the system. Because all your systems depend on secrets, Schoof recommends implementing a high availability architecture to avoid down time of all systems in the event of a failure in the secret management system.
The secrets are encrypted in a database that Schoof calls the ”secret store”. The keys used to encrypt and decrypt the secrets are stored separately in another database called the “master key storage”. These two components are isolated from each other and only accessible by the “secret service”.
Any client, user or computer requiring a secret, must request it to the secret service. The secret service can be a web service that requires authentication from the client and validates the client’s permissions. The service requests the secret from the secret store and the correspondent key from the master key storage. It then decrypts the secret with the key and sends the secret back to the client.
You will also need an administration interface to configure the system. This could be a website, graphical user interface program or a set of command line tools to be used for granting and revoking privileges and configuring access control policies. Contrasting with the secret service which should be accessible by all known clients that can provide authentication, the administration interface, the master key storage and the secret store must have very strict access policies. They should be, for example, only accessible from specific networks, require extra levels of authentication and have a limited number of allowed users.
”Managing systems at scale” talk by Alex Shoof was jam-packed and was one of the highest rated talks at Velocity by attendants using the O’Reilly mobile app or through the Velocity conference web site.