Managing Kubernetes Secrets with the External Secrets Operator

Kubernetes secrets is a mechanism that allows sensitive information to be stored in a central repository called etcd, which is a more secure way to store information than in the pod definition or in a container image itself. However, Kubernetes doesn’t yet have the capabilities to manage the lifecycle of secrets, so sometimes we need external systems to manage this sensitive information. Once the amount of secret information we need to manage increases, we may need additional tools to simplify and better manage the process. In this article we’ll take a detailed look at one of these tools, External Secrets Operator.

What is a Secret?

Secrets are digital credentials used to manage access permissions at both the human-to-application and application-to-application levels. They can be in the form of passwords, encryption keys, tokens, and so on.

What is Secrets Management?

Secrets management is about securely managing the creation, storage, rotation, and revocation of digital credentials while eliminating or at least minimizing human involvement and reducing potential sources of error.

What are Kubernetes Secrets?

Containers require access to sensitive data to perform basic operations such as integrating with databases, APIs and other systems. In terms of Kubernetes, a secret is an object that contains digital credentials such as a password, token, or key. Using a secret eliminates the need to store the sensitive information in a pod specification or container image.

The Problem

We all know the typical pattern of using secrets to connect to external services. Below is a simple solution architecture that shows how we use secrets to connect to a database.

We have a microservice (or a monolith, if you will) that connects to a database using a secret, a username, and a password in this case.

It gets a little harder to manage and synchronize all these secrets when you start supporting multiple environments like development, test, and production.

Now, imagine splitting your application into multiple services, each with its own external dependencies, such as databases, third-party APIs, etc., resulting in an even more complicated architecture.

A setup with multiple services and environments such as the one described above in Kubernetes comes with a number of challenges, including:

• You may need to manage hundreds of secrets.
• It becomes difficult to manage the lifecycle of secrets such as creation, storage, rotation, and revocation.
• It becomes difficult to onboard new services and people with specific access rights.
• You must make considerations for secure distribution of secrets.

For the reasons above, you may consider opting for third-party secrets management tools to offload some of the work related to controlling Kubernetes secrets.

Some of the popular tools and providers to achieve better security and usability for secrets are as follows:

• Cloud provider tools: AWS Secrets Manager, Google Secret Manager, Azure Key Vault, IBM Cloud Secrets Manager, Oracle Key Vault
• Open source tools: HashiCorp Vault

So what we need is a simple solution for at least some of these problems, that will bring the secrets stored in external secrets-management tools into our cluster, and continue using Kubernetes secrets in our applications. This means we need a component to synchronize the external secrets into our cluster. This is what the External Secrets Operator does for us.

Operator Design Pattern

Before we take a closer look at the External Secrets Operator, let's quickly recap what Kubernetes Operators are.

As we already know, each Kubernetes cluster has a desired state. This state determines what workloads (pods, deployments, etc.) should run, what images those workloads should use, and what other resources should be available to those workloads. Controllers are the control loops within the cluster that monitor the current state of the objects, compare it to the desired state, and make changes as needed. We also refer to  these control loops as the reconciliation loops.

Below is a general diagram of how this process works.

This process of managing application and infrastructure resources using declarative state and controllers is called the Operator Design Pattern. . Sometimes, we hear the terms controllers and operators used interchangeably. The difference is that operators have the domain-specific knowledge of how to create and manage the resources by reading the desired spec and updating the cluster by using the controllers.

What is the External Secrets Operator (ESO)?

ESO is a Kubernetes operator that connects to external secrets-management systems like the ones we mentioned above and reads secret information and injects the values into Kubernetes secrets. It is a collection of custom API resources that provide a user-friendly abstraction for the external APIs that manages the lifecycle of the secrets for us.

The Structure of External Secrets Operator

Like all other Kubernetes operators, ESO is composed of some main components:

• Custom Resource Definitions (CRD): These define the data schema of the settings available for the operator, in our case SecretStore and ExternalSecret definitions.
• Programmatic Structures: These define the same data schema as the CRDs above using the programming language of choice, in our case Go.
• Custom Resource (CR): These hold the values for the settings defined by the CRDs and describe the configuration for the operator.
• Controller: This is where the actual work takes place. Controllers act on custom resources and are responsible for creating and managing the resources. They can be created in any programming language, and ESO controllers are created in Go.

External Secret Providers

ESO uses different providers to connect to external secrets management systems and pull the secrets into the cluster. These providers are configured by the SecretStore and ExternalSecret resources that we’ll take a look at shortly. You can find the source code of the currently implemented providers here.

The structure of a secret provider is actually really simple:

type Provider interface{
	//NewClient constructs a SecretsManagerProvider
	NewClient(ctx context.Context, store GenericStore, kube client.Client, namespace string) (SecretsClient, error)

	//ValidateStore checks if the provided store is valid
	ValidateStore(store GenericStore) error
}

As you can see above, each provider has functions to validate the store configuration and instantiate the SecretsClient objects that are going to do the actual work.

SecretsClient instances are responsible for validating the secret configurations and pulling the secrets in various forms:

type SecretsClient interface{
	GetSecret(ctx context.Context, ref ExternalSecretDataRemoteRef) ([]byte, error)

	Validate() (ValidationResult, error)

	GetSecretMap(ctx context.Context, ref ExternalSecretDataRemoteRef) (map[string][]byte, error)

	GetAllSecrets(ctx context.Context, ref ExternalSecretFind) (map[string][]byte, error)

   Close(ctx context.Context) error
}

Let’s have a look at the resource types we mentioned and find out how they all work together to sync the external secrets.

SecretStore Resource

The SecretStore resource allows you to configure which external secret management service you want to access and how to access it by specifying the required configuration for authentication.

Here’s a sample configuration to access AWS Secrets Manager:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
	name: secretstore-sample
spec:
	provider:
		aws:
		service: SecretsManager
		region: us-east-1
		auth:
			secretRef:
				accessKeyIDSecretRef:
					name: awssm-secret
					key: access-key
				secretAccessKeySecretRef:
					name: awssm-secret
					key: secret-access-key

ExternalSecret Resource

Just as the SecretStore defines how secrets can be accessed, ExternalSecret resources define what should be retrieved. It has a reference to a SecretStore so that the controller for the ESO operator can use ExternalSecret resources to create Kubernetes secrets by using the configuration specified by the SecretStore resources.

This is how you connect both of the resources by using the secretStoreRef property:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
	name: example
spec:
	refreshInterval: 1h
	secretStoreRef:
		name: secretstore-sample
		kind: SecretStore
	target:
		name: secret-to-be-created
		creationPolicy: Owner
	data:
	- secretKey: secret-key-to-be-managed
		remoteRef:
			key: provider-key
			version: provider-key-version
			property: provider-key-property
	dataFrom:
	- extract:
			key: remote-key-in-the-provider

As soon as the application runs, all of the providers register themselves to ESO. Registering is simply adding a provider object and its specification to a map. Whenever the ESO controller needs access to a secret store, it uses this map to lookup the store by its name. We’re going to follow the same route when creating our own secret provider.

How does ESO synchronize the secrets?

As we have shown in the Operator Design Pattern section above, the controllers are responsible for reconciling the drift between the current state of the cluster and desired state by running in an infinite loop. ESO controllers are no different. During each reconciliation loop, external secrets controller does the following:

1. Reads the external secret configuration for the current reconciliation loop
2. Retrieves a SecretStore referenced by the external secret's configuration via the secretStoreRef property
3. Looks up the provider map mentioned above to find the provider associated with the secret, using the provider name from the store specification
4. Instantiates a secret client using the store provider name
5. Retrieves secret data from the external system using the secret client
6. If no secret data is returned and the deletion policy is set to Delete, the secret is deleted from the cluster. If the deletion policy is set to Retain, the secret is left as is.
7. Assuming that the external secret was retrieved successfully, the Kubernetes secret is created in the cluster, applying any specified templates.

Creating a Simple ESO Provider

Our goal in this section is to create a very simple ESO provider. Please keep in mind that what we are doing here is definitely not suitable for production. For more elegant and production-ready solutions, you can always take a look at the source code of other providers after you understand the workflow of adding a new provider.

Here are the steps to add a new secret provider to ESO:

1. Add the configuration schema for the new secret provider.
2. Create the type definitions to map the CRD definitions to Go structures.
3. Add the provider implementation.
4. Update register.go to include the new provider.
5. Create and deploy.

A Simple Secrets-Management Service

To keep the tutorial as simple as possible, and also considering that ESO already covers most of the common external systems for managing secrets, we will use a Node.js Express as a secret server for this tutorial.

Below is the implementation of the service:

const express = require('express');
const router = express.Router();
 
const keys = [];
 
/* GET keys listing as a JSON array */
router.get('/', (req, res, next) => {
   res.send(keys);
});
 
/* GET a single key as a JSON object. */
router.get('/:key', (req, res) => {
   const key = keys.find(k => k.key === req.params.key);
   res.send(key);
})
 
 
module.exports = router;

Adding a new CRD definition

We need to let Kubernetes know the configuration for the new provider. This is a minimal definition for the custom resource:

express:
  description: Configuration to sync secrets using Express provider
  properties:
    host:
      type: string
  required:
    - host
  type: object

This definition should be added to deploy/crds/bundle.yaml along with the other CRDs. The new provider has only one configuration property, host, which tells the provider where the secrets service is running.

Creating Types for the Provider Configuration

For the provider to get its configuration from the controller, we also need to add the necessary types so that the configuration can be converted to Go structs.

package v1beta1
 
type ExpressProvider struct {
   Host string `json:"host"`
}

As you can see, the configuration for the CRD and the structure above match. At runtime, the provider receives the configuration in the form of the above structure.

Implementing the Provider

We need to implement the Provider and SecretClient interfaces to make our provider work. Basically, we need to create a SecretClient and return it. The most important function we need to implement is the GetSecret function for the SecretClient. We could also add the validations to check whether the configuration of the store is correct. Below is the basic implementation for the provider and the explanation of each decision:

package express
 
import (
   "context"
   "encoding/json"
   "fmt"
   esv1beta1 "github.com/external-secrets/external-secrets/apis/externalsecrets/v1beta1"
   "io/ioutil"
   "log"
   "net/http"
   "net/url"
   "sigs.k8s.io/controller-runtime/pkg/client"
   "time"
)
 
const (
   errNilStore              = "nil store found"
   errMissingStoreSpec      = "store is missing spec"
   errMissingProvider       = "storeSpec is missing provider"
   errInvalidProvider       = "invalid provider spec. Missing express field in store %s"
   errInvalidExpressHostURL = "invalid express host URL"
)
 
// this struct will hold the keys that the service returns
type keyValue struct {
   Key   string `json:"key"`
   Value string `json:"value"`
}
 
type Provider struct {
   config  *esv1beta1.ExpressProvider
   hostUrl string
}
 
// NewClient this is where we initialize the SecretClient and return it for the controller to use
func (p *Provider) NewClient(ctx context.Context, store esv1beta1.GenericStore, kube client.Client, namespace string) (esv1beta1.SecretsClient, error) {
   config := store.GetSpec().Provider.Express
 
   return &Provider{
       config:  config,
       hostUrl: config.Host,
   }, nil
}
 
func (p *Provider) GetAllSecrets(ctx context.Context, ref esv1beta1.ExternalSecretFind) (map[string][]byte, error) {
   return nil, fmt.Errorf("GetAllSecrets not implemented")
}
 
// GetSecret reads the secret from the Express server and returns it. The controller uses the value here to
// create the Kubernetes secret
func (p *Provider) GetSecret(ctx context.Context, ref esv1beta1.ExternalSecretDataRemoteRef) ([]byte, error) {
   expressClient := http.Client{
       Timeout: time.Second * 5,
   }
 
   req, err := http.NewRequest(http.MethodGet, p.hostUrl+"/keys/"+ref.Key, nil)
   if err != nil {
       log.Fatal(err)
   }
 
   fmt.Printf("Sending request to: %s\n", p.hostUrl+"/keys/"+ref.Key)
 
   res, getErr := expressClient.Do(req)
   if getErr != nil {
       return nil, fmt.Errorf("error getting the secret %s", ref.Key)
   }
 
   if res.Body != nil {
       defer res.Body.Close()
   }
 
   body, readErr := ioutil.ReadAll(res.Body)
   if readErr != nil {
       return nil, fmt.Errorf("error reading secret %s", ref.Key)
   }
   fmt.Printf("body: %s\n", body)
 
   secret := keyValue{}
   jsonErr := json.Unmarshal(body, &secret)
   if jsonErr != nil {
       return nil, fmt.Errorf("bad key format: %s", ref.Key)
   }
   return []byte(secret.Value), nil
}
 
// ValidateStore validates the store configuration to prevent unexpected errors
func (p *Provider) ValidateStore(store esv1beta1.GenericStore) error {
   if store == nil {
       return fmt.Errorf(errNilStore)
   }
 
   spec := store.GetSpec()
   if spec == nil {
       return fmt.Errorf(errMissingStoreSpec)
   }
 
   if spec.Provider == nil {
       return fmt.Errorf(errMissingProvider)
   }
 
   provider := spec.Provider.Express
   if provider == nil {
       return fmt.Errorf(errInvalidProvider, store.GetObjectMeta().String())
   }
 
   hostUrl, err := url.Parse(provider.Host)
   if err != nil {
       return fmt.Errorf(errInvalidExpressHostURL)
   }
 
   if hostUrl.Host == "" {
       return fmt.Errorf(errInvalidExpressHostURL)
   }
 
   return nil
}
 
// registers the provider object to process on each reconciliation loop
func init() {
   esv1beta1.Register(&Provider{}, &esv1beta1.SecretStoreProvider{
       Express: &esv1beta1.ExpressProvider{},
   })
}

Registering the Provider to the Provider List

The next step is to import the provider module in register.go in order for its initialization function to be called:

package register
 
import (
   …
   _ "github.com/external-secrets/external-secrets/pkg/provider/express"
   …
)

Deploying ESO for testing

The steps required to deploy ESO to a Kubernetes cluster are described in the ESO documentation. However, because we are working locally, we can speed up the development and testing process by manually running the tasks defined in the Makefile.

First let’s deploy the CRDs:

make crds.install

Then run ESO locally:

make run

Testing the Provider with Secrets

To test the provider, we need to deploy SecretStore and ExternalSecret configurations to the cluster. The SecretStore configuration will point to the Express server and the ExternalSecret configuration will map the secret stored in the Express server to the Kubernetes secret.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
 name: secretstore-express
spec:
 provider:
   express:
     host: http://express-secrets-service
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
 name: express-external-secret
spec:
 refreshInterval: 1h
 
 secretStoreRef:
   kind: SecretStore
   name: secretstore-express
 
 target:
   name: my-express-secret
   creationPolicy: Owner
 
 data:
   - secretKey: secretKey # Key given to the secret to be created on the cluster
     remoteRef:
       key: my-secret-key

Deploy the manifest above:

kubectl apply -f secret.yaml 

If everything goes as planned, the secret should be created in the Kubernetes cluster.:

kubectl get secret my-express-secret -o yaml

Below is the output we get from Kubernetes API:

apiVersion: v1
data:
  secretKey: dGhpcy1pcy1hLXNlY3JldA==
immutable: false
kind: Secret

Conclusion

In this article, we explained the need for the External Secrets Operator and showed how you can start developing external secret providers. External Secrets Operator is a powerful tool for managing secrets in multi-tenant and multi-service environments that is used by many organizations in production.