Aditya Sutar's Blog

Secure Secret Management in Kubernetes: Exploring Different Approaches

Photo by Michael Dziedzic on Unsplash

Secure Secret Management in Kubernetes: Exploring Different Approaches

Aditya Sutar's photo
Aditya Sutar
·

7 min read

Table of contents

"Security is not a product, but a process." - Bruce Schneier.

Introduction

Secrets are an important aspect of any application, containing sensitive information such as passwords, API keys, and certificates. Storing and managing secrets securely is crucial to maintaining the overall security of an application. While Kubernetes provides a built-in secrets management solution, it has been deemed least secure due to the way it stores secrets in a base64 encoded format. To address this issue, several key management services have been developed, such as AWS KMS, Azure Key Vault, and Hashicorp Vault, that offer a more secure way of storing secrets. However, the challenge remains on how to pass these secrets to the application securely. In this blog, we will explore three different approaches for securely passing secrets to applications in Kubernetes.

Kubernetes Built-in Secrets

Kubernetes has built-in support for secrets management through the Secrets API. This API allows you to store and manage sensitive information such as passwords, API keys, and TLS certificates in your Kubernetes cluster.

However, it's important to note that the default storage format for Kubernetes secrets is base64-encoded, which provides only a basic level of obfuscation. Anyone with access to the Kubernetes API can easily decode the secret and retrieve the sensitive information.

Additionally, Kubernetes secrets have some limitations, such as a maximum size of 1MB per secret and no built-in support for rotating secrets. As a result, using Kubernetes built-in secrets may not be sufficient for more complex security requirements.

For these reasons, it's recommended to consider other options for storing and managing secrets, such as using external key management services. However, even when using external key management services, it's still important to securely pass the secrets to the application. In the following sections, we'll explore three approaches for doing so.

Key Management Services

One of the best ways to securely store secrets is by using Key Management Services (KMS) provided by cloud providers or third-party vendors. KMS provides a centralized and secure way to manage keys, certificates, and secrets. These services allow for the secure storage of secrets and control over who can access them.

AWS KMS, Azure Key Vault, Google Cloud KMS, and HashiCorp Vault are some of the popular key management services that can be integrated with Kubernetes for secure secret management.

Approach 1 - Volumes

One way to securely pass secrets to your application in Kubernetes is to use volumes. This approach involves storing secrets in an external service, such as a Key Management Service (KMS), and then mounting the secret data into a volume in the pod. The application can then read the secrets from the mounted volume.

However, cloud-native applications generally like to receive secrets via environment variables, rather than from files. To address this, Kubernetes provides a CSI (Container Storage Interface) driver to sync Kubernetes secrets with the external storage provider. Using this driver, we can mount the secret data as a volume and expose it as environment variables to the application.

While this approach allows us to keep secrets secure in an external service, it still requires storing the secret data in Kubernetes secrets, which are stored in base64-encoded format and may not be the most secure option.

Approach 2 - Sidecar Container

Another approach to securely pass secrets to an application in Kubernetes is by using sidecar containers. With this approach, a sidecar container is added to the pod that is responsible for retrieving the secrets from an external service and passing them to the main application container via a shared volume or environment variables.

The sidecar container can be configured to retrieve the secrets from a Key Management Service or any other external service that is responsible for securely storing secrets. The main application container can then read the secrets from the shared volume or environment variables provided by the sidecar container.

As an example, we can use the Vault sidecar container with the official HashiCorp Vault Kubernetes integration. The sidecar container is responsible for retrieving the secrets from Vault, and the main application container can read the secrets from the shared volume or environment variables provided by the sidecar container.

This approach ensures that secrets are never stored in the pod's file system and are only passed securely to the main container.

However, it's important to note that while we are passing secrets securely to the container, if someone gains access to the container, they can access the secrets as well. So, it's important to implement additional security measures such as pod security policies and container runtime security to minimize the risk of unauthorized access to the secrets.

Approach 3 - Secret Injector

Another approach to securely pass secrets to applications in Kubernetes is by using a secret injector. This approach involves using a Kubernetes admission controller to automatically inject secrets into pods at runtime. One popular example of this approach is the Azure Key Vault to Kubernetes (akv2k8s) project. The akv2k8s project retrieves secrets from Azure Key Vault and injects them into pods using Kubernetes mutating webhooks.

When a pod is created or updated, the admission controller intercepts the pod creation request and adds the necessary environment variables and volumes to the pod specification. The environment variables and volumes are then populated with the secret values at runtime, ensuring that the application running inside the pod has access to the secrets.

One of the main advantages of this approach is that the secrets are not even accessible by the container running in the pod. Instead, only the application process running inside the container receives the secrets, which are injected at runtime.

This approach provides an extra layer of security as the secrets are never stored in the pod or container, and access to the secrets is strictly controlled by the Kubernetes API server.

However, it's important to note that the security of the secrets still relies on the security of the external service used. Any security issues or vulnerabilities in the external service can compromise the security of the secrets. It's also important to ensure that the Kubernetes admission controller is configured securely to prevent unauthorized access to the secrets during the injection.

Overall, the secret injector approach can be a powerful tool for securely passing secrets to applications in Kubernetes, and the akv2k8s project is a popular and well-maintained implementation of this approach.

Best Practices

While using Kubernetes for secret management, it is important to follow best practices to ensure the security of your secrets. Here are some best practices that you should consider:

  1. Avoid using Kubernetes built-in secrets: As mentioned earlier, Kubernetes built-in secrets are not secure as secret data is stored in base64-encoded format. It is recommended to use a third-party secret management solution like Key Management Services.

  2. Use RBAC for secret access control: Kubernetes provides Role-Based Access Control (RBAC) to control access to secrets. You should use RBAC to restrict access to secrets to only the necessary users or services.

  3. Use short-lived secrets: Avoid using long-lived secrets, as it increases the risk of unauthorized access. Instead, use short-lived secrets that expire after a certain time.

  4. Use secret rotation: Regularly rotate secrets, such as passwords and certificates, to minimize the risk of unauthorized access.

  5. Use TLS for communication: Use Transport Layer Security (TLS) to encrypt communication between Kubernetes components and secret management systems.

  6. Use encryption for secret data at rest: Encrypt the secret data at rest using encryption tools and techniques.

  7. Implement least privilege access: Use the principle of least privilege to grant access to secret management systems, tools, and APIs.

  8. Use container image scanning: Use container image scanning to detect and fix vulnerabilities in container images before deploying them to Kubernetes.

By following these best practices, you can ensure the security and confidentiality of your secrets in Kubernetes.

Conclusion

In conclusion, secret management is an essential aspect of Kubernetes application deployment. While Kubernetes provides a built-in secrets management solution, it is not very secure and lacks some important features that are necessary for secure secret management.

Key Management Services such as AWS KMS, Azure Key Vault, and HashiCorp Vault provide a more secure and reliable way of storing secrets. To pass these secrets securely to the application, we explored three approaches: volumes, sidecar containers, and secret injectors. Each approach has its own advantages and limitations, and the choice of approach depends on the specific use case.

In addition to these approaches, there are also some best practices that can be followed for secure secret management, such as minimizing secret access, rotating secrets regularly, and implementing multi-factor authentication.

Overall, secure secret management is critical for the security and integrity of applications running in Kubernetes, and it is important to choose the right approach and follow best practices to ensure that secrets are kept safe and secure.

References

SecurityKubernetes