Tools · Containers and Kubernetes security · 07 / 20
Kubernetes Secrets are base64, not encrypted: encryption at rest and external secret stores
See why base64 is not protection, configure etcd encryption at rest with an EncryptionConfiguration, and decide when to use an external secrets manager.
What a Secret really is
A Secret is an API object whose data fields are base64 encoded so they can hold binary content. Base64 is an encoding, not encryption: anyone who can get the object can decode it instantly. By default, Secrets are also stored unencrypted in etcd, so a copy of an etcd backup reveals them.
Protecting Secrets therefore means three things: limit who can read them through RBAC, encrypt them at rest, and keep them out of Git and container images.
Encryption at rest
The API server can encrypt resources before writing them to etcd using an EncryptionConfiguration file passed with --encryption-provider-config. The kms v2 provider uses an external key management service and is the recommended option. aescbc and secretbox use a local key in the file. The identity provider means no encryption. Managed services such as EKS, AKS, and GKE offer envelope encryption with their cloud KMS as a setting.
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: [secrets]
providers:
- aescbc:
keys:
- name: key1
secret: <base64-encoded 32-byte key>
- identity: {}Lab: see the difference in etcd
In a kind cluster, create a Secret and read it back directly from etcd using etcdctl inside the control-plane node: the value appears in plain text after the key path. Then enable the encryption config above on the API server static pod, rewrite all Secrets so they are re-encrypted, and read etcd again: the value now starts with k8s:enc:aescbc:v1:key1.
kubectl create secret generic demo --from-literal=password=lab-only-value echo bGFiLW9ubHktdmFsdWU= | base64 -d # inside the control-plane container ETCDCTL_API=3 etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt \ --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key \ get /registry/secrets/default/demo | hexdump -C | head kubectl get secrets -A -o json | kubectl replace -f -
External secret stores
Tools such as External Secrets Operator or the Secrets Store CSI Driver sync or mount values from AWS Secrets Manager, Azure Key Vault, Google Secret Manager, or HashiCorp Vault. That centralizes rotation and auditing. For GitOps, Sealed Secrets or SOPS let you commit encrypted values instead of plaintext manifests.
Diagram
Default Secret versus protected Secret
Encoding is not encryption.
In the manifest. base64
In etcd. Plaintext
Who can read. Anyone with get secrets
In the manifest. Not in Git, or sealed with SOPS
In etcd. Encrypted with KMS
Who can read. Only the workload’s service account
Check yourself
1. Is a Kubernetes Secret’s data encrypted because it looks scrambled in YAML?
2. Which API server setting enables encryption of Secrets in etcd?
3. After enabling encryption, existing Secrets are still plaintext in etcd. How do you fix that?