HackBlue

Tools · Containers and Kubernetes security · 07 / 20

Kubernetes Secrets are base64, not encrypted: encryption at rest and external secret stores

See why base64 is not protection, configure etcd encryption at rest with an EncryptionConfiguration, and decide when to use an external secrets manager.

10 min read · practice only with authorization

What a Secret really is

A Secret is an API object whose data fields are base64 encoded so they can hold binary content. Base64 is an encoding, not encryption: anyone who can get the object can decode it instantly. By default, Secrets are also stored unencrypted in etcd, so a copy of an etcd backup reveals them.

Protecting Secrets therefore means three things: limit who can read them through RBAC, encrypt them at rest, and keep them out of Git and container images.

Encryption at rest

The API server can encrypt resources before writing them to etcd using an EncryptionConfiguration file passed with --encryption-provider-config. The kms v2 provider uses an external key management service and is the recommended option. aescbc and secretbox use a local key in the file. The identity provider means no encryption. Managed services such as EKS, AKS, and GKE offer envelope encryption with their cloud KMS as a setting.

Snippet
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: [secrets]
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded 32-byte key>
      - identity: {}

Lab: see the difference in etcd

In a kind cluster, create a Secret and read it back directly from etcd using etcdctl inside the control-plane node: the value appears in plain text after the key path. Then enable the encryption config above on the API server static pod, rewrite all Secrets so they are re-encrypted, and read etcd again: the value now starts with k8s:enc:aescbc:v1:key1.

Snippet
kubectl create secret generic demo --from-literal=password=lab-only-value
echo bGFiLW9ubHktdmFsdWU= | base64 -d

# inside the control-plane container
ETCDCTL_API=3 etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/demo | hexdump -C | head

kubectl get secrets -A -o json | kubectl replace -f -

External secret stores

Tools such as External Secrets Operator or the Secrets Store CSI Driver sync or mount values from AWS Secrets Manager, Azure Key Vault, Google Secret Manager, or HashiCorp Vault. That centralizes rotation and auditing. For GitOps, Sealed Secrets or SOPS let you commit encrypted values instead of plaintext manifests.

Diagram

Default Secret versus protected Secret

Encoding is not encryption.

Default

In the manifest. base64

In etcd. Plaintext

Who can read. Anyone with get secrets

Protected

In the manifest. Not in Git, or sealed with SOPS

In etcd. Encrypted with KMS

Who can read. Only the workload’s service account

Check yourself

1. Is a Kubernetes Secret’s data encrypted because it looks scrambled in YAML?

2. Which API server setting enables encryption of Secrets in etcd?

3. After enabling encryption, existing Secrets are still plaintext in etcd. How do you fix that?