Kubernetes audit logs: writing an audit policy and finding suspicious API activity

Levels and stages

An audit policy is an ordered list of rules. The first matching rule sets the level: None, Metadata (who, what, when, response code), Request (plus the request body), or RequestResponse (plus the response body). Events are emitted at stages such as RequestReceived and ResponseComplete. Logging Secrets at Request or RequestResponse would write their values into the log, so Secrets are usually logged at Metadata only.

A starting policy

The policy below skips noisy read-only system traffic, records Secret and ConfigMap access as metadata, records full requests for RBAC and pod exec, and falls back to metadata for everything else.

Snippet

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: [RequestReceived]
rules:
  - level: None
    users: [system:kube-proxy]
    verbs: [watch]
  - level: Metadata
    resources: [ { group: '', resources: [secrets, configmaps] } ]
  - level: Request
    resources:
      - { group: rbac.authorization.k8s.io }
      - { group: '', resources: [pods/exec, pods/attach, pods/portforward] }
  - level: Metadata

Lab: enable and query

On a kind cluster, mount the policy into the API server static pod and add --audit-policy-file and --audit-log-path flags. Then perform a few actions: exec into a pod, read a Secret, and create a RoleBinding. Query the JSON lines log with jq to find each one with the user, source IP, and response code.

Snippet

jq -c 'select(.objectRef.subresource=="exec") | {t:.requestReceivedTimestamp,user:.user.username,ns:.objectRef.namespace,pod:.objectRef.name}' audit.log
jq -c 'select(.objectRef.resource=="secrets" and .verb=="get") | {user:.user.username,secret:.objectRef.name,code:.responseStatus.code}' audit.log
jq -c 'select(.objectRef.apiGroup=="rbac.authorization.k8s.io" and (.verb=="create" or .verb=="update")) | {user:.user.username,kind:.objectRef.resource,name:.objectRef.name}' audit.log

What to alert on

Useful detections include exec into pods in production namespaces, list or get of Secrets across namespaces by a service account, new ClusterRoleBindings to cluster-admin, anonymous requests that succeed, and 403 bursts from one identity, which suggest permission probing. Managed clusters expose audit logs through CloudWatch, Azure Monitor, or Cloud Logging.

Diagram

Audit levels

First matching rule wins.

None
Do not log.
Metadata
User, verb, resource, timestamp, response code.
Request
Metadata plus the request body.
RequestResponse
Everything, including the response body. Never for Secrets.

Check yourself

1. Which level is appropriate for Secrets in an audit policy?

2. Two rules match an event. Which one sets the level?

3. A burst of 403 responses from one service account most likely indicates what?