HackBlue

Tools · Containers and Kubernetes security · 20 / 20

Responding to a compromised Kubernetes pod: isolate, preserve, investigate

A practical sequence for a suspicious pod: isolate it with labels and network policy, preserve evidence before it is rescheduled, check token use in audit logs, and rebuild cleanly.

10 min read · practice only with authorization

Do not delete it first

Deleting a pod destroys its filesystem and memory, and its controller will start a fresh copy, possibly with the same weakness. First isolate it so it can do no more harm, then capture evidence, then remove it.

Isolate

Remove the pod from its Deployment’s selector by changing a label, so the controller creates a replacement and stops managing the suspect pod. Apply a NetworkPolicy that selects the quarantine label and denies all ingress and egress. Cordon the node if you suspect node compromise.

Snippet
kubectl -n shop label pod api-7d9f8 app=quarantine --overwrite
cat <<'EOF' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: quarantine, namespace: shop }
spec:
  podSelector: { matchLabels: { app: quarantine } }
  policyTypes: [Ingress, Egress]
EOF
kubectl cordon worker-2

Lab: preserve and investigate in a lab

In a lab cluster, simulate the scenario by running a pod that writes a marker file and makes periodic outbound requests. Isolate it as above, then collect evidence: the pod spec, events, logs from all containers, a list of processes, and a copy of suspicious files. With kubectl debug, attach an ephemeral container that shares the process namespace for inspection without modifying the target image. Search the audit log for API calls made by the pod’s service account during the window.

Snippet
kubectl -n shop get pod api-7d9f8 -o yaml > pod.yaml
kubectl -n shop describe pod api-7d9f8 > describe.txt
kubectl -n shop logs api-7d9f8 --all-containers --timestamps > logs.txt
kubectl -n shop debug -it api-7d9f8 --image=busybox:1.36 --target=api -- ps aux
kubectl -n shop cp api-7d9f8:/tmp/marker ./evidence-marker
jq -c 'select(.user.username=="system:serviceaccount:shop:api") | {t:.requestReceivedTimestamp,verb,res:.objectRef.resource,code:.responseStatus.code}' audit.log

Recover

Determine the entry point, such as a vulnerable dependency, exposed service, or leaked credential. Rotate any Secrets the pod could read and any cloud credentials its service account could assume. Patch and rebuild the image, redeploy, and add the detection that would have caught it earlier. Then delete the quarantined pod and uncordon or replace the node.

Diagram

Pod incident sequence

  1. Isolate

    Relabel the pod out of its controller; apply a deny-all policy.

  2. Preserve

    Save spec, logs, events, processes, and suspicious files.

  3. Investigate

    Trace the entry point and API calls by its service account.

  4. Recover

    Rotate reachable secrets, rebuild the image, redeploy, add detection.

Check yourself

1. Why not delete a suspicious pod immediately?

2. How do you stop a Deployment from managing a suspect pod while keeping it running?

3. Which source shows what the attacker did with the pod’s service account token?