Tools · Cloud security · 11 / 24
Investigating AWS CloudTrail logs: who did what, from where, with which credentials
Read a CloudTrail record field by field, follow an access key through its events, and answer the first questions of a cloud incident with the CLI and jq.
The fields that answer questions
eventTime and eventName say when and what. eventSource is the service, like iam.amazonaws.com. userIdentity says who: type (IAMUser, AssumedRole, Root, AWSService), arn, accessKeyId, and for roles, sessionContext with the session issuer. sourceIPAddress and userAgent say from where and with what tool. errorCode shows denied attempts. requestParameters and responseElements show details, such as which user was created.
accessKeyId is the thread you pull during an incident: it ties every action to one credential.
First questions in a leaked-key case
What did this key do, from which addresses, and when did it start? Did it create anything that would persist, such as new IAM users, access keys, roles, or login profiles? Did it touch data, such as S3 GetObject on sensitive buckets? Did it launch compute, a common sign of crypto-mining? Did it try to disable logging or GuardDuty?
Answer them in that order, writing each answer with the event IDs that support it.
Lab: follow one key
In a sandbox, use a test user’s key to list buckets, create a second IAM user, and attempt an action it lacks permission for. Then use lookup-events filtered by AccessKeyId and pipe through jq to print a timeline. Count events by source IP and user agent, and pull out the persistence events.
For logs in S3, the same jq expressions work on the gzipped JSON files, or use Athena or CloudTrail Lake for larger windows.
KEY=AKIAEXAMPLEKEY123456
aws cloudtrail lookup-events --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=$KEY --max-results 50 \
--query 'Events[].CloudTrailEvent' --output json | jq -r '.[] | fromjson | [.eventTime, .eventSource, .eventName, .sourceIPAddress, (.errorCode // "ok")] | @tsv' | sort
# persistence
aws cloudtrail lookup-events --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=$KEY --query 'Events[].CloudTrailEvent' --output json |
jq -r '.[] | fromjson | select(.eventName | test("CreateUser|CreateAccessKey|CreateLoginProfile|AttachUserPolicy|CreateRole|PutRolePolicy")) | [.eventTime,.eventName,(.requestParameters|tostring)] | @tsv'Containment
Deactivate the key, attach a deny-all policy to the user, and revoke active role sessions with the “Revoke active sessions” option, which adds a policy denying tokens issued before now. Then remove any persistence you found and check other regions, since attackers often launch resources in regions you do not use.
Write down every resource created so cleanup is complete and billing surprises are avoided.
Diagram
A CloudTrail record, annotated
One CreateAccessKey event from a leaked key.
CloudTrail record
{
"eventTime": "2026-11-02T03:14:07Z",
"eventSource": "iam.amazonaws.com",
"eventName": "CreateAccessKey",
"userIdentity": { "type": "IAMUser", "arn": "arn:aws:iam::111122223333:user/ci-bot", "accessKeyId": "AKIAEXAMPLEKEY123456" },
"sourceIPAddress": "203.0.113.77",
"userAgent": "aws-cli/2.17 ...",
"requestParameters": { "userName": "backup-admin" }
}- A CI user created a key for a different user, backup-admin.
- The source address is not your CI provider’s range.
- This is persistence: deactivate both keys and investigate backup-admin.
Check yourself
1. Which field ties every action to one credential?
2. Which event most clearly indicates an attacker creating persistence?
3. Why check regions you do not use?