jq tutorial for security work: filtering, selecting, grouping, and CSV output

Explore first

Start with jq . to pretty-print, keys to list fields, and paths to see the structure of nested objects. For JSON lines, jq processes one object per line automatically; use -c for compact output and -s to slurp everything into an array when you need to group.

Snippet

jq . event.json | head -40
jq 'keys' event.json
jq -c 'paths | join(".")' event.json | sort -u | head -40
head -1 eve.json | jq .

Select and reshape

select keeps objects matching a condition. Build new objects with braces to keep only fields you care about. The // operator supplies defaults for missing fields. test applies a regular expression.

Snippet

jq -c 'select(.event_type == "alert") | {ts: .timestamp, src: .src_ip, dst: .dest_ip, sig: .alert.signature}' eve.json
jq -c 'select(.errorCode? // "" | test("AccessDenied|Unauthorized"))' ct.jsonl
jq -r '.Records[] | select(.eventName == "ConsoleLogin") | [.eventTime, .userIdentity.arn, .sourceIPAddress, (.responseElements.ConsoleLogin // "")] | @tsv' trail.json

Lab: group and export

Using a Suricata eve.json from a lab pcap or a CloudTrail export from a sandbox account, count events by type, find the top alert signatures, and export alerts to CSV for a spreadsheet. Then write a one-line jq filter that answers a question you choose, such as “which IAM principals called DeleteBucket”.

Snippet

jq -r '.event_type' eve.json | sort | uniq -c | sort -rn
jq -s '[.[] | select(.event_type=="alert")] | group_by(.alert.signature) | map({sig: .[0].alert.signature, n: length}) | sort_by(-.n) | .[:10]' eve.json
jq -r 'select(.event_type=="alert") | [.timestamp, .src_ip, .dest_ip, .dest_port, .alert.signature] | @csv' eve.json > alerts.csv

Tips

Use -r for raw strings without quotes, @tsv or @csv for tabular output, --arg to pass shell variables safely instead of string concatenation, and the ? suffix to ignore errors on missing paths. For very large files, avoid -s, which loads everything into memory; use reduce or pipe to sort and uniq instead.

Snippet

jq --arg ip "$IP" -c 'select(.src_ip == $ip)' eve.json

Diagram

jq building blocks

Most filters combine these.

.field.sub
Navigate into objects.
select(cond)
Keep matching objects.
{a: .x, b: .y}
Reshape to the fields you need.
group_by / map / length
Aggregate after -s.
@csv / @tsv with -r
Export rows for spreadsheets.

Check yourself

1. Which jq function keeps only objects matching a condition?

2. How should you pass a shell variable into a jq filter?

3. Why be careful with -s on huge files?