Analyzing JSON lines logs in Python: CloudTrail, Suricata, and app logs without a SIEM

JSON lines

Many tools write one JSON object per line: Suricata eve.json, Zeek in JSON mode, application logs, and exported cloud logs. Reading line by line keeps memory low even for multi-gigabyte files. CloudTrail delivers files containing a Records array, which you can flatten into lines first with jq.

Filter and group

A small helper reads nested fields with a dotted path and a default, so missing keys do not crash the script.

Snippet

import gzip, json, sys, collections

def read(path):
    opener = gzip.open if path.endswith('.gz') else open
    with opener(path, 'rt') as f:
        for line in f:
            if line.strip():
                yield json.loads(line)

def get(obj, dotted, default=None):
    for part in dotted.split('.'):
        if not isinstance(obj, dict) or part not in obj:
            return default
        obj = obj[part]
    return obj

counts = collections.Counter()
for ev in read(sys.argv[1]):
    if get(ev, 'errorCode') in ('AccessDenied', 'UnauthorizedOperation'):
        counts[(get(ev, 'userIdentity.arn', '?'), get(ev, 'eventName'))] += 1
for (who, action), n in counts.most_common(20):
    print(f'{n:5} {action:35} {who}')

Lab: CloudTrail denied calls

Export a day of CloudTrail logs from a sandbox account, flatten them to JSON lines with jq, and run the script to list identities with the most AccessDenied errors by action. Many denials from one role often indicate a misconfigured application or someone probing permissions with stolen credentials. Then adapt the filter for Suricata alerts by changing the field names.

Snippet

aws s3 sync s3://my-trail-bucket/AWSLogs/111122223333/CloudTrail/us-east-1/2026/11/02/ ./ct
for f in ct/*.json.gz; do gzip -dc "$f" | jq -c '.Records[]'; done > ct.jsonl
python3 denied.py ct.jsonl

When to use pandas or DuckDB

For ad hoc questions on a few million rows, DuckDB can query JSON lines files directly with SQL, and pandas.read_json(lines=True) loads them into a DataFrame. Plain Python streaming is best when files are huge or when you need custom parsing logic.

Snippet

duckdb -c "SELECT eventName, count(*) n FROM read_json_auto('ct.jsonl') WHERE errorCode='AccessDenied' GROUP BY 1 ORDER BY n DESC LIMIT 10"

Diagram

Plain Python streaming versus DuckDB

Choose by size and question.

Python streaming

Memory. One line at a time

Best for. Huge files, custom logic

Effort. Write the grouping yourself

DuckDB SQL

Memory. Efficient columnar engine

Best for. Ad hoc questions with GROUP BY

Effort. One SQL query

Check yourself

1. Why read JSON lines files line by line?

2. Why use a helper with defaults to read nested fields?

3. Many AccessDenied errors from one role may indicate what?