awk and sed for log processing: fields, filters, sums, and in-place cleanup

awk in five patterns

awk splits each line into fields $1, $2, and so on, with NF as the number of fields. A pattern selects lines and an action runs on them. Associative arrays make grouping easy, and an END block prints results. -F sets the field separator for CSV or colon-separated data.

Snippet

awk '{print $1, $9}' access.log                       # IP and status
awk '$9 >= 500' access.log                             # server errors
awk '{bytes[$1] += $10} END {for (ip in bytes) print bytes[ip], ip}' access.log | sort -rn | head
awk -F: '$3 == 0 {print $1}' /etc/passwd               # UID 0 accounts
awk -F, 'NR > 1 && $5 == "DENY" {c[$3]++} END {for (k in c) print c[k], k}' fw.csv | sort -rn

sed for extraction and redaction

sed applies edits line by line. s/old/new/g substitutes; -n with p prints only matches; -E enables extended regular expressions. -i edits files in place, so test without -i first and keep a backup with -i.bak. Redaction before sharing logs is a common defensive use.

Snippet

sed -n 's/.*Failed password for \(invalid user \)\?\([^ ]*\) from \([0-9.]*\).*/\2 \3/p' auth.log | sort | uniq -c | sort -rn | head
sed -E 's/([0-9]{1,3}\.){3}[0-9]{1,3}/x.x.x.x/g' access.log > access.redacted.log
sed -E 's/(password|token|apikey)=[^& ]+/\1=REDACTED/Ig' app.log > app.redacted.log

Lab: answer five questions

Using your lab web server access log, answer with awk and sed only: the top 10 client IPs, the number of 404 responses per path, total bytes sent per status code, all requests with a query string containing password= redacted, and the busiest minute. Compare one answer against the same question in a SIEM or Python script if you have one.

Snippet

awk '{print substr($4, 2, 17)}' access.log | sort | uniq -c | sort -rn | head -1

When to switch tools

awk and sed are perfect for line-oriented logs. For JSON, use jq. For quoted CSV with embedded commas, use a real CSV parser such as Python’s csv module or mlr (Miller), because simple field splitting gets it wrong.

Diagram

awk versus sed

Columns and math versus text edits.

awk

Thinks in. Fields and records

Great for. Filtering by column, counting, summing

Example. awk '$9==404 {c[$7]++}'

sed

Thinks in. Lines and regex substitutions

Great for. Extracting, replacing, redacting

Example. sed -E 's/token=[^&]+/token=REDACTED/g'

Check yourself

1. Which awk construct is used to count values per key?

2. What does sed -i do?

3. Which tool should handle CSV with quoted fields containing commas?