HackBlue

Tools · SIEM and detection engineering · 07 / 24

Regular expressions for log analysis: extracting IPs, users, and URLs safely

Build regular expressions step by step to pull fields out of log lines with grep, Splunk rex, and KQL extract, and avoid patterns that are slow or match too much.

10 min read · practice only with authorization

The pieces you need

Most log extraction uses a handful of constructs: character classes like \d and [A-Za-z0-9._-], quantifiers like + and {1,3}, anchors ^ and $, groups ( ) to capture, and non-greedy quantifiers like .*? to stop at the first match. Named groups such as (?<user>...) give extracted fields readable names.

Common patterns

An IPv4 pattern for logs does not need to validate octet ranges perfectly; it needs to match the field reliably. The examples below extract the user and source IP from an OpenSSH failure line in three tools.

Snippet
# sample line
# Nov  2 10:14:03 web1 sshd[2211]: Failed password for invalid user deploy from 203.0.113.45 port 51122 ssh2

grep -oP 'Failed password for (invalid user )?\K\S+(?= from)' /var/log/auth.log | sort | uniq -c | sort -rn
grep -oE 'from ([0-9]{1,3}\.){3}[0-9]{1,3}' /var/log/auth.log | awk '{print $2}' | sort | uniq -c | sort -rn | head

# Splunk
| rex "Failed password for (invalid user )?(?<user>\S+) from (?<src_ip>\d{1,3}(?:\.\d{1,3}){3})"

# KQL
| extend user = extract(@"Failed password for (?:invalid user )?(\S+) from", 1, SyslogMessage)

Lab: build it incrementally

Open regex101.com or use grep on a sample auth log from your own lab VM. Start by matching the literal text “Failed password for”, then add the optional “invalid user ” group, then capture the username, then the IP. After each step, check that every sample line still matches and that no extra text is captured. Finally, write a version for an Apache or nginx access log that captures the client IP, method, path, and status code.

Snippet
# nginx combined log example
# 198.51.100.7 - - [02/Nov/2026:10:20:11 +0000] "GET /login HTTP/1.1" 401 512 "-" "curl/8.5.0"
grep -oE '^[0-9.]+ .*"(GET|POST|PUT|DELETE) [^ ]+ HTTP/[0-9.]+" [0-9]{3}' access.log | head

Pitfalls

Greedy .* swallows more than you expect. Nested quantifiers such as (a+)+ can cause catastrophic backtracking on long lines. Escape dots in IP and domain patterns. When a log has a structured format like JSON, parse it with a JSON parser instead of a regex.

Diagram

Regex building blocks for logs

Most extraction uses only these.

  • \d{1,3}(?:\.\d{1,3}){3}

    An IPv4-looking address.

  • (?<user>\S+)

    A named capture of a run of non-space characters.

  • .*?

    Non-greedy: stop at the first possible match.

  • ^ and $

    Anchor to start and end of line.

  • \K

    In grep -P, drop everything matched so far from the output.

Check yourself

1. Which pattern stops at the first possible match instead of the last?

2. Which Splunk command extracts fields with a regular expression?

3. Logs are in JSON. What is the best way to extract fields?