HackBlue

Tools · Incident response playbooks · 08 / 20

Linux live response triage: commands to capture state before containment

Collect processes, network connections, users, persistence, and recent file changes from a suspect Linux host with trusted binaries, writing output off the system with hashes.

10 min read · practice only with authorization

Use trusted tools and external storage

An attacker with root may have replaced common binaries such as ps or netstat. Where possible, run statically linked tools from read-only media, or collect with an agent such as Velociraptor or a script like UAC (Unix-like Artifacts Collector). Write output to a mounted external volume or stream it over the network, and hash everything.

What to collect

Date and uptime, logged-in users, process list with full command lines and executable paths, open network connections and listening ports, loaded kernel modules, cron and systemd units, SSH authorized_keys, recently modified files in system paths, shell histories, and key logs.

Snippet
OUT=/mnt/ir/$(hostname)-$(date -u +%Y%m%dT%H%M%SZ); mkdir -p "$OUT"
{ date -u; uptime; who -a; last -Fw | head -50; } > "$OUT/system.txt"
ps -eo pid,ppid,user,lstart,cmd --forest > "$OUT/ps.txt"
ls -l /proc/*/exe 2>/dev/null | grep -E 'deleted|/tmp|/dev/shm' > "$OUT/suspicious_exe.txt"
ss -tanp > "$OUT/ss.txt"; ss -lunp >> "$OUT/ss.txt"
lsmod > "$OUT/lsmod.txt"
systemctl list-units --type=service --all > "$OUT/services.txt"; ls -la /etc/cron* /var/spool/cron* > "$OUT/cron.txt" 2>&1
find / -xdev \( -path /proc -o -path /sys \) -prune -o -type f -mtime -3 -print 2>/dev/null > "$OUT/recent_files.txt"
for h in /root /home/*; do cat "$h"/.ssh/authorized_keys "$h"/.bash_history 2>/dev/null | sed "s|^|$h: |"; done > "$OUT/user_artifacts.txt"
cp -a /var/log/auth.log* /var/log/secure* "$OUT/" 2>/dev/null
sha256sum "$OUT"/* > "$OUT/SHA256SUMS"

Lab: run it on a test VM

On a Linux lab VM, attach a second virtual disk, mount it at /mnt/ir, and run the collection. Plant a benign marker first: a cron entry and a running process from /tmp named something obvious like lab-marker. Confirm your collection captured the cron entry, the process with its /tmp path, and the file in recent_files.txt. Then try UAC with its ir_triage profile and compare.

Snippet
git clone https://github.com/tclahr/uac.git && cd uac
sudo ./uac -p ir_triage /mnt/ir

Memory on Linux

For memory, AVML from Microsoft or LiME can acquire RAM. Analysis with Volatility 3 needs a symbol table matching the exact kernel, so record the kernel version with uname -a during collection.

Diagram

Linux triage checklist

Capture before you contain.

  • Processes

    Full command lines, executable paths, deleted binaries.

  • Network

    Established connections and listeners with owning processes.

  • Persistence

    cron, systemd units, authorized_keys, shell profiles.

  • Recent changes

    Files modified in the last few days.

  • Logs and memory

    auth logs, journal, and a RAM image with kernel version.

Check yourself

1. Why prefer trusted or static binaries during live response?

2. Which artifact suggests a process is running from a deleted file?

3. What must you record to analyze a Linux memory image later?