Reading /proc for live response: exe, cmdline, environ, and deleted binaries

A window into each process

/proc is a virtual filesystem the kernel generates on demand. Each running process has a directory /proc/PID with files describing it. exe is a symlink to the executable, cmdline holds arguments separated by null bytes, environ holds environment variables, cwd links to the working directory, fd lists open file descriptors, and maps lists memory mappings and loaded libraries.

Reading these requires root or ownership of the process. They are a reliable source during live response because they come straight from the kernel.

Snippet

PID=1234
sudo ls -l /proc/$PID/exe /proc/$PID/cwd
sudo tr '\0' ' ' < /proc/$PID/cmdline; echo
sudo tr '\0' '\n' < /proc/$PID/environ
sudo ls -l /proc/$PID/fd

Recovering a deleted binary

Malware often deletes its own file after starting so nothing is left on disk. The kernel keeps the inode alive while the process runs, and /proc/PID/exe shows the old path followed by “(deleted)”. Copying from /proc/PID/exe recovers the exact binary for hashing and analysis.

find /proc/*/exe -lname "*(deleted)" lists every process whose executable was removed. Package upgrades can cause this harmlessly, so check names against recent updates before raising an alarm.

Lab: recover and hash

Copy /bin/sleep to /tmp/demo, start it, then delete /tmp/demo. Confirm ls -l /proc/PID/exe shows (deleted). Copy the file out, hash it, and compare the hash to /bin/sleep. They should match, proving the recovery is exact.

In a real case, also save cmdline, environ, maps, and the fd list to an evidence folder with timestamps before doing anything else.

Snippet

cp /bin/sleep /tmp/demo && /tmp/demo 900 & PID=$!
rm /tmp/demo
ls -l /proc/$PID/exe
cp /proc/$PID/exe /root/recovered-demo
sha256sum /root/recovered-demo /bin/sleep
sudo find /proc/*/exe -lname "*(deleted)" 2>/dev/null

Know the limits

A rootkit that loads into the kernel can hide entries from /proc. If you suspect that, capture memory with a dedicated tool and analyze it offline.

environ shows the environment at start time. Changes the process made later are not reflected there.

Diagram

Files under /proc/PID and what they answer

Each file is a different question.

exe
Which binary is really running, even if deleted?
cmdline
What arguments was it started with?
environ
What environment did it inherit?
fd
Which files and sockets are open?
maps
Which libraries are loaded?

Check yourself

1. ls -l /proc/4321/exe shows “/tmp/x (deleted)”. What happened?

2. How do you recover that binary?

3. Why use tr on /proc/PID/cmdline?