HackBlue

Tools · Incident response playbooks · 16 / 20

Playbook: suspected web server or web application compromise

Investigate signs of a web compromise such as unexpected files or outbound connections, preserve logs and disk state, find the entry point, and rebuild safely.

11 min read · practice only with authorization

Signs

Unexpected files in web directories (especially script files with recent timestamps), web server processes spawning shells or network tools, new outbound connections from the server, defacement, unexplained high CPU, or alerts from a web application firewall followed by 200 responses to suspicious requests.

Preserve first

Snapshot the VM or disk through the hypervisor or cloud API. Copy web server access and error logs, application logs, and the database logs off the host. Record running processes and connections. If it is in the cloud, preserve load balancer and WAF logs, which may live outside the instance.

Snippet
find /var/www -type f -newermt '2026-10-28' -name '*.php' -printf '%TY-%Tm-%Td %TT %u %p\n' | sort
ps -eo pid,ppid,user,cmd --forest | grep -A3 -E 'nginx|apache|php-fpm'
ss -tanp | grep -E 'www-data|nginx|php'
grep -E '" (200|500) ' /var/log/nginx/access.log | grep -iE 'upload|\.php\?|cmd=|\.\./' | tail -50

Lab: find the planted file

On a lab web server VM, plant a harmless file named status-check.php containing only a comment and a phpinfo-free echo of the date, with its timestamp set to look old using touch. Then investigate as if you did not know: compare the web root against the deployment package or Git repository with a checksum diff, check inode change times with stat (ctime cannot be set with touch), and search access logs for requests to the new file.

Snippet
cd /var/www/html && git status --porcelain && git diff --stat
stat status-check.php   # Change time shows when the file was really written
grep 'status-check.php' /var/log/nginx/access.log

Eradicate and rebuild

Identify the entry point: a vulnerable plugin, an exposed admin panel, stolen deployment credentials, or file upload abuse. Rebuild the server from a known-good image and redeploy the application from source control, rather than cleaning files in place. Rotate database passwords, API keys, and deployment credentials stored on the server. Patch the vulnerability and add detection for the behavior you found.

Diagram

Where to look

Combine file, process, and log evidence.

  • Web root diff

    Compare against Git or the deployment artifact.

  • ctime and mtime

    touch can fake mtime, not ctime.

  • Process tree

    Web server user spawning shells or downloaders.

  • Access logs

    Requests to new files; suspicious requests with 200 responses.

  • Outbound connections

    Unexpected destinations from the server.

Check yourself

1. Which timestamp is hard for an attacker to fake with touch?

2. What is the most reliable way to find unexpected files in a web root?

3. After a web server compromise, what is the safer recovery?