HackBlue

Tools · Ethical OSINT and privacy self-defense · 07 / 17

Finding old exposure with the Wayback Machine and archived URLs

Use the Internet Archive’s CDX API to list historical URLs on your domain, find old documents, endpoints, and configuration files that may still be live, and request removals where appropriate.

8 min read · practice only with authorization

History matters

Pages and files removed from navigation often remain on servers, and archives preserve what was once public. Old API endpoints, exported spreadsheets, and staging links may still respond. Attackers mine archives for exactly these.

The CDX API

The Wayback Machine CDX API returns captured URLs for a domain with timestamps, status codes, and MIME types. Filter for interesting extensions and collapse duplicates.

Snippet
curl -s 'https://web.archive.org/cdx/search/cdx?url=example.org/*&output=json&fl=timestamp,original,statuscode,mimetype&collapse=urlkey&limit=5000' \
  | jq -r '.[1:][] | @tsv' > wayback.tsv
grep -Ei '\.(pdf|xlsx?|csv|sql|bak|zip|env|json|config)(\?|\t|$)' wayback.tsv | head -50
cut -f2 wayback.tsv | grep -Ei '/api/|/admin|/staging|/test' | sort -u | head

Lab: check whether old URLs still respond

For your own domain, extract historical URLs with interesting extensions or paths, then check which still return 200 today with a HEAD request at a gentle rate. For each live one, decide whether it should exist. Record findings and fix them on your servers.

Snippet
cut -f2 wayback.tsv | grep -Ei '\.(pdf|xlsx|csv|bak|zip)
#x27; | sort -u | while read -r u; do
  code=$(curl -s -o /dev/null -w '%{http_code}' -I --max-time 10 "$u")
  printf '%s\t%s\n' "$code" "$u"; sleep 1
done | grep '^200'

Removal

Fix the live server first. For archived copies of sensitive content, the Internet Archive accepts removal requests from site owners. Archives are also useful for incident timelines, such as proving when a page was defaced or when content appeared.

Diagram

Historical exposure review

  1. List

    CDX API for every captured URL on your domain.

  2. Filter

    Documents, backups, configs, old APIs and staging paths.

  3. Check live

    HEAD requests at a gentle rate.

  4. Fix

    Remove or protect live content; request archive removal if needed.

Check yourself

1. Which interface lists captured URLs for a domain in bulk?

2. What should you check for each interesting historical URL?

3. What should happen before requesting removal of an archived copy?