httpx: find which inventoried hosts actually serve web pages

Which httpx this is

Two tools share the name. The Python httpx package installs a client also called httpx. The one used here is the Go probe from ProjectDiscovery. If httpx -version prints a Python traceback, call the Go binary by its full path or rename it.

Probing an owned list

httpx reads hosts from a file, tries HTTPS and HTTP, and prints one line per responding service. -sc, -title, -server, and -td (tech detect) add the columns most reviews need. -rl caps requests per second; set it so the probe never registers as load on a shared staging box.

Snippet

httpx -l passive.txt -sc -title -server -td -ip -rl 10 -silent -o live.txt

httpx -l passive.txt -sc -title -tls-grab -j -o live.jsonl -rl 10

Reading the lines it prints

Status codes and titles separate real applications from parked pages and default server screens. A default title such as “Welcome to nginx!” on a public name is often a host that was half-built and left.

Snippet

https://www.hackblue-lab.org [200] [HackBlue] [Vercel] [203.0.113.10]
https://api.hackblue-lab.org [401] [] [nginx] [203.0.113.24]
https://staging.hackblue-lab.org [200] [Welcome to nginx!] [nginx/1.18.0] [198.51.100.7]
http://legacy-ftp.hackblue-lab.org [403] [403 Forbidden] [Apache/2.4.29] [192.0.2.45]

Worked check: certificates about to lapse

With -tls-grab, the JSON line carries the certificate’s subject, issuer, and not_after date. A jq filter lists every certificate that expires in the next 30 days, which is a better weekly report than waiting for a browser warning.

Snippet

jq -r 'select(.tls.not_after != null) | [.url, .tls.not_after] | @tsv' live.jsonl | sort -k2

https://staging.hackblue-lab.org  2026-11-19T23:59:59Z

Diagram

A host list versus a probed inventory

Probing turns names into facts.

Name list

Contains. Every name a source ever saw

Tells you. What might exist

Risk. Stale entries look the same as live ones

httpx output

Contains. Services that answered today

Tells you. Status, title, server, certificate

Risk. Only as current as the last run

Check yourself

1. httpx -version prints a Python error. What happened?

2. Which flag keeps the probe from loading a shared staging server?

3. A public name returns “Welcome to nginx!”. What does that usually suggest?