A robots.txt Disallow line is not access control

The line is a crawl hint

robots.txt on your own site, or on an organization that authorized the review, can list Disallow entries. Each entry is a path the file asks cooperating crawlers not to index. It is text in that file. It is not a password, and it is not a rule the web server enforces.

A browser that requests the path is not stopped by the line. If you need to know whether the path responds, that is a separate fetch of your own URL, with the status written down.

The line does not prove the path

A Disallow entry can name a path that returns 404, a path that is public, or a path you retired. The entry alone does not show which. Do not file the path as a live admin panel because the word appears in robots.txt.

Do not file the path as protected because Disallow is present. Protection is access control on the server, which this file does not show.

What to write down

File the robots.txt URL, the Disallow line, and the time you saved the file. If you fetched the path, file that status separately. If you did not, write “path not fetched.”

Lab: a Disallow line and a 404

https://example.org/robots.txt, a site you control, contains Disallow: /lab-notes. Fetching https://example.org/lab-notes returns 404. A draft says the path is a hidden panel and that robots.txt keeps it safe.

The note quotes the Disallow line as a crawl hint and the 404 as the fetch. The path is not described as a live panel, and the file is not described as access control.

robots: https://example.org/robots.txt
line: Disallow: /lab-notes
fetch: https://example.org/lab-notes
status: 404
write: hint, and a 404
not: a hidden panel, and not protection