Robots.txt: crawl instructions, not a lock
Robots.txt gives cooperating crawlers instructions about which URLs they may request. It is not a password, a privacy shield or a reliable way to remove a URL from search. A rule that blocks crawling can also prevent Google from seeing a page’s noindex instruction.
Useful context
- Use the actual public domain, including the right hostname.
- Keep a copy of existing rules before an authorised administrator edits anything.
How it works
Read the existing file first
Open /robots.txt on your own public domain. You may see rules aimed at all crawlers or named ones, plus a sitemap location. Record the content and ask what it is intended to protect before removing rules that seem unfamiliar.
Distinguish preview from production
A development preview may deliberately discourage all crawling. A reviewed public website needs different settings. If a launch seems invisible to search engines, ask whether the preview restriction was carried over; do not simply enable every private route at the same time.
Choose the right kind of control
For an ordinary utility page you want excluded from search, noindex is normally the relevant indexing instruction. Customer briefs and records need authentication as well. Blocking their path in a text file does not stop a person from opening a known address.
Check the intended outcome
After an approved rules change, have the developer check important pages and required resources. Use Search Console evidence for a real public URL where available. Record which rule changed and why so a later deployment does not quietly restore an inappropriate setting.
What to check
- The rules belong to the correct public hostname.
- Useful public content and necessary resources are not accidentally blocked.
- Personal information is protected independently of crawler instructions.
Sources and further reading
Platform features can change. Check the provider’s current guidance before changing a live service.