Guide and best practices
robots.txt is a small file with large visibility consequences. gene.re helps draft directives in the browser, so staging paths, private URL patterns, and planned site structures do not need to be sent to an external generator.
Use it to prepare allow and disallow rules, sitemap references, and crawler-specific notes. Always test final behavior with search engine tools because robots.txt guides crawlers but does not protect private content.
Use cases
- Draft crawl rules for a new site or migration.
- Add sitemap references for SEO handoff.
- Prepare staging exclusions without exposing internal paths.
FAQ
Does robots.txt block access?
No. It asks compliant crawlers not to fetch paths, but it is not authentication.
Are my paths uploaded?
No. Drafting happens locally in the browser.
Where should the file be placed?
It normally lives at the root of the site, such as /robots.txt.