User-Agent parser for robots.txt, X-Robots-tag and Robots-meta-tag rule sets
-
Updated
Apr 4, 2021 - PHP
User-Agent parser for robots.txt, X-Robots-tag and Robots-meta-tag rule sets
The Robots.txt Generator tool helps you to create the Robots.txt file for your website.
Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag
Add a description, image, and links to the robots-tag topic page so that developers can more easily learn about it.
To associate your repository with the robots-tag topic, visit your repo's landing page and select "manage topics."