#

robots-tag

Here is 1 public repository matching this topic...

toimik / RobotsProtocol

Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag

robots-txt robots meta-tags robots-parser robots-exclusion-standard robotstxt robots-tag robots-exclusion-protocol x-robots-tag

Updated May 2, 2024
C#

Improve this page

Add a description, image, and links to the robots-tag topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the robots-tag topic, visit your repo's landing page and select "manage topics."