This package get, fetch, crawl, sitemap pages recursively and fetch all links in between <loc> tag.
Collect links through the sitemap.xml or robots.txt
GoSitemap2Md is a Golang program that generates a sitemap URL in Markdown format and stores the URLs in a urls.json file for easy adding of new URLs. This tool simplifies the process of generating and maintaining a sitemap for your website.
The Firecrawl Toolkit is the easiest way for developers to interact with web content through crawling, scraping, and mapping capabilities.
a python script that crawls website sitemap in a very quick way with multi threading and extract, write SEO based data to CSV file
Add a description, image, and links to the sitemap-crawler topic page so that developers can more easily learn about it.
To associate your repository with the sitemap-crawler topic, visit your repo's landing page and select "manage topics."