spider-rs

All

24 repositories

spider
Public
Web crawler and scraper for Rust
rust crawler automation spider scraping indexer headless-chrome
Rust
•
MIT License
•155•1.9k•2•0•Updated Aug 24, 2025Aug 24, 2025
ua_generator
Public
Pre-compiled random real User-Agents. Updated weekly for windows, mac, linux, and android
useragent user-agent-string ua-generator
Rust
•
MIT License
•3•33•0•0•Updated Aug 24, 2025Aug 24, 2025
spider-clients
Public
Python, Javascript, and Rust libraries for the Spider Cloud API.
crawler scraper ai spider html-to-markdown web-scraping ai-agents supabase ai-scraping llm-webcrawler
Python
•
MIT License
•7•19•0•0•Updated Aug 22, 2025Aug 22, 2025
spider_fingerprint
Public
The fingerprinting library for Rust
emulation headless fingerprint stealth
Rust
•
MIT License
•0•4•0•0•Updated Aug 19, 2025Aug 19, 2025
firewall
Public
Firewall for Rust
rust blacklist networking firewall adblock adguard
Rust
•
MIT License
•0•0•0•0•Updated Jul 19, 2025Jul 19, 2025
case_insensitive_string
Public
A case-insensitive string for Rust
rust string case-insensitive case-insensitive-strings
Rust
•
MIT License
•1•4•0•0•Updated Jul 16, 2025Jul 16, 2025
spider_transformations
Public
HTML transformation library for Rust
html-to-markdown transform-data html-transformation
Rust
•
MIT License
•0•2•0•0•Updated Jul 8, 2025Jul 8, 2025
spider_chrome
Public
Concurrent high-level API to control Chrome over the DevTools Protocol for Rust
automation headless-chrome chrome-devtools-protocol
Rust
•
Apache License 2.0
•2•14•1•0•Updated Jul 8, 2025Jul 8, 2025
headless-browser
Public
Headless browser - reliable high performance setup for cloud providers like Fargate, CloudRun, K8, and more.
chrome headless cdp chrome-api chrome-server
Rust
•
MIT License
•6•57•0•0•Updated Jul 6, 2025Jul 6, 2025
spider_network_blocker
Public
Network blocking lists on ads, scripts, and analytics for Spider
networking
Rust
•
MIT License
•0•1•0•0•Updated Jun 7, 2025Jun 7, 2025
web-crawling-guides
Public
How to guides on web-crawling or scraping
crawler scraper html-to-markdown web-scraping agents ai-agents ai-scraping llm-webcrawler clean-markdown fast-webcrawler
5•22•0•1•Updated Apr 26, 2025Apr 26, 2025
spider_scraper
Public
html5ever web scraper used for some of spider-rs CSS transformations
Rust
•
MIT License
•0•1•0•0•Updated Mar 16, 2025Mar 16, 2025
bad_websites
Public
websites that should be blocked via firewall
1•0•0•0•Updated Mar 13, 2025Mar 13, 2025
html2md
Public
HTML to markdown for Rust
html markdown rust html2md
HTML
•
MIT License
•0•41•0•0•Updated Feb 22, 2025Feb 22, 2025
spider-nodejs
Public
Spider ported to Node.js
nodejs distributed-systems crawler scraper typescript spider indexer headless-chrome
Rust
•
MIT License
•8•51•0•0•Updated Jan 28, 2025Jan 28, 2025
spider-py
Public
Spider ported to Python
python crawler scraper spider web-crawler headless-chrome
Rust
•
MIT License
•14•90•3•0•Updated Jan 28, 2025Jan 28, 2025
spider-wasm
Public
The spider project ported to wasm [wip]
Rust
•
MIT License
•0•0•0•0•Updated Jan 6, 2025Jan 6, 2025
spider-cloud-live-code-viewer
Public
Visual web crawler
spider web-crawler vscode llm ai-crawler
TypeScript
•
MIT License
•1•4•1•0•Updated Dec 21, 2024Dec 21, 2024
auto-encoder
Public
Auto encoding library bytes to strings
encoding string-encoding auto-encoder
Rust
•
MIT License
•1•1•0•0•Updated Dec 1, 2024Dec 1, 2024
proxier
Public
Auto proxy whitelisting for 3rd parties
Rust
•
MIT License
•0•3•0•0•Updated Nov 2, 2024Nov 2, 2024
readability
Public
The readability library for Rust
rust readability data-cleaning clean-data safari-reader llm-training
Rust
•
MIT License
•2•12•0•0•Updated Oct 25, 2024Oct 25, 2024
homebrew-spider-cloud-cli
Public
Ruby
•0•0•0•0•Updated Aug 5, 2024Aug 5, 2024
async_job
Public
Simple async cron jobs for Rust
rust cronjob async-jobs async-task
Rust
•0•3•0•0•Updated Dec 27, 2023Dec 27, 2023
jsdom
Public
JSDOM for Rust [incomplete]
jsdom dom-parser
Rust
•
MIT License
•0•4•0•0•Updated Aug 2, 2023Aug 2, 2023