Domain History Crawler

dezcrwl is a `Web History Osint` tool for gathering URLs, subdomains, JavaScript endpoints, and sensitive information using various OSINT sources like Wayback Machine, Common Crawl, and VirusTotal.**

---

Features ⁀➴

• Fetches URLs from Common Crawl, Wayback Machine, and VirusTotal

• Detects sensitive information such as API keys and JWT tokens

• Generates detailed reports in TXT or JSON format

• Extracts JavaScript endpoints and hidden APIs

• Discovers subdomains automatically

• Supports status code filtering

│
│
🢃

$ Installation For Linux

$ Installation For Windows

$ Usage

$ Configuration

$ Showcase

🢇⁀🐧 ˗˗˗ˏˋ🖳ˎˊ˗˗˗ 🪟⁀🡾

Linux/macOS

Run these commands:

git clone https://github.com/zebbern/dezcrwl.git
cd dezcrawl
chmod +x linuxinstall.sh
./linuxinstall.sh

Windows

Run these commands:

git clone https://github.com/zebbern/dezcrwl.git
cd dezcrawl
windowsinstall.sh

Now run dezcrwl -h anywhere in terminal

Usage:

Run the script with a domain:

python3 dezcrwl.py -t example.com -cw -js -sum -o output.txt

Extract JavaScript endpoints:

python3 dezcrwl.py -t example.com -js

Fetch results from Wayback and VirusTotal:

python3 dezcrwl.py -t example.com -cw -vt YOUR_VIRUSTOTAL_API_KEY

Filter specific file types:

python3 dezcrwl.py -t example.com -f "(\.json|\.env|\.bak|\.sql)"

How its supposed to run & with what:

dezCrwl target.com -dir -cw -js -ws -sum -o output.txt -f "(\.json|\.env|\.bak|\.backup|\.old|\.git|\.svn|\.swp|\.sql|\.db|\.sqlite|\.log|\.txt|\.zip|\.rar|\.tar\.gz|\.7z|\.pdf|\.docx|\.xlsx|\.conf|\.ini|\.yml|\.yaml|\.dump|\.sql\.dump|\.session|\.pem|\.key|\.crt|\.tmp)"

Configuration:

config.yaml:

• verbose: Enables detailed logging (true/false)

• output_format: Choose "txt" or "json"

• API keys: Configure URLScan and CertSpotter API keys

Showcase

Coming....

Developer:

• GitHub: https://github.com/zebbern