Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

eternals sources are not used #70

Open
ogma-sec opened this issue Jun 5, 2019 · 4 comments
Open

eternals sources are not used #70

ogma-sec opened this issue Jun 5, 2019 · 4 comments
Assignees
@Nekmo
Labels
bug Something isn't working
Milestone
v0.7.0

Comments

@ogma-sec
Copy link

ogma-sec commented Jun 5, 2019 •
edited
Loading

Hello,

It seems that there is an issue with external source. Using a proxy I cannot see any of these sources used (no google or virus total and no request to robots.txt). I have done several test with Python2 and Python2 on both master/dev version and I cannot succed to make dirhunt use external sources.

`root@kali:/var/www/html# ls -al
total 24
drwxr-xr-x 4 root root 4096 juin 5 20:38 .
drwxr-xr-x 3 root root 4096 juil. 28 2018 ..
drwxr-xr-x 2 www-data www-data 4096 juin 5 18:02 images
-rw-r--r-- 1 www-data www-data 59 juin 5 18:04 index.html
-rw-r--r-- 1 www-data www-data 19 juin 5 20:36 robots.txt
drwxr-xr-x 2 www-data www-data 4096 juin 5 20:19 secret

root@kali:/var/www/html# cat robots.txt
Disallow: /secret/

root@kali:/var/www/html# python /opt/dirhunt/scripts/dirhunt http://127.0.0.1/robots.txt
Welcome to Dirhunt v0.6.0 using Python 2.7.15
[200] http://127.0.0.1/ (Blank page)
Index file found: index.html
[200] http://127.0.0.1/images/ (Generic)
▎ Finished after 2 seconds
No interesting files detected ¯_(ツ)_/¯

root@kali:/var/www/html# python3 /opt/dirhunt/scripts/dirhunt http://127.0.0.1/robots.txt
Welcome to Dirhunt v0.6.0 using Python 3.7.3
[200] http://127.0.0.1/ (Blank page)
Index file found: index.html
[200] http://127.0.0.1/images/ (Generic)
◥ Finished after 2 seconds
No interesting files detected ¯_(ツ)_/¯ `
@Nekmo
Copy link
Owner

Nekmo commented Jun 5, 2019

Hi, maybe Dirhunt is not waiting for the source process. Normally with a real (and slower) webite this problem should not happen. Maybe that's why I have not detected it in my tests, thank you.

I will add it to fix it to the next version. 👍

@Nekmo Nekmo self-assigned this Jun 5, 2019
@Nekmo Nekmo added the bug Something isn't working label Jun 5, 2019
@Nekmo Nekmo added this to the v0.7.0 milestone Jun 5, 2019
@ogma-sec
Copy link
Author

ogma-sec commented Jun 7, 2019

Hello, thank you for your response. I have executed several other test on real website (my blog) that have a robots.txt and using a BurpSuite Proxy to see request made by the tool and I don't see any robots.txt request (or access to search engin or virus total).

dirhunt https://ogma-sec.fr --proxies http://127.0.0.1:8080

At least I think I should see a request on https://ogma-sec.fr/robots.txt made by dirhunt in order to check its content but I don't see it. So I'm not sure its about the execution time of a local website. Crawling my blog is taking at least +40 second to dirhunt.

By the way thank you and congrats for dirhunt, it saves me a lot of time :)

Regards,

@Nekmo
Copy link
Owner

Nekmo commented Jun 13, 2019 •
edited
Loading

Screenshot_20190613_221336

I am doing tests with your blog and I see that there is a issue with robots parser.

$ curl http://ogma-sec.fr/robots.txt
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://ogma-sec.fr/robots.txt">here</a>.</p>
</body></html>

@Nekmo
Copy link
Owner

Nekmo commented Jun 13, 2019

Ok, I'm still doing tests and it seems that it is detecting the paths.

Screenshot_20190613_222606

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Nekmo Nekmo

Labels
bug Something isn't working
Projects
None yet
Milestone
v0.7.0
Development

No branches or pull requests
2 participants
@Nekmo @ogma-sec