In this little write-up I'll try to explain shortly what I did to be able to get to the ninth place of the SecurityTrails ReconMaster contest.
Throughouth the contest I tried to keep it as simple as possible.
- Find subdomains for not too well-known domains (using assetfinder)
- Extract words from the discovered subdomains (split by dots and dashes; using a custom script)
- Generate permutations based on the found words (using DNSCewl and custom scripts)
- Resolve the generated permutations to either A or CNAME records (using massdns)
My biggest challenge was the lack of memory and storage space required to generate the permutations. This I solved by splitting big files into smaller chunks and and run it in batches. Also switching between my laptop and VPS running different parts of the iteration helped too. To manage this I created a bunch of custom scripts.
As this was a contest for SeucrityTrails x Amass I did keep one thread of amass running. This was going over a list of 2 letter domains using the custom wordlist I generated. This wordlist kept growing while the steps described earlier iterated.
sudo snap install amass
go get -u github.com/tomnomnom/anew
go get -u github.com/tomnomnom/assetfinder
GO111MODULE=on go get -u -v github.com/projectdiscovery/shuffledns/cmd/shuffledns
GO111MODULE=on go get -u -v github.com/projectdiscovery/subfinder/v2/cmd/subfinder
git clone https://github.com/blechschmidt/massdns.git
cd massdns
make
sudo make install
git clone https://github.com/codingo/dnscewl.git
sudo cp dnscewl/DNScewl /usr/bin/DNScewl
I'll go over the scripts that I used now and try to explain them one by one. If you don't have Axiom setup you can comment the axiom line and uncomment the line below:
find_assets.sh (run on vps)
sort -T /mnt/dataset/tmp -u $1 -o $1
axiom-scan $1 -m assetfinder -o assetfinder_$1
# mkdir assetfinder_$f && cat $f | while read d; do /home/op/go/bin/assetfinder $d | tee -a assetfinder_$f/${d}.txt; done
rm -r ~/.axiom/logs/*This small script takes a file as input and sends it to my axiom instances to use assetfinder to find subdomains. Mostly I used this for the big lists of generated fld's.
split_fld_assetfinder_results.sh (run on vps)
mkdir ../flds
for i in {0..9}; do
cat $i* > ../flds/${1}flds_$i
done
for i in {a..z}; do
for j in {a..z}; do
cat ${i}${j}* > ../flds/${1}flds_${i}${j}
done
done
cd ../flds
ls | while read d; do
sort -T /mnt/g/tmp -u $d -o $d
done
cat * > ${1}flds_full
mkdir split
split -b 500M ${1}flds_full split/${1}flds
cd split
ls | while read d; do
gzip $d
doneTo have managable files I sometims used this script to create smaller chunks of files which I could use later on. This was also to save storage. Gzipping the text files created files that where about 20% of the original size.
cewl_files.sh (run local)
mkdir split_small
ls domains_* -Sr | while read d; do
touch words/${d}_words
cat $(ls domains_* |grep -v $d) | extract_words_from_domain | grep -v "^com$\|^net$" |anew -q words/${d}_words
sort -u words/${d}_words -o words/${d}_words
DNScewl -l $d --set-list /mnt/e/contest/cleaned/vhosts.txt > cewl_out/${d}_cewl_set;
DNScewl -l $d --append-list /mnt/e/contest/cleaned/vhosts.txt > cewl_out/${d}_append;
DNScewl -l $d --prepend-list /mnt/e/contest/cleaned/vhosts.txt > cewl_out/${d}_prepend;
DNScewl -l $d --set-list words/${d}_words > cewl_out/${d}_words_cewl_set;
DNScewl -l $d --append-list words/${d}_words > cewl_out/${d}_words_append;
DNScewl -l $d --prepend-list words/${d}_words > cewl_out/${d}_words_prepend;
sort -T /mnt/g/tmp -u cewl_out/${d}_cewl_set -o cewl_out/${d}_cewl_set
sort -T /mnt/g/tmp -u cewl_out/${d}_append -o cewl_out/${d}_append
sort -T /mnt/g/tmp -u cewl_out/${d}_prepend -o cewl_out/${d}_prepend
sort -T /mnt/g/tmp -u cewl_out/${d}_cewl_set -o cewl_out/${d}_words_cewl_set
sort -T /mnt/g/tmp -u cewl_out/${d}_append -o cewl_out/${d}_words_append
sort -T /mnt/g/tmp -u cewl_out/${d}_prepend -o cewl_out/${d}_words_prepend
gzip -f cewl_out/${d}_cewl_set
gzip -f cewl_out/${d}_append
gzip -f cewl_out/${d}_prepend
gzip -f cewl_out/${d}_words_cewl_set
gzip -f cewl_out/${d}_words_append
gzip -f cewl_out/${d}_words_prepend
# rm $d
doneThis is the script I used to generate the permutations using DNSCewl from the generated wordlist and a list of virtual hosts (dev/internal/corp etc.). This script was also responsible for generating the wordlist from the found domains. I exclude .com/.net because otherwise it created too much big files to process at all. I left in other extensions to generate fld's for those.
A small thing that might need explanation is on line 6. This part extracts the words from all domains, except the one it is working on. This is so that DNSCewl wont make permutations for all the known words for the current domain, but makes permutations on positions of words it finds out of the other list and replaces it with all the words from the list.
I also append/prepend with the both lists.
split_big_files_cleanup_small_files.sh (run local)
grep -Hrni "no target" | awk -F':' '{print $1}' | while read f; do
rm $f;
done
find . -size +100M | while read f; do
split -b 100M $f ./${f}_split; rm $f;
doneDNSCewl generated super large files which where to big to consume as is. Also, not all domains contains subdomains so there where some empty files. This little script was responsible for some cleanup. Removing the empty results and splitting large files into consumable chunks.
After cleanup I manually uploaded these files to my VPS again.
functions.sh (run on vps)
cleanupmassdns() {
input=$1
cat $input | awk '{print $1}' | sed 's/.$//' > cleaned
cp cleaned ../cleaned_copy & (cat ../cleaned_copy | extract_fld | anew ../flds && while read fld; do parseflds $fld | tee -a ../assets; done) &
}
sendrequest() {
input_file=$1
sort -u $input_file -o $input_file
cat $input_file | gzip > ${input_file}.tmp
gzip $input_file
input_file=${input_file}.tmp
response=$(curl -X POST --url "https://api.securitytrails.com/v1/submit/hostnames" -H 'Content-Encoding: gzip' --header 'APIKEY: $SECURITYTRAILS_TOKEN' --data-binary "@$input_file");
if [[ $response == *"has been exceeded"* ]]; then
echo "Waiting one hour because:"
echo $response
secs=$((60 * 60))
while [ $secs -gt 0 ]; do
echo -ne "$secs\033[0K\r"
sleep 1
: $((secs--))
done
sendrequest $input_file | tee -a curl_out
else
echo $response
fi;
}
masscheckandsend() {
massdns -w massdns -o S -r /mnt/dataset/lists/resolvers.txt $1
cat massdns | awk '{print $1}' | sed 's/.$//' > cleaned
split -b 50m cleaned split/split
for cf in $(ls split/*); do
sort -u $cf -o $cf
sendrequest $cf
done
rm split/*
}
rundnscewl() {
domain=$1
DNScewl --target-list=$domain --append-list=../../../lists/vhosts.txt > dnscewl/$domain
masscheckandsend dnscewl/$domain
DNScewl --target-list=$domain --prepend-list=../../../lists/vhosts.txt > dnscewl/$domain
masscheckandsend dnscewl/$domain
DNScewl --target-list=$domain --set-list=../../../lists/vhosts.txt > dnscewl/$domain
masscheckandsend dnscewl/$domain
DNScewl --target-list=$domain --set-list=words > dnscewl/$domain
masscheckandsend dnscewl/$domain
rm dnscewl/$domain
rm assets_new
}
I've been playing quite a bit with different methods, but some functionaly was similar every time. For that I used this file to include the functions:
- cleanupmassdns - to get the resolved domains without ip address or cname
- sendrequest - to send the request to securitytrails
- masscheckandsend - resolve the generated domains, this function is later replaced with axiom
- rundnscewl - one of my first functions which I later stopped using, as the files became to big for my vps
parseflds.sh (run local)
input=$1;
d=$(dig +nocmd $input any +short);
if [[ $d ]]; then
echo $input
fiThe cleanupmassdns function uses this file to do a quick resolve on the flds. If it does resolve it is later used in a new iteration to find subdomains and then create permutations again.
scan_test.sh (run on vps)
source ./functions.sh
while read d; do
echo "Running for: $d"
assetfinder $d >> massdns_out
grep $d dnscewl | shuffledns -d $d -wt 100 -r /mnt/dataset/lists/resolvers.txt -silent >> massdns_out
lines=$(wc -l massdns_out | awk '{print $1}')
if [ $lines -ge 30000 ] ; then
split -b 50m massdns_out split/split
for cf in $(ls split/*); do
sort -u $cf -o $cf
sendrequest $cf
done
rm split/*
rm massdns_out
touch massdns_out
fi;
done<fldsThis script was used to try out new methods. I think this version is one of the earlier things I tried as I undo most of my changes every time. Thought it's nice to include in this repository.
scan_top50.sh (run on vps)
source ./functions.sh
for f in $(shuf asset); do
echo "Assetfinder"
assetfinder $f | shuffledns -d $f -wt 100 -r /mnt/dataset/lists/resolvers.txt | tee -a assets_new
echo "Subfinder"
subfinder -silent -d $f | shuffledns -d $f -wt 100 -r /mnt/dataset/lists/resolvers.txt | anew assets_new
cat assets_new | extract_words_from_domain | sort -u >> words
sort -u words -o words
echo "Wordlist"
shuffledns -silent -d $f -w words -wt 100 -r /mnt/dataset/lists/resolvers.txt | anew assets_new
sort -u assets_new -o assets_new
lines=$(wc -l assets_new | awk '{print $1}')
if [ $lines -ge 30000 ] ; then
sendrequest assets_new
rundnscewl assets_new
fi
done
rundnscewl assets_newThis script, also more in the beginning of the contest, was used to go over the alexa top 50 domains which lived in the asset file. For every domain run assetfinder, subfinder and shuffledns. For shuffledns I used a wordlist which was already generated out of the earlier found subdomains, but did not grow yet.
scan_bulk.sh (run on vps)
source ./functions.sh
zcat $1 > ${1}.tmp
f=${1}.tmp
axiom-scan $f -m massdns -o massdns_$f
# sudo /usr/bin/massdns -r /home/op/lists/resolvers.txt -o S $f -w massdns_$f
cleanupmassdns massdns_$f
split --bytes=200M cleaned ../split/split
for s in $(ls ../split/split*); do
sendrequest $s
done
rm massdns_$f
rm ../split/*
rm $f
rm $1
rm -r ~/.axiom/logs/*The scan_bulk is basically my final script which I used througout the most part of the contest. It's small compared to the others, because I executed it via custom while loops directly from the command line, instead of having the loops in the script. The input where files generated using the previous scripts which where run locally.
Most of the work in this script is done via axiom's massdns module, resolving large files with generated domains.
When resolving is ready the result files are cleaned up (while extracting flds' for the next iteration) and the subdomains are send in to securitytrails.
This is the endresult of all the words generated during the contest, sorted and made unique. Use it well.