Nuclei version 3.x large memory allocation due to /etc/hosts entries.

[dinosn]

Nuclei version:

root@pd-srv:/usr/local/bin# nuclei -version
[INF] Nuclei Engine Version: v3.1.5
[INF] Nuclei Config Directory: /root/.config/nuclei
[INF] Nuclei Cache Directory: /root/.cache/nuclei

Current Behavior:

Memory exhaustion on the system, with or without any scan operations, even on simple nuclei execution without any parameter.

Expected Behavior:

Nuclei to be using less memory as per versions 2.9.x

Steps To Reproduce:

Create a large /etc/hosts entry containing several systems. My current one has 900+ entries.

root@pd-srv:~# wc -l /etc/hosts
915 /etc/hosts
root@pd-srv:~# time nuclei

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.1.5

		projectdiscovery.io

[WRN] Found 18 template[s] loaded with deprecated paths, update before v3 for continued support.
[INF] Current nuclei version: v3.1.5 (outdated)
[INF] Current nuclei-templates version: v9.7.2 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 61
[INF] Templates loaded for current scan: 7358
[INF] Executing 7377 signed templates from projectdiscovery/nuclei-templates
[INF] No results found. Better luck next time!

real	3m56.535s
user	3m35.756s
sys	0m11.113s

3m56sec loading time.

root@pd-srv:~# cp /etc/hosts.ok /etc/hosts
root@pd-srv:~# wc -l /etc/hosts
13 /etc/hosts
root@pd-srv:~# time nuclei

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.1.5

		projectdiscovery.io

[WRN] Found 18 template[s] loaded with deprecated paths, update before v3 for continued support.
[INF] Current nuclei version: v3.1.5 (outdated)
[INF] Current nuclei-templates version: v9.7.2 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 61
[INF] Templates loaded for current scan: 7358
[INF] Executing 7377 signed templates from projectdiscovery/nuclei-templates
[INF] No results found. Better luck next time!

real	0m40.354s
user	0m19.210s
sys	0m1.079s

40 seconds load time.

Whilst attempting to run with the host file containing large entry of systems memory allocation on the system will be at:

root@pd-srv:~# free -h
              total        used        free      shared  buff/cache   available
Mem:           15Gi        14Gi       178Mi       128Mi       547Mi       268Mi
Swap:         9.0Gi       5.7Gi       3.2Gi
root@pd-srv:~# 

Anything else:

During normal scan (and whilst /etc/hosts contains a large entry of addresses) with minimal set, 1-2 URLs nuclei will consume ~ 15Gb of RAM ( active + swap ) and be killed eventually with OOM kill.

Regards,
Nicolas

[tarunKoyalwar]

@dinosn , this has been fixed in #4630 (comment)
and it will merged and new release will be created after furthur testing. meanwhile you can use nuclei from linked pr by installing using below command

go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@051f274