Unable to Spider and Scan after Webpage Authentication

[kvkvenugopal]

I want to spider and scan the webpage after authentication (form-based). It's working on the ZAP GUI but not working with zap-cli in the command line. I was able to login and authenticate using context but that's it. As soon as I run the spider or active scan after authentication, it's not working. Kindly Help.

[Grunny]

Hi. The current version of zap-cli doesn't support specifying a context when starting the spider or scan, but I definitely want to add support for that, hopefully I can get to trying to get that into the tool soon. :)

[kvkvenugopal]

Thank you Grunny, actually I'm also working on that with my team. Thank You anyway

[Grunny]

Cool, are you working on it in your own scripts or adding it to zap-cli? If it's for zap-cli, I'd be grateful for a PR. :)

[kvkvenugopal]

We are adding it to Zap-Cli only and of course I'll be giving a PR. One more thing, have u been able to enable a forced user mode using Zap-Cli ? Actually my team has found out that the ZAP API has some issues while spidering after authentication using context. Due to this also we are unable to spider from command line. But the GUI works well for everything. Any help will be great :)

[ddccffvv]

Any progress on this? It would be very helpful!

[isaguimiot]

Hi Grunny, do you plan to work on this issue on a near future ? We want to use zap in our devops pipelines, and we're currently stuck because we can't authenticate before we scan our app...
Thanks !

[Grunny]

@isaguimiot I have most of it done, but I'm not sure of the best way to allow actually setting up an auth method via commandline, so I'm still playing with that.

What I could provide and release now is the ability to import a context that already has authentication set up and then use that to run scans as logged-in users. You could then open ZAP on your local machine, setup the authentication method and users for a context for the site you're running ZAP against, export the context to a file, and then import that file using a command like zap-cli context import /path/to/context/file as part of your devops pipelines. Then scans would be run by also providing that context ID and the user (i.e. zap-cli active-scan --context "ContextName" --user "UserName" -r <URL to scan>).

Would that work for your use case? I should be able to do that fairly quickly. :)

[isaguimiot]

@Grunny : yes, that's exactly what I was trying to do (the "zap-cli context import file"), I already have a context that was manually exported from Zap with authentication information. I just didn't find a way to specify it in the command line. If you can't fix it quickly, I'll be able to test it on my application, and I'll give you some feedback right after.

Thanks !

[Grunny]

@isaguimiot OK, I put together a quick implementation today, which you can find here: https://github.com/Grunny/zap-cli/compare/context-and-user-scans

It was a quick PoC, so I want to clean it up, add some validation, add tests, etc. but did you want to give it a test and see if it meets your needs, and give any feedback on the parameters and so forth? :) You can install it in a virtualenv by cloning the repo, checking out the branch context-and-user-scans, and then installing it in a virtualenv with pip install -e ..

You can check out each of the context commands with zap-cli context --help, but the basic idea is that you can now import a saved context file, and then run the spider or scanner by providing the context name and user name, i.e.

zap-cli context import /path/to/context.context
zap-cli -v open-url "http://localhost/"
zap-cli -v spider --context-name DevTest --user-name GrunnyTest "http://localhost"
zap-cli -v active-scan -c DevTest -u GrunnyTest "http://localhost"
zap-cli -v quick-scan --spider -c DevTest -u GrunnyTest "http://localhost"

The ZAP API in general requires the context ID and user ID (both integers), but I figured using the plain text names was easier for users, so zap-cli gets the IDs based on the names when you provide them.

Let me know what you think. :)

[isaguimiot]

Oh that's perfect, I have some time this afternoon to test it, I come back to you when it's done :)

[isaguimiot]

Hi again @Grunny , I tried your new version, and I have an error when I launch the zap-cli context import whatever :

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 386, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 382, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.5/http/client.py", line 1198, in getresponse
    response.begin()
  File "/usr/local/lib/python3.5/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.5/http/client.py", line 266, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 649, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.5/site-packages/requests/packages/urllib3/util/retry.py", line 376, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
requests.packages.urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=8090): Max retries exceeded with url: http://zap/JSON/context/action/importContext/?apikey=&contextFile=xx (Caused by ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response',)))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/zap-cli", line 11, in <module>
    load_entry_point('zapcli', 'console_scripts', 'zap-cli')()
  File "/usr/local/lib/python3.5/site-packages/click/core.py", line 664, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/click/core.py", line 644, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.5/site-packages/click/core.py", line 991, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.5/site-packages/click/core.py", line 991, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.5/site-packages/click/core.py", line 837, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.5/site-packages/click/core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/click/decorators.py", line 26, in new_func
    return ctx.invoke(f, ctx.obj, *args[1:], **kwargs)
  File "/usr/local/lib/python3.5/site-packages/click/core.py", line 464, in invoke
    return callback(*args, **kwargs)
  File "/opt/zap-cli/zapcli/commands/context.py", line 104, in context_import
    zap_helper.import_context(file_path)
  File "/opt/zap-cli/zapcli/zap_helper.py", line 454, in import_context
    result = self.zap.context.import_context(file_path, apikey=self.api_key)
  File "/usr/local/lib/python3.5/site-packages/zapv2/context.py", line 108, in import_context
    return six.next(six.itervalues(self.zap._request(self.zap.base + 'context/action/importContext/', {'contextFile': contextfile, 'apikey': apikey})))
  File "/usr/local/lib/python3.5/site-packages/zapv2/__init__.py", line 157, in _request
    data = self._request_api(url, get)
  File "/usr/local/lib/python3.5/site-packages/zapv2/__init__.py", line 147, in _request_api
    return self.session.get(url, params=query, proxies=self.__proxies, verify=False)
  File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 501, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/requests/adapters.py", line 485, in send
    raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPConnectionPool(host='127.0.0.1', port=8090): Max retries exceeded with url: http://zap/JSON/context/action/importContext/?apikey=&contextFile=xx (Caused by ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response',)))

There must be a way to set the self.zap.base , because apparently it points to http://zap/JSON , which is incorrect... Could you help me with this ?

[Grunny]

http://zap/JSON is correct as long as the proxy is working properly (your proxy is set to 127.0.0.1:8090 judging by the logs), as the zap URL works when proxied through ZAP.

The error you're getting is: requests.exceptions.ProxyError: HTTPConnectionPool(host='127.0.0.1', port=8090): Max retries exceeded with url: http://zap/JSON/context/action/importContext/?apikey=&contextFile=xx (Caused by ProxyError('Cannot connect to proxy.', RemoteDisconnected('Remote end closed connection without response',))), so it's struggling to connect to ZAP on 127.0.0.1:8090. Did you start ZAP with zap-cli or separately?

Is ZAP definitely running on port 8090? You can change the port to use with zap-cli from the default 8090 using the ZAP_PORT environment variable (or passing the -p option with all commands). You can find which port ZAP on your system listens to by opening the ZAP GUI and going to Tools > Options > Local Proxy, and their should show the port.

I also noticed your API key is missing in the logs, which you may have just redacted when pasting. If it is missing, I'd make sure it's set in the ZAP_API_KEY environment variable so zap-cli uses it when interacting with the ZAP API.

Let me know if any of that helps. :)

[isaguimiot]

Hi @Grunny , thanks for your answer, the issue was the API key, I didn't know it was mandatory.

Now I can launch everything without any error, but it seems that the scan still doesn't parse the pages that need authentication to be accessed...

I'm sorry I'm not a ZAP expert, just discovering it, and I noticed that in the ZAP full app, I have to manually browse to the login page and effectively log into it with my credentials, if I want the whole website to be attacked by ZAP, even if I set up everything in the context, authentication thru a form, user, login URL with POST args, etc etc. Perhaps I have to do something before I run the zap-cli scan ? Perhaps the context file isn't enough ? I'm sorry for those dummy questions, and I hope you can help me...

Thanks ! :)

[Grunny]

Does the form-based authentication you have configured work for your site?

A simple way to test if it works is to:

1. Open ZAP and have your browser proxied through it (with a new session in ZAP, and not being logged in to the site in your browser)
2. Import your context with the saved authentication method
3. Ensure you have the user credentials for at least one account configured in the context
4. Enable "Forced User Mode" (there's an option in the top menu)
5. Refresh a page on your site and see if you are now logged in

If it's configured properly, then ZAP should log you in to the configured user credentials using the form-based authentication you set up. If it doesn't, something is wrong with the authentication setup.

If it's not working, I'd suggest reading over https://github.com/zaproxy/zap-core-help/wiki/HelpStartConceptsAuthentication and also check if form-based authentication is sufficient for the site you're testing. For example, if the login form has CSRF protection it won't work, and you may need to write a script for script-based authentication.

I hope that helps!

[isaguimiot]

Hi @Grunny , yes I confirm that everything is fine in ZAP, I tested it again this morning : start with a new session, load the context, force the user, and then browse to the page I want to test : I'm logged in !

Is there a way to enable the "forced user mode" in zap-cli ? is it supposed to be enabled by default ?

Here is an extract of my context :

<authentication>
<type>2</type>
<form>
<loginurl>https://mycompany.com/myapp/api/authentication/login</loginurl>
<loginbody>login={%username%}&amp;password={%password%}</loginbody>
</form>
</authentication>
<users>
<user>1;true;Y2l1c2Vy;2;Y2l1c2Vy~encryptedpass~</user>
<user>2;true;YWRtaW4=;2;YWRtaW4=~encryptedpass~</user>
</users>
<forceduser>1</forceduser>
<session>
<type>0</type>
</session>
<authorization>
<type>0</type>
<basic>
<header/>
<body/>
<logic>AND</logic>
<code>-1</code>
</basic>
</authorization>

Any clue ?

Thanks ! :)

[Grunny]

There isn't currently an option to enable forced user mode in zap-cli, but I can add one easily enough I think. But, if that works, then the user option should work if zap-cli was installed from the https://github.com/Grunny/zap-cli/compare/context-and-user-scans as long as you provide the correct context name and the correct username, i.e.

zap-cli -v spider --context-name DevTest --user-name GrunnyTest "http://localhost"

Does that work? Or is that not logged in when it runs? It works for me locally at least, so I'd need to figure out what might be breaking when run with your config.

[isaguimiot]

Yes I ran the spider and scan commands with the context name and user name, and the verbose mode doesn't give a lot of details, but it runs much faster than thru the ZAP application, and when I generate the report, it's shorter and doesn't seem to refer any issue from any authenticated page, so I guess that that authentication doesn't work...

If you could easily add the "force user mode", perhaps it would fix this issue ?

Thanks for your quick responses :)

[lokori]

It would be awesome to get this feature soon merged into main branch. I abandoned writing my own python wrapper for ZAP because zap-cli appeared to have everything I was interested in. Apart from this.

[lokori]

I don't know if I bumped into what @isaguimiot mentioned, but I tried running the scan with imported context and compared to "active scan" run from the GUI using the same context it is indeed much faster. The reason is that it generates significantly smaller number of requests and doesn't detect as much problems.

I have not yet figured out the reason for this.

[lokori]

How foolish of me.. I copied the command from this discussion thread for active scan and it lacks the option -r and therefore does not do a recursive scan. Given that option it seems that everything works for me like this:

zap-cli context import /zap/flask-vuln.context 
zap-cli open-url "http://localhost:5000"
zap-cli -v spider --context-name flask-vuln --user-name foo@bar.tld "http://localhost:5000"
zap-cli -v active-scan -r --context-name flask-vuln -u foo@bar.tld "http://localhost:5000"

My test was based on a small target application so I might have missed something.

[isaguimiot]

@lokori you're probably right, I think I forgot the -r, because with the new version of zap-cli, I had more vulnerabilities than before, but not as many as thru owsap zap, this might be an explanation...

I will test it again this week and I'll tell you if it works... :)

[Grunny]

Sorry for the slow response here!

Ah, yeah, I really need to make recursive scanning the default and just use a flag to turn recursive scanning off. I can do that as part of releasing this change.

@isaguimiot Let me know if that makes it work for you, and I'll release the changes. :)

[isaguimiot]

Hi @Grunny
I made some tests yesterday, and by using the "--recursive" option, everything works fine ! :)
Can you make a release with your latest commits ?
Thanks for your help ! :)

[Grunny]

OK, I've released the changes for this in version 0.9.0 and it's available on PyPI now. I also added a brief explanation to the README. Let me know if there are any issues. :)

[pexus]

I am trying to scan using zap-cli. I imported the context. The context clearly shows the URL pattern is included in the context : e.g.

<incregexes>https://NNN.19.67.106:8080.*</incregexes>

However when I try to scan with the following command, it gives ERROR saying the URL is not in the context.

zap-cli spider --context-name contextName --user-name user https://NNN.19.106:8080

[INFO] Running spider...
[ERROR] Error running spider: "The provided url is not in the required context"