Order of operations for Capture vs Asserts

[ad8lmondy]

TL;DR: what is the order of operations for a failure to be picked up by the status code, asserts, and captures, and is it controllable?

Part of my hurl file looks like this:

POST {{auth0_url}}/usernamepassword/login
referer: {{landing_url}}
{
    "client_id":"{{client_id}}",
    "redirect_uri":"{{redirect_url}}",
    "tenant":"{{tenant}}",
    "state":"{{state}}",
    "connection":"DB",
    "username":"{{username}}",
    "password":"{{password}}",
    "_csrf":"{{csrf}}"
}
HTTP 200
[Captures]
action: regex "action=\"(.*)\""
wa_val: regex "name=\"wa\" value=\"(.*)\">"
wresult_val: regex "(?s)\"wresult\"[^a-z]*value=\"(.*?)\">"
wctx_val: regex "name=\"wctx\" value=\"(.*)\">" htmlUnescape

This works great if the HTTP status code is 200, but if the username and password are not correct, the status code is 401, and hurl says:

* Response: (received 115 bytes in 200 ms)
*
< HTTP/2 401
< content-type: application/json; charset=utf-8
< content-length: 115
...shortened for brevity...
< server: cloudflare
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
<
* Response body:
* {"name":"ValidationError","code":"invalid_user_password","description":"Wrong email or password.","statusCode":400}
error: No query result
  --> permission_tests/test.hurl:52:9
   |
52 | action: regex "action=\"(.*)\""
   |         ^^^^^^^^^^^^^^^^^^^^^^^ The query didn't return any result
   |

This makes sense, in that the Capture has not been able to resolve the regex asked for - but I was expecting the status code to trigger the error, rather than the Capture block, to get something like:

error: Assert status code
  --> permission_tests/test.hurl:50:6
   |
50 | HTTP 200
   |      ^^^ actual value is <401>
   |

In doing some testing, just to explore the issue, I changed the hurl file to this:

HTTP 401
[Asserts]
status == 200
[Captures]
action: regex "action=\"(.*)\""
wa_val: regex "name=\"wa\" value=\"(.*)\">"
wresult_val: regex "(?s)\"wresult\"[^a-z]*value=\"(.*?)\">"
wctx_val: regex "name=\"wctx\" value=\"(.*)\">" htmlUnescape

To translate: expect the 401, but assert that the status is 200 - but this also fails on the same No query result issue.

My question is then: is this the intended behaviour, and if so, is there some way to control it? The reason I ask is because I had assumed that my tests were failing because Auth0 was returning something different, but it was only when I ran hurl with --very-verbose, and I could see the returned data did I realise that the issue was authentication. I was expecting the status code to tip me off for that :)

A potentially annoying caveat: I'm running hurl built from master, because it has the htmlEscape filter I needed.

Thanks, love your work!

[jcamiel]

Yes you're exactly right: we compute first the captures, then the asserts because the asserts can use captured variables.
The status code being an assert we compute it after the captures.

That said, you're also right the status code is very important and its value can really impact the response structure. For instance, if you're asserting a 200 OK and you received a 302 Redirect, you can be sure that all your asserts will fail and add a lot of noise to the error diagnostic.

So, I agree with you, we'll improve it:

• first test version HTTP / status code: if it is not OK, displayed error and stop the test
• then compute all captures
• then checks the remaining asserts

(BTW, as you seem to really use Hurl, I'm curious: what's is your feedbacks on it ? What do you like / dislike (can be anything file format, IDE support, documentation etc...))

[ad8lmondy]

Hey, thanks for making the change - it works as I expect!

In terms of feedback:

• Very happy with the activity and engagement on GitHub ❤️
• Docs are quite good, but as minor nitpick: they might benefit from making sure each feature has more than one example to demonstrate its usage.
• Every so often, I want to use another tool on my system, like base64 or something like that, as filters. I think it's because I'm used to using curl in a shell script. It might be cool to have a shell filter, which allows someone to do something like wctx_val: regex "name=\"wctx\" value=\"(.*)\">" shell "sed 's/remove_me//g'". However, in saying that - I can see that this might be problematic, both for implementation (e.g., windows?) and perhaps for the intended use of hurl - I can't imagine it would be good for someone to be shoving tons of bash in a filter, when maybe they should be using a different tool in the first place.

[jcamiel]

Thanks for the feedbacks, noted for the docs! For the shell script, effectively many users ask for the same feature. For the moment, we'll not address it because:

• platform issues with Windows as you guess
• we want to enable features like launching test in parallel (see add the --parallel option to run *.hurl files in parallel instead of sequentially #87 add the --concurrency option to launch multiple runs of *.hurl files instead of one #88) and doesn't want to be block by this kind on features