/admin returns 422 on login

[androidacy-user]

Subject of the issue

Admin dashboard returns 422

Deployment environment

• vaultwarden version: 1.24.0

• Install method: docker-compose

• Clients used: browser

• Reverse proxy and version: Nginx 1.21.0, behind cloudflare tunnel (previously known as argo tunnel) which is behind cloudflare itself

• MySQL/MariaDB or PostgreSQL version: n/a

• Other relevant details: IP_HEADER variable is set accordingly since we are running behind cloudflare

Steps to reproduce

1. Go to /admin
2. Input correct password
3. See:

422: Unprocessable Entity
The request was well-formed but was unable to be followed due to semantic errors.

Rocket

Expected behaviour

Login and shows the dashboard

Actual behaviour

The 422 error is returned.

Troubleshooting data

complete `docker-compose logs`

Attaching to bitwardenrs
�[36mbitwardenrs    |�[0m /--------------------------------------------------------------------\
�[36mbitwardenrs    |�[0m |                        Starting Vaultwarden                        |
�[36mbitwardenrs    |�[0m |                           Version 1.24.0                           |
�[36mbitwardenrs    |�[0m |--------------------------------------------------------------------|
�[36mbitwardenrs    |�[0m | This is an *unofficial* Bitwarden implementation, DO NOT use the   |
�[36mbitwardenrs    |�[0m | official channels to report bugs/features, regardless of client.   |
�[36mbitwardenrs    |�[0m | Send usage/configuration questions or feature requests to:         |
�[36mbitwardenrs    |�[0m |   https://vaultwarden.discourse.group/                             |
�[36mbitwardenrs    |�[0m | Report suspected bugs/issues in the software itself at:            |
�[36mbitwardenrs    |�[0m |   https://github.com/dani-garcia/vaultwarden/issues/new            |
�[36mbitwardenrs    |�[0m \--------------------------------------------------------------------/
�[36mbitwardenrs    |�[0m 
�[36mbitwardenrs    |�[0m [INFO] No .env file found.
�[36mbitwardenrs    |�[0m 
�[36mbitwardenrs    |�[0m [WARNING] The following environment variables are being overriden by the config file,
�[36mbitwardenrs    |�[0m [WARNING] please use the admin panel to make changes to them:
�[36mbitwardenrs    |�[0m [WARNING] SIGNUPS_ALLOWED, ADMIN_TOKEN, IP_HEADER, SMTP_HOST, SMTP_SSL, SMTP_PORT, SMTP_FROM, SMTP_USERNAME, SMTP_PASSWORD
�[36mbitwardenrs    |�[0m 
�[36mbitwardenrs    |�[0m Listening for new connections on 0.0.0.0:3012.
�[36mbitwardenrs    |�[0m Rocket has launched from http://0.0.0.0:80
�[36mbitwardenrs    |�[0m Accepted a new tcp connection from 172.18.0.1:39474.
�[36mbitwardenrs    |�[0m Accepted a new tcp connection from 172.18.0.1:39476.
�[36mbitwardenrs    |�[0m GET /api/accounts/revision-date
�[36mbitwardenrs    |�[0m GET /api/accounts/revision-date (revision_date) => 200 OK
�[36mbitwardenrs    |�[0m GET /admin
�[36mbitwardenrs    |�[0m GET /admin [2] (admin_login) => 200 OK
�[36mbitwardenrs    |�[0m POST /admin
�[36mbitwardenrs    |�[0m POST /admin (post_admin_login) => 303 See Other
�[36mbitwardenrs    |�[0m POST /admin
�[36mbitwardenrs    |�[0m POST /admin (post_admin_login) => 422 Unprocessable Entity
�[36mbitwardenrs    |�[0m Accepted a new tcp connection from 172.18.0.1:39480.
�[36mbitwardenrs    |�[0m Accepted a new tcp connection from 172.18.0.1:39482.
�[36mbitwardenrs    |�[0m GET /api/accounts/revision-date
�[36mbitwardenrs    |�[0m GET /api/accounts/revision-date (revision_date) => 200 OK

[BlackDex]

I can't see that paste , i need to login before i can see it it looks like.

Have you tried to see what happens if you take cloudflare out of the equation?

[androidacy-user]

I can't see that paste , i need to login before i can see it it looks like.

Have you tried to see what happens if you take cloudflare out of the equation?

You can login with any account, it's just that pastebinit makes them private

And no, taking cloudflare out of the equation is not an option I will even consider. Seeing as our webserver isn't actually exposed to the internet whatsoever with our current setup, it'd require quite a bit of configuration reworking.

[BlackDex]

That is not what i asked. I asked what happens if you bypass it. I guess that you as the admin would be able to do that.

The thing is, there is something send to Vaultwarden which is not understood. And the thing in between is cloudflare, and that could be an issue.

[androidacy-user]

Let me rephrase again
We can't

Our setup would have to be almost ENTIRELY reworked to allow us to take cloudflare out of the equation. Even for an admin.

As it stands now, the firewall doesn't allow non cloudflare connections, nginx is only listening on localhost, and all web requests have to come through cloudflare tunnel.

This would also require us to issue a valid SSL certificate on the servers, as the only certificate installed is the cloudflare origin certificate which is only trusted by cloudflare, and require us to even temporarily expose our servers to the internet.

[BlackDex]

Then we would need to see the actual http request coming in into Vaultwarden. Either via a tcpdump/wireshark or by enabling trace logging.

Also double check then nginx config compared with the examples in the wiki.

[androidacy-user]

There is no magic bypass button in our case.

It would be really helpful if logs told us why the request isn't processable.

[androidacy-user]

Then we would need to see the actual http request coming in into Vaultwarden. Either via a tcpdump/wireshark or by enabling trace logging.

Also double check then nginx config compared with the examples in the wiki.

Is trace logging referring to setting EXTENDED_LOGGING=true? Because that didn't seem to output anything extra related to the admin requests.

[BlackDex]

There is no magic bypass button in our case.

It would be really helpful if logs told us why the request isn't processable.

That is done by the webserver library (Rocket), which you can trigger to be a bit more verbose by setting logging to trace.

[BlackDex]

LOG_LEVEL=trace

[BlackDex]

Let me rephrase again We can't

Our setup would have to be almost ENTIRELY reworked to allow us to take cloudflare out of the equation. Even for an admin.

As it stands now, the firewall doesn't allow non cloudflare connections, nginx is only listening on localhost, and all web requests have to come through cloudflare tunnel.

This would also require us to issue a valid SSL certificate on the servers, as the only certificate installed is the cloudflare origin certificate which is only trusted by cloudflare, and require us to even temporarily expose our servers to the internet.

You could use an ssh tunnel maybe or ssh socks proxy to connect to it. There probably is a way to connect to it without even nginx in front of it.

[androidacy-user]

The only one on the pastebinit list that works and isn't private by default

[androidacy-user]

You could use an ssh tunnel maybe or ssh socks proxy to connect to it. There probably is a way to connect to it without even nginx in front of it.

Ssh also goes through cloudflare, through their zero trust solution.

Nonetheless logs with the logging level set to trace are above, albeit with a few extra ASCII color codes in the mix.

[BlackDex]

Please check your nginx config. It looks like it is sending a header to do an upgrade which isn't right.

Also, try if you can filter the cf-* headers in nginx so that they do not pas on to Vaultwarden.

[BlackDex]

Then it still is ssh. Not sure if it then supports socks5 proxy or port tunneling. But if that is the case you could use that to bypass cloudflare.

[androidacy-user]

Please check your nginx config. It looks like it is sending a header to do an upgrade which isn't right.

Also, try if you can filter the cf-* headers in nginx so that they do not pas on to Vaultwarden.

The only Upgrade headers i see (CTRL-F'ing the log) are browser headers. NGINX only attempts the connection over http1.1 and http, and up until it hits cloudflare http2/3 doesn't even come into play.

[BlackDex]

http1.1 supports upgrade.
But again, please try to filter the cloudflare headers and double check the nginx config with the examples in the wiki.

[androidacy-user]

Okay, the only place upgrade headers are set is /notifications/hub

And the config is originally copy-pasted from the wiki. The only modifications are to work with cloudflare tunnel and cloudflare's origin cert, plus CSP, Access-Control-*, referrer-policy, and x-content-type-options.

[androidacy-user]

Just for giggles, I recopied the config since it was a while ago when we initially set this up, only changing the listen, ssl, and adding one location directive for internal use, and still same result.

We're shooting in the dark here, because the logs even at higher levels, don't say what isn't processable. Just that something isn't.

[BlackDex]

Well, a 422 is mostly because something is malformed. That could be headers, or the request body. For example, no correct new lines after headers or weird characters in the headers or body.

But, most of the time it is malformed body, so incorrect json for example. It could even be charset issues, or something which should be an integer and suddenly is a string, or the other way around. It could even be a simple thing as a difference between lower or uppercase characters if that is changed somewhere in between.

And that is why i asked to try and remove cloudflare from the equations to rule that out, since cloudflare is known to do some transformations sometimes with requests.

I can take a closer look in the morning if i can find anything more specific in the posted logs.

[BlackDex]

@androidacybot , I'm looking at it right now.
But may i suggest that you change your admin password, since that is exposed now.

[BlackDex]

Well, it looks like the client you are using, or CloudFlare is doing something to let your browser redirect using a POST instead of GET after a possible correct login.

I see the POST to login, i see that Vaultwarden responds with a Location header using 303 See Other as it's code, which should trigger the client to always use GET. But your client doesn't, it keeps the POST.

And that is what is causing the issues, since Vaultwarden expect that there is a token provided for a POST and causes the 422 error to happen.

This is not something wrong on the Vaultwarden side, since Vaultwarden (and the Rocket library) is very strict in what they receive and verify the contents, if it doesn't matched what is programmed it will disallow the request.

[androidacy-user]

@BlackDex which I suppose is why it worked PERFECTLY FINE before I suppose.

Sorry to bother you. Obviously the only solution you are willing to pursue is 'taking cloudflare out of the equation".

[BlackDex]

@androidacybot, I'm sorry you feel that way.

I did an investigation of the trace logs you send, i put time and effort in it, and i gave you my conclusion that it looks like your browser, or something in between is not responding correctly to the 303 See Other Location header Vaultwarden sends. It keeps using a POST, which is wrong, it should use GET, but something on your side is not using GET, but POST which is wrong.

I do not say that you need to take it out by default, but i say that you should see what happens. That is something totally different.
I even tried to see if i could use CloudFlare, but i do not want to change my NS records, so that was not an option for me.

For me it feels like you think this could only be an error on the Vaultwarden side, which could have been an option, but looking at the logs, it just isn't. I can't change the outcome of that, it just is what the logs show.

[androidacy-user]

I hit this exact same issue with DataUnlocker. They solved it on their end, albeit I don't know how.

I'm sorry but it really probably is on Vaultwarden's end. Neither cloudflare, nginx, nor chrome has any reason to not follow a 303 - and the 303 never makes it to chrome and it doesn't look like it even makes it to nginx. Chrome POSTS to /admin which immediately returns a 422.

[androidacy-user]

You can't just blame things on cloudflare - I don't think they'd break a spec like you're suggesting they are.

[BlackDex]

Well¸ then i suggest to read the logs a bit better your self and make your own conclusions!

I will give you a step-by-step instructions on what to do:
In the pastbin you send do the following.

1. Look for the text Location Headers.append_raw( "Location", [104, 116
   There you see at the line above that POST /admin (post_admin_login) => 303 See Other
2. Now a bit back in the logs you will see a line which states try_parse([80,
3. Copy all those decimal numbers which are between []
4. Now you need to convert all those decimals to ASCII which will show you your POST and your login token.
   You can use a site like https://www.rapidtables.com/convert/number/ascii-hex-bin-dec-converter.html or something.
5. Now, scroll a bit below, so after the Location header, there you will see an other try_parse, do the exact same as above.
6. There you will see an other POST without any body attached to it, which is wrong, and not allowed.

If you say that that specific Location redirect is not reaching your browser, that i can only conclude that something in between is doing something for you, and that something unfortunately still is either nginx or CloudFlare.

If you have a link or something to a bug report of that DataUnlocker tool you are talking about that i can take a look at, I'm willing to do that and see if I can figure out what they did, but without it, i really can't help any further for a couple of reasons.

1. I do not have CloudFlare, so i can't test it for you.
2. During my investigation on your logs it looks like it is use POST and not GET for the redirected request which is wrong.
3. It works for all other users, we have not received any other reports regarding this issue at all, which would be strange if this was a major issue on the Vaultwarden side.

Don't get me wrong, I really want to help and fix the issue. But i have nothing to go on besides the logs, and the logs tell me it is not an issue on the Vaultwarden side which i can fix.

[BlackDex]

You can't just blame things on cloudflare - I don't think they'd break a spec like you're suggesting they are.

And you can't just keep saying that it really isn't the issue, while you haven't tried taking it out of the loop. That is just bad investigation on your side and making assumptions that it is anything but cloudflare or nginx.

In my experience you always need to checkout each step in between, and not just blame the end product!

[androidacy-user]

@BlackDex by any chance does Vaultwarden try to redirect http to https on it's own? Asking because the last time this was an issue the service behind nginx was trying to redirect http to https because up until nginx the request is done over https, but nginx to the backend service is http (it's a local connection at that point so it doesn't hurt anything)

[BlackDex]

That should produce the exact same redirect, but this time it should show you a message in the browser:

I think it is strange to, and maybe there are some headers we can set to trick cloudflare (or any other CDN/Proxy) to do something special. But i can't find anything regarding something like that. And i only see an incorrect response from the server to the browser if i try it, and an incorrect request at the Vaultwarden side. I have no clue what is going on in between there which seems to do the redirect for you, but in a wrong way.

[BlackDex]

Also, i seem to miss the cookies Vaultwarden sets. You should see something like set-cookie: VW_ADMIN=eyJ0eXAiO in the response headers. Which i also miss at your site.

And if there was a wrong login it should set a cookie like this:
set-cookie: _flash=5%3AerrorInvalid%20admin%20token,%20please%20try%20again.; SameSite=Strict; Path=/; Max-Age=300

[androidacy-user]

Here's a relevant entry from nginx debugging. It looks like the 303 doesn't set a correct VW_ADMIN cookie but something about too many requests.

2022/04/30 14:36:05 [debug] 1045202#1045202: *95612 http proxy status 303 "303 See Other"
2022/04/30 14:36:05 [debug] 1045202#1045202: *95612 http proxy header: "Location: https://bw.androidacy.com/admin"
2022/04/30 14:36:05 [debug] 1045202#1045202: *95612 http proxy header: "Set-Cookie: _flash=5%3AerrorToo%20many%20requests,%20try%20again%20l
ater.; Path=/; Max-Age=300"
2022/04/30 14:36:05 [debug] 1045202#1045202: *95612 http proxy header: "Server: Rocket"
2022/04/30 14:36:05 [debug] 1045202#1045202: *95612 http proxy header: "Permissions-Policy: accelerometer=(), ambient-light-sensor=(), autop
lay=(), camera=(), encrypted-media=(), fullscreen=(), geolocation=(), gyroscope=(), magnetometer=(), microphone=(), midi=(), payment=(), pic
ture-in-picture=(), sync-xhr=(self "https://haveibeenpwned.com" "https://2fa.directory"), usb=(), vr=()"
2022/04/30 14:36:05 [debug] 1045202#1045202: *95612 http proxy header: "Referrer-Policy: same-origin"
2022/04/30 14:36:05 [debug] 1045202#1045202: *95612 http proxy header: "X-Frame-Options: SAMEORIGIN"
2022/04/30 14:36:05 [debug] 1045202#1045202: *95612 http proxy header: "X-Content-Type-Options: nosniff"
2022/04/30 14:36:05 [debug] 1045202#1045202: *95612 http proxy header: "X-XSS-Protection: 1; mode=block"
2022/04/30 14:36:05 [debug] 1045202#1045202: *95612 http proxy header: "Content-Security-Policy: frame-ancestors 'self' chrome-extension://n
ngceckbapebfimnlniiiahkandclblb chrome-extension://jbkfoedolllekgbhcbcoahefnbanhhlh moz-extension://* ;"
2022/04/30 14:36:05 [debug] 1045202#1045202: *95612 http proxy header: "Cache-Control: no-cache, no-store, max-age=0"
2022/04/30 14:36:05 [debug] 1045202#1045202: *95612 http proxy header: "Access-Control-Allow-Origin: https://bw.androidacy.com"
2022/04/30 14:36:05 [debug] 1045202#1045202: *95612 http proxy header: "Content-Length: 0"
2022/04/30 14:36:05 [debug] 1045202#1045202: *95612 http proxy header: "Date: Sat, 30 Apr 2022 14:36:05 GMT"
2022/04/30 14:36:05 [debug] 1045202#1045202: *95612 http proxy header done

Further requests show that cloudflare is following the redirect itself....which is highly odd. It returns the expected redirect just fine for wordpress and our api - wordpress using a 302 and we use a 302 or 303 depending on the request.

[BlackDex]

That now probably is because you attempted this a lot of times and that triggers the brute-force login blocker build-in into Vaultwarden.
You can restart Vaultwarden to clear that (Since it is only in memory).
But it might be wise to also configure this to higher values:
https://github.com/dani-garcia/vaultwarden/blob/main/.env.template#L290-L293=

[BlackDex]

A 302 should only be sent as a response to a GET request, never as a response to a POST or PUT request.
That is where 303 is for which should change a POST or PUT to a GET and should act as a confirmation (or failure) of the request is is responding to.
A 301 is totally out of the question here, since it's not a permanently relocated request.

[androidacy-user]

Okay, I confirmed a suspicion but I'm no closer to solving.

It has to do with an interesting interaction with Cloudflare Workers.

Everything BUT 303 is being handled correctly - we bypass requests if the method is not GET and response type is not text/html. For some reason, the 303 is being followed by the worker itself - and never getting sent to the client.

The interesting part is the worker is not correctly transitioning from POST to GET.

One workaround on VW's side would be to use a 302 - that seems to be handled correctly, and WordPress uses a 302 in much the same way (login page -> POST to login page -> 302 to final destination, with appropriate cookies now in place). Otherwise i'll try to get in touch with cloudflare to see why this is happening, and update you if we find out anything.

[BlackDex]

Well, a 302 on a POST request would indicate it needs to send the request again as a POST to a different location. But since it isn't a different location but the exact same one, that will fail and would probably cause a redirect loop, and in the end still fails.

And, we also do not need to to send the login token again, since it was already validated, and probably found valid (or invalid for that matter).

A 302 would be ok maybe if the endpoint would be different, but that isn't the case, and we also can not change this on our side, for one, since it just is incorrect, and i know there are people who have scripts which use these endpoints to login and automate stuff, so that will probably break those setups.

Also, setting a response type on a redirect response would be strange, but that isn't set at all currently. Not sure if i can even configure that. But if not setting a content-type means CloudFlare assumes text/html that could cause the issue for you i think right?

[BlackDex]

And, i know other users also use cloudflare for Vaultwarden and they do not seem to have any issues at all. Since this is the first issue reported regarding something like this.

[BlackDex]

@androidacy-user I have created a PR which should probably solve your issue reported here.
Instead of using FlashMessages and Redirects the admin page will (when the PR #2729 is merged) return the correct status codes, and no more redirects, instead during a logout. So, 401 when unauthorized, 429 when too many attempts are done, and just a normal 200 and the correct HTML when allowed to login.

I think this should fix the issues you had regarding this.