organization vault and members pages hang; 500 internal error

[ebdavison]

Vaultwarden Support String

Your environment (Generated via diagnostics page)

• Vaultwarden version: v1.32.6
• Web-vault version: v2024.6.2c
• OS/Arch: linux/x86_64
• Running within a container: true (Base: Debian)
• Database type: SQLite
• Database version: 3.46.0
• Environment settings overridden!: false
• Uses a reverse proxy: true
• IP Header check: true (X-Real-IP)
• Internet access: true
• Internet access via a proxy: false
• DNS Check: true
• Browser/Server Time Check: true
• Server/NTP Time Check: true
• Domain Configuration Check: false
• HTTPS Check: true
• Websocket Check: true
• HTTP Response Checks: false

Config & Details (Generated via diagnostics page)

Show Config & Details

Failed HTTP Checks:

2FA Connector calls:
Header: 'x-frame-options' is present while it should not

Config:

{
  "_duo_akey": null,
  "_enable_duo": true,
  "_enable_email_2fa": true,
  "_enable_smtp": true,
  "_enable_yubico": true,
  "_icon_service_csp": "",
  "_icon_service_url": "",
  "_ip_header_enabled": true,
  "_max_note_size": 10000,
  "_smtp_img_src": "cid:",
  "admin_ratelimit_max_burst": 3,
  "admin_ratelimit_seconds": 300,
  "admin_session_lifetime": 20,
  "admin_token": null,
  "allowed_iframe_ancestors": "",
  "attachments_folder": "data/attachments",
  "auth_request_purge_schedule": "30 * * * * *",
  "authenticator_disable_time_drift": false,
  "data_folder": "data",
  "database_conn_init": "",
  "database_max_conns": 10,
  "database_timeout": 30,
  "database_url": "***************",
  "db_connection_retries": 15,
  "disable_2fa_remember": false,
  "disable_admin_token": true,
  "disable_icon_download": false,
  "domain": "*****://**************************************",
  "domain_origin": "*****://**************************************",
  "domain_path": "",
  "domain_set": true,
  "duo_context_purge_schedule": "30 * * * * *",
  "duo_host": null,
  "duo_ikey": null,
  "duo_skey": null,
  "duo_use_iframe": false,
  "email_2fa_auto_fallback": false,
  "email_2fa_enforce_on_verified_invite": false,
  "email_attempts_limit": 3,
  "email_change_allowed": true,
  "email_expiration_time": 600,
  "email_token_size": 6,
  "emergency_access_allowed": true,
  "emergency_notification_reminder_schedule": "0 3 * * * *",
  "emergency_request_timeout_schedule": "0 7 * * * *",
  "enable_db_wal": true,
  "enable_websocket": true,
  "enforce_single_org_with_reset_pw_policy": false,
  "event_cleanup_schedule": "0 10 0 * * *",
  "events_days_retain": null,
  "experimental_client_feature_flags": "fido2-vault-credentials",
  "extended_logging": true,
  "helo_name": null,
  "hibp_api_key": null,
  "http_request_block_non_global_ips": true,
  "http_request_block_regex": null,
  "icon_blacklist_non_global_ips": true,
  "icon_blacklist_regex": null,
  "icon_cache_folder": "data/icon_cache",
  "icon_cache_negttl": 259200,
  "icon_cache_ttl": 2592000,
  "icon_download_timeout": 10,
  "icon_redirect_code": 302,
  "icon_service": "internal",
  "incomplete_2fa_schedule": "30 * * * * *",
  "incomplete_2fa_time_limit": 3,
  "increase_note_size_limit": false,
  "invitation_expiration_hours": 120,
  "invitation_org_name": "Vaultwarden",
  "invitations_allowed": true,
  "ip_header": "X-Real-IP",
  "job_poll_interval_ms": 30000,
  "log_file": "/data/bitwarden.log",
  "log_level": "warn",
  "log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f",
  "login_ratelimit_max_burst": 10,
  "login_ratelimit_seconds": 60,
  "org_attachment_limit": null,
  "org_creation_users": "",
  "org_events_enabled": false,
  "org_groups_enabled": false,
  "password_hints_allowed": true,
  "password_iterations": 600000,
  "push_enabled": false,
  "push_identity_uri": "https://identity.bitwarden.com",
  "push_installation_id": "***",
  "push_installation_key": "***",
  "push_relay_uri": "https://push.bitwarden.com",
  "reload_templates": false,
  "require_device_email": false,
  "rsa_key_filename": "data/rsa_key",
  "send_purge_schedule": "0 5 * * * *",
  "sendmail_command": null,
  "sends_allowed": true,
  "sends_folder": "data/sends",
  "show_password_hint": false,
  "signups_allowed": true,
  "signups_domains_whitelist": "*****************************",
  "signups_verify": false,
  "signups_verify_resend_limit": 6,
  "signups_verify_resend_time": 3600,
  "smtp_accept_invalid_certs": false,
  "smtp_accept_invalid_hostnames": false,
  "smtp_auth_mechanism": null,
  "smtp_debug": false,
  "smtp_embed_images": true,
  "smtp_explicit_tls": null,
  "smtp_from": "*****************",
  "smtp_from_name": "Vaultwarden",
  "smtp_host": "********************",
  "smtp_password": "***",
  "smtp_port": 587,
  "smtp_security": "starttls",
  "smtp_ssl": null,
  "smtp_timeout": 15,
  "smtp_username": "*****************",
  "templates_folder": "data/templates",
  "tmp_folder": "data/tmp",
  "trash_auto_delete_days": null,
  "trash_purge_schedule": "0 5 0 * * *",
  "use_sendmail": false,
  "use_syslog": false,
  "user_attachment_limit": null,
  "user_send_limit": null,
  "web_vault_enabled": true,
  "web_vault_folder": "web-vault/",
  "yubico_client_id": null,
  "yubico_secret_key": null,
  "yubico_server": null
}

Vaultwarden Build Version

1.32.6

Deployment method

Official Container Image

Custom deployment method

No response

Reverse Proxy

nginx 1.22.1

Host/Server Operating System

Linux

Operating System Version

Fedora 35

Clients

Web Vault

Client Version

No response

Steps To Reproduce

1. Login
2. Go to Admin Console
3. Click on either Vault or Members
5. Internal 500 error appears in JS console

Expected Result

vault items and/or member list to show up

Actual Result

erros in console

Failed to load resource: the server responded with a status of 500 ()
https://bw.myservercom:48080/api/organizations/2b3b81e1-4e83-43d1-a2ec-527ddd661165/users?includeCollections=true

Logs

Nothing in the docker logs
Nothing in the nginx logs

Screenshots or Videos

Additional Context

No response

[BlackDex]

It shows an internal server error. Could you provide the Vaultwarden logs please.
And if you see nothing strange the set the LOG_LEVEL to debug and try again.

You should at least have some logs of Vaultwarden which shows the request is being made. Same btw for nginx, the request should be seen there.

[ebdavison]

Nothing in the vaultwarden logs. I will set the LOG_LEVEL and see what shows up.

[ebdavison]

[2024-12-19 14:22:58.580][request][INFO] GET /api/organizations/2b3b81e1-4e83-43d1-a2ec-527ddd661165/users?includeCollections=true
[2024-12-19 14:22:58.584][request][INFO] GET /api/organizations/2b3b81e1-4e83-43d1-a2ec-527ddd661165/collections
[2024-12-19 14:22:58.586][response][INFO] (get_org_collections) GET /api/organizations/<org_id>/collections => 200 OK
[2024-12-19 14:22:58.591][panic][ERROR] thread 'rocket-worker-thread' panicked at 'called `Option::unwrap()` on a `None` value': src/db/models/organization.rs:445
   0: vaultwarden::init_logging::{{closure}}
   1: std::panicking::rust_panic_with_hook
   2: std::panicking::begin_panic_handler::{{closure}}
   3: std::sys::backtrace::__rust_end_short_backtrace
   4: rust_begin_unwind
   5: core::panicking::panic_fmt
   6: core::panicking::panic
   7: core::option::unwrap_failed
   8: vaultwarden::db::models::organization::UserOrganization::to_json_user_details::{{closure}}
   9: vaultwarden::api::core::organizations::get_org_users::into_info::monomorphized_function::{{closure}}
  10: rocket::server::<impl rocket::rkt::Rocket<rocket::phase::Orbit>>::route::{{closure}}
  11: rocket::server::hyper_service_fn::{{closure}}::{{closure}}
  12: tokio::runtime::task::raw::poll
  13: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  14: tokio::runtime::scheduler::multi_thread::worker::run
  15: tokio::runtime::task::raw::poll
  16: std::sys::backtrace::__rust_begin_short_backtrace
  17: core::ops::function::FnOnce::call_once{{vtable.shim}}
  18: std::sys::pal::unix::thread::Thread::new::thread_start
  19: <unknown>
  20: __clone

[2024-12-19 14:22:58.614][rocket::server::_][ERROR] Handler get_org_users panicked.
[2024-12-19 14:22:58.615][rocket::server::_][WARN] A panic is treated as an internal server error.
[2024-12-19 14:22:58.615][rocket::server::_][WARN] No 500 catcher registered. Using Rocket default.
[2024-12-19 14:22:58.616][response][INFO] (get_org_users) GET /api/organizations/<org_id>/users?<data..> => 500 Internal Server Error

[ebdavison]

and the matching nginx logs:

/var/log/nginx/access.log:1.2.3.4 - - [19/Dec/2024:19:22:58 +0000] "GET /api/organizations/2b3b81e1-4e83-43d1-a2ec-527ddd661165/collections HTTP/2.0" 200 715 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "-" "bw.myserver.com" sn="bw.retirementprosperitygroup.com" rt=0.005 ua="127.0.0.1:8080" us="200" ut="0.005" ul="1331" cs=-
/var/log/nginx/access.log:1.2.3.4 - - [19/Dec/2024:19:22:58 +0000] "GET /api/organizations/2b3b81e1-4e83-43d1-a2ec-527ddd661165/users?includeCollections=true HTTP/2.0" 500 169 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "-" "bw.retirementprosperitygroup.com" sn="bw.myserver.com" rt=0.038 ua="[::1]:8080" us="500" ut="0.037" ul="169" cs=-

[BlackDex]

That is a very strange issue. While the code could made a bit more safe, this actually could never happen unless the database is corrupted or someone deleted a user manually with foreign keys disabled.

[BlackDex]

Since this would cause inaccurate data and seems like a serious database issue on which Vaultwarden could not do anything it self. We have Foreign Keys enabled for a reason to prevent these kind of issue.

And, this is not an actual bug in Vaultwarden per say, as we would probably still panic here or error out, I'm going to move this to discussions.

I would suggest to check the integerty of the database and try to figure out which user is still a member of the org but not in the users table anymore.

[ebdavison]

Ok, so there was a user that I deleted because they forgot their master password. There seems to be no other way to delete a user from vaultwarden if they are not part of the organization or not invited to the organization as the user is just visible. They kept getting "user already exists" but I could not seem them in the web UI. Looks like something I did here. I can check the database and fix it with a little more data.

I was not aware of this relationship between tables but since I am using sqlite3, there was no error on the delete for foreign keys as I would have expected from PostgreSQL or MySQL.

Is the scheme published somewhere with the foreign keys noted? That would help.

[BlackDex]

There is a Vaultwarden Admin Backend /admin where you can manage this.

And the user was part of an org else this error wouldn't exist actually.

You should check for any user in the organization_users table and find user_uuid's which do not exist anymore, and then delete those

[ebdavison]

I was able to find the entry in users_organization with select uuid, user_uuid, org_uuid from users_organizations where user_uuid not in (select uuid from users); and fix it. Thank you very much for the tips and assistance.

[Gerardv514]

Seeing you had to fix this in the databsase backend, what would have caused this issue in the first place?

If I’m following correctly, when deleting a user belonging to an org in the ui , it will causes the issue you experienced? If so, does something need code fix here because that seems like a basic operation?

[BlackDex]

If the Vaultwarden admin had been used this wouldn't have happened.

Also, foreign keys must have been disabled to delete the record of this user, otherwise the sql engine should have returned an error.

[Gerardv514]

So by design, a user must be deleted from the Vaultwarden admin page which is /admin?

Not from the ui?

[ebdavison]

I very much appreciate your feedback and data on this so I can resolve it.

[ebdavison]

I will check out the /admin interface. I thought that the Admin Console (bottom left of the web UI) was the admin interface. I see now that it is not and am happy to find this new admin interface!