Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up expired SSL certificates in order to prevent web timeouts #2316

Open
yeah opened this issue Oct 17, 2023 · 5 comments · May be fixed by #2410
Open

Clean up expired SSL certificates in order to prevent web timeouts #2316

yeah opened this issue Oct 17, 2023 · 5 comments · May be fixed by #2410

Comments

@yeah
Copy link
Contributor

yeah commented Oct 17, 2023

We have 53 domains and are running MIAB since 2017. By now, our /home/user-data/ssl had grown to over 1.000 old expired certificate files.

Since get_ssl_certificates() from management/ssl_certificates.py is looping over all of these old certs, reading and analyzing them at least once per domain for most web requests of the management interface, we had a situation where the web interface had become unusable due to request timeouts (like in #1966). Another result of this was that DNSSEC RRSIG updates were not working properly anymore resulting in our domains not being resolved properly anymore by many DNS servers. (This is due to /etc/cron.daily/mailinabox-dnssec using ~/tools/dns_update which in turn makes web requests.

The simple make-it-work-now solution is to clean up /home/user-data/ssl for old files.

But there should be a mechanism in MIAB that removes obsolete certificates or at least disregards them when looping in get_ssl_certificates().

@jvolkenant
Copy link
Contributor

jvolkenant commented Oct 17, 2023

To a degree, that's probably already done, but likely the time it takes to go through all the certs still exceeds the proxy timeout. Like I mentioned in #1966, extending the proxy timeout might fix this. Or as you say deleting old certs. If that isn't added in ssl_certificates.py, something like find /home/user-data/ssl/*-*.pem -maxdepth 1 -mtime +365 -delete may work

@ringe
Copy link

ringe commented Oct 26, 2023

I deleted old certs and that made the status panel appear again.

@jvolkenant
Copy link
Contributor

jvolkenant commented Oct 26, 2023

It must take longer than 60s to pull the data from all the certs. The default is proxy_read_timeout 60s; https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout

You can set the timeout to 5 minutes to test and see if that works any better, it would confirm the issue is related to nginx proxy timeout setting.

In /etc/nginx/conf.d/local.conf add proxy_read_timeout 300s; to the PRIMARY_DOMAIN's server block location /admin/ section. and restart nginx

...
server {
    server_name <PRIMARY_DOMAIN>;
    ...
    location /admin/ {
    ...
    proxy_read_timeout 300s;
    }
    ...
}

@yeah
Copy link
Contributor Author

yeah commented Oct 26, 2023

I've done this and can confirm that increasing the nginx timeout will resolve the issue. It's not a real fix though. 2 years down the line, or with a couple more domains/certs, the timeout will eventually be have to increased to 10 minutes, etc.

I've debugged management/ssl_certificates.py and have pointed to where the core issue is. It's an easy fix.

IMHO, increasing the nginx timeout is just a bandaid, or more water in a leaking bucket so to speak :-)

@yeah
Copy link
Contributor Author

yeah commented Jul 22, 2024

Reminder to ideally revert de5a060 / #2407 as this issue is closed.

yeah added a commit to yeah/mailinabox that referenced this issue Jul 22, 2024
yeah added a commit to yeah/mailinabox that referenced this issue Jul 22, 2024
yeah added a commit to yeah/mailinabox that referenced this issue Sep 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants