construct a single PanDomainAuthSettingsRefresher per app instance

[andrew-nowak]

What is the value of this and can you measure success?

Fewer (hopefully no) unhealthy preview instance replacements.

What does this change?

We're seeing frequent healthcheck failures and unhealthy instances from preview since #27012
There haven't been conclusive error logs, but since that was merged there has been a substantial increase of "request timeout" and "connection refused" logs, which indicate some sort of resource exhaustion. Going through that PR I noticed that the construction of a PanDomainAuthSettingsRefresher is declared as a def (ie. function) rather than a val - which means every time the settings is referenced a new instance is constructed. Each instance on construction schedules updates to itself on a new thread pool, which means this quickly mounts up and we'll have lots of schedulers attempting to update their own instance of the refresher, clogging up memory, threads and sockets.

Instead declare as val to ensure the refresher is only constructed once per EC2 instance.

Checklist

• Tested locally, and on CODE if necessary
• Will not break dotcom-rendering
• Will not break our database – if updating CAPI, updated and committed the database files
• Meets our accessibility standards
  • Tested with screen reader
  • Navigable with keyboard
  • Colour contrast passed

[bryophyta]

great spot! I note that there are a couple of other uses of def for pandomainSettings in other projects (though not many) so assuming this fixes the issue here we should possibly look at those uses, too?

[prout-bot]

Seen on FRONTS-PROD (merged by @andrew-nowak 12 minutes and 18 seconds ago)

• Check your changes on www.theguardian.com ✔️
• Keep an eye on the deploy dashboard 📉

[prout-bot]

Overdue on ADMIN-PROD (merged by @andrew-nowak 30 minutes and 7 seconds ago) What's gone wrong?

[prout-bot]

Seen on ADMIN-PROD (merged by @andrew-nowak 100 hours, 37 minutes and 56 seconds ago)

• Check your changes on www.theguardian.com ✔️
• Keep an eye on the deploy dashboard 📉

[rtyley]

I note that there are a couple of other uses of def for pandomainSettings in other projects (though not many) so assuming this fixes the issue here we should possibly look at those uses, too?

Panda v6 goes a bit further by including this PR:

• Fix creation of superfluous PanDomainAuthSettingsRefresher instances, make panDomainSettings a val pan-domain-authentication#155

...which forces any code overriding the panDomainSettings field in the AuthActions trait to use val rather than def, so they no longer have the option to do wrong thing 😄