Alternative silo limit

[corneliusroemer]

preview URL: https://alternative-silo-limit.loculus.org

Summary

• Make silo cronjob pod fail after 3600s (configurable)
• Only let jobs start within 60s of when it should have started
• Don't restart failed jobs, it's ok if they fail, there'll be a new one in <1min

resolves #1020 (at least partially)

[theosanderson]

Sorry I was confused - I thought this was my PR and these were your comments and I didn't know how to do them :)

[corneliusroemer]

In my nomenclature the deadline is conf

But same for all instances IIRC

[theosanderson]

Lots of nice ideas!

[corneliusroemer]

Alright, I've made it go back to the roots - but with correct syntax :D

[corneliusroemer]

Changed base to main after @theosanderson's closing of #1022

[theosanderson]

Code looks good. Could be worth doing a bit of testing over repeated cycles, which I haven't. But code LGTM. Thanks for fixing up my syntax :)

[theosanderson]

And we can consider per-organism limits for post-MVP - it wouldn't be hard (but I also can't face doing it myself this sec :) )

[corneliusroemer]

Code looks good. Could be worth doing a bit of testing over repeated cycles, which I haven't. But code LGTM. Thanks for fixing up my syntax :)

I haven't found a way to get yaml schema to work with kubernetes, would be great help for vscode.

I'll notice bugs on main - I mean things were broken for 18h and this can't be worse.