Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[infra] dev namespaces should scale down #14020

Closed
danking opened this issue Nov 21, 2023 · 0 comments · Fixed by #14025
Closed

[infra] dev namespaces should scale down #14020

danking opened this issue Nov 21, 2023 · 0 comments · Fixed by #14025

Comments

@danking
Copy link
Contributor

danking commented Nov 21, 2023

What happened?

dev namespace scaling is doing a good job of keeping costs down over the weekend and over night, but if devs don't k delete deployments -n NAMESPACE and k delete statefulsets -n NAMESPACE, these deployments stick around all day every day. It seems, in practice, our db's get an entire 2 core node to themselves.

Let's eliminate the scale-up job. We will keep the cronjob that scales down namespaces at the end of each workday. We will provide a scale-up command to either the make targets or devbin/functions.sh. hailctl dev deploy, of course, would also force a scale up (while also blowing away whatever was there before).

Version

0.2.126

Relevant log output

No response

@daniel-goldstein daniel-goldstein self-assigned this Nov 27, 2023
danking pushed a commit that referenced this issue Nov 29, 2023
Resolves #14020.

I'll manually delete the scale up cron jobs from the dev namespaces once
this goes in.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants