Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(k8s): Add resource limits to all per-instance resources #1842

Merged
merged 13 commits into from
May 10, 2024

Conversation

corneliusroemer
Copy link
Contributor

@corneliusroemer corneliusroemer commented May 10, 2024

preview URL: https://limits.loculus.org

Summary

Setting resource limits is good for 2 reasons:

  • Java containers set the heap size based off how much memory they have available. No limits mean they think they've got the whole machine. Setting limits fixes this effectively. Backend memory usage goes down from 3GB to 500MB. Finally things get garbage collected.
  • Step towards limiting swapping which currently can happen. We want to prioritize central services and non-kubernetes resources over preview deployments.

I set memory requests and limits for all, but only cpu requests, as cpu limits are not very useful (we are not CPU limited usually and throttling works well usually).

@corneliusroemer corneliusroemer added the preview Triggers a deployment to argocd label May 10, 2024
Copy link
Member

@theosanderson theosanderson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@corneliusroemer corneliusroemer changed the title Add resource limits to prevent swapping feat(k8s): Add resource limits to all per-instance resources May 10, 2024
@corneliusroemer corneliusroemer merged commit ead6afc into main May 10, 2024
10 checks passed
@corneliusroemer corneliusroemer deleted the limits branch May 10, 2024 14:40
corneliusroemer added a commit that referenced this pull request May 10, 2024
Setting resource limits is good for 2 reasons:
- Java containers set the heap size based off how much memory they have available. No limits mean they think they've got the whole machine. Setting limits fixes this effectively. Backend memory usage goes down from 3GB to 500MB. Finally things get garbage collected.
- Step towards limiting swapping which currently can happen. We want to prioritize central services and non-kubernetes resources over preview deployments.

I set memory requests and limits for all, but only cpu requests, as cpu limits are not very useful (we are not CPU limited usually and throttling works well usually).

(cherry picked from commit ead6afc)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
preview Triggers a deployment to argocd
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants