-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provide memory usage limits on session creation from startup params #545
Comments
Could we start with a hard-coded limit? |
Yes, I was thinking we could do:
|
@scsmithr I like what's described here:
I think we can implement that in cloud and actually include that information to accomplish https://github.com/GlareDB/cloud/issues/578 I imagine seeing this possibly show up in the header, and for sure in the settings (now that we have them 😎 |
How do we actually limit resource utilization? |
Garrett: Can we infer it from disk usage? |
Mem and disk more important than cpu |
I think we can start by stress testing and getting an reasonable estimate for number of people we can roll alpha out to without causing problems. Again what matters most is memory and disk. |
Sean: What if we attached a disk to each node |
Garrett: Does it need to be shared? |
2023-04-17: Adding the param is easy, but we need to be able to actually limit the memory too. DF?
|
Datafusion has a runtime config that allows for setting a memory pool: We'll likely want to use the And we would construct the |
Really appreciate the guidance here 👍 |
Summary
When making a request to Cloud for connection authentication, Cloud will respond back with that database ID. Eventually Cloud should respond back with an optional set of resource limits as well. For example, a session connecting to a database on a shared node should be limited to .5G of memory. This will then be used during session creation to create a
RuntimeEnv
with this limit.We'll want to specify disk/cpu usage in the future, but I'm not yet sure how we accomplish that at the session level yet.
See what we're currently doing for the database id:
https://github.com/glaredb/glaredb/blob/8d758de55c8112623e13c24de9341b29ea63e5ce/crates/pgsrv/src/handler.rs#L105-L116
Specifications
Rationale
Limit session resource usage on shared nodes.
Impact
Cloud needs the expand the endpoint to optionally include the limits when it sends back a response to pgsrv.
cc @greyscaled @f0ssel
Tasks
Related issues:
apache/datafusion#587
apache/datafusion#3941
The text was updated successfully, but these errors were encountered: