Large Data Querys - Receives an Gateway Shutdown on Session

[MarcosBan]

Hey,

Im running a POC of the self hosted solution in a GKE cluster, trying to retrieve large tables from our databases. I`ve run a demo query, and after running for about 35 minutes the log off this session says:

level: "info"
logger: "audit/audit.go:180"
msg: "closing session, reason=gateway shutdown"
sid: "44cca287-*****"
timestamp: "2024-08-07T12:47:47Z"

There`s a way to optmize this large data processing? The current implementation resources, this resources has no HPA implemented yet, during the processing the agent reach the 85% of the cpu:

Hoop gateway:

    Limits:
      cpu:                1250m
      ephemeral-storage:  1Gi
      memory:             1280Mi
    Requests:
      cpu:                1250m
      ephemeral-storage:  1Gi
      memory:             1280Mi

Hoop Agent:

    Limits:
      cpu:                500m
      ephemeral-storage:  1Gi
      memory:             2Gi
    Requests:
      cpu:                500m
      ephemeral-storage:  1Gi
      memory:             2Gi

[mtmr0x]

Hi @MarcosBan, we have two main use cases for accessing resources. One-offs (through hoop exec CLI and our webapp) and tunneling that uses our hoop connect CLI. The first one might face spikes when accessing large databases because it doesn't stream the information like the second one. But still, for both methods, depending on what you are accessing, the amount of data processed will impact CPU and memory usage. Nevertheless, there's plan to improve this, it's only not expected to this semester.