Out of Memory Error on Trino Pod Despite Memory Limits Configured

[AlakmarShafin]

Hello Trino Community,

I am encountering an Out of Memory (OOM) error with my Trino deployment on Azure Kubernetes Service (AKS). Despite configuring several memory limits, my pod was restarted because it breached the total node memory limit of 62 GiB. Here’s the setup I have:

AKS Node Memory: 62 GiB
Pod Memory Limits:
CPU Limit: 7000m
Memory Limit: 58.5 GiB
Soft Memory Limit: 46 GiB configured in the resource group.
Query Max Memory: 18 GiB

Configuration Files

resource-group.json:

{
  "rootGroups": [
    {
      "name": "global",
      "softMemoryLimit": "46GB",
      "hardConcurrencyLimit": 40,
      "maxQueued": 1000,
      "schedulingPolicy": "fair",
      "jmxExport": false
    }
  ],
  "selectors": [
    {
      "group": "global"
    }
  ]
}

jvm.config:

-server
-Xmx54G
-XX:InitialRAMPercentage=20
-XX:MaxRAMPercentage=90
-XX:G1HeapRegionSize=32M
-XX:+UseG1GC
-XX:MaxGCPauseMillis=200
-XX:ParallelGCThreads=4
-XX:+ExitOnOutOfMemoryError
-XX:+HeapDumpOnOutOfMemoryError
-XX:-OmitStackTraceInFastThrow
-XX:ReservedCodeCacheSize=512M
-XX:PerMethodRecompilationCutoff=10000
-XX:PerBytecodeRecompilationCutoff=10000
-Djdk.attach.allowAttachSelf=true
-Djdk.nio.maxCachedBufferSize=2000000
-XX:+UnlockDiagnosticVMOptions
-XX:+UseAESCTRIntrinsics

Issue:

Despite these configurations, the pod still encountered an OOM error and was restarted after breaching the 62 GiB node limit.

Questions:

Could you help me understand why the pod would breach the 62 GiB limit even with the defined soft and hard memory limits?
Are there any additional configurations or best practices I should consider to prevent this issue in the future?
Thank you for your assistance!

[hashhar]

Resource groups only perform admission control. The config you have doesn't mean queries will fail if they use more than 46GB of memory, it means that newly submitted queries will have to wait until memory consumption falls below 46GB.

To limit memory usage see max-memory-per-node and max-memory from https://trino.io/docs/current/admin/properties-resource-management.html#query-max-memory-per-node

[AlakmarShafin]

But I have mentioned Query Max Memory: 18 GiB

[hashhar]

that's per query, the total memory usage across multiple queries can be higher.

What error do you get when the crash happens? Is it a Java OOM or a kernel OOM?