Why does paketo-libjvm set direct memory to an arbitrary 10MB by default?

[postalservice14]

Just curious the reasoning behind setting -XX:MaxDirectMemorySize to 10M by default? The Java default seems to have always been set it to max memory. 10M as a default is quite a bit different than max memory.

[dmikusa]

I didn't make the original decision, but a lot of the Java decisions like this were made based on history and compatibility with the Cloud Foundry Java buildpack. So when you ask why, the answer is probably because that's how the CF Java Buildpack did it.

As for why the CF Java Builidpack picks this seemingly arbitrary value, I'm not sure. That predates my involvement. I do trust that the maintainers had their reasons though, likely based on research and user feedback.

Personally, I'd like to see some research into how the JVM has evolved in this area and see what we might be able to move out of the memory calculator and just let the JVM do its thing. Originally, the memory calculator was put in because it prevents container OOMEs, and the JVM had zero awareness of container limits. The JVM has been improving in this area, so maybe we can, based on the JVM version, start scaling back what the memory calculator does without risking OOMEs. I feel like when we use JVM default behaviors, that's a better user experience because it's what people naturally expect, however, preventing OOMEs is more important than that so we want to ensure we're giving users all the protection we can there. Anyway, talking out loud here.

If this is an area where you're interested in doing a little research, that would certainly help. Feel free to post findings here. Beyond that, we'd probably need an RFC to propose changes of this nature. The research I mentioned would certainly help drive that RFC.

[postalservice14]

Yeah, I hear ya. Thanks @dmikusa for your thoughts on this. Let me think on this a bit and I'll open an RFC if we end up suggesting a change.

[jukkaleh-atoz]

Tl;dr: if -XX:MaxDirectMemorySize flag is not given by default, the amount of native memory used for Direct Byte Buffers is limited to the maximum heap size. If the flag is given it is fixed and will OOM if full. We need an option to skip setting this flag. If larger than default size is used. Buildpack will subtract that from heap space leading to wasted ram and difficulty starting the container in azure for example. Plans pool the ram somehow and setting too large of an XX:MaxDirectMemorySize can lead to calculator to think there is not a enough of ram and fail to start.

We had a spring boot build with bootBuildImage gradle task. It was build to upload and download small 0-10MB pdf files to/from azure blob storage which internally uses netty to upload/download files.

This -XX:MaxDirectMemorySize=10M flag causes OOMs out of the blue. You can't upload/download simple files even though there is gigs of heap available.

java.lang.OutOfMemoryError: Cannot reserve 10282 bytes of direct buffer memory (allocated: 10478225, limit: 10485760)
   at java.base/java.nio.Bits.reserveMemory(Unknown Source) ~[na:na]
   at java.base/java.nio.DirectByteBuffer.<init>(Unknown Source) ~[na:na]
   at java.base/java.nio.ByteBuffer.allocateDirect(Unknown Source) ~[na:na]
   at io.netty.buffer.UnpooledDirectByteBuf.allocateDirect(UnpooledDirectByteBuf.java:104) ~[netty-buffer-4.1.111.Final.jar:4.1.111.Final]

The fix was to update the code not to load whole file into byte array before sending it but I think we will also migrate away from buildpacks as this shouldn't have happened with the amount memory available.

From

public String writeFile(otherParams... final byte[] content) throws IOException {
        final Resource resource = resourceLoader.getResource(...params);
        try (final OutputStream os = ((WritableResource) resource).getOutputStream()) {
            os.write(content);
        }
        return resource.getFilename();
    }

Into

public String writeFile(otherParams... final Resource file) throws IOException {
        final Resource resource = resourceLoader.getResource(...params);
        try (final OutputStream os = ((WritableResource) resource).getOutputStream()) {
            file.getInputStream().transferTo(os);
        }
        return resource.getFilename();
    }

[dmikusa]

Tl;dr: if -XX:MaxDirectMemorySize flag is not given by default, the amount of native memory used for Direct Byte Buffers is limited to the maximum heap size.

This is exactly why we set this flag and put a limit on direct memory. The JVM is not container-friendly in this regard. If you were to go with the defaults, what would happen is that you'd wildly over-allocate memory in your container. Let's say you have a 1G limit on your container, you set the heap to 512M and non-heap except direct memory to 512M. Since you've not set direct memory to anything, the JVM will happily continue to allocate memory right up until the OOM killer steps in and kills your app.

Putting a limit on direct memory prevents this. The tradeoff is that you need to change the limit for direct memory if you need more than the default. It is very simple to override the default, just set JAVA_TOOL_OPTIONS='-XX:MaxDirectMemorySize=20M' or whatever you need and restart your container.

If the flag is given it is fixed and will OOM if full. We need an option to skip setting this flag. If larger than default size is used. Buildpack will subtract that from heap space leading to wasted ram and difficulty starting the container in azure for example.

This is intentional. Again, the memory calculator does this to prevent OOMEs. If you have a 1G memory limit on your container, the memory calculator will configure the JVM to use the full 1G. If you then go and tell it to change the direct memory limit from 10M to 100M, of course, it has to change something else. That memory has to come from somewhere, otherwise, you over-allocate memory and you are now susceptible to having sporadic and unpredictable OOME's in your app.

If you need to increase direct memory and not decrease heap then you need to 1.) change the default direct memory size and 2.) increase your container's memory limit by the difference (90M in the example of going from 10M to 100M).

This -XX:MaxDirectMemorySize=10M flag causes OOMs out of the blue. You can't upload/download simple files even though there is gigs of heap available.

This is not an issue with the buildpack or memory calculator configuration. Your app is using direct memory so it needs direct memory. It doesn't matter how much heap is available, it will fail without enough direct memory.

The reason this works for you elsewhere is simply because you are relying on the ability to over-commit your container or system's resources. That's not good practice though, because it can lead to very difficult-to-debug and painful crashes in your application. Especially in container environments because the container, and all context about the app state, goes away when the OOM killer kills the app.

The buildpack and memory calculator do not allow you to over-commit memory, it is a safety and stability feature.

[jukkaleh-atoz]

The reason this works for you elsewhere is simply because you are relying on the ability to over-commit your container or system's resources.

Containers do have hard memory limit that is true. How you use that memory in practice should not matter, long as it doesn't run out.
If I have container with 16GB of rams I'm totally relying it not to OOM when dealing with dozen of megabytes. What I don't want is to manually joggle between direct and heap memory allocations. One day it can be 10GB of direct memory and 6GB of heap or 8gb direct and 8gb of direct memory depending on the workload. Goes totally against what Java automatic memory management is about

What I'm trying to imply is that there is no 1 fixed MaxDirectMemorySize that will work. Set it 10,20,30,40... it will always crash on some workload. Not having the flag will make the OOMs go away. Kinda nobrainer for any production workload.

it is a safety and stability feature.

Theoretical OOM prevention technique that actually causes OOMs

[dmikusa]

If I have container with 16GB of rams I'm totally relying it not to OOM when dealing with dozen of megabytes. What I don't want is to manually joggle between direct and heap memory allocations. One day it can be 10GB of direct memory and 6GB of heap or 8gb direct and 8gb of direct memory depending on the workload. Goes totally against what Java automatic memory management is about

I respectfully disagree. I think that each app has its own unique memory profile. It may take a small bit of observation and tuning to dial in what that app needs, but unless an app's behavior changes drastically, the memory profile shouldn't change drastically either.

Case in point. Your app had a memory leak. You were loading an entire file into memory. That went against the expected memory profile. This setting allowed to you see that there was a problem and fix it. Your app now performs better and more memory efficient than before.

What I'm trying to imply is that there is no 1 fixed MaxDirectMemorySize that will work. Set it 10,20,30,40... it will always crash on some workload. Not having the flag will make the OOMs go away. Kinda nobrainer for any production workload.

Again, respectfully disagree. We have this conversation come up where some user's apps are impacted by this setting, but by in large many apps work with the 10M default. Where it needs to be adjusted is when someone is either a.) doing something unique in their app with direct memory like a lot of caching or b.) there's a memory leak. This helps you detect b.) faster, which is a good thing.