[Android] Runtime/BCL native libs need support for devices with 16k memory pages (arm64 and x64)

[grendello]

Wiki: https://github.com/xamarin/xamarin-android/wiki/Android-support-for-devices-with-16k-pages

Summary

Android is moving towards supporting devices with 16k memory page size, for the x64 and arm64 architectures. This affects all the native components of the application. In our case it means the MonoVM runtime, the BCL support libraries and .NET for Android native runtime and libraries.

Google plan to make support for 16k devices a requirement for Google Play application submissions next year.

A lot of work needs to be done.

Google have an article with more technical details.

How it affects the runtime

• All the native libraries must be rebuilt with flags enabling 16k alignment of data and code
• Code review of native (and maybe some managed?) code needs to be done in order to make sure we don't assume 4k page size anywhere. This includes MonoVM runtime/JIT/interpreter code.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

[steveisok]

/cc @vitek-karas

[janvorli]

• Code review of native (and maybe some managed?) code needs to be done in order to make sure we don't assume 4k page size anywhere

coreclr should work on variable page sizes. macOS ARM64 uses 16kB pages, we have tested Linux arm64 with 64kB pages in the past too. I am not sure about the x64 though, there might be some stuff expecting 4kB page size for this architecture.

[janvorli]

Ah, sorry, this issue is mono specific, right?

[steveisok]

Ah, sorry, this issue is mono specific, right?

Yeah, but it's also important to understand how coreclr would behave.

[grendello]

@janvorli CoreCLR is used to build apps that run on Android too, so as @steveisok said, it's important to make sure it works too.

[janvorli]

Ok, I am glad the details I have added above are useful then.

[janvorli]

Actually, as for x64, the hardware only supports 4kB, 2MB and 1GB large pages. See the Intel® 64 and IA-32 Architectures
Software Developer’s Manual.

[jkotas]

for x64, the hardware only supports 4kB, 2MB and 1GB large pages

They have kernel that simulates 16kB page sizes on x64 for testing purposes: https://source.android.com/docs/core/architecture/16kb-page-size/getting-started-cf-x86-64-pgagnostic

[grendello]

Yesterday we realized that the runtime (and the native BCL libraries) will have to be built and shipped with both 16k and 4k alignment. This is necessary in order for us to be able to support 4k emulators/devices (especially with x86_64) as not all developers will need to deploy their packages to Google Play Store. Developers working on in-house devices, IoT devices and other embedded 64-bit devices running Android will most likely keep using 4k-aligned hosts.

This requirement applies to both .so libraries and .a archives.

It would be ideal if both 16k and 4k versions were shipped in the same nuget, since both versions of libraries might be required to be present at all times on the developer's machine. However, this would require "mangling" file names for the affected libraries.
Would naming them by adding .16k/.4k before the extension be acceptable?

We also must have this implemented in time for the NET9 release, since it will be supported throughout next year and Google are going to start requiring 16k alignment for Play Store deployment next year. /cc @steveisok

[grendello]

for x64, the hardware only supports 4kB, 2MB and 1GB large pages

They have kernel that simulates 16kB page sizes on x64 for testing purposes: https://source.android.com/docs/core/architecture/16kb-page-size/getting-started-cf-x86-64-pgagnostic

There's also now an preview x86_64 system image for the Android emulator (the traditional one, not cuttlefish) based on Android 15 which has this kernel.

[jkotas]

Yesterday we realized that the runtime (and the native BCL libraries) will have to be built and shipped with both 16k and 4k alignment.

What's the downside of using 16k alignment everywhere? I assume that 16k aligned binaries work fine on the existing Android phones. If the 16k is fine for Google Play Store, it should be fine for everybody out there as well.

[steveisok]

What's the downside of using 16k alignment everywhere? I assume that 16k aligned binaries work fine on the existing Android phones. If the 16k is fine for Google Play Store, it should be fine for everybody out there as well.

Agreed - I think this is a product decision and something we should discuss internally. I'm not excited about having to support 4 more runtime packs with another toggle switch.

[grendello]

Yesterday we realized that the runtime (and the native BCL libraries) will have to be built and shipped with both 16k and 4k alignment.

What's the downside of using 16k alignment everywhere? I assume that 16k aligned binaries work fine on the existing Android phones. If the 16k is fine for Google Play Store, it should be fine for everybody out there as well.

Yes, they will work fine on devices with 4k pages. The main downside is more memory use.

[grendello]

What's the downside of using 16k alignment everywhere? I assume that 16k aligned binaries work fine on the existing Android phones. If the 16k is fine for Google Play Store, it should be fine for everybody out there as well.

Agreed - I think this is a product decision and something we should discuss internally. I'm not excited about having to support 4 more runtime packs with another toggle switch.

Two more, this is 64-bit only.

[grendello]

My main concern with just shipping the 16k-aligned binaries is embedded device developers - if we don't ship 4k-aligned binaries, they would be forced to use the larger binaries (with bigger memory consumption), which on embedded devices with little memory may have a lot of importance.

[jkotas]

How much is the increase in memory consumption?

I understand that there is a bit higher address space consumption (the padding takes address space). This does not matter - address space consumption that we talk about here costs close to nothing.

I do not see why this have to cause measurable increase in physical memory consumption. ELF loader and Linux kernel should map the pages where the sections overlap to same physical memory page(s).

[grendello]

How much is the increase in memory consumption?

I understand that there is a bit higher address space consumption (the padding takes address space). This does not matter - address space consumption that we talk about here costs close to nothing.

It depends on the library. However, the consumption will be real, since what is "wasted" is the space between the end of a section and the beginning of the next one. The preceding section "spills over" into its last page, so the page is actually mapped and using real memory. The difference between the used space and the page size alignment is the wasted memory. Granted, it's not that much for most devices in most cases, but for embedded devices it may be important.

I do not see why this have to cause measurable increase in physical memory consumption. ELF loader and Linux kernel should map the pages where the sections overlap to same physical memory page(s).

I'm not sure they will do that, because then the alignment requirement wouldn't be satisfied for the beginning of one of the sections.

Anyway, after some discussion with @jonpryor, I think you and @steveisok are right - we don't need to ship 4k and 16k binaries right now. If anyone needs them, they will let us know and we can cross the bridge if/when that happens.

So for NET9 release the thing that we'd need is to build the runtime with 16k alignment. I can open a PR later this week which will implement that, unless somebody beats me to it.

[janvorli]

the page is actually mapped and using real memory.

Unless that extra virtual memory range is accessed, it should not increase the resident set and thus it should only consume virtual address space.

[grendello]

PR is up #104577