DPS binary doesn't work on non 4k page size linux arm64 kernels

[pschiffe]

What is Happening

DPS binary, distributed either as docker image or from dns-proxy-server-linux-aarch64-3.19.3-snapshot.tgz archive, doesn't work on non 4k page size linux arm kernels.

RHEL 8 variants for arm64 has only 64k page size kernel:

$ getconf PAGESIZE
65536

RHEL 9 variants for arm64 have 2 kernel versions (4k and 64k page size) and you can choose. DPS works on the 4k version, but not on the 64k version.

I didn't tested it, but this will probably be issue also when running on linux on apple's arm, as that kernel has 16k page size.

When you try to run the binary from release archive of from docker, it won't start, you just get the error:
Fatal error: Failed to create the main Isolate. (code 8)

When I try to run the DPS as jar, it works:

$ java -version
openjdk version "22.0.1" 2024-04-16
OpenJDK Runtime Environment (Red_Hat-22.0.1.0.8-1) (build 22.0.1+8)
OpenJDK 64-Bit Server VM (Red_Hat-22.0.1.0.8-1) (build 22.0.1+8, mixed mode)

$ java -jar ./dns-proxy-server.jar 
20:42:31.229 [main           ] INF c.m.dnsproxyserver.config.dataprovider.JsonConfigsl=75   m=createDefault                   status=createdDefaultConfigFile, path=/root/conf/config.json
20:42:31.308 [main           ] DEB c.m.d.config.dataprovider.ConfigDAOJson           l=39   m=find                            configPath=/root/conf/config.json
20:42:31.657 [main           ] INF c.m.d.s.d.a.DpsDockerEnvironmentSetupService      l=32   m=setup                           status=binding-docker-events, connectedToDocker=true
20:42:31.657 [main           ] INF c.m.d.s.d.a.DpsDockerEnvironmentSetupService      l=44   m=setupNetwork                    status=dpsNetwork, active=false
20:42:31.657 [main           ] INF c.m.d.s.docker.application.DpsContainerService    l=102  m=tRunningContainersToDpsNetwork  status=autoConnectDpsNetworkDisabled, dpsNetwork=false, dpsNetworkAutoConnect=false
20:42:31.657 [main           ] INF c.m.d.solver.docker.entrypoint.EventListener      l=32   m=onStart                         status=containerAutoConnectToDpsNetworkDisabled
20:42:31.660 [main           ] INF com.mageddo.dnsserver.UDPServerPool               l=31   m=start                           Starting UDP server, addresses=/0.0.0.0:53
....

I think this is some java issue, but I'm not sure how to fix it. Google shows some reports like this for other java sw.

Specs

• Docker Version:

Client: Docker Engine - Community
 Version:           26.1.3
 API version:       1.45
 Go version:        go1.21.10
 Git commit:        b72abbb
 Built:             Thu May 16 08:34:00 2024
 OS/Arch:           linux/arm64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.3
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       8e96db1
  Built:            Thu May 16 08:33:12 2024
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.6.32
  GitCommit:        8b3b7ca2e5ce38e8f31a34f35b2b68ceb8470d89
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

• DPS Version: defreitas/dns-proxy-server:3.19.3-snapshot-aarch64
• Attach DPS Log file

Fatal error: Failed to create the main Isolate. (code 8)

• OS: Rocky Linux 8.10 (Green Obsidian)

[mageddo]

Hey @pschiffe thanks for your report. I have no idea of how to fix it, I will search how to improve on that, any help is welcome.

It is probably related to the docker file

dns-proxy-server/Dockerfile.builder.linux-aarch64

Line 1 in 2dd5290

FROM defreitas/tools_graalvm-22.3_java-19_debian-9_aarch64:0.1.3

Or the deploy specs

dns-proxy-server/.github/workflows/actions-deploy.yml

Line 69 in 2dd5290

linux_arm_release:

[mageddo]

Maybe it's related to this one oracle/graal#7513, Looks like I will have to upgrade qemu version, maybe graal version too.

[mageddo]

Hey, @pschiffe , I've upgraded GraalVM and Qemu versions, can you check if dps 3.22.0-snapshot fix your usecase?

[pschiffe]

Hi @mageddo, so far no joy, though the error code is little bit different (and .jar stopped working):

# cat /etc/os-release | grep PRETTY
PRETTY_NAME="Rocky Linux 8.10 (Green Obsidian)"

# getconf PAGESIZE
65536

# uname -r
4.18.0-553.5.1.el8_10.aarch64

# docker version | grep -A 2 Server
Server: Docker Engine - Community
 Engine:
  Version:          26.1.3

# docker run -d defreitas/dns-proxy-server:3.22.0-snapshot-aarch64
# docker logs d6928c2b6735
Fatal error: Failed to create the main Isolate. (code 24)

When trying binary dns-proxy-server-linux-aarch64-3.22.0-snapshot.tgz:

# ./dns-proxy-server 
./dns-proxy-server: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by ./dns-proxy-server)
./dns-proxy-server: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by ./dns-proxy-server)

# rpm -q glibc
glibc-2.28-251.el8_10.2.aarch64

Trying .jar file:

# java -version
openjdk version "21.0.3" 2024-04-16 LTS
OpenJDK Runtime Environment (Red_Hat-21.0.3.0.9-1) (build 21.0.3+9-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-21.0.3.0.9-1) (build 21.0.3+9-LTS, mixed mode, sharing)

# java -jar ./dns-proxy-server.jar 
Exception in thread "main" java.lang.NoClassDefFoundError: org/graalvm/nativeimage/ImageInfo
	at com.mageddo.utils.Runtime.getRunningDir(Runtime.java:34)
	at com.mageddo.dnsproxyserver.config.dataprovider.ConfigDAOJson.buildConfigPath(ConfigDAOJson.java:59)
	at com.mageddo.dnsproxyserver.config.dataprovider.ConfigDAOJson.find(ConfigDAOJson.java:33)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:510)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
	at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
	at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
	at com.mageddo.dnsproxyserver.config.application.ConfigService.findConfigs(ConfigService.java:35)
	at com.mageddo.dnsproxyserver.config.application.ConfigService.findCurrentConfig(ConfigService.java:29)
	at com.mageddo.dnsproxyserver.config.application.Configs.lambda$getInstance$0(Configs.java:19)
	at com.mageddo.commons.lang.Singletons.lambda$createOrGet$0(Singletons.java:19)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
	at com.mageddo.commons.lang.Singletons.createOrGet(Singletons.java:19)
	at com.mageddo.commons.lang.Singletons.createOrGet(Singletons.java:15)
	at com.mageddo.dnsproxyserver.config.application.Configs.getInstance(Configs.java:18)
	at com.mageddo.dnsproxyserver.App.findConfig(App.java:53)
	at com.mageddo.dnsproxyserver.App.start(App.java:36)
	at com.mageddo.dnsproxyserver.App.main(App.java:25)
Caused by: java.lang.ClassNotFoundException: org.graalvm.nativeimage.ImageInfo
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
	... 23 more

[mageddo]

alright, I will have to try it out

[mageddo]

Hey @pschiffe , Can you check if DPS 3.24.0-snapshot fixes your usecase? The jar also should be working again.

• Fixed Exception in thread "main" java.lang.NoClassDefFoundError: org/graalvm/nativeimage/ImageInfo
• Probably fixed Fatal error: Failed to create the main Isolate. (code 24), I really tried to emulate rockylinux on aarch but as I don't have an aarch machine, I had not success until now
• ./dns-proxy-server: /lib64/libc.so.6: version GLIBC_2.32 not found (required by ./dns-proxy-server) will implicate to downgrade the glibc version at the builder machine or upgrade the glibc at your machine. I got how downgrade required libc version to 2.28 at Downgrade necessary libc version to run aarch binary #507

[pschiffe]

Hi @mageddo, awesome job! This seems to be fixed in all my use cases. Here's what I've tested (with 3.24.0-snapshot):

• Rocky 8 on ARM64 with 64k page size: docker image, binary & jar all working
• Rocky 9 on ARM64 with 4k page size: docker image, binary & jar all working
• Rocky 9 on x86_64 with 4k page size: docker image, binary, binary static & jar all working

It's perfect, thank you!

Reg glibc version, it's usually not possible to upgrade the version within the distribution. RHEL 8 & derivates are currently running on glibc-2.28, and these will be supported until 2029. Anything else supported is probably running on newer versions, so you don't need to support older versions than that. For now, supporting glibc-2.28 and newer is perfect.

This issue can be closed from my side, thanks again.

[mageddo]

Cheers, thanks for your help.