[glibc] p_vaddr p_offset skew unequal w.r.t. host page size

[gzz2000]

Hello jart, and thanks for this nice project. I am trying to execute a very simple statically linked linux program using blink on arm (m1) macos. It is reporting this error:

% ./o/blink/blink ~/Desktop/a.linux
I2023-01-02T21:59:35.837793:blink/loader.c:92:59351 p_vaddr p_offset skew unequal w.r.t. host page size
% file ~/Desktop/a.linux 
a.linux: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=98243f5fb9c6f72c28efb8dbfb68de795f990d24, for GNU/Linux 3.2.0, not stripped

The program is very simple, like this:

#include <cstdio>

int main() {
  puts("Hello, world from x86_64");
  int a, b; scanf("%d%d", &a, &b);
  printf("%d\n", a + b);
  return 0;
}

Compiled using: g++ a.cpp -o a -static.
If you would like the compiled binary I can later upload it to google drive.

[jart]

Works OK for me. Your distro must have some kind of strange toolchain configuration. The way to troubleshoot this kind of issue is to use readelf -Wa a which might be useful to post here. The solution might be to use -Wl,-z,common-page-size=16384,-z,max-page-size=16384. Could you give that flag a try and report back?

[gzz2000]

Thank you for your reply. Using -Wl,-z,max-page-size=16384 as you provided does the trick and the program executes successfully:

% ./o/blink/blink ~/Desktop/a.fix 
I2023-01-03T20:40:03.896284:blink/syscall.c:2539:81426 missing syscall 0x111
I2023-01-03T20:40:03.897141:blink/syscall.c:2539:81426 missing syscall 0x14e
I2023-01-03T20:40:03.897720:blink/syscall.c:1787:81426 getrandom() flags not supported yet
I2023-01-03T20:40:03.899523:blink/xlat.c:490:81426 atf 4096 not supported yet
I2023-01-03T20:40:03.900205:blink/xlat.c:490:81426 atf 4096 not supported yet

1 2
Hello, world from x86_64
3

The output of readelf -Wa a on the old binary is here: https://pastebin.com/rj0vBW2j (too long so put on the pastebin). My compiler is the default g++ of Ubuntu 22.04:

$ g++ --version
g++ (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Running readelf on the new, working binary outputs this: https://pastebin.com/8kfiEFSY

[jart]

I'm glad that flag fixed things. Since there's a workaround, I'm going to mark this "contributions welcome". My resources are primarily focused on making Blink work great for Cosmopolitan Libc. I see you're using a Glibc development stack and I want to support that really well too (including dynamic executables). What will make that happen fastest is if we get pull requests adding whatever support you need, which I'm happy to review.

I2023-01-03T20:40:03.896284:blink/syscall.c:2539:81426 missing syscall 0x111 [set_robust_list]
I2023-01-03T20:40:03.897141:blink/syscall.c:2539:81426 missing syscall 0x14e [rseq]
I2023-01-03T20:40:03.897720:blink/syscall.c:1787:81426 getrandom() flags not supported yet
I2023-01-03T20:40:03.899523:blink/xlat.c:490:81426 atf 4096 not supported yet [AT_EMPTY_PATH]

These should be added to the burndown list too. Glibc tracks very modern Linux-specific system calls. Expanding our support for modern Linux system calls that Glibc needs, is also very welcome and I'm happy to review / merge. Some of the stuff listed above isn't supported yet, because supporting it is nontrivial. If at all possible, I highly recommend compiling your binaries using Cosmopolitan Libc or Musl Libc, which are less likely to cause friction, since they both use a much more conservative set of easily portable kernel ABIs.

[gzz2000]

Thanks!

[jedisct1]

Same issue with anything compiled with Zig (both with the x86_64-linux-musl and x86_64-linux-gnu targets).

Unfortunately, the linker doesn't allow changing max-page-size.

[jart]

I did some Googling a few days ago and I think Zig has been having problems in general with Apple M1's page size requirements. This issue probably needs to be solved on our end. I don't yet know for certain if I'm going about loading ELF images in the most proper way. Help is definitely wanted, if anyone here has time to dig into how other ELF loaders do it and can help us do the same.

[jart]

Speaking of which, could someone here who encountered this problem please attach to this issue the specific hello world binary that's causing problems? That could be helpful in reproducing the problem and regression testing once it's solved.

[jedisct1]

Sure:

blink-test.zip

[jedisct1]

There's indeed an issue with the Zig standard library, that is being tracked here.

But that may not be related. The same problem happens when compiling pure C code (just with zig cc --target=x86_64-linux), that only uses the musl C library.

[jart]

I think I discovered better compiler flags to use, than the ones I recommended before.

Could everyone try this?

cc -static -Wl,-z,common-page-size=16384,-z,max-page-size=16384 -o hello hello.c

Using these flags, the linker will generate more sane program headers that look like this:

  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  LOAD           0x000000 0x0000000000400000 0x0000000000400000 0x000230 0x000230 R   0x4000
  LOAD           0x004000 0x0000000000404000 0x0000000000404000 0x000799 0x000799 R E 0x4000
  LOAD           0x008000 0x0000000000408000 0x0000000000408000 0x000054 0x000054 R   0x4000
  LOAD           0x00bfe0 0x000000000040ffe0 0x000000000040ffe0 0x000030 0x0002d8 RW  0x4000
  NOTE           0x000200 0x0000000000400200 0x0000000000400200 0x000030 0x000030 R   0x8
  GNU_PROPERTY   0x000200 0x0000000000400200 0x0000000000400200 0x000030 0x000030 R   0x8
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RW  0x10
  GNU_RELRO      0x00bfe0 0x000000000040ffe0 0x000000000040ffe0 0x000020 0x000020 R   0x1

[easrng]

I'm running blink in wasm and I'm getting this error even with the suggested compiler flags. It works when blink is compiled for x86_64 or x86 though (I haven't tested other architectures)

(I had a comment already but GitHub wasn't letting me edit it (something's probably wrong with one of my browser extensions) and I realized I hadn't pulled changes, so I deleted it while I pulled and rebuilt.)

[jart]

Wasm has a 64kb page size, correct? In that case, the executables you launch inside Blink will most likely need to be compiled using -Wl,-z,common-page-size=65536,-z,max-page-size=65536

[easrng]

Yep, that fixed the error. Now I have more different errors to work on :)