ci: move tests from x86_64-gnu-llvm-19 job to aarch64

[marcoieni]

This PR splits the tests of the x86_64-gnu-llvm-19 job in multiple jobs, using two free runners and one 8 core ARM large runner instead of one 16 core x86 runner.

One job is still on x86 because it is fast enough.

We are effectively replacing a 16 core large runner with an 8 core, which is more efficient.

[marcoieni]

error: Command cd "/checkout/obj/build/aarch64-unknown-linux-gnu/gcc/build" && "make" "--silent" "-j4" (failure_mode=Exit) has failed. Rerun with -v to see more details.

Not sure what's the issue. I'm removing ENABLE_GCC_CODEGEN: "1"

[Kobzol]

Oh, right. I think that GCC currently only works for x64.

[marcoieni]

it's nice that x86_64-gnu-llvm-19 finished in only 34 minutes!

• Probably we could use a 4 core runner.

but I don't know why the arm runner is failing now 😵

[jieyouxu]

but I don't know why the arm runner is failing now 😵

#128973

You'll probably need to add a //@ ignore-aarch64-unknown-linux-gnu (gdb tries to read from 0x0; FIXME: #128973) to ignore that debuginfo test.

[marcoieni]

Thanks, I'll do that. At the same time I wonder why this works

[jieyouxu]

I wonder if they're using different gdb versions, if you want to find out exactly you'll probably need to compare the two gdb versions used, but that test is already kinda broken for aarch64-unknown-linux-gnu / windows-gnu anyway.

[marcoieni]

this is the failing test: https://github.com/rust-lang/rust/blob/master/tests/run-make/mte-ffi/bar_float.c

[jieyouxu]

I'm surprised that test doesn't fail on aarch64-gnu, did that test never manage to run?

[marcoieni]

I'm also surprised. yes, it runs.
check the run of b5eb989 (raw log here).

[jieyouxu]

I'm also surprised. yes, it runs.

can you slap a FIXME pointing to #141600

[bors]

⌛ Testing commit 1b7a927 with merge 40311c4...

[bors]

☀️ Test successful - checks-actions
Approved by: Kobzol
Pushing 40311c4 to master...

[github-actions]

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing 77101fe (parent) -> 40311c4 (this PR)

Test differences

Show 1 test diff

Stage 2

• [debuginfo-gdb] tests/debuginfo/by-value-non-immediate-argument.rs: ignore (ignored when the GDB version is lower than 13.0) -> ignore (ignored when the target is aarch64-unknown-linux-gnu ((gdb tries to read from 0x0; FIXME: gdb on aarch64-unknown-linux-gnu/windows-gnu cannot print unused by-value non-immediate arguments #128973))) (J0)

Job group index

• J0: aarch64-gnu

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
    test-dashboard 40311c4dcf666e8bddcec4a8059e7d9f12bbd363 --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

1. x86_64-apple-2: 4791.8s -> 5985.6s (24.9%)
2. dist-x86_64-apple: 8899.7s -> 7075.7s (-20.5%)
3. dist-aarch64-linux: 5393.7s -> 6255.4s (16.0%)
4. aarch64-gnu-debug: 4784.3s -> 4048.1s (-15.4%)
5. x86_64-apple-1: 6886.6s -> 7928.8s (15.1%)
6. dist-apple-various: 7273.4s -> 7970.5s (9.6%)
7. aarch64-apple: 4964.7s -> 5392.9s (8.6%)
8. x86_64-rust-for-linux: 2628.6s -> 2797.9s (6.4%)
9. dist-x86_64-freebsd: 4984.9s -> 5304.6s (6.4%)
10. dist-x86_64-msvc: 6088.0s -> 6463.5s (6.2%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

[rust-timer]

Finished benchmarking commit (40311c4): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results (secondary -4.7%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-4.7%	[-4.7%, -4.7%]	1
All ❌✅ (primary)	-	-	0

Cycles

Results (secondary -2.0%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-2.0%	[-2.0%, -2.0%]	1
All ❌✅ (primary)	-	-	0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 777.676s -> 778.629s (0.12%)
Artifact size: 368.53 MiB -> 368.47 MiB (-0.02%)

[drzymalanet]

@marcoieni @jieyouxu

but I don't know why the arm runner is failing now 😵

I'm surprised that test doesn't fail on aarch64-gnu, did that test never manage to run?

MTE is an ARM thingy. Probably you can skip the mte_ffi on x86_64.

[marcoieni]

I see, thanks! The issue we opened ( #141600 ) is about this test not failing in ARM, so probably the issue remains.

[drzymalanet]

The point is, this test should pass only on ARM. It should fail on all other cpus. By fixing it for x86 you broke the test.

[jieyouxu]

The point is, this test should pass only on ARM. It should fail on all other cpus. By fixing it for x86 you broke the test.

As far as I can't tell, the test never ran for x86 because it was gated with //@ only-aarch64-unknown-linux-gnu. Additionally, the test didn't fail on x86, it failed on aarch64 (on the new aarch64-gnu-llvm-19-1 job introduced in this PR):

---- [run-make] tests/run-make/mte-ffi stdout ----

error: rmake recipe failed to complete
status: exit status: 1
command: cd "/checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make/mte-ffi/rmake_out" [..] TARGET="aarch64-unknown-linux-gnu" [..] "/checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make/mte-ffi/rmake"
--- stdout -------------------------------
int test...
float test...
string test...
------------------------------------------
--- stderr -------------------------------
command failed at line 35
Command { cmd: "gcc" "bar_string.c" "libfoo.so" "-o" "test" "-lm" "-lrt" "-ldl" "-lpthread" "-march=armv8.5-a+memtag", stdin_buf: None, stdin: None, stdout: None, stderr: None, drop_bomb: DropBomb { command: "gcc", defused: true, armed_location: Location { file: "/checkout/tests/run-make/mte-ffi/rmake.rs", line: 30, col: 5 } }, already_executed: true }
output status: `exit status: 1`
=== STDOUT ===



=== STDERR ===
bar_string.c: In function ‘main’:
bar_string.c:36:9: error: assignment to ‘char *’ from incompatible pointer type ‘unsigned int *’ [-Wincompatible-pointer-types]
   36 |     ptr = (unsigned int *)((uintptr_t)ptr | 0x1fl << 56);
      |         ^
------------------------------------------

[drzymalanet]

Let me show you something:

A pointer in x86 CPU:                                                    0b00000000000000000000000000000000
A pointer in x86_64 CPU (a.k.a. AMD64):  0b0000000000000000000000000000000000000000000000000000000000000000
A pointer in aarch64 (a.k.a. ARM64):     0b........00000000000000000000000000000000000000000000000000000000
TBI bits (Top Byte Ignore bits):         0b11111111........................................................
MTE bits (Memory Tagging Extension)      0b....1111........................................................

Observe the following:

1. x86 is only 32 bit wide, so the MTE test will just do nothing because it changes bits 59-56.
2. x86_64 uses the whole 64 bits for addressing, so when you change MTE bits, you mess up the address. Test has to fail.
3. Finally, aarch64 ignores the TBI bits, so even if you change the MTE bits, the address will not change. Test should pass.

References:
https://en.wikichip.org/wiki/arm/tbi
https://www.kernel.org/doc/html/v5.10/arm64/memory-tagging-extension.html

[drzymalanet]

As far as I can't tell, the test never ran for x86 because it was gated with //@ only-aarch64-unknown-linux-gnu.

Good. Leave the gate in place, it is correct.

Additionally, the test didn't fail on x86

No wonder. The test touches bits outside of a 32bit CPU range.

it failed on aarch64 (on the new aarch64-gnu-llvm-19-1 job introduced in this PR)

This feature requires ARMv8.5 or higher. And it has to be enabled. Maybe the CI job runner does not support this extension?

[jieyouxu]

This feature requires ARMv8.5 or higher. And it has to be enabled. Maybe the CI job runner does not support this extension?

Looks like the runner hardware does not have not MTE extension enabled, at all.

aarch64-gnu-llvm-19-1

gcc_version: Using built-in specs.
COLLECT_GCC=aarch64-linux-gnu-gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/aarch64-linux-gnu/14/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: aarch64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 14.2.0-4ubuntu2' --with-bugurl=file:///usr/share/doc/gcc-14/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2,rust --prefix=/usr --with-gcc-major-version-only --program-suffix=-14 --program-prefix=aarch64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/libexec --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-libstdcxx-backtrace --enable-gnu-unique-object --disable-libquadmath --disable-libquadmath-support --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --enable-fix-cortex-a53-843419 --disable-werror --enable-offload-targets=nvptx-none=/build/gcc-14-T7YiXd/gcc-14-14.2.0/debian/tmp-nvptx/usr --enable-offload-defaulted --without-cuda-driver --enable-checking=release --build=aarch64-linux-gnu --host=aarch64-linux-gnu --target=aarch64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-serialization=2
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 14.2.0 (Ubuntu 14.2.0-4ubuntu2) 
 
lscpu: Architecture:                         aarch64
CPU op-mode(s):                       32-bit, 64-bit
Byte Order:                           Little Endian
CPU(s):                               4
On-line CPU(s) list:                  0-3
Vendor ID:                            ARM
Model name:                           Neoverse-N2
Model:                                0
Thread(s) per core:                   1
Core(s) per socket:                   4
Socket(s):                            1
Stepping:                             r0p0
BogoMIPS:                             2000.00
Flags:                                fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16
L1d cache:                            256 KiB (4 instances)
L1i cache:                            256 KiB (4 instances)
L2 cache:                             4 MiB (4 instances)
L3 cache:                             128 MiB (1 instance)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-3
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; CSV2, BHB
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

aarch64-gnu

gcc_version: Using built-in specs.
COLLECT_GCC=aarch64-linux-gnu-gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/aarch64-linux-gnu/11/lto-wrapper
Target: aarch64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 11.4.0-1ubuntu1~22.04' --with-bugurl=file:///usr/share/doc/gcc-11/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-11 --program-prefix=aarch64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libquadmath --disable-libquadmath-support --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --enable-fix-cortex-a53-843419 --disable-werror --enable-checking=release --build=aarch64-linux-gnu --host=aarch64-linux-gnu --target=aarch64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-serialization=2
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04) 

lscpu: Architecture:                         aarch64
CPU op-mode(s):                       32-bit, 64-bit
Byte Order:                           Little Endian
CPU(s):                               4
On-line CPU(s) list:                  0-3
Vendor ID:                            ARM
Model:                                0
Thread(s) per core:                   1
Core(s) per socket:                   4
Socket(s):                            1
Stepping:                             r0p0
BogoMIPS:                             2000.00
Flags:                                fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16
L1d cache:                            256 KiB (4 instances)
L1i cache:                            256 KiB (4 instances)
L2 cache:                             4 MiB (4 instances)
L3 cache:                             128 MiB (1 instance)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-3
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; CSV2, BHB
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

[jieyouxu]

The test failure is most likely due to GCC version differences.

• aarch64-gnu-llvm-19-1
  • Uses gcc version 14.2.0 (Ubuntu 14.2.0-4ubuntu2)
  • Based on lscpu output, the hardware that was used for this runner does not have MTE enabled.
• aarch64-gnu
  • Uses gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
  • Based on lscpu output, the hardware that was used for this runner does not have MTE enabled.