[mono] Linux arm64 AOT build failing on perf jobs

[matouskozak]

Since 20th of December (https://dev.azure.com/dnceng/internal/_build/results?buildId=2605880&view=results), linux arm64 AOT builds on perf runs are failing with

2024-12-27T08:08:32.3117042Z   Running 'PYTHONPATH=/__w/1/s/.packages/runtime.linux-x64.microsoft.netcore.runtime.mono.llvm.libclang/19.1.0-alpha.1.24569.1/tools/bindings/python python3 /__w/1/s/src/mono/mono/offsets/offsets-tool.py --abi=aarch64-linux-gnu --targetdir="/__w/1/s/artifacts/obj/mono/linux.arm64.Release" --monodir="/__w/1/s/src/mono" --nativedir="/__w/1/s/src/native" --outfile="/__w/1/s/artifacts/obj/mono/linux.arm64.Release/cross/offsets-aarch-linux-gnu.h" --libclang="/__w/1/s/.packages/runtime.linux-x64.microsoft.netcore.runtime.mono.llvm.libclang/19.1.0-alpha.1.24569.1/tools/libclang.so" --libclang-headers="/__w/1/s/.packages/runtime.linux-x64.microsoft.netcore.runtime.mono.llvm.libclang/19.1.0-alpha.1.24569.1/tools/include" --prefix="/crossrootfs/arm64/usr/lib/gcc/aarch64-linux-gnu/5" --sysroot="/crossrootfs/arm64"'
2024-12-27T08:08:33.1176807Z   provided path via --prefix ("/crossrootfs/arm64/usr/lib/gcc/aarch64-linux-gnu/5") doesn't exist.
2024-12-27T08:08:33.1252035Z /__w/1/s/src/mono/mono.proj(1000,5): error MSB3073: The command "python3 /__w/1/s/src/mono/mono/offsets/offsets-tool.py --abi=aarch64-linux-gnu --targetdir="/__w/1/s/artifacts/obj/mono/linux.arm64.Release" --monodir="/__w/1/s/src/mono" --nativedir="/__w/1/s/src/native" --outfile="/__w/1/s/artifacts/obj/mono/linux.arm64.Release/cross/offsets-aarch-linux-gnu.h" --libclang="/__w/1/s/.packages/runtime.linux-x64.microsoft.netcore.runtime.mono.llvm.libclang/19.1.0-alpha.1.24569.1/tools/libclang.so" --libclang-headers="/__w/1/s/.packages/runtime.linux-x64.microsoft.netcore.runtime.mono.llvm.libclang/19.1.0-alpha.1.24569.1/tools/include" --prefix="/crossrootfs/arm64/usr/lib/gcc/aarch64-linux-gnu/5" --sysroot="/crossrootfs/arm64"" exited with code 1.
2024-12-27T08:08:33.1302978Z ##[error]src/mono/mono.proj(1000,5): error MSB3073: (NETCORE_ENGINEERING_TELEMETRY=Build) The command "python3 /__w/1/s/src/mono/mono/offsets/offsets-tool.py --abi=aarch64-linux-gnu --targetdir="/__w/1/s/artifacts/obj/mono/linux.arm64.Release" --monodir="/__w/1/s/src/mono" --nativedir="/__w/1/s/src/native" --outfile="/__w/1/s/artifacts/obj/mono/linux.arm64.Release/cross/offsets-aarch-linux-gnu.h" --libclang="/__w/1/s/.packages/runtime.linux-x64.microsoft.netcore.runtime.mono.llvm.libclang/19.1.0-alpha.1.24569.1/tools/libclang.so" --libclang-headers="/__w/1/s/.packages/runtime.linux-x64.microsoft.netcore.runtime.mono.llvm.libclang/19.1.0-alpha.1.24569.1/tools/include" --prefix="/crossrootfs/arm64/usr/lib/gcc/aarch64-linux-gnu/5" --sysroot="/crossrootfs/arm64"" exited with code 1.

The range of dotnet/runtime commits when the regression was introduced cf4d2b0...57ab984

I wonder if the update to newer build images from #110198 is triggering:

2024-12-27T08:08:33.1176807Z provided path via --prefix ("/crossrootfs/arm64/usr/lib/gcc/aarch64-linux-gnu/5") doesn't exist.

and we might need to change

runtime/src/mono/mono.proj

Line 817 in c969265

<MonoAotOffsetsPrefix Condition="'$(Platform)' == 'arm64'">$(MonoCrossDir)/usr/lib/gcc/aarch64-linux-gnu/5</MonoAotOffsetsPrefix>

.

Last non-failing build: https://dev.azure.com/dnceng/internal/_build/results?buildId=2605380

[dotnet-policy-service]

Tagging subscribers to this area: @akoeplinger, @matouskozak
See info in area-owners.md if you want to be subscribed.

[kotlarmilos]

runtime/src/mono/mono.proj

Line 817 in c969265

<MonoAotOffsetsPrefix Condition="'$(Platform)' == 'arm64'">$(MonoCrossDir)/usr/lib/gcc/aarch64-linux-gnu/5</MonoAotOffsetsPrefix>

Yes, I think this can be removed since gcc-7 is available in the new images.

[matouskozak]

Yes, I think this can be removed since gcc-7 is available in the new images.

Verified on internal perf job and removing it fixes the crash #110965.

Worryingly we didn't catch this before on the dotnet/runtime CI even though we are building cross compilers in e.g., https://dev.azure.com/dnceng-public/public/_build/results?buildId=902899&view=logs&j=89bc6db5-401c-5796-8771-4042af6a0738&t=6778479a-ccf3-52c8-867d-5fb6df451ae3.

@akoeplinger do you know why BuildMonoCross task is only limited to BuildMonoAOTCrossCompiler?

runtime/src/mono/mono.proj

Lines 778 to 780 in 194ad16

	<!-- Build AOT cross compiler (if available) -->
	<Target Name="BuildMonoCross" Condition="'$(BuildMonoAOTCrossCompiler)' == 'true'" DependsOnTargets="BuildMonoRuntime">

I would expect that when BuildMonoAOTCrossCompilerOnly=true, the AOT cross compiler would get build as well or is there something I'm missing...
I'm trying to understand what is the difference between BuildMonoAOTCrossCompiler and BuildMonoAOTCrossCompilerOnly is and when we should use one and when the other.

[akoeplinger]

@matouskozak I think the issue why we didn't see it in that job is because it sets /p:DotNetBuildMonoCrossAOT=true and then we have this:

runtime/Directory.Build.props

Lines 198 to 202 in 2eb59bf

	<PropertyGroup Condition="'$(DotNetBuildMonoCrossAOT)' == 'true'">
	<BuildMonoAOTCrossCompilerOnly>true</BuildMonoAOTCrossCompilerOnly>
	<MonoCrossAOTTargetOS>android+browser+wasi</MonoCrossAOTTargetOS>
	<MonoCrossAOTTargetOS Condition="'$(TargetOS)' == 'osx'">$(MonoCrossAOTTargetOS)+tvos+ios+maccatalyst</MonoCrossAOTTargetOS>
	</PropertyGroup>

That sets the AOT compiler to target the mobile platforms and they use different offsets files and don't run into the issue because they use a different sysroot.