ARM-CI: Make the Linux ARM emulator check automatic

[prajwal-aithal]

• Previously the Linux ARM emulator build had to be triggered manually
• Making this check automatic based on the stability of emulator build

Signed-off-by: Prajwal A N an.prajwal@samsung.com

[prajwal-aithal]

Merging this PR depends on #5209 and confirmation that the Linux ARM emulator build works correctly.

Dont merge until further notice

[prajwal-aithal]

@jashook @leemgs Please review this. Lets merge this after #5209 and confirming that the Linux ARM emulator build works correctly.

[prajwal-aithal]

@dotnet-bot test Linux ARM emulator build

[prajwal-aithal]

@dotnet-bot test Linux ARM emulator build

[prajwal-aithal]

Update: The previous PRs have not yet resolved the errors in Linux ARM emulator build. Merging this PR will have to wait until the errors are resolved.

[prajwal-aithal]

@dotnet-bot test Linux ARM emulator build

[mkborg]

21:46:42 + sudo chroot /opt/linux-arm-emulator-root/ /bin/bash -x
21:46:43 chroot: failed to run command ‘/bin/bash’: Exec format error

Recently I was experimenting with ARM 'chroot' in 'docker' container and got similar issue. In my case it failed because '/proc/sys/fs/binfmt_misc/' was not mounted.

[prajwal-aithal]

@mkborg I checked this issue on the CI machines. This error was caused as the qemu-arm-static binary was not installed correctly. I have resolved this error, but I dont know if the new emulator rootfs has been activated in the CI machines after that.

@jashook Could you please confirm if the CI machines are using the latest emulator image?

[prajwal-aithal]

@dotnet-bot test Linux ARM emulator build

[prajwal-aithal]

@jashook I have made a change in the timeout of the job (extending from previous 2 hours to 4 hours). I think with this change everything should work well.

This PR also makes the CI check automatic. Can you please review this PR and merge it as soon as possible?

[myungjoo]

LTGM.

Acked-by: MyungJoo Ham

I've been waiting for this long enough :)

[jashook]

@prajwal-aithal can you hoist out the timeout change into its own PR?

Also, the inner dev loop is expected to run well within this timeout by design. Changing the timeout for the jobs that run on PR is not something we are willing to do. In the meantime this is still ok to run post check-in. Why is this build so long? Does it have to be done in the emulated environment? Can we not do cross builds to solve this problem?

/cc @mmitche @RussKeldorph @russellhadley

[prajwal-aithal]

@dotnet-bot test Linux ARM emulator Debug Build

[prajwal-aithal]

@dotnet-bot test Linux ARM emulator Release Build

[prajwal-aithal]

@dotnet-bot test Linux ARM emulator Debug build

[mkborg]

04:16:43 + sudo cp -R ./ /opt/linux-arm-emulator-root/home/coreclr`

I would suggest to use mount -o bind instead of copying CoreCLR:
sudo mount . /opt/linux-arm-emulator-root/home/coreclr -o bind

[mkborg]

04:16:46 + ./build.sh arm clean verbose skipmscorlib

I suppose emulator is mainly for ARM soft FP. Then why arm instead of arm-softfp?

[jashook]

@dotnet-bot test CI Please

[jashook]

@dotnet-bot test Linux ARM emulator Debug build

[prajwal-aithal]

@jashook Going by the name of the job, I think the PR #5786 has not yet been included in the CI system (The name of the job is supposed to look like Linux ARM Emulator Cross {configuration} Build). When can we test this job again including the changes made by PR #5786 ?

[jashook]

@dotnet-bot test Linux ARM emulator Debug build

[jashook]

@dotnet-bot test Linux ARM emulator Release build

[prajwal-aithal]

@jashook The builds are failing. However, the errors seem to be caused by the CI system rather than the emulator build itself. Could you please look at the errors?

[jashook]

@prajwal-aithal I have found the problem, working on it now.

[jashook]

@dotnet-bot test Linux ARM emulator Release build

[prajwal-aithal]

@jashook The build is failing with the following error

17:29:49 + sudo mount /opt/linux-arm-emulator/platform/rootfs-t30.ext4 /opt/linux-arm-emulator-root
17:29:49 mount: mount point /opt/linux-arm-emulator-root does not exist

I remember creating this directory (/opt/linux-arm-emulator-root/) on the CI machine. Is a new image being used in the CI system? If yes, could you please create the directory (mkdir /opt/linux-arm-emulator-root) in the new image?

[jashook]

@dotnet-bot test Linux ARM emulator Release build

[jashook]

@dotnet-bot test Linux ARM emulator Debug build

[jashook]

@prajwal-aithal looks like the problem still exists with the bash script. I am unable to reproduce that error manually on the machine or through the script environment in jenkins.

[jashook]

@mmitche PTAL

[prajwal-aithal]

@jashook That error is caused by the absence of the the binutils-arm-linux-gnueabi package on the Ubuntu machine. I did install this on the CI machine that we were working on. Should this be done on all machines (or is the same image used in all CI machines)?

[jashook]

@prajwal-aithal I see, we will need to capture that vm and use it in the ci system.

[prajwal-aithal]

@jashook Ohh, can this be done soon or would this take a lot of time?

[jashook]

@prajwal-aithal hopefully soon, working on it now.

[prajwal-aithal]

@jashook Thank you. Looking forward to see the result of the cross build on the updated VM.

[jashook]

@prajwal-aithal the CI machine change has been merged. It should take anywhere between 5 to 25 min to take effect.

[prajwal-aithal]

@dotnet-bot test Linux ARM Emulator Cross Debug Build

[prajwal-aithal]

@dotnet-bot test Linux ARM emulator Debug build

[prajwal-aithal]

@dotnet-bot test Linux ARM emulator Release build

[prajwal-aithal]

@jashook The builds succeeded but the final step of archiving the results is failing due to incorrect path. I have initiated PR #5941 to resolve this error.

[jashook]

@dotnet-bot test Linux ARM emulator Release build

[jashook]

@dotnet-bot test Linux ARM emulator Debug build

[prajwal-aithal]

@jashook The CI system has not yet been updated with the changes of PR #5941. I think we will have to wait for some time and try the emulator builds again.

[prajwal-aithal]

@jashook There is typo in the result archive path (/bin/Product/** against the correct path bin/Product/**)

I have initiated PR #5942 to resolve the same. I am very sorry for this duplication of effort. Could you please review that PR and merge it. I will trigger the final (hopefully) tests around 20 minutes after that PR is merged.

[prajwal-aithal]

@dotnet-bot test CI please

[jashook]

@dotnet-bot test Linux ARM emulator Debug build

[prajwal-aithal]

@dotnet-bot test Linux ARM emulator Release build

[prajwal-aithal]

@jashook Both tests have passed successfully.

Thanks a lot for your efforts in integrating the Linux ARM emulator checks into the CI systems.

[prajwal-aithal]

@jashook Can we make the Linux ARM emulator build tests automatic on a per PR basis by merging this PR? (CI has been tested already)

[jashook]

@prajwal-aithal lgtm. Thank you for the work. We still have an outstanding problem that the existing arm32 ubuntu cross build does not exist. I believe this is because the name of the arm build is used for both the emulator and the cross build. Either way, this can be addressed later.