[ARM32] qemu: Unsupported syscall: 389 when running console app in Docker

[MichaelSimons]

While testing the new arm32v7 Docker images for .NET Core I'm seeing qemu: Unsupported syscall: 389 written to the output whenever I run a console application in an arm Docker container on a Windows 10 machine. I don't see this if I run the same Docker image on a physical arm32 device (e.g. raspberry pi).

Steps to reproduce

0. From a Windows 10 machine with Docker installed (Docker version 17.06.0-ce, build 02c1d87)
1. Create a new console app - dotnet new console
2. Add the following Dockerfile to your project

FROM microsoft/dotnet-nightly:2.0-sdk AS build-env
WORKDIR /app

# copy csproj and restore as distinct layers
COPY *.csproj ./
RUN dotnet restore

# copy everything else and build
COPY . ./
RUN dotnet publish -c Release -o out

# build runtime image
FROM msimons/dotnet-nightly:2-runtime-stretch-arm32v7
WORKDIR /app
COPY --from=build-env /app/out ./
ENTRYPOINT ["dotnet", "arm.dll"]

Note: The Dockerfile for msimons/dotnet-nightly:2-runtime-stretch-arm32v7 is contained in dotnet/dotnet-docker-nightly#328
3. Run the following from your project dir docker build -t armapp .
4. Run the image docker run --rm armapp

Expected Output of App

Hello World!

Output of App

qemu: Unsupported syscall: 389
Hello World!

Notes

1. The app's return code is 0
2. I tried running the image with the --privileged and --security-opt="seccomp=unconfined" options but see no difference in behavior.

[BruceForstall]

@dotnet/arm32-contrib @janvorli

[janvorli]

The syscall 389 on ARM32 seems to be the membarrier. According to the docs, it was added in Linux 4.11. dotnet core doesn't use it, so I assume one of our 3rd party dependencies try to call it to see if it is present. So I would assume the message is benign.

[mikedn]

Looks like this happens in CI too: https://ci.dot.net/job/dotnet_coreclr/job/master/job/arm_cross_debug_ubuntu16.04_prtest/4680/

FAILED   - JIT/Directed/coverage/importer/ldelemnullarr2/ldelemnullarr2.sh
               BEGIN EXECUTION
               /home/coreclr/bin/tests/Windows_NT.x64.Debug/Tests/coreoverlay/corerun ldelemnullarr2.exe
               qemu: Unsupported syscall: 389
               qemu: uncaught target signal 11 (Segmentation fault) - core dumped
               ./ldelemnullarr2.sh: line 243:   169 Segmentation fault      (core dumped) $_DebuggerFullPath "$CORE_ROOT/corerun" $ExePath $CLRTestExecutionArguments
               Expected: 100
               Actual: 139
               END EXECUTION - FAILED

[Tragetaschen]

membarrier has been in the ARM kernel since Linux 4.3: torvalds/linux@208473c

It looks like some tools like qemu (or in my case valgrind) don't yet have support for that.

[okodron]

Is it possible to use the latest version of ARM qemu with membarrier() support?

[davmason]

@RussKeldorph I am seeing this issue repeatedly in dotnet/coreclr#17142, it is blocking me from having a clean CI.

What is the guidance here? I have reset it 3 times and each time it takes hours to run and then fails. Should I start ignoring the armrel legs of the CI?

[RussKeldorph]

@davmason dotnet/coreclr#17028 is possibly more relevant. Yes, for now you can ignore these failures. I hope to have the job out of the default trigger soon since there is no outlook for a fix.

[BruceForstall]

Looks like dotnet/dotnet-docker-nightly#328 despite causing QEMU failures?

[MichaelSimons]

@janvorli - I am seeing a more severe variant of this with the new ARM SDK builds.

Steps to reproduce

1. Using Docker on a Windows 10 machine running in Linux container mode or a macOS.
2. Build the following Dockerfile - docker build -t dotnet-arm-sdk .

FROM arm32v7/buildpack-deps:stretch-scm

# Install .NET CLI dependencies
RUN apt-get update \
    && apt-get install -y --no-install-recommends \
        libc6 \
        libcurl3 \
        libgcc1 \
        libgssapi-krb5-2 \
        libicu57 \
        liblttng-ust0 \
        libssl1.0.2 \
        libstdc++6 \
        libunwind8 \
        zlib1g \
    && rm -rf /var/lib/apt/lists/*

# Install .NET Core SDK
ENV DOTNET_SDK_VERSION 2.1.300-preview2-008380

RUN curl -SL --output dotnet.tar.gz https://dotnetcli.blob.core.windows.net/dotnet/Sdk/$DOTNET_SDK_VERSION/dotnet-sdk-$DOTNET_SDK_VERSION-linux-arm.tar.gz \
    && dotnet_sha512='3f8276cd982f309236d5cac6656c9364cb190d1cb1ac0474acd67e16f35b8bb5d66fcb220342f0edb80c548a073d5e7f7330ceeeebdc88bceb8ea4836cd1abc6' \
    && echo "$dotnet_sha512 dotnet.tar.gz" | sha512sum -c - \
    && mkdir -p /usr/share/dotnet \
    && tar -zxf dotnet.tar.gz -C /usr/share/dotnet \
    && rm dotnet.tar.gz \
    && ln -s /usr/share/dotnet/dotnet /usr/bin/dotnet

# Configure Kestrel web server to bind to port 80 when present
ENV ASPNETCORE_URLS=http://+:80 \
    # Enable detection of running in a container
    DOTNET_RUNNING_IN_CONTAINER=true \
    # Enable correct mode for dotnet watch (only mode supported in a container)
    DOTNET_USE_POLLING_FILE_WATCHER=true \
    # Skip extraction of XML docs - generally not useful within an image/container - helps perfomance
    NUGET_XMLDOC_MODE=skip

# Trigger first run experience by running arbitrary cmd to populate local package cache
RUN dotnet help

Results:

...
Step 10/10 : RUN dotnet help
 ---> Running in ae71e039abc8
qemu: Unsupported syscall: 389

Welcome to .NET Core!
---------------------
Learn more about .NET Core: https://aka.ms/dotnet-docs
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli-docs

Telemetry
---------
The .NET Core tools collect usage data in order to help us improve your experience. The data is anonymous and doesn't include command-line arguments. The data is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.

Read more about .NET Core CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry
qemu: Unsupported syscall: 345

ASP.NET Core
------------
Successfully installed the ASP.NET Core HTTPS Development Certificate.
To trust the certificate (Windows and macOS only) first install the dev-certs tool by running 'dotnet install tool dotnet-dev-certs -g --version 2.1.0-preview1-final' and then run 'dotnet-dev-certs https --trust'.
For more information on configuring HTTPS see https://go.microsoft.com/fwlink/?linkid=848054.
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault (core dumped)
The command '/bin/sh -c dotnet help' returned a non-zero code: 139

You can also reproduce this on Linux when using qemu but the setup is more complex.

[northtyphoon]

I hit the issue when trying to build an image based on microsoft/dotnet:2.1-runtime-stretch-slim-arm32v in Ubuntu 18.04 amd64 (qemu). Any suggestion?

https://github.com/Azure/dotnet-template-azure-iot-edge-module/blob/master/content/dotnet-template-azure-iot-edge-module/CSharp/Dockerfile.arm32v7

[Jacobh2]

Also seeing this issue on OSX 10.13.6 using image microsoft/dotnet:2.1-sdk-stretch-arm32v7 on step RUN dotnet restore -nowarn:msb3202,nu1503 -p:RestoreUseSkipNonexistentTargets=false. It gives the following output:

Step 9/18 : RUN dotnet restore -nowarn:msb3202,nu1503 -p:RestoreUseSkipNonexistentTargets=false
 ---> Running in 4cf6b676bcf0
qemu: Unsupported syscall: 389
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault

[eshvatskyi]

Same error :(

[janvorli]

@eshvatskyi which runtime version have you tried it with? We have fixed a bunch of ARM issues in 2.1.5.

[eshvatskyi]

@eshvatskyi which runtime version have you tried it with? We have fixed a bunch of ARM issues in 2.1.5.

FROM microsoft/dotnet:2.1-sdk-stretch-arm32v7
...
RUN dotnet restore -r linux-arm

[stephenpope]

Also seeing this issue for microsoft/dotnet:2.1.500-sdk-bionic-arm32v7 + microsoft/dotnet:2.1.500-sdk-stretch-arm32v7

FROM microsoft/dotnet:2.1.500-sdk-bionic-arm32v7
...
RUN dotnet restore

Error :

Step 5/21 : RUN dotnet restore
 ---> Running in 657b266d8fb2
qemu: Unsupported syscall: 389

Extra info:

Client: Docker Engine - Community
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:47:51 2018
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:55:00 2018
  OS/Arch:          linux/amd64
  Experimental:     false

[eshvatskyi]

Any news?

[Dekorg]

same ARG REPO=mcr.microsoft.com/dotnet/core/runtime-deps
FROM $REPO:2.2-bionic-arm32v7

RUN apt-get update
&& apt-get install -y --no-install-recommends
curl
&& rm -rf /var/lib/apt/lists/*

Install ASP.NET Core

ENV ASPNETCORE_VERSION 2.2.2

RUN curl -SL --output aspnetcore.tar.gz https://dotnetcli.blob.core.windows.net/dotnet/aspnetcore/Runtime/$ASPNETCORE_VERSION/aspnetcore-runtime-$ASPNETCORE_VERSION-linux-arm.tar.gz
&& aspnetcore_sha512='8ea208b141841df06b614fe2f2b68d4753775aa6cdce9d35a7870ec513a063ef6a6696724c28503284ba8d173eed65fa5355be301dd9d4f7509cdbad93cb4da7'
&& echo "$aspnetcore_sha512 aspnetcore.tar.gz" | sha512sum -c -
&& mkdir -p /usr/share/dotnet
&& tar -zxf aspnetcore.tar.gz -C /usr/share/dotnet
&& rm aspnetcore.tar.gz
&& ln -s /usr/share/dotnet/dotnet /usr/bin/dotnet

[qJake]

This is still an issue with mcr.microsoft.com/dotnet/core/aspnet:3.0-buster-slim-arm32v7 (Preview 9) and Qemu (building the container on Windows with Linux Docker containers).

For me, the issue actually happens on dotnet restore of trying to build my project for ARM32.

Strangely enough, this is ONLY an issue on ARM32v7, the same build runs perfectly on mcr.microsoft.com/dotnet/core/aspnet:3.0-buster-slim-arm64v8 with only the image tags changed (FROM ... as base and FROM ... as build).

[MihaMarkic]

Looks like the same as dotnet/dotnet-docker#1512. Still present in 3.1 containers :(

[jkotas]

As @okodron suggested, try upgrading to latest qemu that should have the syscall implemented.

We do not test or support .NET Core on qemu. We have tried using qemu to run tests in CI for a while, but we have found quemu to be too unstable for what .NET Core needs.

If you are using docker just for building, .NET Core 3.1 tooling supports cross-compilation that allows dotnet publish linux arm32 binaries from linux x64.

[MihaMarkic]

@jkotas So, what you are saying is to build within x86/x64 container, target linux_arm and just copy published files to correct arm based image? That should work, thanks.