membarrier + C# ASP.NET Core Container issue

[razfriman]

I have run into an issue with gvisor while trying to deploy a containerized C# ASP.NET Core application to Google Cloud Run.

I have used the default .NET Core 3 templates and dockerized the application. It works locally via docker/docker-compose.

When attempting to deploy to Google Cloud Run, the application deploys and seems to start successfully according to the application logs, however, the application fails to report healthy and eventually fails the deployment.

Here is logs from the Google Log Viewer:

{
 insertId: "5dxxxxxxxxxxxxxxxxbf6a"  
 labels: {
  instanceId: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"   
 }
 logName: "projects/web/logs/run.googleapis.com%2Fvarlog%2Fsystem"  
 receiveTimestamp: "2019-10-17T21:14:40.257442779Z"  
 resource: {
  labels: {…}   
  type: "cloud_run_revision"   
 }
 severity: "DEBUG"  
 textPayload: "Container Sandbox Limitation: Unsupported syscall membarrier(0x0,0x0,0xc,0x0,0xc,0x1). Please, refer to https://gvisor.dev/c/linux/amd64/membarrier for more information."  
 timestamp: "2019-10-17T21:14:40.245710214Z"  
}

I have tried to compile the container with various flags and fail with the same issue everytime.

I assume this is a gvisor issue account to the error message. Do you have any suggestions?

---

TODO: Figure out how to get real debug logs.

For all other issues, please attach debug logs. To get debug logs, follow the
instructions here: https://gvisor.dev/docs/user_guide/debugging/

Other useful information to include is:

• runsc -v
• docker version or docker info if more relevant
• uname -a - git describe
• Detailed reproduction steps

[razfriman]

It seems like it is saying that membarrier is not implemented yet - Which I see a related issue here: #267

[prattmic]

Hi, thanks for the report. I've got a few more questions.

Could you link to an example Docker image / Dockerfile that has this problem? You've described it, but being unfamiliar with .NET applications I'm not sure I'd create one correctly.

Also, what is the behavior you actually see? Are there additional relevant log messages? i.e., is the application crashing, failing to respond to requests, etc?

If possible, could you try running the container under gVisor locally (https://gvisor.dev/docs/user_guide/docker/) to see if the issue reproduces there?

Thanks!

[razfriman]

Sure:

I have created a sample project that attempts to reproduce the issue with minimal amount of code.
GitHub Repository: https://github.com/razfriman/csharp-cloud-run
Hosted Docker Image: https://hub.docker.com/r/razfriman/cloud-run-web-app
Note: I created this sample app using the .NET CLI (dotnet new webapp) and added a Dockerfile.

Then I created the Cloud Run using the above docker image by adding it to my current projects GCR registry.

When I run the same container in GCE, everything works as expected. The application seems like it is running correctly, but gvisor is mentioning a message about membarrier and it never marks the container as healthy.

Logs:

2019-10-18T17:51:04.787Z Cloud Run CreateService csharp-cloud-run raz@gmail.com {"@type":"type.googleapis.com/google.cloud.audit.AuditLog","authenticationInfo":{"principalEmail":"raz@gmail.com"},"requestMetadata":{"callerIp":"111.220.68.158","callerSuppliedUserAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865… I  Cloud Run CreateService csharp-cloud-run raz@gmail.com
2019-10-18T17:51:39.476936827Z Container Sandbox Limitation: Unsupported syscall membarrier(0x0,0x0,0x41,0x1,0x800000,0x1). Please, refer to https://gvisor.dev/c/linux/amd64/membarrier for more information. D 
2019-10-18T17:51:42.952902Z �[40m�[1m�[33mwarn�[39m�[22m�[49m: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60] A 
2019-10-18T17:51:42.953213Z       Storing keys in a directory '/home/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed. A 
2019-10-18T17:51:43.142357Z �[40m�[1m�[33mwarn�[39m�[22m�[49m: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35] A 
2019-10-18T17:51:43.142387Z       No XML encryptor configured. Key {4f315068-50c0-4063-a839-4d181277ece7} may be persisted to storage in unencrypted form. A 
2019-10-18T17:51:43.615818Z �[40m�[32minfo�[39m�[22m�[49m: Microsoft.Hosting.Lifetime[0] A 
2019-10-18T17:51:43.615867Z       Now listening on: http://[::]:80 A 
2019-10-18T17:51:43.617115Z �[40m�[32minfo�[39m�[22m�[49m: Microsoft.Hosting.Lifetime[0] A 
2019-10-18T17:51:43.617146Z       Application started. Press Ctrl+C to shut down. A 
2019-10-18T17:51:43.617156Z �[40m�[32minfo�[39m�[22m�[49m: Microsoft.Hosting.Lifetime[0] A 
2019-10-18T17:51:43.617164Z       Hosting environment: Production A 
2019-10-18T17:51:43.617172Z �[40m�[32minfo�[39m�[22m�[49m: Microsoft.Hosting.Lifetime[0] A 
2019-10-18T17:51:43.617181Z       Content root path: /app A 

[razfriman]

I can attempt to run gvisor locally, but I do not have a machine with Linux so that may take some time

[razfriman]

@prattmic - It turns out that Cloud Run needed some extra configuration that most .NET apps do not require.

The warning from gvisor was a red herring as I still see it with the corrected configuration, however, the application starts up as expected.

Reference: https://stackoverflow.com/questions/55663172/docker-container-upload-to-gcp-cloud-run-with-core-web-api-app-not-working

Closing