Executables restored from NuGet packages are no longer executable with .NET Core "2.0"

[eerhardt]

Details about Problem

NuGet product used (NuGet.exe | VS UI | Package Manager Console | dotnet.exe):

Erics-MBP:cli eerhardt$ ./.dotnet_stage0/x64/dotnet --info
.NET Command Line Tools (2.0.0-alpha-004622)

Product Information:
 Version:            2.0.0-alpha-004622
 Commit SHA-1 hash:  1f44cabf4a

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.12
 OS Platform: Darwin
 RID:         osx.10.11-x64

Worked before? If so, with which NuGet version: Yes, with dotnet restore version 1.0.0-rc3-004530.

Detailed repro steps so we can see the same problem

1. Install a "2.0" CLI from https://github.com/dotnet/cli/tree/master#installers-and-binaries on a non-Windows machine.

2. Restore a package that contains a tool/executable. For example, Microsoft.NETCore.Runtime.CoreCLR version 1.2.0-beta-24904-03.

3. Try to execute the crossgen executable at .nuget/packages/runtime.osx.10.10-x64.microsoft.netcore.runtime.coreclr/1.2.0-beta-24904-03/tools/crossgen

Expected result:

With "1.0", the executable would run.

Actual result:

15:57:11.104     1>/Users/dotnet-bot/j/workspace/dotnet_cli/master/release_osx_x64_prtest/build/Microsoft.DotNet.Cli.Compile.targets(225,7): error MSB6003: The specified task executable "crossgen" could not be run. Operation not permitted [/Users/dotnet-bot/j/workspace/dotnet_cli/master/release_osx_x64_prtest/build.proj]

Notes

The regression is caused by System.IO.Compression in corefx. See https://github.com/dotnet/corefx/issues/15516. In .NET Core 2.0, the Zip APIs are no longer setting the "x" mod on files unzipped.

This is going to be an issue for all tooling using NuGet to restore executables on non-Windows platforms. We shouldn't expect every place that is executing programs from NuGet packages to have to ensure the file has an "x" permission on it.

[rrelyea]

Any reason this is a shipstopper for rtm?

[eerhardt]

Not a shipstopper. This only affects "2.0". "1.0" works fine.

[eerhardt]

@rrelyea - when VS 2017 is out the door, can we get this issue triaged? This is going to affect a lot of "2.0" scenarios, so I'd like to get it fixed early in "2.0".

[eerhardt]

For example, this is breaking the dotnet cache feature on non-Windows machines in 2.0 because the crossgen executable is not marked with the "x" permission.

@gkhanna79 @ramarag

[eerhardt]

Shell scripts need to be marked executable as well.

/cc @livarcocc

[eerhardt]

@rrelyea - as we are getting more people using .NET Core 2.0, this is becoming more of an issue. See https://github.com/dotnet/cli/issues/6286. Can we get this looked into?

[eerhardt]

/cc @livarcocc

[Eilon]

@rrelyea - this issue blocks ASP.NET Core (and presumably other) apps from running after running the publish operation. Do we have an idea when we might see a fix for this?

[muratg]

👀

[rrelyea]

From thread with NetCore team...i think they are addressing this.
However, Eric said in that thread:
I kind of think this will have to get resolved in NuGet regardless. The reasoning being that we already have a bunch of .nupkgs that were built with 1.0 that don’t have permission information inside of them.

[eerhardt]

From thread with NetCore team...i think they are addressing this.

The NetCore team is not addressing this. It is an intentional breaking change from netcoreapp1.x to netcoreapp2.0 in the System.IO.Compression area.

[tmds]

In 1.x, files created by .NET Core had the x-bit set. In 2.x, files created by .NET Core no longer have the x-bit set. This change is not specific to System.IO.Compression.

A change made in ZipArchive is making the permission bits accessible (dotnet/corefx#18565) but no-one is using those yet.

Packages created so far, don't have the x-bit set for their files. Extracting packages, resulted in the x-bit being set for all files on 1.x and for no files on 2.x. This means executable files will no longer be executable when extracting with 2.x.

When support is added for maintaining permissions when packing and unpacking, there is still an issue for existing packages that don't have the x-bit set and relied on the 1.x behavior to set it.

Most packages contain only dlls and so files, they seem to work fine without the x-bit. Packages containing executables&scripts are affected.

[kichalla]

@tmds Hmm, did you test with the repro scenario I mentioned here? I did clear the nuget cache doing dotnet nuget locals all --clear and followed the repro steps in the link above, but I still see the execute permission not set. Just want to make sure if we both are testing the same scenario.

[tmds]

@kichalla the quest continues. The x-bit is not set on the application. However, it is set in the package folder upon extracting!

$ ls -lah ~/.nuget/packages/runtime.linux-x64.microsoft.netcore.dotnetapphost/2.0.0-preview2-25407-01/runtimes/linux-x64/native/apphost 
-rwxrw-rw-. 1 tmds tmds 96K Jun  7 13:04 /home/tmds/.nuget/packages/runtime.linux-x64.microsoft.netcore.dotnetapphost/2.0.0-preview2-25407-01/runtimes/linux-x64/native/apphost

[tmds]

@kichalla If I followed the breadcrumbs correctly, the execute bit gets lost here: https://github.com/dotnet/sdk/blob/master/src/Tasks/Microsoft.NET.Build.Tasks/EmbedAppNameInHost.cs#L62.

[gkhanna79]

Is this onky happening for standalone scenario' s entryloint executable only? Adding @eerhardt to chime in as well. Thanks, Gaurav

…

-------- Original message -------- From: Tom Deseyn <notifications@github.com> Date: 6/9/17 12:08 PM (GMT-08:00) To: NuGet/Home <Home@noreply.github.com> Cc: "Gaurav Khanna (CLR)" <Gaurav.Khanna@microsoft.com>, Mention <mention@noreply.github.com> Subject: Re: [NuGet/Home] Executables restored from NuGet packages are no longer executable with .NET Core "2.0" (#4424) @kichalla<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkichalla&data=02%7C01%7CGaurav.Khanna%40microsoft.com%7C557ead296d2946b63ac608d4af6add56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636326320893010305&sdata=6zPycgrC5U2emf%2FAL5b4eqTF5lOzygnOl%2FlWRPO9Q0A%3D&reserved=0> If I followed the breadcrumbs correctly, the execute bit gets lost here: https://github.com/dotnet/sdk/blob/master/src/Tasks/Microsoft.NET.Build.Tasks/EmbedAppNameInHost.cs#L62<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fsdk%2Fblob%2Fmaster%2Fsrc%2FTasks%2FMicrosoft.NET.Build.Tasks%2FEmbedAppNameInHost.cs%23L62&data=02%7C01%7CGaurav.Khanna%40microsoft.com%7C557ead296d2946b63ac608d4af6add56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636326320893010305&sdata=%2BMJG0kUCrQYrOluYFWABVfbL2XnT5bqVYcNVW9OPtkM%3D&reserved=0>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNuGet%2FHome%2Fissues%2F4424%23issuecomment-307474563&data=02%7C01%7CGaurav.Khanna%40microsoft.com%7C557ead296d2946b63ac608d4af6add56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636326320893020313&sdata=blMj6MjR39iXzB4N88Dei98JZqCKFFCOxX55Hypqs7c%3D&reserved=0>, or mute the thread<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJA3wWStReHWkMAmPN5Sr1sBhlt25Yvpks5sCZgVgaJpZM4LwlX3&data=02%7C01%7CGaurav.Khanna%40microsoft.com%7C557ead296d2946b63ac608d4af6add56%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636326320893020313&sdata=8izC6ZlJm090nxhYa9y6drs7N8%2BHR7xrExudSmZKwqc%3D&reserved=0>.

[gkhanna79]

If the possible cause identified by @tmds is correct, then this issue has been present for standalone publish in Preview1 as well and fix would be to set the X bit once the modified binary is published.

@steveharter can you PTAL?

[gkhanna79]

@kichalla Can you please file an issue in SDK for your repro above and assign it to @eerhardt and @steveharter?

[gkhanna79]

@steveharter Possbile fix needs to be like https://github.com/dotnet/sdk/blob/ddfd75cacfd5fb94bebc442441c75852ca4b1ee2/test/Microsoft.NET.TestFramework/CommandExtensions.cs#L16.

[tmds]

or alternatively, the apphost can be placed in the publish folder using File.Copy and renamed using File.Move.

[gkhanna79]

Apphost needs to be modified as part of publish operation so simply copying is not going to work.

[tmds]

You can open the copied file

[gkhanna79]

@livarcocc When CLI published a standalone app, does it ever mark its entrypoint executable with the "X" bit on non-Windows?

[steveharter]

Taking a look now

[steveharter]

My understanding:

• Only the modified apphost executable needs the +x; other executables are now properly restored from nuget.
• This only occurs (the x is lost) during publish of a standalone app, when copying the source apphost to the destination\modified apphost

I am still going through the repro steps; clearing the nuget cache and then restore is being very slow...

[gkhanna79]

Yes, that sounds about right @steveharter. Thanks, Gaurav

…

-------- Original message -------- From: Steve Harter <notifications@github.com> Date: 6/9/17 2:37 PM (GMT-08:00) To: NuGet/Home <Home@noreply.github.com> Cc: "Gaurav Khanna (CLR)" <Gaurav.Khanna@microsoft.com>, Mention <mention@noreply.github.com> Subject: Re: [NuGet/Home] Executables restored from NuGet packages are no longer executable with .NET Core "2.0" (#4424) My understanding: * Only the modified apphost executable needs the +x; other executables are now properly restored from nuget. * This only occurs (the x is lost) during publish of a standalone app, when copying the source apphost to the destination\modified apphost I am still going through the repro steps; clearing the nuget cache and then restore is being very slow... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNuGet%2FHome%2Fissues%2F4424%23issuecomment-307505749&data=02%7C01%7CGaurav.Khanna%40microsoft.com%7C52edb22c45164e2ebadb08d4af7fba44%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636326410504448082&sdata=5ChvKqVNNmIjiHrPkR5WGWPGxUAWMIpz8ANuqr%2FVi5Q%3D&reserved=0>, or mute the thread<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJA3wQelNjndMwnP3VtbuWha2Yw2VAVwks5sCbsKgaJpZM4LwlX3&data=02%7C01%7CGaurav.Khanna%40microsoft.com%7C52edb22c45164e2ebadb08d4af7fba44%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636326410504448082&sdata=ABN5C0qZ3hWegSovCaRerK2bvkSti1pFETezgrqT0lo%3D&reserved=0>.

[livarcocc]

If the file is signed in the NuGet package, why is it losing that permission? It seems like these permissions should not be lost when copying/moving the file. And explicitly setting it in the SDK seems like a workaround for an issue somewhere else.

[kichalla]

@kichalla Can you please file an issue in SDK for your repro above and assign it to @eerhardt and @steveharter?

dotnet/sdk#1331

[tmds]

If the file is signed in the NuGet package, why is it losing that permission? It seems like these permissions should not be lost when copying/moving the file.

@livarcocc This is the code which is 'copying' the app host: https://github.com/dotnet/sdk/blob/master/src/Tasks/Microsoft.NET.Build.Tasks/EmbedAppNameInHost.cs#L62. It is creating a new file from bytes in memory.