Signing: enable signed package verification on Linux by default in .NET 6 SDK

[dtivel]

Bug

Fixes: NuGet/Home#11264, NuGet/Home#11263

Regression? Last working version: N/A

Description

This change enables signed package verification by default on Linux in .NET 6 SDK (6.0.4xx). This PR a re-do of #4585 by targeting .NET 6 SDK instead of .NET 7 SDK.

Windows

No changes. Signed package verification continues to be enabled.

macOS

Signed package verification is disabled by default in .NET 5 SDK onwards; however, currently users can opt in by setting the DOTNET_NUGET_SIGNATURE_VERIFICATION environment variable to TRUE since #3986. In .NET 6 SDK, opting in will leverage a fallback certificate bundle as the X.509 trust store (versus the system's X.509 trust store). This trust store is a subset of the Windows CTL that is valid for both code signing and timestamping and will be distributed as part of the .NET 6 SDK. (The fallback certificate bundle was added to the dotnet/sdk repo via dotnet/sdk#25687.)

We have a couple concerns on macOS for which we don't currently have a great story, so signed package verification remains disabled by default on macOS:

1. Apple recently explicitly distrusted some certificate chains. Because every package on NuGet.org has at least a timestamp from a deprecated Symantec timestamping service, verification would fail for these packages on macOS.
2. Apple removed CRL support in macOS 10.15, which means that CRL-based revocation checks will not work on macOS.

With this change, if a user chooses to opt in to signed package verification in the .NET 6 SDK despite these concerns, it means that NuGet will use the new fallback certificate bundle and not the system's X.509 trust store. Issues 1 & 2 would still apply and would prevent affected packages from installing successfully.

Linux

With this change, signed package verification will be enabled by default starting with .NET 6 SDK, but users can opt out by setting the DOTNET_NUGET_SIGNATURE_VERIFICATION environment variable to FALSE. When signed package verification is enabled, NuGet will prefer a system-provided certificate bundle valid for both code signing and timestamping, if available; otherwise, NuGet will use the fallback certificate bundle in the .NET 6 SDK. Details can be found here.

The ability to opt out of signed package verification in the .NET 6 SDK on Linux is an escape hatch.

Testing

Existing tests (plus newly added ones) pass, though a handful which no longer make sense with the new default were removed on Linux. I've also done manual testing using a .NET 6 SDK patched with NuGet CI assemblies.

Linux

On Linux (Ubuntu 20.04.4 LTS), I tested using a patched .NET 6 SDK (6.0.400-preview.22322.20). Restoring signed packages with increased verbosity yields the following additional log message:

X.509 certificate chain validation will use the fallback certificate bundle at '/home/orange/testsdk/sdk/6.0.400-preview.22322.20/trustedroots/codesignctl.pem'.

macOS

On macOS 12.4 (21F79), I tested using a patched .NET 6 SDK (6.0.400-preview.22322.20). By default, signature verification is skipped and restoring signed packages succeeds.

On macOS, opting in to signed package verification causes verification to fail (as expected):

detailed restore output

Determining projects to restore...
/Users/dtivel/Desktop/test/test.csproj : warning NU3028: Package 'NuGet.Packaging 6.2.1' from source 'https://api.nuget.org/v3/index.json': The author primary signature's timestamp found a chain building issue: ExplicitDistrust: The trust setting for this policy was set to Deny.
/Users/dtivel/Desktop/test/test.csproj : warning NU3028: Package 'NuGet.Common 6.2.1' from source 'https://api.nuget.org/v3/index.json': The author primary signature's timestamp found a chain building issue: ExplicitDistrust: The trust setting for this policy was set to Deny.
/Users/dtivel/Desktop/test/test.csproj : warning NU3028: Package 'NuGet.Frameworks 6.2.1' from source 'https://api.nuget.org/v3/index.json': The author primary signature's timestamp found a chain building issue: ExplicitDistrust: The trust setting for this policy was set to Deny.
/Users/dtivel/Desktop/test/test.csproj : warning NU3028: Package 'NuGet.Versioning 6.2.1' from source 'https://api.nuget.org/v3/index.json': The author primary signature's timestamp found a chain building issue: ExplicitDistrust: The trust setting for this policy was set to Deny.
/Users/dtivel/Desktop/test/test.csproj : warning NU3028: Package 'NuGet.Configuration 6.2.1' from source 'https://api.nuget.org/v3/index.json': The author primary signature's timestamp found a chain building issue: ExplicitDistrust: The trust setting for this policy was set to Deny.
Package 'System.Security.Cryptography.Cng 5.0.0' from source 'https://api.nuget.org/v3/index.json': 
Signature type: Author
Package 'System.Security.Cryptography.Cng 5.0.0' from source 'https://api.nuget.org/v3/index.json':   Subject Name: CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
Package 'System.Security.Cryptography.Cng 5.0.0' from source 'https://api.nuget.org/v3/index.json':   SHA256 hash: 3F9001EA83C560D712C24CF213C3D312CB3BFF51EE89435D3430BD06B5D0EECE
Package 'System.Security.Cryptography.Cng 5.0.0' from source 'https://api.nuget.org/v3/index.json':   Valid from: 2/25/2018 4:00:00 PM to 1/27/2021 4:00:00 AM
/Users/dtivel/Desktop/test/test.csproj : error NU3037: Package 'System.Security.Cryptography.Cng 5.0.0' from source 'https://api.nuget.org/v3/index.json': The author primary signature validity period has expired.
/Users/dtivel/Desktop/test/test.csproj : warning NU3028: Package 'System.Security.Cryptography.Cng 5.0.0' from source 'https://api.nuget.org/v3/index.json': The author primary signature's timestamp found a chain building issue: ExplicitDistrust: The trust setting for this policy was set to Deny.
Package 'System.Security.Cryptography.Cng 5.0.0' from source 'https://api.nuget.org/v3/index.json': 
Signature type: Repository
Package 'System.Security.Cryptography.Cng 5.0.0' from source 'https://api.nuget.org/v3/index.json':   Subject Name: CN=NuGet.org Repository by Microsoft, O=NuGet.org Repository by Microsoft, L=Redmond, S=Washington, C=US
Package 'System.Security.Cryptography.Cng 5.0.0' from source 'https://api.nuget.org/v3/index.json':   SHA256 hash: 0E5F38F57DC1BCC806D8494F4F90FBCEDD988B46760709CBEEC6F4219AA6157D
Package 'System.Security.Cryptography.Cng 5.0.0' from source 'https://api.nuget.org/v3/index.json':   Valid from: 4/9/2018 5:00:00 PM to 4/14/2021 5:00:00 AM
/Users/dtivel/Desktop/test/test.csproj : error NU3037: Package 'System.Security.Cryptography.Cng 5.0.0' from source 'https://api.nuget.org/v3/index.json': The repository countersignature validity period has expired.
/Users/dtivel/Desktop/test/test.csproj : warning NU3028: Package 'System.Security.Cryptography.Cng 5.0.0' from source 'https://api.nuget.org/v3/index.json': The repository countersignature's timestamp found a chain building issue: ExplicitDistrust: The trust setting for this policy was set to Deny.
Package 'System.Security.Cryptography.Pkcs 5.0.0' from source 'https://api.nuget.org/v3/index.json': 
Signature type: Author
Package 'System.Security.Cryptography.Pkcs 5.0.0' from source 'https://api.nuget.org/v3/index.json':   Subject Name: CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
Package 'System.Security.Cryptography.Pkcs 5.0.0' from source 'https://api.nuget.org/v3/index.json':   SHA256 hash: 3F9001EA83C560D712C24CF213C3D312CB3BFF51EE89435D3430BD06B5D0EECE
Package 'System.Security.Cryptography.Pkcs 5.0.0' from source 'https://api.nuget.org/v3/index.json':   Valid from: 2/25/2018 4:00:00 PM to 1/27/2021 4:00:00 AM
/Users/dtivel/Desktop/test/test.csproj : error NU3037: Package 'System.Security.Cryptography.Pkcs 5.0.0' from source 'https://api.nuget.org/v3/index.json': The author primary signature validity period has expired.
/Users/dtivel/Desktop/test/test.csproj : warning NU3028: Package 'System.Security.Cryptography.Pkcs 5.0.0' from source 'https://api.nuget.org/v3/index.json': The author primary signature's timestamp found a chain building issue: ExplicitDistrust: The trust setting for this policy was set to Deny.
Package 'System.Security.Cryptography.Pkcs 5.0.0' from source 'https://api.nuget.org/v3/index.json': 
Signature type: Repository
Package 'System.Security.Cryptography.Pkcs 5.0.0' from source 'https://api.nuget.org/v3/index.json':   Subject Name: CN=NuGet.org Repository by Microsoft, O=NuGet.org Repository by Microsoft, L=Redmond, S=Washington, C=US
Package 'System.Security.Cryptography.Pkcs 5.0.0' from source 'https://api.nuget.org/v3/index.json':   SHA256 hash: 0E5F38F57DC1BCC806D8494F4F90FBCEDD988B46760709CBEEC6F4219AA6157D
Package 'System.Security.Cryptography.Pkcs 5.0.0' from source 'https://api.nuget.org/v3/index.json':   Valid from: 4/9/2018 5:00:00 PM to 4/14/2021 5:00:00 AM
/Users/dtivel/Desktop/test/test.csproj : error NU3037: Package 'System.Security.Cryptography.Pkcs 5.0.0' from source 'https://api.nuget.org/v3/index.json': The repository countersignature validity period has expired.
/Users/dtivel/Desktop/test/test.csproj : warning NU3028: Package 'System.Security.Cryptography.Pkcs 5.0.0' from source 'https://api.nuget.org/v3/index.json': The repository countersignature's timestamp found a chain building issue: ExplicitDistrust: The trust setting for this policy was set to Deny.
Package 'System.Formats.Asn1 5.0.0' from source 'https://api.nuget.org/v3/index.json': 
Signature type: Author
Package 'System.Formats.Asn1 5.0.0' from source 'https://api.nuget.org/v3/index.json':   Subject Name: CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US
Package 'System.Formats.Asn1 5.0.0' from source 'https://api.nuget.org/v3/index.json':   SHA256 hash: 3F9001EA83C560D712C24CF213C3D312CB3BFF51EE89435D3430BD06B5D0EECE
Package 'System.Formats.Asn1 5.0.0' from source 'https://api.nuget.org/v3/index.json':   Valid from: 2/25/2018 4:00:00 PM to 1/27/2021 4:00:00 AM
/Users/dtivel/Desktop/test/test.csproj : error NU3037: Package 'System.Formats.Asn1 5.0.0' from source 'https://api.nuget.org/v3/index.json': The author primary signature validity period has expired.
/Users/dtivel/Desktop/test/test.csproj : warning NU3028: Package 'System.Formats.Asn1 5.0.0' from source 'https://api.nuget.org/v3/index.json': The author primary signature's timestamp found a chain building issue: ExplicitDistrust: The trust setting for this policy was set to Deny.
Package 'System.Formats.Asn1 5.0.0' from source 'https://api.nuget.org/v3/index.json': 
Signature type: Repository
Package 'System.Formats.Asn1 5.0.0' from source 'https://api.nuget.org/v3/index.json':   Subject Name: CN=NuGet.org Repository by Microsoft, O=NuGet.org Repository by Microsoft, L=Redmond, S=Washington, C=US
Package 'System.Formats.Asn1 5.0.0' from source 'https://api.nuget.org/v3/index.json':   SHA256 hash: 0E5F38F57DC1BCC806D8494F4F90FBCEDD988B46760709CBEEC6F4219AA6157D
Package 'System.Formats.Asn1 5.0.0' from source 'https://api.nuget.org/v3/index.json':   Valid from: 4/9/2018 5:00:00 PM to 4/14/2021 5:00:00 AM
/Users/dtivel/Desktop/test/test.csproj : error NU3037: Package 'System.Formats.Asn1 5.0.0' from source 'https://api.nuget.org/v3/index.json': The repository countersignature validity period has expired.
/Users/dtivel/Desktop/test/test.csproj : warning NU3028: Package 'System.Formats.Asn1 5.0.0' from source 'https://api.nuget.org/v3/index.json': The repository countersignature's timestamp found a chain building issue: ExplicitDistrust: The trust setting for this policy was set to Deny.
Package 'System.Security.Cryptography.ProtectedData 4.4.0' from source 'https://api.nuget.org/v3/index.json': 
Signature type: Repository
Package 'System.Security.Cryptography.ProtectedData 4.4.0' from source 'https://api.nuget.org/v3/index.json':   Subject Name: CN=NuGet.org Repository by Microsoft, O=NuGet.org Repository by Microsoft, L=Redmond, S=Washington, C=US
Package 'System.Security.Cryptography.ProtectedData 4.4.0' from source 'https://api.nuget.org/v3/index.json':   SHA256 hash: 0E5F38F57DC1BCC806D8494F4F90FBCEDD988B46760709CBEEC6F4219AA6157D
Package 'System.Security.Cryptography.ProtectedData 4.4.0' from source 'https://api.nuget.org/v3/index.json':   Valid from: 4/9/2018 5:00:00 PM to 4/14/2021 5:00:00 AM
/Users/dtivel/Desktop/test/test.csproj : error NU3037: Package 'System.Security.Cryptography.ProtectedData 4.4.0' from source 'https://api.nuget.org/v3/index.json': The repository primary signature validity period has expired.
/Users/dtivel/Desktop/test/test.csproj : warning NU3028: Package 'System.Security.Cryptography.ProtectedData 4.4.0' from source 'https://api.nuget.org/v3/index.json': The repository primary signature's timestamp found a chain building issue: ExplicitDistrust: The trust setting for this policy was set to Deny.
Failed to restore /Users/dtivel/Desktop/test/test.csproj (in 1.43 sec).

A lot of test infrastructure changes were required to make this work. In particular, the TrustedTestCert<T> class, which was hard-coded to add/remove certificates to the system trust store, needed to be reworked to enable adding/removing certificates to a test-only, in-memory fallback certificate bundle. This approach is pretty cool and if used more broadly, could eliminate the need for elevation to modify system trust stores on test machines. Overall, this would improve speed and robustness of signing tests on all platforms. This PR doesn't do that though.

Also, .NET SDK tests on non-Windows machines need to both modify in-memory trusted roots in order to generate test data (i.e.: a package signed with a test certificate) AND the on-disk trusted roots (in the test .NET SDK) in order to respect those same test roots when running in a separate process.

A new test fixture (X509TrustTestFixture), when added to a test class/collection, automatically enables in-memory fallback certificate bundle support on non-Windows platforms.

Specifications

The specifications for this change are:

• Re-enabling signed NuGet package verification
• Re-enabling Cross-Platform NuGet Signed Package Verification Technical Specification

PR Checklist

• PR has a meaningful title

• PR has a linked issue.

• Described changes

• Tests

  • Automated tests added
  • OR
  • Test exception
  • OR
  • N/A

• Documentation

  • Documentation PR or issue filled: https://github.com/NuGet/docs.microsoft.com-nuget/issues/2794
  • OR
  • N/A

CC @richlander

[jeffkl]

LGTM!

[richlander]

X.509 certificate chain validation will use the fallback certificate bundle at '/home/orange/testsdk/sdk/6.0.400-preview.22322.20/trustedroots/codesignctl.pem'.

Good call. I never thought about that. If a non-fallback location is used, will there be a similar message?

Opting in to signed package verification causes verification to fail (as expected):

It would be good to reinforce that this is on macOS. It might be good to do the same thing ahead of that on Linux. That would then present a nice contrast. I was initially confused about all the failures and thought it was some weird set of test packages but then saw that it wasn't, and then realized what was intended.

[dtivel]

@richlander, I updated the comment, hopefully clarifying matters.

These messages will be logged with verbose logging level. The 3 columns on the right show which operating systems you might see the messages on.

Message	Windows	Linux	macOS
X.509 certificate chain validation will use the default trust store selected by .NET.	Y	N	N
X.509 certificate chain validation will use the fallback certificate bundle at '<file path>'.	N	Y	Y
X.509 certificate chain validation will use the system certificate bundle at '<file path>'.	N	Y	N
X.509 certificate chain validation will not have any trusted roots. Chain building will fail with an untrusted status.	N	Y	Y

The last message would happen if, say, the fallback certificate bundle is missing or unreadable.

[richlander]

The chart helps. Thanks and LGTM.

[nkolev92]

👏

[heng-liu]

LGTM.

[loic-sharma]

@dtivel is there a plan for revocation checks on macOS? Will nuget.org require that all certificates have OCSP responders?

[richlander]

We need to have a separate and broader plan for macOS (which includes this OCSP topic). We've briefly discussed that already (although not including OCSP).