Certificate trust list needs to be backported to 6.0.3xx (and maybe 6.0.1xx)

[bradwilson]

Describe the bug

.NET SDK 6.0.4xx introduced a new embedded certificate trust list. Older SDK versions (including LTS 6.0.3xx and 6.0.1xx) rely primarily on the certificate trust list provided by the operating system (which of course varies from OS to OS).

There are two bugs related to this functionality:

1. The new certificate trust list includes the Microsoft Time Stamping service certificate trust, which does not exist as a trust in any OS. Therefore, attempting to verify a signed package using this certificate fails on all operating systems unless the user upgrades to a new enough 6.0.4xx SDK. This affects new packages signed by the new .NET Foundation code signature process, since it uses the Microsoft Time Stamping service. (Users will experience this as NU3003)
2. The certificate used for repository countersignature in nuget.org has shifted from Symantec to DigiCert, related to the feature discussion for the new certificate trust list. Unfortunately, some OSes (like Ubuntu 22.04) have already revoked their trust for the Symantec root certificate. This affects older packages countersigned by NuGet before the move to DigiCert. (Users will probably experience this as NU3028)

In my opinion, all LTS versions need this functionality, not just 6.0.4xx.

To Reproduce

Microsoft Time Stamping service

1. Download xunit package version 2.5.0: https://www.nuget.org/api/v2/package/xunit/2.5.0
2. Run dotnet nuget verify --all xunit.2.5.0.nupkg

Here are the results with 6.0.314 (latest as of this writing):

And here are the results with 6.0.411 (latest as of this writing):

Example images are shown from Linux (as it was the easiest way to quickly move between versions), but this is also reproducible with Windows.

Symantec Time Stamping service

On a non-Windows machine:

1. Download xunit package version 2.4.2: https://www.nuget.org/api/v2/package/xunit/2.4.2
2. Run dotnet nuget verify --all -v normal xunit.2.4.2.nupkg

Here are the results with 6.0.314 on Linux:

And here are the results with 6.0.411 on Linux:

[bradwilson]

Also worth mentioning: nuget.info is broken right now because of this.

https://nuget.info/packages/xunit/2.4.2

https://nuget.info/packages/xunit/2.5.0

[ghost]

Thanks for creating this issue! We believe this issue is related to NuGet tooling, which is maintained by the NuGet team. Thus, we closed this one and encourage you to raise this issue in the NuGet repository instead. Don’t forget to check out NuGet’s contributing guide before submitting an issue!

If you believe this issue was closed out of error, please comment to let us know.

Happy Coding!

[bradwilson]

This was closed in error.

This is related to NuGet packages, but the core problem is in the .NET SDK.

[richlander]

I think we made the decision to not backport this capability to .NET 6. Is there a specific reason why the .NET 7 (and soon) .NET 8 SDK are not sufficient?

@dtivel

[bradwilson]

@richlander You did put this in 6.0.4xx, so that isn't a correct statement.

The person who is reluctant to use any non-LTS SDK would not be satisfied by "upgrade to 7 or a preview of 8".

Additionally, you have another year of support for VS 2022 17.2 and .NET SDK 6.0.3xx, so this would be an entire year of this being broken for LTS users. Even once .NET SDK 8 is released, that's still not necessarily going to be supported by the LTS version of VS 2022.

[teo-tsirpanis]

another year of support for VS 2022 17.2

6 months actually.

[bradwilson]

You're right, January 2024. My brain remembered July.

[bradwilson]

.NET 6.0 is supported until November 2024. When will be the next version of an LTS of that SDK?

[ghost]

Thanks for creating this issue! We believe this issue is related to NuGet tooling, which is maintained by the NuGet team. Thus, we closed this one and encourage you to raise this issue in the NuGet repository instead. Don’t forget to check out NuGet’s contributing guide before submitting an issue!

If you believe this issue was closed out of error, please comment to let us know.

Happy Coding!

[bradwilson]

Can someone tell the bot to cut it out, please? @jaredpar @richlander

[jaredpar]

I have no power over our bot overlords.

[bradwilson]

Thank you for at least re-opening it again @jaredpar 😂

[marcpopMSFT]

Thanks for reaching out to us. I want to clarify expectations around the .NET SDK lifecycle and how it connects to the LTS and STS versions of both .NET and Visual Studio. Below is a high level summary of our principles.

The .NET SDK has feature bands that enable feature stability
The .4xx feature band is supported for the life of the matching .NET Runtime
The other feature bands track Visual Studio releases and go out of support when those VS releases go out of support.

Additional details:

Each SDK band ships with a specific version of Visual Studio and includes tooling that matches that Visual Studio (NuGet, MSBuild, Roslyn, etc).
Each SDK can build any .NET application targeting the major version it shipped with and all previously shipped .NET versions.

We are working on improvements to our documentation to make this more clear. You can find more information here

I will let @JonDouglas from the NuGet team reply with an explanation around the NuGet verify functionality you have identified and why it was considered a feature and did not meet the bar for backporting to the older .NET 6 SDKs.

[bradwilson]

@marcpopMSFT Your reply sounds like you don't intend to fix this. If that's the case, then I am disappointed. At the end of the day, I actually don't care about the reasoning if you don't intend to fix it.

I've already reverted back to the DigiCert Time Stamp service for future builds. (I had no knowledge of the switch to--nor any personal desire to use--the Microsoft Time Stamp service. It's a net-negative benefit to my end users, so there's no reason I can think of to switch back, even once .NET 8 is widely available.)

Regardless of the reasoning here, the combination of decisions from NuGet/.NET/.NET Foundation have yielded a terrible experience for a non-zero number of our users. I can't fix 2.5.0 because it's already shipped, but I've committed to shipping a 2.5.1 ASAP so that people will stop being blocked by these decisions.

[JonDouglas]

Hey Brad,

I know these responses disappoint you and I know this response is probably moot to you at this point.

Long story short, we decided to revert supporting verification on Linux for .NET 6 in 2021. We were put between a rock and hard place for the upcoming LTS release and had to turn it off entirely until we could figure out some next steps given the Symantec distrust situation. While we did backport to 6.0.4xx, that is because we identified it as the right vehicle(latest feature band, Linux/macOS as the main platforms, etc).

We encourage users to use a 6.0.4xx+ or 7.0.xxx+ SDK that has this functionality backported (still disabled by default) or 8.0.xxx SDK that also has this functionality (enabled by default).

The desired experience and functionality is available in the next .NET LTS this November which is .NET 8. We apologize profusely for the growing pains over these last couple years, it's been a long journey.

[JonDouglas]

Hey Brad,

I'm going to keep collecting some issues I'm seeing come in over the last few days. We're discussing some options to see if we can address the bug portions of this issue, but I do want to be clear that the feature availability will likely unchange here.

While they are all somewhat related, it seems that the specific challenge here is enabling support for Microsoft timestamps as .NET Foundation's signtool has a new timestamp default that the client tooling does not understand on these older .NET 6.0 SDKs. We may not be able to address this bug on every older .NET 6.0 SDK, but it may be possible on the ones people are largely reporting from given we're seeing Windows being affected.

Here's a quick PSA for anyone seeing this:

Impact:

Developers using .NET 6.0.3xx or earlier on Windows get a warning (NU3003) when referencing any package using the .NET Foundation SignTool to sign and timestamp their package. This is because the SignTool uses the Microsoft Time Stamp service by default, which uses a cert attribute that the aforementioned .NET SDK versions fail to parse. (Digicert doesn't embed the same attribute.)

Workaround:

Developers can NoWarn NU3003 or use an older version of an impacted package.

[marcpopMSFT]

I wanted to update @bradwilson and others. After some offline discussions, @dtivel realized that the fix to correctly verify the certificate of the xunit package on windows in 6.0.3xx did not involve backporting feature work from 6.0.4xx but rather picking up a fix that was already available in a 6.0 version of the System.Security.Cryptograph.Pkcs binary (as the SDK was using a 5.0.0 version). The SDK was still using the old version of that package as NuGet was targeting 5.0.0 and MSBuild was targeting 4.7.0 (so the build chose the higher version of the two transitive dependencies). MSBuild in 17.3 moved to 6.0.1 which included the fix and that's why 6.0.4xx worked.

Backporting this single change is much lower risk than the feature work we originally thought would be needed. I've put out a PR (thanks to @rainersigwald ) that adds a direct reference to the new version of the package as the least impactful change. We're still reviewing the change for risk and testing (and we wouldn't be able to ship a fix until September) and will still need approval but this may be a possible option. There is a separate effort exploring whether NPE can be updated as well.

[bradwilson]

@marcpopMSFT Would this work for all OSes, or just Windows?

[dtivel]

There are 2 distinct issues here:

1. Missing trust: The root certificate bundles that contain the Microsoft timestamping root exist only in .NET 6.0.4xx SDK and later. Using an older .NET 6 SDK (like 6.0.1xx or 6.0.3xx) means that package signature verification will use the system's root certificate store. On Windows, that's fine, since package signature verification on Windows always uses the Windows root certificate store. However, on Linux, the system root certificate store is generally not appropriate, and you'll run into problems (like missing roots). That's because the Linux system root certificate store is typically not curated for code signing and timestamping purposes.

   On macOS, I recommend against verifying signed packages on any version of the .NET SDK. See this for more information.

2. Parsing failure: The Microsoft timestamping service does something unusual but within standards. There were a couple of bugs in System.Security.Cryptography.Pkcs where .NET failed to parse these attribute certificates, but those bugs were fixed, and the fixes were backported to the 6.0 runtime. However, some .NET 6.0 SDK feature bands didn't pick up the newer version of System.Security.Cryptography.Pkcs. @marcpopMSFT has 2 PR's out to fix this.

Fixing # 2 means that this parsing will not fail on any OS. No more NU3003 warnings (or errors if you treat warnings as errors). After that, whether or not the timestamp (or any other certificate, for that matter) is trusted depends on whether you're using a pre-6.0.4xx release of .NET SDK on non-Windows or not.

[bradwilson]

So to clarify: the parsing solution only fixes Windows, and this will remain broken on Linux/macOS. As this won't completely fix this issue, it's up to the team do decide whether this partial fix is worth doing.

I pushed a 2.5.1 prerelease build last night that is time stamped with DigiCert which will solve it for xUnit.net. The rest of the .NET Foundation projects will need to do this themselves as well unless dotnet/sign#638 get resolved.