Improve pruned package data loading and handling

[dsplaisted]

Improves how package pruning data is supplied in the .NET SDK.

• Prune any later patches of a pruned package (by setting the patch value to 32767). See Should platform package pruning data represent GA of a framework or the serviced version? #44566
• Correctly supply pruned package data for .NET Framework, .NET Standard, and framework references using profiles (such as WPF and Windows Forms)
• Use pruned package data from targeting packs for .NET 10 and higher.

Targeting packs include a PackageOverrides.txt which provides the data we can use to determine the packages that should be pruned. However, this data was not entirely accurate for previous versions of .NET, as it was meant as an optimization so it didn't matter much if a few packages were missing or the versions listed weren't up to date. As of .NET 10, that should be fixed, so we will use the PackageOverrides.txt data as the source for package pruning for .NET 10 and higher.

For earlier versions of .NET, and for .NET Standard and .NET Framework, we will use data that has been calculated based on running conflict resolution and curated to be correct. This data is encapsulated in the FrameworkPackages source files.

During the .NET SDK build, targeting packs are downloaded for previous versions of .NET that are version 10.0 or higher, and the PackageOverrides.txt files from those targeting packs are laid out in a PrunePackageData folder in the SDK layout. Since there aren't currently any "previous versions of .NET that are version 10.0 or higher", this won't actually do anything until .NET 11.

The GetPackagesToPrune task will load data from the FrameworkPackages, the PrunePackageData folder, or the targeting packs for the current version of .NET that are bundled with the SDK, depending on the target framework and version.

[Copilot]

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

[ViktorHofer]

One important use-case are repositories that build shared frameworks (targeting packs) and consume them in their live build. These repositories need to be able to tell the SDK to use their live built targeting pack PackageOverrides.txt file. The current use of NetCoreTargetingPackRoot seems to make this hard. Such repos only build one targeting pack, i.e. Microsoft.NETCore.App.Ref. Therefore overriding the property wouldn't be correct as it would also impact the resolution of the pruned package data of the other non live-built targeting packs.

FWIW runtime uses this hook to tell the SDK to use the locally built packs: https://github.com/dotnet/runtime/blob/0004083b20a30dbb030aeb7e977feca0187c856c/eng/targetingpacks.targets#L119-L132

[dsplaisted]

One important use-case are repositories that build shared frameworks (targeting packs) and consume them in their live build. These repositories need to be able to tell the SDK to use their live built targeting pack PackageOverrides.txt file. The current use of NetCoreTargetingPackRoot seems to make this hard. Such repos only build one targeting pack, i.e. Microsoft.NETCore.App.Ref. Therefore overriding the property wouldn't be correct as it would also impact the resolution of the pruned package data of the other non live-built targeting packs.

FWIW runtime uses this hook to tell the SDK to use the locally built packs: https://github.com/dotnet/runtime/blob/0004083b20a30dbb030aeb7e977feca0187c856c/eng/targetingpacks.targets#L119-L132

For repositories that build shared frameworks, is it important to prune packages? Do those repos have transitive references to packages that should be pruned?

Is this problem specific to package overrides, or is it a general problem for using a repo-built targeting pack for one shared framework but also referencing another shared framework? How is that handled today?

[ViktorHofer]

For repositories that build shared frameworks, is it important to prune packages? Do those repos have transitive references to packages that should be pruned?

Yes, let me speak for runtime but this should also apply to aspnetcore and windowsdesktop. Runtime builds the live shared framework and then uses that to build the out-of-band package projects and tests. OOBs and test projects have external dependencies that bring transitives in that overlap with the framework assemblies. Those already caused CG issues in the past and it would be good to be able to use package pruning here.

Is this problem specific to package overrides, or is it a general problem for using a repo-built targeting pack for one shared framework but also referencing another shared framework? How is that handled today?

This is specific to the PackageOverrides.txt file. The SDK offers all the hooks needed to point to live built packs. I.e. the PlatformManifest.txt and other data files are used from the live built SDK.

[ericstj]

I'll add that we want our repos to use the product in the best way possible -- that way our builds test what we ship to customers. The more custom we need to make our solution the more we have to maintain and we lose valuable coverage of the SDK features.

[dsplaisted]

artifacts/obj/extracted-dotnet-sdk/sdk/10.0.100-ci/Containers/build/Microsoft.NET.Build.Containers.targets(6,9): error MSB4184: The expression "[MSBuild]::VersionGreaterThan('', 7.0.100)" cannot be evaluated. Version string was not in a correct format. [/__w/1/vmr/src/scenario-tests/artifacts/obj/generatedtests/SdkTemplateTestsComplex_Console_CSharp/SdkTemplateTestsComplex_Console_CSharp.csproj] [/__w/1/vmr/src/scenario-tests/src/Microsoft.DotNet.ScenarioTests.SdkTemplateTests/Microsoft.DotNet.ScenarioTests.SdkTemplateTests.csproj]

@dotnet/product-construction @baronfel Can someone take a look at this VMR build issue? Is it a known issue showing up elsewhere? Thanks.

[richlander]

Someone once said to me that "the SDK cannot build the SDK" and I did not initially understand what was meant by that simple but provocative statement. It has stuck with me.

[dsplaisted]

artifacts/obj/extracted-dotnet-sdk/sdk/10.0.100-ci/Containers/build/Microsoft.NET.Build.Containers.targets(6,9): error MSB4184: The expression "[MSBuild]::VersionGreaterThan('', 7.0.100)" cannot be evaluated. Version string was not in a correct format. [/__w/1/vmr/src/scenario-tests/artifacts/obj/generatedtests/SdkTemplateTestsComplex_Console_CSharp/SdkTemplateTestsComplex_Console_CSharp.csproj] [/__w/1/vmr/src/scenario-tests/src/Microsoft.DotNet.ScenarioTests.SdkTemplateTests/Microsoft.DotNet.ScenarioTests.SdkTemplateTests.csproj]

@dotnet/product-construction @baronfel Can someone take a look at this VMR build issue? Is it a known issue showing up elsewhere? Thanks.

@dotnet/product-construction @dotnet/source-build Ping on this, can someone take a look at why the source build / VMR builds here are failing?

[mthalman]

Can someone take a look at this VMR build issue? Is it a known issue showing up elsewhere? Thanks.

It's not a known issue. Somehow these changes are causing the NetCoreSdkVersion property to be empty in the SDK produced with these changes. It's probably worthwhile to grab the built SDK from the pipeline artifacts and examine it (Windows_x64_Artifacts_Attempt1/assets/Release/Sdk).

[dsplaisted]

Can someone take a look at this VMR build issue? Is it a known issue showing up elsewhere? Thanks.

It's not a known issue. Somehow these changes are causing the NetCoreSdkVersion property to be empty in the SDK produced with these changes. It's probably worthwhile to grab the built SDK from the pipeline artifacts and examine it (Windows_x64_Artifacts_Attempt1/assets/Release/Sdk).

Thanks, it turned out this was my mistake but it was in the generation of the bundled versions file which currently isn't applied for our normal test runs.

[dsplaisted]

@ViktorHofer @ericstj I've updated this so that you can set a PrunePackageTargetingPackRoots property, which can be a semicolon-separated list of where targeting packs can be found.

Would this work for you? It essentially allows you to specify multiple targeting pack roots just for the purposes of looking up the PackageOverrides.txt file to get the prune package data. You'll still have to have the full folder layout, so under the root you'd need the package contents to be under something like Microsoft.NETCore.App.Ref\<PackageVersion>.

[ViktorHofer]

@dsplaisted I assume we can't depend on the ResolveFrameworkReferences target to get that information? Asking as that's the hook that we use in runtime and SBRP to set the local paths: https://github.com/dotnet/runtime/blob/4c6a7068e9024696cb0bb0f996b89cb0f03adb32/eng/targetingpacks.targets#L120-L122

That target probably doesn't run during restore but I still wanted to ask.

[dsplaisted]

@mthalman @dotnet/product-construction This is failing for the VMR build in the aspnetcore build.repotasks build. It's failing in new code I'm adding in this PR, and it looks like it's failing because the targeting packs aren't included in the layout of the SDK it's using.

Do we expect the targeting packs to be in the packs directory for the SDK that's being used here?

Is there a way to see the source build layout? I don't seem to see it in the build artifacts.

[dsplaisted]

@dsplaisted I assume we can't depend on the ResolveFrameworkReferences target to get that information? Asking as that's the hook that we use in runtime and SBRP to set the local paths: https://github.com/dotnet/runtime/blob/4c6a7068e9024696cb0bb0f996b89cb0f03adb32/eng/targetingpacks.targets#L120-L122

That target probably doesn't run during restore but I still wanted to ask.

Yes, ResolveFrameworkReferences needs the targeting packs to be downloaded, so it can only run after restore. GetPackagesToPrune has to run before restore because it generates the pruned packages list which is needed for restore. So GetPackagesToPrune has to duplicate some of the logic that ResolveFrameworkReferences has.

[mthalman]

@mthalman @dotnet/product-construction This is failing for the VMR build in the aspnetcore build.repotasks build. It's failing in new code I'm adding in this PR, and it looks like it's failing because the targeting packs aren't included in the layout of the SDK it's using.

Do we expect the targeting packs to be in the packs directory for the SDK that's being used here?

Is there a way to see the source build layout? I don't seem to see it in the build artifacts.

The build that's failing is the stage 2 source build leg. In that same pipeline, the leg that's passing is stage 1. In stage 1, we use the Microsoft-built SDK to build the VMR in source build mode. In stage 2, we take the SDK produced by stage 1 and build the VMR again using that SDK. This verifies that the source-built SDK can be used to build the VMR. So the issue seems to be with how the SDK layout is produced by stage 1. You can find that SDK in the pipeline artifacts: CentOSStream9_Online_MsftSdk_x64_Artifacts/assets/Release/dotnet-sdk-<version>-centos.9-x64.tar.gz

[dsplaisted]

The build that's failing is the stage 2 source build leg. In that same pipeline, the leg that's passing is stage 1. In stage 1, we use the Microsoft-built SDK to build the VMR in source build mode. In stage 2, we take the SDK produced by stage 1 and build the VMR again using that SDK. This verifies that the source-built SDK can be used to build the VMR. So the issue seems to be with how the SDK layout is produced by stage 1. You can find that SDK in the pipeline artifacts: CentOSStream9_Online_MsftSdk_x64_Artifacts/assets/Release/dotnet-sdk-<version>-centos.9-x64.tar.gz

@mthalman Thanks, that's helpful. I looked in the stage 1 layout, and there is a packs\Microsoft.NETCore.App.Ref\10.0.0-preview.3.25154.11 folder in it. However, in the stage 2 build it doesn't seem to be finding that folder. I added some logging to the GetPackagesToPrune task:

Looking for targeting packs in /__w/1/vmr/src/aspnetcore/.dotnet/packs/Microsoft.NETCore.App.Ref
Targeting pack folder /__w/1/vmr/src/aspnetcore/.dotnet/packs/Microsoft.NETCore.App.Ref does not exist

Should the /__w/1/vmr/src/aspnetcore/.dotnet/ folder have the same contents as the stage 1 layout from the .tar.gz file?

[ViktorHofer]

/__w/1/vmr/src/aspnetcore/.dotnet/ shouldn't exist. When building inside the VMR, the SDK is restored once in the VMR root and then used to build all the repositories. That would /__w/1/vmr/.dotnet/ in this example.

Something points to the wrong location. What is that?

[ViktorHofer]

This worries me. I submitted various PRs in the last month that removed these in-built restores. That has multiple downsides:

• The build doesn't work without network connection after an initial build.cmd/sh -restore. "Air plane mode"
• With a coming SFI requirement, the times when we can hit the network will be restricted.
• Increased complexity and the build is slower because of the intermediate projects.

We hit that same limitation in SBRP (https://github.com/dotnet/source-build-reference-packages/blob/232bcf31aad21949f80d6706720540b85e43fff3/src/packageSourceGenerator/PackageSourceGenerator.proj#L98C9-L104) and decided to introduce an inline task that can run before restore.

Now that I see this additional use-case, I would prefer if we move the CollatePackageDownloads task into the Arcade SDK which makes it available before project restore and solves this recurring problem for our own repositories (SBRP, SDK and potentially others).

FWIW I'm not sure why NuGet requires customers to collate the items in the first place. Why can't NuGet parse such items:

<PackageDownload Include="x" Version="[7.0.0]" />
<PackageDownload Include="x" Version="[8.0.0]" />

[dsplaisted]

It's not just the collating, there's also the GeneratePackagePruneDataDownloads task.

It would be nice if we can find a good pattern for running custom tasks that help determine what needs to be restored.

[ViktorHofer]

Right. Looking at the GeneratePackagePruneDataDownloads task, I don't think that needs to be a build task at all. It should be able to express what the task is doing in pure msbuild syntax with even less code with item batching.

[ViktorHofer]

@ViktorHofer @ericstj I've updated this so that you can set a PrunePackageTargetingPackRoots property, which can be a semicolon-separated list of where targeting packs can be found.
Would this work for you? It essentially allows you to specify multiple targeting pack roots just for the purposes of looking up the PackageOverrides.txt file to get the prune package data. You'll still have to have the full folder layout, so under the root you'd need the package contents to be under something like Microsoft.NETCore.App.Ref<PackageVersion>.

Yes that will work to repos that build live targeting pakcs. Thanks for responding to my feedback. Just curious, why is that a property instead of an item?

[ViktorHofer]

Is this a no-op today with .NET 10 as netVersion is equal to maxNetVersion?

[MiYanni]

I just looked over the files that changed since the last time I reviewed. Looks good!