Skip to content

Commit

Permalink
Enable restore for ref and src projects in libs (#33553)
Browse files Browse the repository at this point in the history 
- Use RestoreUseStaticGraphEvaluation which improves no-op restore by 10-15x down to 10-20 seconds.
- .builds msbuild files renamed to .proj as RestoreUseStaticGraphEvaluation throws for non .proj files without an env var set.
- Introducing subsets for libraries and mono and replacing -buildtests switch which was only working for libraries in favor of the subset switch -subset tests which works consistently.
- Fixing the Microsoft.DotNet.CodeAnalysis analyzer which wasn't running and adding missing exclusions.
- Separating restore and build phases in different parts in the repo (ie for installer.tasks) as generated props and targets need to be imported which requires a reevaluation in the build phase.
- Fix eng/docker/build-docker-sdk.ps1 by using the official build entrypoints (cc @alnikola)
- Remove a few depprojs in favor of project restore (faster restore :))
- Fix root code coverage measurement not working correctly
- Traversal support instead of dir.traversal.targets or manual build target defines.
- Introduce a root Build.proj entrypoint which is responsible for building and restoring the repository. This is necessary to enable the new NuGet fast restore which works best and fastest with a single entrypoint.
- Avoid binclashes in libraries and between libraries and installer (netstandard.depproj vs netstandard.csproj)
- Upgrading the SDK to 5.0 latest
- Code cleanup
  • Loading branch information
@ViktorHofer
ViktorHofer committed Apr 6, 2020
1 parent 59ca590 commit 42183b1
Show file tree
Hide file tree
Showing 105 changed files with 798 additions and 1,161 deletions.
2 changes: 1 addition & 1 deletion .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ bld/
msbuild.log
msbuild.err
msbuild.wrn
msbuild.binlog
*.binlog
.deps/
.dirstamp
.libs/
Expand Down
80 changes: 80 additions & 0 deletions Build.proj
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
<Project Sdk="Microsoft.Build.Traversal">

<!--
Subsets are already imported by Directory.Build.props.
Reference the projects for traversal build. Ordering matters here.
-->
<ItemGroup>
<ProjectReference Include="@(CoreClrProject)" />
<ProjectReference Include="@(MonoProject)" />
<ProjectReference Include="@(LibrariesProject)" />
<ProjectReference Include="@(InstallerProject)" />
</ItemGroup>

<!--
Exclude installer depproj and pkgproj from static graph restore. We restore them below.
Remove when https://github.com/NuGet/Home/issues/9398 is fixed.
-->
<ItemGroup Condition="'$(MSBuildRestoreSessionId)' != ''">
<ProjectReference Remove="@(DepprojProjectToBuild)" />
<ProjectReference Remove="@(PkgprojProjectToBuild)" />
<ProjectReference Remove="@(BundleProjectToBuild)" />
</ItemGroup>

<!-- Custom arcade target which isn't available in Microsoft.Build.Traversal. -->
<Target Name="Rebuild" DependsOnTargets="Clean;Build" />

<Import Project="$(RepositoryEngineeringDir)SubsetValidation.targets" />

<!-- Upfront restore hooks -->
<Import Project="$(RepositoryEngineeringDir)restore\docs.targets" Condition="'$(DotNetBuildFromSource)' != 'true'" />
<Import Project="$(RepositoryEngineeringDir)restore\optimizationData.targets" Condition="'$(DotNetBuildFromSource)' != 'true' and '$(EnableNgenOptimization)' == 'true'" />
<Import Project="$(RepositoryEngineeringDir)restore\runtimeprops.targets" />

<!--
Use synthetic inputs/outputs to avoid building it all the time. This should let devs build with
MSBuild node reuse enabled (the Arcade default). If it were built every time, it would hit file
locking issues vs. the persistent nodes that loaded the task DLL for the previous build. It
isn't particularly accurate, but better than nothing.
-->
<Target Name="BuildRepoTasks"
DependsOnTargets="GetRepoTasksSrc"
BeforeTargets="Restore"
Inputs="@(RepoTasksSrc)"
Outputs="$(RepoTasksOutputFile)">
<ItemGroup>
<RepoTaskProjects Include="$(RepoTasksDir)**\*.csproj" />
</ItemGroup>

<MSBuild Projects="@(RepoTaskProjects)"
Properties="MSBuildRestoreSessionId=$([System.Guid]::NewGuid());Configuration=Debug;Platform=AnyCPU"
Targets="Restore"/>

<MSBuild Projects="@(RepoTaskProjects)"
Properties="Configuration=Debug;Platform=AnyCPU"
Targets="Build"/>

<WriteLinesToFile File="$(RepoTasksOutputFile)"
Lines="$(RepoTasksOutputFile)"
Overwrite="true" />
</Target>

<Target Name="GetRepoTasksSrc">
<PropertyGroup>
<RepoTasksDir>$(RepoTasksDir)</RepoTasksDir>
<RepoTasksOutputFile>$(ArtifactsObjDir)runtime.tasks\Debug\build-semaphore.txt</RepoTasksOutputFile>
</PropertyGroup>

<ItemGroup>
<RepoTasksSrc Include="$(RepoTasksDir)**\*.cs*" />
</ItemGroup>
</Target>

<Target Name="RestoreWithoutStaticGraph"
BeforeTargets="Restore">
<MSBuild Projects="@(LibrariesRestoreProject);@(DepprojProjectToBuild);@(PkgprojProjectToBuild);@(BundleProjectToBuild)"
Properties="MSBuildRestoreSessionId=$([System.Guid]::NewGuid());RestoreUseStaticGraphEvaluation=false"
Targets="Restore" />
</Target>

</Project>
9 changes: 1 addition & 8 deletions Directory.Build.props
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,7 @@
<ArtifactsObjDir>$([MSBuild]::NormalizeDirectory('$(ArtifactsDir)', 'obj'))</ArtifactsObjDir>
</PropertyGroup>

<!-- The TFMs to build and test against. -->
<PropertyGroup>
<NETCoreAppCurrentVersion>5.0</NETCoreAppCurrentVersion>
<NetCoreAppCurrentTargetFrameworkMoniker>.NETCoreApp,Version=v$(NETCoreAppCurrentVersion)</NetCoreAppCurrentTargetFrameworkMoniker>
<NetCoreAppCurrent>netcoreapp$(NETCoreAppCurrentVersion)</NetCoreAppCurrent>
<NetFrameworkCurrent>net472</NetFrameworkCurrent>
</PropertyGroup>

<!-- Configuration properties which are needed in both the (isolated) restore and build phases. -->
<Import Project="$(RepositoryEngineeringDir)Configurations.props" />

<!--
Expand Down
162 changes: 2 additions & 160 deletions docs/coding-guidelines/package-projects.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,164 +55,6 @@ Sample `System.Collections.Concurrent.pkgproj`
</Project>
```

### Framework-specific library
Framework specific libraries are effectively the same as the previous example. The difference is that the src project reference **must** refer to the `.builds` file which will provide multiple assets from multiple projects.

Sample System.Net.Security.pkgproj
```
<Project DefaultTargets="Build">
<Import Project="$([MSBuild]::GetPathOfFileAbove(Directory.Build.props))" />
<ItemGroup>
<ProjectReference Include="..\ref\System.Net.Security.builds">
<SupportedFramework>net463;netcoreapp1.1;$(AllXamarinFrameworks)</SupportedFramework>
</ProjectReference>
<ProjectReference Include="..\src\System.Net.Security.builds" />
</ItemGroup>
<ItemGroup>
<InboxOnTargetFramework Include="MonoAndroid10" />
<InboxOnTargetFramework Include="MonoTouch10" />
<InboxOnTargetFramework Include="xamarinios10" />
<InboxOnTargetFramework Include="xamarinmac20" />
<InboxOnTargetFramework Include="xamarintvos10" />
<InboxOnTargetFramework Include="xamarinwatchos10" />
<NotSupportedOnTargetFramework Include="netcore50">
<PackageTargetRuntime>win7</PackageTargetRuntime>
</NotSupportedOnTargetFramework>
</ItemGroup>
<ItemGroup>
</ItemGroup>
<Import Project="$([MSBuild]::GetPathOfFileAbove(Directory.Build.targets))" />
</Project>
```

Sample \ref .builds file defining a constant used to filter API that were added on top of the netstandard1.7 ones and are available only in netcoreapp1.1:

```
<Project DefaultTargets="Build">
<Import Project="$([MSBuild]::GetPathOfFileAbove(Directory.Build.props))" />
<PropertyGroup>
<OutputType>Library</OutputType>
<NuGetTargetMoniker>.NETStandard,Version=v1.7</NuGetTargetMoniker>
<DefineConstants Condition="'$(TargetFramework)' == 'netcoreapp1.1'">$(DefineConstants);netcoreapp11</DefineConstants>
</PropertyGroup>
<ItemGroup>
<Compile Include="System.Net.Security.cs" />
<Compile Include="System.Net.Security.Manual.cs" />
</ItemGroup>
<ItemGroup>
<None Include="project.json" />
</ItemGroup>
<Import Project="$([MSBuild]::GetPathOfFileAbove(Directory.Build.targets))" />
</Project>
```

Conditional compilation using the above-mentioned constant (from `ref\System.Net.Security.cs`):

```
#if NETCOREAPP
public virtual void AuthenticateAsClient(string targetHost, System.Security.Cryptography.X509Certificates.X509CertificateCollection clientCertificates, bool checkCertificateRevocation) { }
#endif
```

Sample \src .builds file (in this case the implementation is the same in both netcoreapp1.1 and netstandard1.7):

```
<Project DefaultTargets="Build">
<Import Project="$([MSBuild]::GetPathOfFileAbove(Directory.Build.props))" />
<ItemGroup>
<Project Include="System.Net.Security.csproj">
<OSGroup>Unix</OSGroup>
</Project>
<Project Include="System.Net.Security.csproj">
<OSGroup>Windows_NT</OSGroup>
</Project>
<Project Include="System.Net.Security.csproj">
<TargetFramework>net463</TargetFramework>
</Project>
</ItemGroup>
<Import Project="$(RepositoryEngineeringDir)dir.traversal.targets" />
</Project>
```

Tests can be similarly filtered grouping the compilation directives under:
```
<ItemGroup Condition="'$(TargetFramework)'=='netcoreapp1.1'">
```
(from `\tests\FunctionalTests\System.Net.Security.Tests.csproj`)

### Platform-specific library
These packages need to provide a different platform specific implementation on each platform. They do this by splitting the implementations into separate packages and associating those platform specific packages with the primary reference package. Each platform specific package sets `PackageTargetRuntime` to the specific platform RID that it applies.

Sample `System.IO.FileSystem.pkgproj`
```
<Project DefaultTargets="Build">
<Import Project="$([MSBuild]::GetPathOfFileAbove(Directory.Build.props))" />
<ItemGroup>
<ProjectReference Include="..\ref\System.IO.FileSystem.csproj">
<SupportedFramework>net46;netcore50;netcoreapp1.0</SupportedFramework>
</ProjectReference>
<ProjectReference Include="..\src\Facade\System.IO.FileSystem.csproj" />
<ProjectReference Include="win\System.IO.FileSystem.pkgproj" />
<ProjectReference Include="unix\System.IO.FileSystem.pkgproj" />
<InboxOnTargetFramework Include="MonoAndroid10" />
<InboxOnTargetFramework Include="MonoTouch10" />
<InboxOnTargetFramework Include="xamarinios10" />
<InboxOnTargetFramework Include="xamarinmac20" />
</ItemGroup>
<Import Project="$([MSBuild]::GetPathOfFileAbove(Directory.Build.targets))" />
</Project>
```

`win/System.IO.FileSystem.pkgproj`
```
<Project DefaultTargets="Build">
<Import Project="$([MSBuild]::GetPathOfFileAbove(Directory.Build.props))" />
<PropertyGroup>
<PackageTargetRuntime>win7</PackageTargetRuntime>
<PreventImplementationReference>true</PreventImplementationReference>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\..\src\System.IO.FileSystem.builds">
<AdditionalProperties>OSGroup=Windows_NT</AdditionalProperties>
</ProjectReference>
<!-- No implementation on platforms where our P-Invokes are not allowed -->
<NotSupportedOnTargetFramework Include="win8" />
<NotSupportedOnTargetFramework Include="wp8" />
<NotSupportedOnTargetFramework Include="wpa81" />
<!-- don't use the dotnet implementation for any version of desktop, it's implementation comes from the reference package -->
<ExternalOnTargetFramework Include="net" />
</ItemGroup>
<Import Project="$([MSBuild]::GetPathOfFileAbove(Directory.Build.targets))" />
</Project>
```
`unix/System.IO.FileSystem.pkgproj`
```
<Project DefaultTargets="Build">
<Import Project="$([MSBuild]::GetPathOfFileAbove(Directory.Build.props))" />
<PropertyGroup>
<PackageTargetRuntime>unix</PackageTargetRuntime>
<PreventImplementationReference>true</PreventImplementationReference>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\..\src\System.IO.FileSystem.builds">
<AdditionalProperties>OSGroup=Linux</AdditionalProperties>
</ProjectReference>
</ItemGroup>
<Import Project="$([MSBuild]::GetPathOfFileAbove(Directory.Build.targets))" />
</Project>
```

## Asset selection

The makeup of a package folder is primarily a grouping of project references to the projects that compose that package. Settings within each referenced project determines where that asset will be placed in the package. For example, reference assembly projects will be placed under the `ref/{targetMoniker}` folder in the package and implementations will be under either `lib/{targetMoniker}` or `runtimes/{rid}/lib/{targetMoniker}`. Whenever NuGet evaluates a package in the context of a referencing project it will choose the best compile time asset (preferring `ref`, then falling back to `lib`) and runtime asset (preferring `runtimes/{rid}/lib` and falling back to `lib`) for every package that is referenced. For more information see http://docs.nuget.org/.
Expand All @@ -221,8 +63,8 @@ Asset projects (`.csproj`, `.vbproj`, or `.depproj`) can control their `{targetM

The primary thing that the library author needs to do in order to ensure the correct asset selection is:

1. Configure the correct projects in your library's `.builds` file.
2. Reference the `.builds` file from the package project.
1. Configure the correct projects in your library's `.proj` file.
2. Reference the `.proj` file from the package project.
3. Provide a default PackageTargetFramework for empty-BuildTargetFramework builds in the library's `.csproj` or `.vbproj`.
```
<PackageTargetFramework Condition="'$(PackageTargetFramework)' == ''">dotnet5.4</PackageTargetFramework>
Expand Down
4 changes: 2 additions & 2 deletions docs/coding-guidelines/project-guidelines.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ once before you can iterate and work on a given library project.
- Netstandard Library - Copy to `bin\ref\netstandard2.0`
- NetFx targeting pack - Copy to `bin\ref\net472`
- Build targeting pack
- Build src\ref.builds which builds all references assembly projects. For reference assembly project information see [ref](#ref)
- Build src\libraries\ref.proj which builds all references assembly projects. For reference assembly project information see [ref](#ref)
- Build product
- Build src\src.builds which builds all the source library projects. For source library project information see [src](#src).
- Build src\libraries\src.proj which builds all the source library projects. For source library project information see [src](#src).
- Sign product
- Build src\sign.proj

Expand Down
4 changes: 2 additions & 2 deletions docs/project/library-servicing.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,9 @@ Where the `AssemblyVersion` is set to the old version before updating. To determ

If you incremented the `AssemblyVersion` in the last step, you'll also need to add an entry to [packageIndex.json](https://github.com/dotnet/runtime/blob/master/src/libraries/pkg/Microsoft.Private.PackageBaseline/packageIndex.json). Find the entry for your library in that file (again, making sure you're in the correct release branch), then find the subsection labeled `AssemblyVersionInPackageVersion`. There, add an entry that maps your new `AssemblyVersion` to your new `PackageVersion`. For an example, see [this PR](https://github.com/dotnet/runtime/commit/d0e4dcc7ebf008e7b6835cafbd03878c3a0e75f8#diff-ec9fd7a62cb0c494d86029014940382cR107), where we bumped the `PackageVersion` of `Microsoft.Diagnostics.Tracing.EventSource` from `2.0.0` to `2.0.1`, and bumped the `AssemblyVersion` from `2.0.0.0` to `2.0.1.0`. Therefore, we added an entry to `packageIndex.json` of the form `"2.0.1.0": "2.0.1"`.

## Add your package to packages.builds
## Add your package to libraries-packages.proj

In order to ensure that your package gets built, you need to add it to [packages.builds](https://github.com/dotnet/runtime/blob/master/src/libraries/packages.builds). In the linked example, we were building `System.Drawing.Common`. All you have to do is add a `Project` block inside the linked ItemGroup that matches the form of the linked example, but with `System.Drawing.Common` replaced by your library's name. Again, make sure to do this in the right servicing branch.
In order to ensure that your package gets built, you need to add it to [libraries-packages.proj](https://github.com/dotnet/runtime/blob/master/src/libraries/libraries-packages.proj). In the linked example, we were building `System.Drawing.Common`. All you have to do is add a `Project` block inside the linked ItemGroup that matches the form of the linked example, but with `System.Drawing.Common` replaced by your library's name. Again, make sure to do this in the right servicing branch.

## Test your changes

Expand Down
6 changes: 3 additions & 3 deletions docs/workflow/building/libraries/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -81,9 +81,9 @@ The build settings (BuildTargetFramework, TargetOS, Configuration, Architecture)

For more details on the build settings see [project-guidelines](../../../coding-guidelines/project-guidelines.md#build-pivots).

If you invoke the `build` script without any actions, the default action chain `-restore -build` is executed. You can chain multiple actions together (e.g., `-restore -build -buildtests`) and they will execute in the appropriate order. Note that if you specify actions like `-build` explicitly, you likely need to explicitly add `-restore` as well.
If you invoke the `build` script without any actions, the default action chain `-restore -build` is executed.

By default the `build` script only builds the product libraries and none of the tests. If you want to build the tests you can add the flag `-buildtests`. If you want to run the tests you can add the flag `-test`. To build and run the tests combine both arguments: `-buildtests -test`. To specify just the libraries, use `-subcategory libraries`.
By default the `build` script only builds the product libraries and none of the tests. If you want to include tests, you want to add the subset `-subset libtests`. If you want to run the tests you want to use the `-test` action instead of the `-build`, e.g. `build.cmd/sh -subsetcategory libraries -test`. To specify just the libraries, use `-subcategory libraries`.

**Examples**
- Building in release mode for platform x64 (restore and build are implicit here as no actions are passed in)
Expand All @@ -93,7 +93,7 @@ By default the `build` script only builds the product libraries and none of the

- Building the src assemblies and build and run tests (running all tests takes a considerable amount of time!)
```bash
./build.sh -subsetCategory libraries -restore -build -buildtests -test
./build.sh -subsetCategory libraries -restore -build -test
```

- Building for different target frameworks (restore and build are implicit again as no action is passed in)
Expand Down
6 changes: 3 additions & 3 deletions docs/workflow/debugging/libraries/debugging-packages.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Assuming the current directory is `\src\contractname\`:

Check the logs for output such as:
```
Project "S:\c1\src\System.Net.ServicePoint\ref\System.Net.ServicePoint.builds" (1) is building "S:\c1\src\System.Net.ServicePoint\ref\System.Net.ServicePoint.csproj" (2:3) on node 1
Project "S:\c1\src\System.Net.ServicePoint\ref\System.Net.ServicePoint.csproj" (1) is building "S:\c1\src\System.Net.ServicePoint\ref\System.Net.ServicePoint.csproj" (2:3) on node 1
(Build target(s)).
[...]
Expand All @@ -37,7 +37,7 @@ CopyFilesToOutputDirectory:
[...]
Project "S:\c1\src\System.Net.ServicePoint\ref\System.Net.ServicePoint.builds" (1) is building "S:\c1\src\System.Net.ServicePoint\ref\System.Net.ServicePoint.csproj" (2:4) on node 1
Project "S:\c1\src\System.Net.ServicePoint\ref\System.Net.ServicePoint.csproj" (1) is building "S:\c1\src\System.Net.ServicePoint\ref\System.Net.ServicePoint.csproj" (2:4) on node 1
(Build target(s)).
[...]
Expand Down Expand Up @@ -86,7 +86,7 @@ Ensure that the right `BuildTargetFramework` (what we're testing) is set.
To identify which of the combinations failed, search for the following pattern in the output:

```
Project "S:\c1\src\System.Net.ServicePoint\tests\System.Net.ServicePoint.Tests.builds" (1) is building "S:\c1\src\System.Net.ServicePoint\tests\System.Net.ServicePoint.Tests.csproj"
Project "S:\c1\src\System.Net.ServicePoint\tests\System.Net.ServicePoint.Tests.csproj" (1) is building "S:\c1\src\System.Net.ServicePoint\tests\System.Net.ServicePoint.Tests.csproj"
(2:5) on node 1 (Build target(s)).
ResolvePkgProjReferences:
Resolved compile assets from .NETStandard,Version=v2.0: S:\c1\bin\ref\System.Net.ServicePoint\4.0.0.0\System.Net.ServicePoint.dll
Expand Down
10 changes: 4 additions & 6 deletions docs/workflow/testing/libraries/testing.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ To build the tests and run them you can call the libraries build script.
**Examples**
- The following shows how to build only the tests but not run them:
```
libraries -buildtests
libraries -subset libtests
```

- The following builds and runs all tests in release configuration:
Expand All @@ -20,17 +20,15 @@ libraries -test -c Release
libraries -test /p:WithoutCategories=IgnoreForCI
```

Unless you specifiy `/p:TestNoBuild=true`, test assemblies are implicitly built when invoking the `Test` target.
Unless you specifiy `-testnobuild`, test assemblies are implicitly built when invoking the `Test` action.
- The following shows how to only test the libraries without building them
```
libraries -test /p:TestNoBuild=true
libraries -test -testnobuild
```

## Running tests on the command line

To build tests you need to pass the `-buildtests` flag to `build.cmd/sh` or run `libraries -restore -build -buildtests` (note that you need to specify `-restore` and `-build` additionally as those are only implicit if no action is passed in).

If you are interested in building and running the tests only for a specific library, then there are two different ways to do it:
To build tests you need to specify the `test` subset when invoking build.cmd/sh: `libraries -subset libtests`.

The easiest (and recommended) way to build and run the tests for a specific library, is to invoke the `Test` target on that library:
```cmd
Expand Down
Loading

0 comments on commit 42183b1

Please sign in to comment.