Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crossgen2 support for System.Private.CoreLib compilation in CoreCLR build #44090

Merged
merged 24 commits into from
Nov 12, 2020
Merged

Crossgen2 support for System.Private.CoreLib compilation in CoreCLR build #44090

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
4307360
Stop calling crossgen-corelib.cmd / sh from the proj script
trylek Oct 25, 2020
50f1f74
Linux fixes for SPC crossgenning
trylek Oct 25, 2020
b81cf2d
Add call to setup_vs_tools to the step using crossgen to emit PDB
trylek Oct 30, 2020
1af6ad0
Add cross-building support; clean up PDB & PerfMap conditionals
trylek Oct 31, 2020
2df5b13
Fix folder name for cross-targeting Crossgen
trylek Oct 31, 2020
7ff71a0
Adjust cross-build host architecture specifications
trylek Oct 31, 2020
839acc3
Fix CrossComponentBinDir construction; shorten to CrossDir
trylek Oct 31, 2020
7023447
Typo
trylek Oct 31, 2020
63a2fff
Fix CrossBuild treatment (to fix FreeBSD run skipping Crossgen2)
trylek Oct 31, 2020
03011bb
Typo
trylek Oct 31, 2020
a9c08b6
Make CrossBuild automatic on Windows
trylek Oct 31, 2020
9019e9d
One more clarification w.r.t. cross-build specification
trylek Oct 31, 2020
25583a3
One more
trylek Oct 31, 2020
5ea9a82
Add support for Crossgen2
trylek Oct 31, 2020
c6db712
Fix x86 Windows build using Crossgen2
trylek Nov 1, 2020
125eaa0
Fix hosting architecture for arm CG2 cross-build to x64
trylek Nov 1, 2020
8594a6a
Fix several bugs in Crossgen1 mode; making it the default to test
trylek Nov 1, 2020
ef1cf4b
Undo local instrumentation unconditionally enabling PerfMap
trylek Nov 2, 2020
21aa174
Fix CG1 cross-build for ARM
trylek Nov 2, 2020
a286dde
Fix ARM cross-targeting on Windows Crossgen1
trylek Nov 3, 2020
4501b48
Put back CG2 as the default SPC compiler
trylek Nov 3, 2020
b7ecf5d
Reflect Anirudh's change renaming TargetOS to just Windows
trylek Nov 4, 2020
19aae26
Simplify usage of UseCrossgen2 property per Anton's PR feedback
trylek Nov 9, 2020
c81441e
Make Crossgen1 the default before the libraries failure is fixed
trylek Nov 11, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 10 additions & 8 deletions eng/Subsets.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@
</PropertyGroup>

<PropertyGroup>
<DefaultCoreClrSubsets>clr.runtime+clr.jit+clr.alljits+linuxdac+clr.corelib+clr.nativecorelib+clr.tools+clr.packages</DefaultCoreClrSubsets>
<DefaultCoreClrSubsets>clr.runtime+clr.jit+clr.alljits+linuxdac+clr.corelib+clr.tools+clr.nativecorelib+clr.packages</DefaultCoreClrSubsets>
AntonLapounov marked this conversation as resolved.
Show resolved Hide resolved

<DefaultMonoSubsets Condition="'$(MonoEnableLLVM)' == 'true' and '$(MonoLLVMDir)' == ''">mono.llvm+</DefaultMonoSubsets>
<DefaultMonoSubsets Condition="'$(MonoAOTEnableLLVM)' == 'true' and '$(MonoAOTLLVMDir)' == ''">mono.llvm+</DefaultMonoSubsets>
Expand Down Expand Up @@ -191,17 +191,19 @@
<ProjectToBuild Include="$(CoreClrProjectRoot)src\.nuget\Microsoft.CrossOsDiag.Private.CoreCLR\Microsoft.CrossOsDiag.Private.CoreCLR.proj" Category="clr" />
</ItemGroup>

<ItemGroup Condition="$(_subset.Contains('+clr.nativecorelib+'))">
<ProjectToBuild Include="$(CoreClrProjectRoot)crossgen-corelib.proj" Category="clr" />
</ItemGroup>

<ItemGroup Condition="$(_subset.Contains('+clr.tools+'))">
<ProjectToBuild Include="$(CoreClrProjectRoot)src\tools\runincontext\runincontext.csproj;
$(CoreClrProjectRoot)src\tools\r2rdump\R2RDump.csproj;
$(CoreClrProjectRoot)src\tools\dotnet-pgo\dotnet-pgo.csproj;
$(CoreClrProjectRoot)src\tools\r2rtest\R2RTest.csproj" Category="clr" BuildInParallel="true" />
<ProjectToBuild Include="$(CoreClrProjectRoot)src\tools\aot\crossgen2\crossgen2.csproj" Category="clr" />
<ProjectToBuild Include="$(CoreClrProjectRoot)src\tools\aot\ILCompiler.TypeSystem.ReadyToRun.Tests\ILCompiler.TypeSystem.ReadyToRun.Tests.csproj" Test="true" Category="clr" Condition="'$(__DistroRid)' != 'linux-musl-x64'"/>
$(CoreClrProjectRoot)src\tools\r2rtest\R2RTest.csproj;
$(CoreClrProjectRoot)src\tools\aot\crossgen2\crossgen2.csproj" Category="clr" />

<ProjectToBuild Include="$(CoreClrProjectRoot)src\tools\aot\ILCompiler.TypeSystem.ReadyToRun.Tests\ILCompiler.TypeSystem.ReadyToRun.Tests.csproj"
Test="true" Category="clr" Condition="'$(__DistroRid)' != 'linux-musl-x64'"/>
</ItemGroup>

<ItemGroup Condition="$(_subset.Contains('+clr.nativecorelib+'))">
<ProjectToBuild Include="$(CoreClrProjectRoot)crossgen-corelib.proj" Category="clr" />
</ItemGroup>

<ItemGroup Condition="$(_subset.Contains('+clr.packages+'))">
Expand Down
102 changes: 91 additions & 11 deletions src/coreclr/crossgen-corelib.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,21 +3,101 @@
<Import Project="Directory.Build.targets" />

<Target Name="Build">
<ItemGroup>
<_CoreClrBuildArg Condition="'$(TargetArchitecture)' != ''" Include="-$(TargetArchitecture)" />
<_CoreClrBuildArg Include="-$(Configuration.ToLower())" />
<_CoreClrBuildArg Condition="!$([MSBuild]::IsOsPlatform(Windows))" Include="-os $(TargetOS)" />
<_CoreClrBuildArg Condition="'$(CrossBuild)' == 'true'" Include="-cross" />
<_CoreClrBuildArg Condition="'$(PortableBuild)' != 'true'" Include="-portablebuild=false" />
</ItemGroup>
<PropertyGroup>
<!-- Default for using Crossgen2 when not set externally -->
<UseCrossgen2 Condition="'$(UseCrossgen2)' == ''">false</UseCrossgen2>

<OSPlatformConfig>$(TargetOS).$(TargetArchitecture).$(Configuration)</OSPlatformConfig>
<RootBinDir>$(RepoRoot)\artifacts</RootBinDir>
<LogsDir>$(RootBinDir)\log</LogsDir>
<BinDir>$(RootBinDir)\bin\coreclr\$(OSPlatformConfig)</BinDir>
<IntermediatesDir>$(RootBinDir)\obj\coreclr\$(OSPlatformConfig)</IntermediatesDir>
<CrossGenCoreLibLog>$(LogsDir)\CrossgenCoreLib_$(TargetOS)__$(TargetArchitecture)__$(Configuration).log</CrossGenCoreLibLog>
<ExeExtension Condition="'$(OS)' == 'Windows_NT'">.exe</ExeExtension>
<DotNetCli>$(RepoRoot)/dotnet.sh</DotNetCli>
<DotNetCli Condition="'$(OS)' == 'Windows_NT'">$(RepoRoot)\dotnet.cmd</DotNetCli>

<CoreLibAssemblyName>System.Private.CoreLib</CoreLibAssemblyName>
<CoreLibInputPath>$(BinDir)\IL\$(CoreLibAssemblyName).dll</CoreLibInputPath>
<CoreLibOutputPath>$(BinDir)\$(CoreLibAssemblyName).dll</CoreLibOutputPath>

<CrossDir></CrossDir>
<CrossDir Condition="'$(TargetArchitecture)' == 'arm' or '$(TargetArchitecture)' == 'arm64'">x64</CrossDir>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect this change broke the native build on arm64 and arm because now when we try to run ./build.sh clr.nativecorelib -arch arm64 -c Debug -runtimeconfiguration Debug we get:

  Generating native image of System.Private.CoreLib for OSX.arm64.Debug. Logging to /Users/seandree/git/runtime/artifacts/log/CrossgenCoreLib_OSX__arm64__Debug.log
  /Users/seandree/git/runtime/artifacts/bin/coreclr/OSX.arm64.Debug/x64/crossgen /nologo  /Platform_Assemblies_Paths "/Users/seandree/git/runtime/artifacts/bin/coreclr/OSX.arm64.Debug/IL" /out "/Users/seandree/git/runtime/artifacts/bin/coreclr/OSX.arm64.Debug/System.Private.CoreLib.dll" "/Users/seandree/git/runtime/artifacts/bin/coreclr/OSX.arm64.Debug/IL/System.Private.CoreLib.dll"
  /var/folders/wq/w3ybc8px5xgb0r5k7g6tkkdm0000gp/T/tmp7fd421fa5bce4bacb6316a02f0b68db9.exec.cmd: line 2: /Users/seandree/git/runtime/artifacts/bin/coreclr/OSX.arm64.Debug/x64/crossgen: No such file or directory

we should not use CrossDir if HostArch is the same as the target.

<CrossDir Condition="'$(TargetArchitecture)' == 'arm' and '$(UseCrossgen2)' != 'true' and '$(OS)' == 'Windows_NT'">x86</CrossDir>
<CrossDir Condition="'$(TargetArchitecture)' == 'x86' and '$(BuildArchitecture)' != '$(TargetArchitecture)' and '$(UseCrossgen2)' == 'true'">$(BuildArchitecture)</CrossDir>

<BuildDll>true</BuildDll>
<BuildDll Condition="'$(CrossBuild)' == 'true' and '$(CrossDir)' == ''">false</BuildDll>

<BuildPdb>false</BuildPdb>
<BuildPdb Condition="$(BuildDll) and '$(OS)' == 'Windows_NT' and '$(TargetOS)' == 'Windows'">true</BuildPdb>

<BuildPerfMap>false</BuildPerfMap>
<BuildPerfMap Condition="$(BuildDll) and '$(TargetOS)' == 'Linux'">true</BuildPerfMap>

<CrossGen1Cmd>$(BinDir)\$(CrossDir)\crossgen$(ExeExtension)</CrossGen1Cmd>
<CrossGen1Cmd>$(CrossGen1Cmd) /nologo</CrossGen1Cmd>
<CrossGen1Cmd>$(CrossGen1Cmd) <!-- IbcTuning --></CrossGen1Cmd>
<CrossGen1Cmd>$(CrossGen1Cmd) /Platform_Assemblies_Paths "$(BinDir)\IL"</CrossGen1Cmd>
</PropertyGroup>

<MakeDir
Directories="$(BinDir);$(IntermediatesDir);$(LogsDir)" />

<Message Importance="High"
Text="Generating native image of System.Private.CoreLib for $(OSPlatformConfig). Logging to $(CrossGenCoreLibLog)" />

<PropertyGroup>
<_CoreClrBuildScript Condition="$([MSBuild]::IsOsPlatform(Windows))">crossgen-corelib.cmd</_CoreClrBuildScript>
<_CoreClrBuildScript Condition="!$([MSBuild]::IsOsPlatform(Windows))">crossgen-corelib.sh</_CoreClrBuildScript>
<CrossGenDllCmd>$(DotNetCli) $(BinDir)\$(CrossDir)\crossgen2\crossgen2.dll</CrossGenDllCmd>
<CrossGenDllCmd>$(CrossGenDllCmd) -o:$(CoreLibOutputPath)</CrossGenDllCmd>
<CrossGenDllCmd>$(CrossGenDllCmd) -r:$(BinDir)\IL\*.dll</CrossGenDllCmd>
<CrossGenDllCmd>$(CrossGenDllCmd) --targetarch:$(TargetArchitecture)</CrossGenDllCmd>
<CrossGenDllCmd>$(CrossGenDllCmd) -O</CrossGenDllCmd>
<CrossGenDllCmd>$(CrossGenDllCmd) $(CoreLibInputPath)</CrossGenDllCmd>
</PropertyGroup>

<!-- Use IgnoreStandardErrorWarningFormat because Arcade sets WarnAsError and there's an existing warning in the native build. -->
<Exec Command="&quot;$(MSBuildThisFileDirectory)$(_CoreClrBuildScript)&quot; @(_CoreClrBuildArg->'%(Identity)',' ')" />
<PropertyGroup Condition="'$(UseCrossgen2)' != 'true'">
<CrossGenDllCmd>$(CrossGen1Cmd) /out "$(CoreLibOutputPath)"</CrossGenDllCmd>
<CrossGenDllCmd>$(CrossGenDllCmd) "$(CoreLibInputPath)"</CrossGenDllCmd>
</PropertyGroup>

<!-- For now we're using Crossgen1 for generating the perf map as Crossgen2 doesn't yet implement it: -->
<!-- https://github.com/dotnet/runtime/issues/44123 -->
<PropertyGroup Condition="$(BuildPerfMap)">
<CrossGenPerfMapCmd>$(CrossGen1Cmd) /CreatePerfMap "$(BinDir)"</CrossGenPerfMapCmd>
<CrossGenPerfMapCmd>$(CrossGenPerfMapCmd) "$(CoreLibOutputPath)"</CrossGenPerfMapCmd>
</PropertyGroup>

<PropertyGroup Condition="$(BuildPdb)">
<CrossGenPdbCmd>$(DotNetCli) $(BinDir)\r2rdump\r2rdump.dll</CrossGenPdbCmd>
<CrossGenPdbCmd>$(CrossGenPdbCmd) --create-pdb</CrossGenPdbCmd>
<CrossGenPdbCmd>$(CrossGenPdbCmd) --pdb-path:$(BinDir)\PDB</CrossGenPdbCmd>
<CrossGenPdbCmd>$(CrossGenPdbCmd) --in:$(CoreLibOutputPath)</CrossGenPdbCmd>
</PropertyGroup>

<PropertyGroup Condition="$(BuildPdb) and '$(UseCrossgen2)' != 'true'">
<VsSetupCmd>call $(RepoRoot)\src\coreclr\setup_vs_tools.cmd &amp;&amp;</VsSetupCmd>

<CrossGenPdbCmd>$(VsSetupCmd) $(CrossGen1Cmd) /CreatePdb "$(BinDir)\PDB"</CrossGenPdbCmd>
<CrossGenPdbCmd>$(CrossGenPdbCmd) "$(CoreLibOutputPath)"</CrossGenPdbCmd>
</PropertyGroup>

<Message Condition="$(BuildDll)" Importance="High" Text="$(CrossGenDllCmd)" />

<Exec Condition="$(BuildDll)" Command="$(CrossGenDllCmd)" />

<Message Condition="$(BuildPdb)" Importance="High" Text="$(CrossGenPdbCmd)" />

<Exec Condition="$(BuildPdb)" Command="$(CrossGenPdbCmd)" />

<Message Condition="$(BuildPerfMap)" Importance="High" Text="$(CrossGenPerfMapCmd)" />

<Exec Condition="$(BuildPerfMap)" Command="$(CrossGenPerfMapCmd)" />

<Copy Condition="!$(BuildDll)" SourceFiles="$(CoreLibInputPath)" DestinationFiles="$(CoreLibOutputPath)" UseHardlinksIfPossible="true" />

<Message Importance="High" Text="Crossgenning of System.Private.CoreLib succeeded. Finished at $(TIME)" />
<Message Importance="High" Text="Product binaries are available at $(BinDir)" />
</Target>

<Target Name="Restore" />
Expand Down
4 changes: 2 additions & 2 deletions src/coreclr/src/tools/r2rdump/R2RDump.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,15 @@
<AssemblyVersion>1.0.0.0</AssemblyVersion>
<AllowUnsafeBlocks>true</AllowUnsafeBlocks>
<OutputType>Exe</OutputType>
<Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
<Platforms>x64;x86;arm64;arm</Platforms>
<PlatformTarget>AnyCPU</PlatformTarget>
<AssemblyKey>Open</AssemblyKey>
<IsDotNetFrameworkProductAssembly>true</IsDotNetFrameworkProductAssembly>
<TargetFramework>$(NetCoreAppToolCurrent)</TargetFramework>
<CLSCompliant>false</CLSCompliant>
<NoWarn>8002,NU1701</NoWarn>
<RuntimeIdentifiers>win-x64;win-x86</RuntimeIdentifiers>
<OutputPath>$(RuntimeBinDir)/R2RDump</OutputPath>
<Platforms>AnyCPU;x64</Platforms>
</PropertyGroup>

<Import Project="$(RepositoryEngineeringDir)DiaSymReaderNative.targets" />
Expand Down