-
Notifications
You must be signed in to change notification settings - Fork 517
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[msbuild] Add ILStrip'ing for net6 applications. Fixes #11445. #12563
Conversation
- Controled by EnableAssemblyILStripping which defaults to true - Integration test included Before - https://gist.github.com/chamons/c7886f7bacbc2e5ac5966e4251d13e71 After - https://gist.github.com/chamons/148e1bef22fa336f953f3d02dcf20667 859,136 -> 527,872 managed
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Those from #12558 see the last commit for changes. I've now run this locally in sim and device, and hand tested the follow cases:
|
msbuild/Xamarin.MacDev.Tasks.Core/Tasks/ParseBundlerArgumentsTaskBase.cs
Outdated
Show resolved
Hide resolved
@@ -53,7 +53,11 @@ build-dotnet: $(TARGETS) | |||
$(DOTNET6) build size-comparison/MySingleView/dotnet/MySingleView.csproj --runtime ios-arm64 $(COMMON_ARGS) /bl:$@.binlog $(MSBUILD_VERBOSITY) | |||
|
|||
run-dotnet: $(TARGETS) | |||
$(DOTNET6) build -t:Run size-comparison/MySingleView/dotnet/MySingleView.csproj --runtime ios-arm64 $(COMMON_ARGS) | |||
$(DOTNET6) build -t:Run size-comparison/MySingleView/dotnet/MySingleView.csproj --runtime ios-arm64 $(COMMON_ARGS) /bl:$@.binlog $(MSBUILD_VERBOSITY) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Convince mostly. Writing out log is almost free, and being able to test sim was nice.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Target strip-dotnet:
can also be removed now :)
and the README updated to document how to preserve the IL for analysis (it's already done for legacy)
You know, let me try removing that and see what needs to change. |
So turns out my more generic conditions work in either case, so I could just remove it. |
@@ -130,6 +133,10 @@ public override bool Execute () | |||
xml = new List<string> (); | |||
xml.Add (value); | |||
break; | |||
case "nostrip": | |||
// Output is EnableAssemblyILStripping so we enable if --nostring=false and disable if true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
typo: nostring
-> nostrip
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
@@ -53,7 +53,11 @@ build-dotnet: $(TARGETS) | |||
$(DOTNET6) build size-comparison/MySingleView/dotnet/MySingleView.csproj --runtime ios-arm64 $(COMMON_ARGS) /bl:$@.binlog $(MSBUILD_VERBOSITY) | |||
|
|||
run-dotnet: $(TARGETS) | |||
$(DOTNET6) build -t:Run size-comparison/MySingleView/dotnet/MySingleView.csproj --runtime ios-arm64 $(COMMON_ARGS) | |||
$(DOTNET6) build -t:Run size-comparison/MySingleView/dotnet/MySingleView.csproj --runtime ios-arm64 $(COMMON_ARGS) /bl:$@.binlog $(MSBUILD_VERBOSITY) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Target strip-dotnet:
can also be removed now :)
and the README updated to document how to preserve the IL for analysis (it's already done for legacy)
<_StrippedAssemblies Include="@(_AssembliesToBeStripped->'%(OutputPath)')" > | ||
</_StrippedAssemblies> | ||
</ItemGroup> | ||
<ILStrip Assemblies="@(_AssembliesToBeStripped)" /> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When building from Windows this won't be executed remotely because it's missing the SessionId property. Does this need to be executed on a Mac? Do we own this task?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The runtime team owns it: https://github.com/dotnet/runtime/pull/57359/files
And yes, this has to be executed on the Mac (it happens after the AOT compiler has executed, but before we sign the final .app)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, this won't work then, for executing a task remotely we need to add the SessionId property and the logic to execute it remotely based on that value. We had the same problem with ILLink, and we ended up inheriting from the original task to add support for remoting it, see https://github.com/xamarin/xamarin-macios/blob/main/msbuild/Xamarin.iOS.Tasks/Tasks/ILLink.cs and https://github.com/xamarin/xamarin-macios/blob/main/msbuild/Xamarin.iOS.Tasks.Core/Tasks/ILLinkBase.cs. The base class is adding input and output properties for all the files that need to be copied to the Mac and the output ones that need to be created on Windows. I'm not sure if there's a better way for doing this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Taking a stab at this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dalexsoto I'll be out next week, but @mauroa should be able to help with any questions. Please keep in mind he has many things in his plate already, but he'll try to test this from Windows once the changes are ready.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is fixed.
❌ [PR Build] Tests failed on Build ❌Tests failed on Build. API diff✅ API Diff from stable View API diffAPI & Generator diff✅ API Diff (from PR only) (no change) GitHub pagesResults can be found in the following github pages (it might take some time to publish): Test results3 tests failed, 140 tests passed.Failed tests
Pipeline on Agent XAMBOT-1109.BigSur' |
<_StrippedAssemblies Include="@(_AssembliesToBeStripped->'%(OutputPath)')" > | ||
</_StrippedAssemblies> | ||
</ItemGroup> | ||
<ILStrip Assemblies="@(_AssembliesToBeStripped)" /> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The runtime team owns it: https://github.com/dotnet/runtime/pull/57359/files
And yes, this has to be executed on the Mac (it happens after the AOT compiler has executed, but before we sign the final .app)
@@ -1062,6 +1062,7 @@ Copyright (C) 2018 Microsoft. All rights reserved. | |||
<Output TaskParameter="Registrar" PropertyName="_BundlerRegistrar" /> | |||
<Output TaskParameter="Verbosity" PropertyName="_BundlerVerbosity" /> | |||
<Output TaskParameter="XmlDefinitions" ItemName="_BundlerXmlDefinitions" /> | |||
<Output TaskParameter="NoStrip" PropertyName="EnableAssemblyILStripping" /> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This task should take EnableAssemblyILStripping
as input as well, otherwise it won't be possible for users to override the default by setting EnableAssemblyILStripping
to something.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure this is correct.
I hacked up a test project: https://gist.github.com/chamons/abe82c128d909df53a1780c792394dab to check
In it, I invoke ParseBundlerArguments with some _BundlerArguments and EnableAssemblyILStripping preset. I print before and after, and I don't think it overrides (with 0f13eb3).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thoughts on ^?
<PropertyGroup> | ||
<_StrippedAssemblyDirectory>$(DeviceSpecificIntermediateOutputPath)\stripped</_StrippedAssemblyDirectory> | ||
</PropertyGroup> | ||
<MakeDir Directories="$(_StrippedAssemblyDirectory)" /> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is missing the SessionId parameter and the IsMacEnabled condition, so when building from Windows this will only be executed on Windows. Like this:
<MakeDir SessionId="$(BuildSessionId)" Condition="'$(IsMacEnabled)' == 'true'" Directories="$(_AppResourcesPath)" /> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
🔥 Tests failed catastrophically on Build (no summary found). 🔥Result file $(TEST_SUMMARY_PATH) not found. |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
@dalexsoto @chamons @rolfbjarne I don't see new packages for the latest changes/fixes in the CI build. Did something go wrong? I would need to have them in order to test VS |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
@mauroa our CI isn't in a nice mood 😒 I started another attempt, hopefully it works this time (third time's the charm!) |
❌ [PR Build] Tests failed on Build ❌Tests failed on Build. API diff✅ API Diff from stable View API diffAPI & Generator diff✅ API Diff (from PR only) (no change) GitHub pagesResults can be found in the following github pages (it might take some time to publish): Test results3 tests failed, 134 tests passed.Failed tests
Pipeline on Agent XAMBOT-1101.BigSur' |
❌ [PR Build] Tests failed on Build ❌Tests failed on Build. API diff✅ API Diff from stable View API diffAPI & Generator diff✅ API Diff (from PR only) (no change) GitHub pagesResults can be found in the following github pages (it might take some time to publish): Test results1 tests failed, 136 tests passed.Failed tests
Pipeline on Agent XAMBOT-1094.BigSur' |
@@ -24,6 +25,10 @@ | |||
<!-- We need the net472 impl, otherwise the Build agent needs to be a net5.0 app --> | |||
<HintPath>$(PkgMicrosoft_NET_ILLink_Tasks)\tools\net472\ILLink.Tasks.dll</HintPath> | |||
</Reference> | |||
<Reference Include="ILStrip"> | |||
<!-- We need the net472 impl, otherwise the Build agent needs to be a net5.0 app --> | |||
<HintPath>$(PkgMicrosoft_NET_Runtime_MonoTargets_Sdk)\tasks\net472\ILStrip.dll</HintPath> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FYI, for net7 I'm looking at consolidating the MonoTargets tasks into a single assembly dotnet/runtime#59720
1 test failure is known: https://github.com/xamarin/maccore/issues/2443 The other two seem obviously related. |
@mauroa - I just had to re-merge due to a test file conflict but hopefully the last build is good enough to test today? |
❌ [PR Build] Tests failed on Build ❌Tests failed on Build. API diff✅ API Diff from stable View API diffAPI & Generator diff✅ API Diff (from PR only) (no change) GitHub pagesResults can be found in the following github pages (it might take some time to publish): Test results2 tests failed, 135 tests passed.Failed tests
Pipeline on Agent XAMBOT-1101.BigSur' |
@chamons @rolfbjarne @dalexsoto I tested with the latest build and I was able to successfully build a net6 ios app with ILStrip enabled. I think we are good to signoff and merge the PR if everything else is ok |
Test failures are unrelated
|
- In a late minute change to the ILStrip PR (xamarin#12563) a change XVS support broke execution of Apps that were stripped - Applications were broken because none of the stripped assemblies were actually copied into the bundle - However, the tests still passed, because all assemblies that were there had no IL (zero assemblies total) Now why did this happen? - The stripped assemblies were changed to return via an msbuidl Output Element - Output Element can return an Property or ItemGroup, depending if you use the PropertyName or ItemName attributes - Unfortuntely I used PropertyName, when I expected an ItemGroup. So I silently had a property created instead. - Thus zero items were added to the list of files to copy into the bundle - Which was undetected as the test did not confirm files were copied in, and manual tests were not run so late into the PR (3 weeks after PR was opened) How was it fixed? - Correctly using ItemName on Output created a valid item group to reference - However, that still failed with an absurdly confusing error: PATH/Microsoft.NET.Publish.targets(277,5): error MSB3024: Could not copy the file FILE to the destination file PATH, because the destination is a folder instead of a file. To copy the source file into a folder, consider using the DestinationFolder parameter instead of DestinationFiles. - After a splunking through netcore targets, I found the metadata on these assemblies references really matters. Without it, they are not processed correctly at all. - Thus, I updated ILStripBase to clone the existing metadata when changing the orignal assembly reference to the stripped path - Finally, I corrected the test to assert that required files are copied in. I also manually ran our device test.
- In a late minute change to the ILStrip PR (xamarin#12563) a change to support XVS support broke execution of Apps that were stripped - Applications were broken because none of the stripped assemblies were actually copied into the bundle - However, the tests still passed, because all assemblies that were there had no IL (zero assemblies total) Now why did this happen? - The stripped assemblies were changed to return via an msbuild Output Element - Output Element can return an Property or ItemGroup, depending if you use the PropertyName or ItemName attributes - Unfortunately I used PropertyName, when I expected an ItemGroup. So I silently had a property created instead. - Thus zero items were added to the list of files to copy into the bundle - Which was undetected as the test did not confirm files were copied in, and manual tests were not run so late into the PR (3 weeks after PR was opened) How was it fixed? - Correctly using ItemName on Output created a valid item group to reference - However, that still failed with an absurdly confusing error: PATH/Microsoft.NET.Publish.targets(277,5): error MSB3024: Could not copy the file FILE to the destination file PATH, because the destination is a folder instead of a file. To copy the source file into a folder, consider using the DestinationFolder parameter instead of DestinationFiles. - After a splunking through netcore targets, I found the metadata on these assemblies references really matters. Without it, they are not processed correctly at all. - Thus, I updated ILStripBase to clone the existing metadata when changing the original assembly reference to the stripped path - Finally, I corrected the test to assert that required files are copied in. I also manually ran our device test.
* [net6] Fix ILStrip'ed apps to actually work again - In a late minute change to the ILStrip PR (#12563) a change to support XVS support broke execution of Apps that were stripped - Applications were broken because none of the stripped assemblies were actually copied into the bundle - However, the tests still passed, because all assemblies that were there had no IL (zero assemblies total) Now why did this happen? - The stripped assemblies were changed to return via an msbuild Output Element - Output Element can return an Property or ItemGroup, depending if you use the PropertyName or ItemName attributes - Unfortunately I used PropertyName, when I expected an ItemGroup. So I silently had a property created instead. - Thus zero items were added to the list of files to copy into the bundle - Which was undetected as the test did not confirm files were copied in, and manual tests were not run so late into the PR (3 weeks after PR was opened) How was it fixed? - Correctly using ItemName on Output created a valid item group to reference - However, that still failed with an absurdly confusing error: PATH/Microsoft.NET.Publish.targets(277,5): error MSB3024: Could not copy the file FILE to the destination file PATH, because the destination is a folder instead of a file. To copy the source file into a folder, consider using the DestinationFolder parameter instead of DestinationFiles. - After a splunking through netcore targets, I found the metadata on these assemblies references really matters. Without it, they are not processed correctly at all. - Thus, I updated ILStripBase to clone the existing metadata when changing the original assembly reference to the stripped path - Finally, I corrected the test to assert that required files are copied in. I also manually ran our device test. Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
- In a late minute change to the ILStrip PR (xamarin#12563) a change to support XVS support broke execution of Apps that were stripped - Applications were broken because none of the stripped assemblies were actually copied into the bundle - However, the tests still passed, because all assemblies that were there had no IL (zero assemblies total) Now why did this happen? - The stripped assemblies were changed to return via an msbuild Output Element - Output Element can return an Property or ItemGroup, depending if you use the PropertyName or ItemName attributes - Unfortunately I used PropertyName, when I expected an ItemGroup. So I silently had a property created instead. - Thus zero items were added to the list of files to copy into the bundle - Which was undetected as the test did not confirm files were copied in, and manual tests were not run so late into the PR (3 weeks after PR was opened) How was it fixed? - Correctly using ItemName on Output created a valid item group to reference - However, that still failed with an absurdly confusing error: PATH/Microsoft.NET.Publish.targets(277,5): error MSB3024: Could not copy the file FILE to the destination file PATH, because the destination is a folder instead of a file. To copy the source file into a folder, consider using the DestinationFolder parameter instead of DestinationFiles. - After a splunking through netcore targets, I found the metadata on these assemblies references really matters. Without it, they are not processed correctly at all. - Thus, I updated ILStripBase to clone the existing metadata when changing the original assembly reference to the stripped path - Finally, I corrected the test to assert that required files are copied in. I also manually ran our device test.
…rk again (#13106) * [net6] Fix ILStrip'ed apps to actually work again - In a late minute change to the ILStrip PR (#12563) a change to support XVS support broke execution of Apps that were stripped - Applications were broken because none of the stripped assemblies were actually copied into the bundle - However, the tests still passed, because all assemblies that were there had no IL (zero assemblies total) Now why did this happen? - The stripped assemblies were changed to return via an msbuild Output Element - Output Element can return an Property or ItemGroup, depending if you use the PropertyName or ItemName attributes - Unfortunately I used PropertyName, when I expected an ItemGroup. So I silently had a property created instead. - Thus zero items were added to the list of files to copy into the bundle - Which was undetected as the test did not confirm files were copied in, and manual tests were not run so late into the PR (3 weeks after PR was opened) How was it fixed? - Correctly using ItemName on Output created a valid item group to reference - However, that still failed with an absurdly confusing error: PATH/Microsoft.NET.Publish.targets(277,5): error MSB3024: Could not copy the file FILE to the destination file PATH, because the destination is a folder instead of a file. To copy the source file into a folder, consider using the DestinationFolder parameter instead of DestinationFiles. - After a splunking through netcore targets, I found the metadata on these assemblies references really matters. Without it, they are not processed correctly at all. - Thus, I updated ILStripBase to clone the existing metadata when changing the original assembly reference to the stripped path - Finally, I corrected the test to assert that required files are copied in. I also manually ran our device test. * Update tests/dotnet/UnitTests/PostBuildTest.cs Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com> Co-authored-by: Chris Hamons <chris.hamons@xamarin.com> Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Before - https://gist.github.com/chamons/c7886f7bacbc2e5ac5966e4251d13e71
After - https://gist.github.com/chamons/148e1bef22fa336f953f3d02dcf20667
859,136 -> 527,872 managed
Fixes #11445.