Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to distribute xcframework with nugets ? #10819

Closed
spouliot opened this issue Mar 9, 2021 · 11 comments
Closed

How to distribute xcframework with nugets ? #10819

spouliot opened this issue Mar 9, 2021 · 11 comments
Labels
iOS Issues affecting Xamarin.iOS macOS Issues affecting Xamarin.Mac question The issue is a question
Milestone

Comments

@spouliot
Copy link
Contributor

spouliot commented Mar 9, 2021

Follow up to questions inside #10774

Both nugets and xcframeworks are meant to distribute binary code that can work on several platforms.

This raise the issue on how a nuget can publish efficiently bindings to an XCFramework.

Before XCFramework a nuget could ship one, or more, assemblies for different platforms.

Those assemblies, when used for bindings, would have (generally embedded) native libraries.

Now we do not want /iOS/Bindings.dll to embed an XCFramework that includes binaries for iOS, tvOS, macOS, watchOS and MacCatalyst (...). Even more if the same nuget supports more platforms, e.g. tvOS/Bindings.dll ...

Ideally there would be a single .xcframework inside the nuget. Symlinks comes to mind but might not be Windows-friendly.

@spouliot spouliot added question The issue is a question macOS Issues affecting Xamarin.Mac iOS Issues affecting Xamarin.iOS labels Mar 9, 2021
@spouliot spouliot added this to the Future milestone Mar 9, 2021
@dkornev
Copy link

dkornev commented Apr 7, 2021

@leonluc-dev was trying to figure out how to make a NuGet package work with binding to an XCFramework in #10774
Since you created a different issue to follow up, I'll post the steps on how I made it work here.

On Mac I am still working on getting this side-car setup to work in NuGet packages. Creating a NuGet package using the project pack command without a nuspec file only adds the dll to the package, so a custom nuget package needs to be made to add the .resources folder as well.
I've tried to add the .resources file to the lib/xamarinios10 folder and the content folder in the package but neither option allows the Xamarin tooling to 'find' the .resources folder once the package is added to a project.
Looking in the folder documentation for NuGet it seems the content folder would be the best way to add the resources folder, since it allows arbitrary content. But I'm having a hard time figuring out how to get the Xamarin tooling to 'pick it up'.

So far as I know the easiest way to copy the .resources folder into the output dictionary is with the MSBuild .targets file.
I created one for my https://github.com/dkornev/TwilioVideoXamarinIOS binding:

Twilio.Video.iOS.csproj 

  <ItemGroup>
    <None Include="Twilio.Video.XamarinBinding.targets">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
  </ItemGroup>

In my .nuspec file, I set the .targets file created to be copied into the package build/ folder. Also added the binding itself (.dll) as lib/ and .resources folder as content/ The .targets file is automatically inserted into the project that uses our NuGet package:

twilio-video.nuspec

    <files>
        <file src="bin/Release/Twilio.Video.iOS.dll" target="lib/xamarinios10/Twilio.Video.iOS.dll" />
        <file src="bin/Release/Twilio.Video.iOS.resources/**" target="content/Twilio.Video.iOS.resources" />
        <file src="Twilio.Video.XamarinBinding.targets" target="build/Twilio.Video.XamarinBinding.targets" />
    </files>

The .targets file copies the .resources folder into the output folder and tells the Xamarin tooling where to 'find' the .resources:

Twilio.Video.XamarinBinding.targets

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <Target Name="BeforeCompile">
        <ItemGroup>
            <BindingResources Include="$(MSBuildThisFileDirectory)../content/Twilio.Video.iOS.resources/**/*.*" />
        </ItemGroup>
        <Copy SourceFiles="@(BindingResources)" DestinationFolder="$(TargetDir)/Twilio.Video.iOS.resources/%(RecursiveDir)" ContinueOnError="false" />
        <ItemGroup>
            <NativeReference Include="$(TargetDir)/Twilio.Video.iOS.resources\TwilioVideo.xcframework">
                <Kind>Framework</Kind>
                <SmartLink>False</SmartLink>
            </NativeReference>
        </ItemGroup>
    </Target>
</Project>

@jkasten2
Copy link

jkasten2 commented Nov 22, 2021

@dkornev Thanks for posting this!, I was able to follow these steps to get a different native .xcframework to work with my nuget package. I have few follow up comments and questions.

Improvement suggestions

  1. Important to note that in your .nuspec the .target file must have a target= value of build/*/<package_id>.targets.
    • Replace <package_id> with what you have as an id in .nuget file.
    • source
  2. The target= value should include the path Xamarin.iOS10/ so it doesn't get added to non-iOS projects when the NuGet package is consumed.
    • Example <file src="Twilio.Video.XamarinBinding.targets" target="build/Xamarin.iOS10/Twilio.Video.XamarinBinding.targets" />
  3. I would recommend also including here that the iOS binding project requires:
    • Bindings to XCFrameworks requires the native reference(s)
    • Bindings to XCFrameworks requires <NoBindingEmbedding>true</NoBindingEmbedding>
    • source

Questions

  1. @dkornev Why did you add the .target to your Twilio.Video.iOS.csproj? Copying this file to the root of the project doesn't seem needed, why not just use the full path for src in the .nuspec?
  2. Anyway the .targets file can be shared between the binding project and the consumer?
    • I don't like the idea of having maintain two entries for NativeReference. (one in my .csproj and one in this .targets consumer file)
  3. Is this workaround using anything that is deprecated? My concern is how fragile is this solution?

jkasten2 added a commit to OneSignal/OneSignal-Xamarin-SDK that referenced this issue Nov 22, 2021
Xamarin has limited support for `.xcframework` files, they do not have a
built in working solution for NuGet packages yet.
This works around the issue by including `OneSignal.xcframework` as-is
in the `.nupkg`. It also includes a `.target` file which is setting to
add the `OneSignal.xcframework` as a native reference.
See the comment in the `Com.OneSignal.nuspec` file for more details.

The items under `<NativeReference>` in `Com.OneSignal.targets` is from
`OneSignal.iOS.Binding.csproj` in this repo.

This was added based on the recommendations from:
   - xamarin/xamarin-macios#10819 (comment)
jkasten2 added a commit to OneSignal/OneSignal-Xamarin-SDK that referenced this issue Nov 22, 2021
Xamarin has limited support for `.xcframework` files, they do not have a
built in working solution for NuGet packages yet.
This works around the issue by including `OneSignal.xcframework` as-is
in the `.nupkg`. It also includes a `.target` file which is setting to
add the `OneSignal.xcframework` as a native reference.
See the comment in the `Com.OneSignal.nuspec` file for more details.

The items under `<NativeReference>` in `Com.OneSignal.targets` is from
`OneSignal.iOS.Binding.csproj` in this repo.

This was added based on the recommendations from:
   - xamarin/xamarin-macios#10819 (comment)
@leonluc-dev
Copy link

leonluc-dev commented Nov 22, 2021

@dkornev Combined with some of the tips by @jkasten2 this solution works!

It's a shame the official documentation on this is so minimal to non-existent. Especially since over time more third-party libraries are switching to xcframework.

A few recommendations for stuff I encountered:

Path length

  • Since XCFrameworks have many directory levels the path of their files can end up quite long already. To prevent hitting the max path length (especially on Windows based NuGet, which is often limited to 255 characters), it's recommended to keep the names of the directories in the content folder short.
    <file src="bin/Release/Twilio.Video.iOS.resources/**" target="content/Twilio.Video.iOS.resources" />
    to
    <file src="bin/$configuration$/Twilio.Video.iOS.resources/**" target="content/res" />

  • Make sure to change this in the targets file as well:
    <BindingResources Include="$(MSBuildThisFileDirectory)../content/res/**/*.*" />

Output folder

  • This is more of a "keep a clean target folder" thing. If you don't want a unused copy of the xcframework to end up in your bin/Release folder, you can adjust the targets file to copy them to the obj folder instead (before being embedded into the app itself).
    <Copy SourceFiles="@(BindingResources)" DestinationFolder="$(IntermediateOutputPath)/Twilio.Video.iOS/%(RecursiveDir)" ContinueOnError="false" />

  • Of course, make sure this is reflected in the native references in your targets file as well

<NativeReference Include="$(IntermediateOutputPath)/Twilio.Video.iOS/TwilioVideo.xcframework">
    <Kind>Framework</Kind>
    <SmartLink>False</SmartLink>
</NativeReference>

@rolfbjarne
Copy link
Member

This will be fixed in .NET, where we'll support dotnet pack for binding projects to create a NuGet with the *.xcframework (or *.framework / *.a files) in a location inside the NuGet where we know to find them at build time.

@jkasten2
Copy link

jkasten2 commented Feb 4, 2022

@rolfbjarne Thanks for the update, good to hear this will be supported! Is there a beta or preview where we can try this out now?
Doesn't seem live yet as I don't see any new details on XCFrameworks on this page:

@rolfbjarne
Copy link
Member

@jkasten2 that page won't get any updates about XCFrameworks, which is Apple-specific, and that page is more generic.

You can try the latest .NET 6 + MAUI preview and it should work there.

@nrudnyk
Copy link

nrudnyk commented Mar 7, 2022

This will be fixed in .NET, where we'll support dotnet pack for binding projects to create a NuGet with the *.xcframework (or *.framework / *.a files) in a location inside the NuGet where we know to find them at build time.

@rolfbjarne is there any ETA? or maybe there's a proper known workaround for NuGet while using .xcframework?
any input is greatly appreciated, thanks

@rolfbjarne
Copy link
Member

@nrudnyk ETA for .NET is second quarter in 2022 (https://github.com/dotnet/maui/wiki/roadmap).

or maybe there's a proper known workaround for NuGet while using .xcframework?

Other people in this issue seem to have been able to find a solution (#10819 (comment) for instance).

@nrudnyk
Copy link

nrudnyk commented Mar 8, 2022

@rolfbjarne that's exactly what I've tried, but when trying to consume Nuget Package, I've got the following issue
image

@nrudnyk
Copy link

nrudnyk commented Mar 8, 2022

@leonluc-dev do you have link for final .targets file? or maybe can share one (which includes fixes from above mentioned comments). Thanks

@LeadAssimilator
Copy link

LeadAssimilator commented Apr 11, 2022

For what it's worth, I've come up with a different solution which allows one to use the newer compressed binding resource package support and still have it work with the legacy Xamarin.iOS10 target framework when consuming the resulting nuget package from Windows. It was based off of the scheme xamarin/FacebookComponents used with some changes to support the compressed resources being used remotely. It works both locally on a Mac, remotely from Windows and within multi-targeting packages that support both legacy xamarin. and newer net6.0 frameworks, though each target framework will have the same xcframework compressed repeatedly.

The Xamarin.iOS10 target xcframework support is rather incomplete/broken in that it doesn't properly support zipped frameworks when used remotely. It is also a giant mess with some work performed by msbuild, MTouch and other tasks rather than all in one place. Zipped frameworks are crucial to avoid long path and file names issues on Windows and for macOS and MacCatalyst to preserve the symlinks since NuGet doesn't pack them properly and thus will fail during codesign.

The basic idea is to turn off binding embedding, enable compressing of the binding resources, include a build/buildTransitive targets file in the nuget, import it in the binding project and add a placeholder native reference. The targets file does the work of resolving the placeholder native reference into a real native reference based on the build context to either the xcframework or an inner arch specific framework, and copies the resources zip to a well known location that will get extracted automatically by MTouch during build. Some of the complexity could get simplified if you don't mind duplicating the native reference params in both the binding project and targets file. And most of that complexity only applies to legacy xamarin target frameworks, as the newer net6.0 ones just work when compressing the binding resource package.

MyBindingAssembly.csproj:

...
  <PropertyGroup>
    <NoBindingEmbedding>true</NoBindingEmbedding>
    <CompressBindingResourcePackage>true</CompressBindingResourcePackage>
  </PropertyGroup>

  <Import Project="MyBindingAssembly.targets" />
  <ItemGroup>
    <_NativeReference Include="MyBindingAssembly.xcframework">
      <_Id>$(_MyBindingAssemblyId)</_Id>
    </_NativeReference>
  </ItemGroup>
...

MyBindingAssembly.targets:

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <_MyBindingAssemblyId>MyBindingAssembly</_MyBindingAssemblyId>
  </PropertyGroup>

  <Target Name="_MyBindingAssemblyResolveNativeReferences" BeforeTargets="ResolveNativeReferences" Condition="('$(TargetFrameworks)' == '' And '$(TargetFramework)' == '') Or '$(IsBindingProject)' == 'true'">
    <PropertyGroup>
      <_MyBindingAssemblyXCFArch Condition="'$(_PlatformName)' == 'iOS' And '$(_SdkIsSimulator)' == 'False'">ios-arm64_armv7</_MyBindingAssemblyXCFArch>
      <_MyBindingAssemblyXCFArch Condition="'$(_PlatformName)' == 'iOS' And '$(_SdkIsSimulator)' == 'True'">ios-arm64_i386_x86_64-simulator</_MyBindingAssemblyXCFArch>
      <_MyBindingAssemblyXCFArch Condition="'$(_PlatformName)' == 'tvOS' And '$(_SdkIsSimulator)' == 'False'">tvos-arm64</_MyBindingAssemblyXCFArch>
      <_MyBindingAssemblyXCFArch Condition="'$(_PlatformName)' == 'tvOS' And '$(_SdkIsSimulator)' == 'True'">tvos-arm64_x86_64-simulator</_MyBindingAssemblyXCFArch>
      <_MyBindingAssemblyXCFArch Condition="'$(_PlatformName)' == 'MacCatalyst'">ios-arm64_x86_64-maccatalyst</_MyBindingAssemblyXCFArch>
    </PropertyGroup>

    <ItemGroup Condition="('$(OutputType)' != 'Library' OR '$(IsAppExtension)' == 'True')">
      <_NativeReference Include="$(DeviceSpecificIntermediateOutputPath)mtouch-cache\MyBindingAssembly.resources\MyBindingAssembly.xcframework\$(_MyBindingAssemblyXCFArch)\MyBindingAssembly.framework">
        <_Id>$(_MyBindingAssemblyId)</_Id>
      </_NativeReference>
    </ItemGroup>

    <ItemGroup>
      <_NativeReference Update="@(_NativeReference)" Condition="'%(_NativeReference._Id)' == '$(_MyBindingAssemblyId)'">
        <Kind>Framework</Kind>
        <SmartLink>True</SmartLink>
        <ForceLoad>True</ForceLoad>
        <LinkerFlags>-ObjC</LinkerFlags>
      </_NativeReference>
      <NativeReference Include="@(_NativeReference)" />
    </ItemGroup>

    <Ditto
      SessionId="$(BuildSessionId)"
      Condition="'$(IsMacEnabled)' == 'true' And ('$(OutputType)' != 'Library' Or '$(IsAppExtension)' == 'True')"
      Source="$(MSBuildThisFileDirectory)..\lib\xamarinios10\MyBindingAssembly.resources.zip"
      Destination="$(DeviceSpecificOutputPath)MyBindingAssembly.resources.zip"
    />
  </Target>
</Project>

This whole scheme works because MTouch checks for and unzips a resources.zip file that matches the assembly name used for references. It does this to read an embedded manifest file to add additional native references to link with. Unfortunately it does nothing when it encounters an xcframework because it expects an msbuild task ResolveNativeReferences to do that. But we can at least leverage this partial support to automatically unzip the resources for us as long as we can put the resources.zip in the right location. The Ditto task will do just that and does so correctly when run both locally on the Mac and remotely from Windows. Alternatively, the implicit MTouch behavior could be replaced with the Exec task that has SessionId set appropriately and Command set to unzip the resources.zip.

Now there is a big problem with the ResolveNativeReferences task in that it doesn't know we have a NativeReference in a nuget package reference that isn't embedded. That is actually because such nuget references are intentionally excluded. The typical workaround is to use the build/buildTransitive targets import to inject NativeReference in the project being built. Alternatively the nuget package references could get forcibly added back to the References ItemGroup. Unfortunately neither option will work because the xcframework is compressed in the resources.zip. If we force it, it even attempts to run /usr/bin/unzip on Windows rather than on the connected Mac, in an effort to inspect the frameworks archs and their info plist files within the zip to perform the resolve, though it would work when run locally on a Mac.

The only viable workaround is a variant of the typical one to inject a NativeReference during the build but here we must perform the resolution of an xcframework into a framework ourselves by knowing the available architectures the xcframework supports and checking the various available build properties to select it. Once the correct architecture is determined, it can then be emitted as a NativeReference per usual, avoiding the need for the compressed xcframework to be extracted and bypassing the work ResolveNativeReferences normally performs.

Everything would work so much better if the ResolveNativeReferences task was fixed. Why this was never implemented properly in the first place just shows how the xamarin devs don't actually make real apps with their tools and/or fundamentally misunderstand real world development workflows.

While the above scheme works, multi-targeting can lead to package bloat. This could potentially be solved by augmenting the _CreateBindingResourcePackage target to exclude the xcframework or using a separate nuspec rather than the pack target and separately zip and include the xcframework in another location. Doing so would also require injecting the NativeReference items for all target frameworks, including the newer net6.0 ones as the automatic support that looks for the assembly.resources.zip with an embedded manifest would no longer function correctly in that case.

@ghost ghost locked as resolved and limited conversation to collaborators May 19, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
iOS Issues affecting Xamarin.iOS macOS Issues affecting Xamarin.Mac question The issue is a question
Projects
None yet
Development

No branches or pull requests

7 participants