iOS/OSX universal binaries handling

[SSE4]

universal (fat) binaries are pretty common for Apple platforms - that's just binaries for several architectures combined together.

sorry for the long text - but I am trying to describe my use case as detailed as possible :)
e.g. for OSX it's common to have x86 + x64 universal binaries, while for iOS it's usually armv7 + armv7s + arm64/armv8 combined together, plus optionally x86 and x64, if iOS simulator is requied (so from 3 to 5 architectues). same story for other Apple platforms, such as watchOS and tvOS.
usually, universal binaries are produced by running multiple builds and then running lipo utility on resulting shared/static libraries. there is alternate approach to run single universal build, but it tends to be complicated and error-prone with configure (autotools) style projects - e.g. such configure scripts need to detect size of pointer, which is ambiguous in case of universal binaries (sizeof(void*) - 4 or 8?).

I need some advice on how to proceed with universal binaries in conan. there are several approaches how it could be done:

1. conan receipt invokes N builds for N architectures and runs lipo to combine as postprocessing.
   this is the way I am currently using. typical conan file may look like:

if self.settings.os == "Windows":
   self.build_with_msvc()
else:
   self.build_with_configure()

and then for OSX/iOS it will become much more complicated, like:

if self.settings.os == "Windows":
   self.build_with_msvc()
elif self.settings.os == "iOS":
   arches = ["armv7", "armv7s", "arm64", "i386", "x86_64"]
   for arch in arches :
        self.build_with_configure(arch, "iphoneos")
    self.lipo(arches)
elif self.settings.os == "Macos":
   arches = ["i386", "x86_64"]
   for arch in arches :
        self.build_with_configure(arch, "macos")
    self.lipo(arches)
else:
   self.build_with_configure()

there are few disadvantages of such approach:

• conanfile has very special path for iOS and Macos, although build process is same as for other NIX* platforms
• lots of code copy-pasted from package to package for universal binaries handling
• package "arch" setting is no longer showing the truth, moreover, it's confusing field in case of universal binaries

2. build N conan packages for each architecture and then combine them somehow
   the idea is to build conan packages for each architecture (armv7, arm64, etc) as usual and then somehow combine them into aggregated multi-architecture conan package. I am not sure if it's even possible now to have some approach for conan package to support multiple settings at the same time, but probably there are more use cases for such combined packages (e.g. Windows tools which may run on both x86 and x64, but not on ARM might have combined x86+x64 indication). probably this approach ends up in some new conan command line, like "conan combine" or "conan lipo" which will run lipo for all libraries from packages to be combined.

some unclear points about this approach:

• while binaries are in general easy to handle, headers might be tricky part if they are different for different arches (in my practice I just copy headers from sources, but some headers might be generated from the build process and contain definitions like SIZE_OF_VOIDP)
• anyway, conan somehow needs to know which binaries to process and where are they located (but in worst case, conan still can wildcard *so *a *dylib)
• some convention on how to identify and describe combined packages shall be developed
• new conan command will be too OS-specific which might be not good (or useless for other arches?)
• probably issues with conflict resolution may appear - e.g. if asked for x86 arch, what to install, x86 or x86+x64 package?

[memsharded]

cc/ @tru

[memsharded]

package "arch" setting is no longer showing the truth, moreover, it's confusing field in case of universal binaries

Yes, I think in that case (approach 1), the arch setting should be removed from the settings. Exactly the same, as if packaging a multi-config (debug/release) package, you should remove the build_type from settings.

Logic for having packages that matches different settings can be done in the package_id() method, for example:

def package_id(self):
      if "arm" in self.settings.arch:
             self.info.settings.arch = "AnyARM"

This will build one package for any arm architecture, and one package for x86, and another for x86_64 and so on.

I like the approach (1), it is explicit, reads well. You say it is much more complex, but the intention is so evident and explicit, you can perfectly know what the recipe is doing. And it is just a few lines. So IMHO, it will be more robust and easy to maintain that other approaches. If sharing code, maybe reusing such code via shared python packages could be an option.

You can see that the explicit build of different configurations is what I am proposing for multi-config (debug/release) packages in https://github.com/conan-io/docs/pull/161/files#diff-e4c22b2beb97bec45bf31f8a032893afR120 (PR to the docs, with some new 0.20.0 features)

[SSE4]

@memsharded sounds good, I am going to try approach with package_id()
for the code sharing, I am not sure if such python package can be useful outside of conan, what do you think if I contribute to conan.tools adding several helper functions for dealing with Apple-specific stuff, just like tools.vcvars_command?

[memsharded]

Yes, those utility functions can be added to tools, are the same concept as vcvars_command helper, so they will be easy to use and useful for more people there. So, sounds just perfect, go ahead.

[tru]

I am very interested in this considering I was pestering @memsharded about this the other day. I think the tricky parts are the following:

1. how to save away the binaries between each build phase to have them easily available for the lipo step
2. how to clean out the source directory so that nothing is left to break the next build. Ideally we would be able to have one build dir per arch - that would solve that.

Just spitballing here, but the "perfect" setup would actually be if conan could do the following:

• run source once
• for arch in archs:
  • copy source to build-arch
  • run build
  • run package function (to copy all lib headers etc)
• run combination method that runs lipo on all the artifacts collected in package function.

This could be done by adding a new property variants at a toplevel of the conanfile and then set self.current_variant or something like that between each run. And then add a new post_package function to allow for lipo combination or other post-processing before it's packaged up.

This would make the conanfile much much nicer since it can reuse the same methods without a lot of extra code for one specific platform.

If self.current_variant is not good enough I think we could allow build and package to take a new variable if variants are defined. There would have to be some runtime checking of code to make sure it's backwards compatible.

[tru]

Here is a mock example of how it could look:

class MultipleVariant(ConanFile):
  name = "testmultiple"
  version = "1.0"
  settings = "os", "compiler", "build_type", "arch"

  def configure(self):
    if self.settings.os == "iOS":
      self.variants = ["arm7", "arm64", "x86_64"]
    elif self.settings.os == "MacOS":
      self.variants = ["x86", "x86_64"]

  def source(self):
    tools.download("http://foo.com/bar.bz", "bar.bz")
    tools.unzip("bar.bz")

  def build(self, variant):
    self.run("./configure --arch={0}".format(variant))
    self.run("make")

  def package(self, variant):
    self.copy("*.dylib", src="lib", dst="tmp/" + variant)

  def combine_package(self):
    if not self.variants:
      return

    libs = [l for l in os.listdir("tmp/x86_64") if l.endswith(".dylib")]
    for lib in libs:
      self.run("lipo -o lib/{0} {1}".format(lib, [os.path.join("tmp", var, lib) for var in self.variants]))

    shutil.rmtree("tmp")

[SSE4]

@tru that sounds like a good addition as well. currently in my scripts I am building into the $TMPDIR/{arch}, which is obviously outside of conan source directory. adding variants and post-processing method to the conanfile sounds like a great idea.

[tru]

This can be used for much more than just iOS/MacOS binaries as well. I have at least one other package where I need to build it twice right now that could benefit.

[tru]

For extra points I guess variants could be set on a global level in the profile or something and the packages can choose to use it by adding the variant argument to the build function. Since most likely it's usually controlled on a global level.

[lasote]

This can be used for much more than just iOS/MacOS binaries as well. I have at least one other package where I need to build it twice right now that could benefit.

Why do you need to build twice that package? Remember that we support in 0.20.0 the build_id method. http://docs.conan.io/en/latest/reference/conanfile.html#build-id

For extra points I guess variants could be set on a global level in the profile or something and the packages can choose to use it by adding the variant argument to the build function. Since most likely it's usually controlled on a global level.

I'm not sure about a parallel implementation to support variant overriding. It could be done with envionment variable, remember that now you can use profiles to assign package level variables:

Profile:

[env]
testmultiple:TESTMULTIPLE_IOS_VARIANTS = "armv8,x86_64"
testmultiple:TESTMULTIPLE_MACOS_VARIANTS = "x86"

Conanfile:

def configure(self):
    if self.settings.os == "iOS":
      self.variants = (os.environ.get("TESTMULTIPLE_IOS_VARIANTS", None) or "arm7,arm64, x86_64").split(",")
    elif self.settings.os == "MacOS":
      self.variants = (os.environ.get("TESTMULTIPLE_MACOS_VARIANTS", None) or  "x86,x86_64").split(",")

I like the idea, but I need to think about possible problems implementing this variants feature.

[memsharded]

This seems like the opposite behavior of the one implemented with the new build_id(), which basically allows to build once, package many. The request would be, build many, package once. Will think about it too.

[lasote]

HI, I've been talking with @memsharded and we are not sure about the variants. Please, look at this example code and tell me the problems that you see:

class MultipleVariant(ConanFile):
    name = "testmultiple"
    version = "1.0"
    settings = "os", "compiler", "build_type", "arch"

    def package_id(self):
        if self.settings.arch in self.variants:
            self.info.settings.arch = ",".join(self.variants)

    def source(self):
        tools.download("http://foo.com/bar.bz", "bar.bz")
        tools.unzip("bar.bz")

    def build(self):
        for variant in self.variants:
            variant_dir = self.variant_dir(variant)
            mkdir(variant_dir)
            copy_build_files_to(variant_dir, exclude="tmp_*")  # <= Pseudocode, could provide some tool
            with tools.chdir(variant_dir):
                self.run("./configure --arch={0}".format(variant))
                self.run("make")

    def package(self, variant):
        for variant in self.variants:
            variant_dir = self.variant_dir(variant)
            self.copy("*.dylib", src="%s/lib" % variant_dir, dst="tmp/" + variant)
        self.combine_package()

    @property
    def variants(self):
        if self.settings.os == "iOS":
            return ["arm7", "arm64", "x86_64"]
        elif self.settings.os == "MacOS":
            return ["x86", "x86_64"]
        return [self.settings.arch]

    def variant_dir(self, variant):
        return "tmp_%s" % variant

    def combine_package(self):
        if not self.variants:
            return

        libs = [l for l in os.listdir("tmp/x86_64") if l.endswith(".dylib")]
        for lib in libs:
            self.run("lipo -o lib/{0} {1}".format(lib, [os.path.join("tmp", var, lib) for var in self.variants]))

        shutil.rmtree("tmp")

We think that it's more explicit and do not introduce more complexity in the conan model/processes.
I didn't touch your combine_package method, the variants property is almost the same code, the package_id method is needed also in your example, and the build method is not too much complex, we could provide some tool to ease the sources copy. WDYT?

[tru]

Yeah I don't see a huge problem with what you layed out there. The only thing that I need to dig into is how easy it is to copy the sources without doing it a great many times. I could centralize some of those functions into my own conantools package as well so I think it will be OK.

[lasote]

Ok, feel free to share with us your tool to copy the sources, we can put it in the conans.tools package.

[SSE4]

by the way, usually it even is't required to copy sources - e.g. with most configure-style projects "--prefix" can be used to ensure "make install" copies headers, libraries and other stuff into the required directory, and "make clean"/"make distclean" ensures that working copy has no leftover after the previous build(s).

[tru]

It's a great theory - but I have seen many problems with distclean not cleaning out extra files. I probably want to copy sources anyway to be 100% sure.

[SSE4]

yeah - that's okay and I think is up to package author whether to copy sources or just use --prefix. sometimes it's just waste of time that should be avoided (e.g. in case of boost, which also provides b2 clean).

[lasote]

That's another reason to keep it explicit, we could provide some tool to copy sources, but if any package only needs to change the prefix (or any other mechanism) won't be forced to copy again the sources.

[tru]

Related question to this: I am setting my CFLAGS from my profile, but now when we build all ARCH's in one "go" I wonder what the best way to have variables in the profiles could be, consider this from my profile:

CFLAGS=-arch arm64

changed to:

CFLAGS=-arch {IOS_ARCH}

now I need a way to expand that variable, one idea is to do:

oldenv = os.environ
for env in os.environ:
  newenv[env] = os.environ[env].format(IOS_ARCH=variant)
os.environ=newenv
build(...)
os.environ=oldenv

anyone can think of anything better?

[SSE4]

I would leave CFLAGS in profile without arch, and append arch in conanfile, e.g.:

cflags = os.environ["CFLAGS"] + " -arch=%s" % variant
build(cflags)

actually, there are more flags to be set depends on variant - e.g. -isysroot is usually required

[tru]

Yeah that might be better.

[MathPlayer]

It passed quite some time since last comment and yet there are no changes related to this issue in the tools/apple.py file. Maybe I'm missing the place with related changes? Is anybody still interested/working on adding some helpers for handling universal/fat binaries? I can contribute for such helpers if no one else is working on this.

[memsharded]

Recently, @SSE4 was doing some great work related to apple tools in: conan-io/conan#1815.

I think there is still interest, but the problem might be more challenging than seems at first sight, as the packaging many-to-one flow is not defined in conan so far. So depending on what would be the approach, it might be very difficult. Having some helpers to build all the required variants in the same package, that seems doable.

[MathPlayer]

I analyzed this issue for a bit and yes, it looks quite hard to find a generic solution.
Since I need this kind of functionality only when building for Apple platforms, I think the best thing to do is to add it in a package like https://github.com/theodelrieu/conan-darwin-toolchain

[anton-matosov]

Looks like CMake has direct support for fat binaries now https://cmake.org/cmake/help/latest/variable/CMAKE_OSX_ARCHITECTURES.html
I have not tried it yet, but looks promising.
If Conan would allow array of target architectures instead of single value it would match cmake (and Xcode) approach and make it easy to integrate

[canatella]

This kind of many to one flow is still needed on Android when building with the ndk provided cmake toolchain (which is required if you want to build for a recent NDK).

Another approach would be to allow requirements to specify settings. So I could create a parent package that would require a child with android abi armeabi-v7a and the same child with arm64-v8a. Then assemble them in an aar file for Android distribution.

so I could do a

self.build_requires('mypackage/1.0@repo/stable', settings={ 'os.abi': ['armeabi-v7a', 'arm64-v8a']})

[tonyarnold]

Sorry to necropost, but did this issue ever end up bearing fruit?

How are other Conan users generating universal binaries containing x86_64 and arm64 code for platforms like macOS?

[sashao]

For my project, we solved this by using conan profiles, and adding missing symbols manually.

1. have one conandfile.txt with list of dependencies
2. create 2 profiles for macos arm and x86_64 architectures and install them in separate folders.
3. in this subfolders in separate CMakeLists.txt to not bloat global context load package info for each arch:
   include( ${CMAKE_CURRENT_SOURCE_DIR}/conan_paths.cmake )
   and
   SET (CONAN_LIBS
   ${CONAN_GTEST_ROOT}/lib/libgtest.a
   ${CONAN_GTEST_ROOT}/lib/libgtest_main.a
   ${CONAN_GTEST_ROOT}/lib/libgmock.a)
   SET(CONAN_LIBS_ARM ${CONAN_LIBS})
4. append target_link_libraries with this variables:
   target_link_libraries(
   gtestsample
   GTest::gtest_main
   ${CONAN_LIBS_ARM} ${CONAN_LIBS_X86}
   )
5. set(CMAKE_OSX_ARCHITECTURES "arm64;x86_64")

This allows universal binaries building for macos.

[crazydef]

In the end I opted to spend a day writing my own package manager that provides us with the functionality we need without all this hassle. :)

[gmeeker]

@memsharded my goal (which I imagine is a common one) is to build an application with many CCI dependencies. The app's recipe should then generate an Xcode project with CMake that will link a fat binary. Xcode can sign the application bundle and upload to the app store (especially for iOS developers). I think having a multi-arch Xcode project will make this more familiar.

My pre-2.0 attempt (which is a huge hack) worked like this:

• Run conan install for each arch.
• Run lipo to make fat packages under ~/.conan/data (under a new hash)
• Patch the output of the generators to point to the fat packages (library paths, etc.)
• Use a CMake toolchain so the app builds as fat

Here's the code. The lipo stuff is pretty useful. The rest is a mess but it does work.
https://github.com/gmeeker/BuildConan.git

I'm not sure if this is the workflow others envision but I can try to move my lipo code to a deployer. I'm not sure how to communicate these fat library paths to the CMake tool.

[gegles]

@gmeeker, Thank you so much! We are desperate for something like this and your biggest cheerleaders! ;-)

If it helps at all, we have 2 very different use cases that would both benefit:

1. on macOS, we have a pretty big CMake Qt App, that depends on a 25-30 conan deps (including openssl).. It also builds sophisticated installers using CPack... We currently provide 2 separate versions of the App, (arm64 and x86_64) but our hope would be to one day provide a single universal app & installer... thanks to your work! ;-)

2. We have a low-level C SDK that we wrap in Swift. It's also a CMake project and depends on a lot of conan deps (including openssl). The end goal is an iOS/iOS-Simulator/macOS universal XCFramework... so once again, your work would be key for us to get there...

[zhangyiant]

Is it possible to define new arch type for universal binary? For example: universal_x86_64_arm64. And generate CMake toolchain file with correct CMAKE_OSX_ARCHITECTURES setting.
I don't think there will be many variants. On macOS, only one universal (x86_64 + arm64) is necessary for now.
This could avoid building twice and using lipo.

[memsharded]

Thanks for the suggestion @zhangyiant but I don't think it would be possible. I think I have seen users doing fat binaries for more configurations too, like should they bundle debug and release too? ios/watchos? It it is only one and exactly one configuration for all Mac users we could consider it, but I am not sure it will be the case.

[crazydef]

@memsharded Regardless of what you think you've seen people doing, fat binaries are designed to combine different architectures. Not platforms. Not configurations. Just architectures. And if Conan isn't willing to support that, then it's not fit for purpose. Forcing people to complicate their pipelines and build processes to fit the limitations of a single tool will very quickly ensure that tool is replaced.

[hjmallon]

Fat binaries are fundamentally squashed together binaries indexed by cpu type. There have been lots of theoretically useful selections (list is not exhaustive):

• OSX PowerPC / Intel
• OSX Intel 32 / 64 bit
• macOS intel 64bit / arm 64bit
• iOS various flavours of arm 32 and arm 64
• iOS plus simulator (iOS arm 64 / iOS simulator intel 64)

where I have ios you can sub in tvOS or similar

However, most of these combinations are not useful any more in the general case.

• iOS has been exclusively arm 64 for long enough that dropping arm 32 is valid.
• Intel 32 and powerpc are historically interesting only.
• Since macOS now has arm 64 you can no longer make fat binaries with ios and simulator since you need two arm 64 slices (one ios and one ios simulator) which is not supported.

To solve the iOS plus simulator use case apple introduced a folder structure called xcframework which has multiple binaries for each platform (this is how you can do multiplatform now). But it is not supported by CMake yet (maybe some other build systems have special support).

So as far as I’m concerned there is one remaining important universal binary platform, which is universal 2 (intel 64 plus arm 64 macOS)

[zhangyiant]

Yes. For macOS, only universal 2(x86_64 and arm64) is useful nowadays I think.

[SSE4]

iOS + iOS Simulator is still used nowadays (at least on x86_64 Macs, as not everyone upgraded to M1 yet).

[DoDoENT]

With conan v2.0, you can use the compatibility plugin to define that your arch macos_universal is binary compatible with armv8 and x86_64. Then, such a package could be consumed also by packages that have an armv8 or x86_64 arch set (you can even use None instead of custom macos_universal).
However, this still doesn't resolve the issue of building such a package. Unfortunately, as far as I know, you need to have custom logic for building that. CMAKE_OSX_ARCHITECTURES works correctly only with CMake projects and with the Xcode generator (even CMake+Ninja results with rebuildable code for most projects if this variable is set). This, however, means that your package's build script needs to correctly handle the Xcode IDE generator for CMake, something that is not common for most open-source projects. I think this is the main reason why conan-center packages are officially not supporting creating universal-arch binaries.

[memsharded]

I think there is a big difference between this is "the most common/important/mainstream way nowadays" and "this is the only way nowadays".

Given the very large user base, and from my experience talking to many hundreds of users, there will still be some of them that are dealing with other variants rather than x86_64 + arm64. Maybe not many, but even if this is a 2-5% of users, multiplied by thousands, it will still be a relevant number. I also think that it is even likely that there might be other future variants, like M1, M2, maybe there will be M3 or something like that, that introduces a new architecture in the equation.

If any of that happens, then this approach would be quite poor, as the combinatorics and evolution to take into account other architectures will make it a dirty hack. If we then try to deprecate it, we will get the fire from users relying on that, and not concerned that the solution is bad for other users. If we don't, then the complexity starts to pile up, becomes technical debt, slows down the team and affects morale... You don't have to deal with it, but we do, and it is not a pleasant experience. The larger the Conan user base, the most important is to carefully consider the alternatives, approaches and solutions and make sure they scale to the wider community, and they are not just partial or incomplete, because those rotten very quickly.

I am fine with doing an experimental try of adding this combined architecture for apple, and see how it goes, I'll take it to the team for discussion.

[gegles]

Is anybody actively working on this? I am happy to be a guinea pig if it helps. This is a huge need for us at multiple levels... both to create a single universal set of libs, exes and installers... for a macOS app, but also for mobile SDKs... Ideally I'd love to even see another extension to wrap multiple Frameworks into an xcframework (i.e. macOS + iOS + ...) but that's maybe wishful thinking.. ;-)

[memsharded]

Hi @gegles

Sorry, we haven't had the time to prioritize this yet. Still in our roadmap, but team still over capacity with other priorities.

After sharing with the team and carefully analyzing all the feedback, it seems that managing internal Conan binaries as universal binaries is not a viable approach, and doesn't provide much value in the developer stage, while adding a ton of complexities to Conan internal, and it will be likely fragile and problematic to scale. So we will focus on the automation for creating universal binaries out of the Conan single-architecture binaries, in an easy way that allows Conan users to create and deliver their binaries (outside of Conan) as universal binaries.

The first development stage would probably be an extension (likely a deployer, maybe a custom command) in https://github.com/conan-io/conan-extensions repo. If anyone wants to start contributing something in this line, it would be welcomed.

[gmeeker]

I've had other priorities, but I'm planning to investigate this soon, maybe over the summer. See my comment in March 4th. I think my previous work could be partially adapted to a 2.0 deployer and custom command. However, my goal was to make the final app recipe build as universal and consume lipo'd packages, not build the app and then lipo it. The idea was to generate an Xcode project that could code sign, submit to the app store, etc. I'll focus on the lipo deployer first and that much might be useful to others.

[NoWiseMan]

We are in the process of upgrading our current packages to conan 2, with this issue still giving me the biggest headache here. Currently we build all our libraries as fat libs, so we have custom Python code in each package (not a good approach) and we still have the problem that we currently can't support the iOS simulator on ARM-based Macs (the new default development machines).

A deployer could be a solution if:

1. it is possible to create a fat lib (for macOS)
2. it is possible to create an xcframework (for iOS/watchOS/etc)
3. it could be combined with a generator to create *.cmake files to find the "deployed" libs/xcframeworks

I currently haven't seen a way for a deployer to consume the libs from different configurations, or for a deployer to create the needed information to consume the deployed artifacts.
Maybe I am missing something here.

[gmeeker]

I've finally made some progress on the first stage here. A custom deployer isn't terribly useful because it's getting passed a single arch from conan. Instead I wrote a custom command that acts like conan install --deploy=full_deploy but it runs once per arch and then uses lipo to make a universal version under the folder 'full_deploy'.
https://github.com/gmeeker/conan-deploy-lipo

If this seems like a useful design, I can contribute it to the conan-extensions repo above. (I think the _lipo.py helper should end up in mainline conan under tools.apple so I'm leaving it in a separate file for now.) I haven't looked at iOS at all.

Note that this does not integrate with generators in any way, but at least it seems to fit in with conan's design goals. I do plan to pursue that next, but it's probably still going to involve some ugly hacks like in my BuildConan repo above, such as patching the output files of generators to point to the universal libraries. I haven't been able to get Conan 2.0's CMake or Xcode tools to allow multiple archs (like the previous hack to use a custom toolchain).

In the meantime, look at the xcode example in conan-deploy-lipo. You deploy the universal binaries using conan and then you develop using a standalone Xcode project. It might be enough some purposes (very small number of conan dependencies or a project that isn't cross platform).

[ssrobins]

@gmeeker, thanks for working on this!

Here's the hack I've been using to support multiple architectures on iOS in Conan 2.0 CMake:
https://github.com/ssrobins/conan-recipes/blob/58f01ea7435cfa8c6c1b8caba82f556ad57db9b5/recipes/zlib/conanfile.py#L44
I've been maintaining my own copy of recipes for open-source projects, though I've been informed my free-tier artifactory is going away so that may not work for long. At work, I'm pre-patching recipes before they get uploaded to a private artifactory server. The main advantage of this is no other customization to the Conan workflow is needed,conan install just works as expected, though the arch setting is not the whole truth.

@memsharded I know Conan is supposed to be build-system agnostic, but if there was a way to pass a CMAKE_OSX_ARCHITECTURES CMake variable value via conan install and have it be used for recipes that use CMake to build, that lets us leverage the existing automation in the CMake-generated build systems themselves to produce the universal binaries. It would also make it easy to support universal binaries with https://github.com/conan-io/cmake-conan/tree/develop2. I would expect no guarantee of recipe compatibility when using this setting (I would happily contribute fixes to https://github.com/conan-io/conan-center-index as I find problems) and I certainly wouldn't expect any universal binaries to be available on Conan Center. GitHub Actions caching is actually a pretty serviceable replacement for having binaries on artifactory. It would still be way easier than what I do today.

Anyway, whichever method is used to fix this, I'll help where I can!

[gmeeker]

Yikes! Patching individual recipes in CCI sounds like quite a rabbit hole. I looked into that, and other options, but we have too many dependencies.

I'll put up a PR in conan-extensions soon for the deployment custom command.

What I did here:
https://github.com/gmeeker/BuildConan.git
(Conan 1.0 and a big messy hack but working and doesn't require changing recipes in dependencies)

1. Run conan to build dependencies for each arch
2. Lipo each dependencies and store in local cache (by injecting new settings). This could be separate cache to avoid messing with conan too much, but it's nice to reuse the cache management.
3. Patch all the paths in the generated cmake files to point to the universal binaries.
4. Use a CMake toolchain to set the archs (which is no longer possible in 2.0 I believe, except maybe by patching files after the generator runs?)

My next plan is to try rewriting that for 2.0 and see what still needs to access conan internals. Hopefully it's cleaner this time, and if so maybe could be considered for conan-extensions.

[gmeeker]

@ssrobins Oh, thanks for the tip for setting CMAKE_OSX_ARCHITECTURES! I'll try to spend some time on the rest of it soon.

[DoDoENT]

I know Conan is supposed to be build-system agnostic, but if there was a way to pass a CMAKE_OSX_ARCHITECTURES CMake variable value via ...

Unfortunately, CMAKE_OSX_ARCHITECTURES works correctly with only the Xcode generator, and this requires the project's CMake script to be written in a way to support IDE generators, which is truly rare - usually the CMake script of open source project expects single-config generator, such as Ninja or Make, and optionally has Visual Studio support.

I also had to patch a lot of open-source projects' build systems in order to get a correctly built universal binary. The problem is that CMake will "support" multi-arch build even with ninja, but this does not generate separate build tasks for each architecture - instead, it simply puts both architectures' flags to the clang invocation. This works OK for generic code, but doesn't work for source files that have specific intel or ARM optimizations, like those in OpenSSL, libjpeg-turbo, OpenCV, and similar high-performance libraries.

Even if conan supported passing this variable, it would work only with packages that use CMake as their build system and whose CMake script is written in a "modern" way using generator expressions rather than depending on CMAKE_BUILD_TYPE variable.

[ssrobins]

Unfortunately, CMAKE_OSX_ARCHITECTURES works correctly with only the Xcode generator, and this requires the project's CMake script to be written in a way to support IDE generators, which is truly rare - usually the CMake script of open source project expects single-config generator, such as Ninja or Make, and optionally has Visual Studio support.

I also had to patch a lot of open-source projects' build systems in order to get a correctly built universal binary. The problem is that CMake will "support" multi-arch build even with ninja, but this does not generate separate build tasks for each architecture - instead, it simply puts both architectures' flags to the clang invocation. This works OK for generic code, but doesn't work for source files that have specific intel or ARM optimizations, like those in OpenSSL, libjpeg-turbo, OpenCV, and similar high-performance libraries.

Interesting point, I was not aware of that.

Even if conan supported passing this variable, it would work only with packages that use CMake as their build system and whose CMake script is written in a "modern" way using generator expressions rather than depending on CMAKE_BUILD_TYPE variable.

I've not run into issues with CMAKE_OSX_ARCHITECTURES in single-config generators, how does that affect architecture settings?

[DoDoENT]

I've not run into issues with CMAKE_OSX_ARCHITECTURES in single-config generators, how does that affect architecture settings?

It depends on the project. Ninja/Make will add both -arch x86_64 and -arch armv8 to the clang invocation and this will work as long as:

• apple clang is used (I haven't tested with LLVM clang from homebrew)
• no other arch-specific flags are given (e.g. -mavx2 for enabling Haswell instructions on Intel)
• the code is written in a way that every source file compiles for every architecture

The last point is especially important for projects like libjpeg-turbo and OpenSSL where the source contains dedicated source files that only need to be compiled on Intel and dedicated source files for ARM. CMake will include both sets of files into both architectures and the build will fail. This will also happen with Xcode generator, so you usually need to patch the cmake script of such projects. For example, I used the following patch for libjpeg-turbo in my fork:

set_target_properties( simd PROPERTIES
    XCODE_ATTRIBUTE_EXCLUDED_SOURCE_FILE_NAMES[arch=x86_64] "${ARM_COMMON_SRCS} ${ARM64_SRCS} ${ARM_SRCS} ${INTEL32_SRCS}"
    XCODE_ATTRIBUTE_EXCLUDED_SOURCE_FILE_NAMES[arch=i386]   "${ARM_COMMON_SRCS} ${ARM64_SRCS} ${ARM_SRCS} ${INTEL64_SRCS}"
    XCODE_ATTRIBUTE_EXCLUDED_SOURCE_FILE_NAMES[arch=armv7]  "${INTEL64_SRCS} ${INTEL32_SRCS} ${intel_generic_sources} ${ARM64_SRCS}"
    XCODE_ATTRIBUTE_EXCLUDED_SOURCE_FILE_NAMES[arch=armv7s] "${INTEL64_SRCS} ${INTEL32_SRCS} ${intel_generic_sources} ${ARM64_SRCS}"
    XCODE_ATTRIBUTE_EXCLUDED_SOURCE_FILE_NAMES[arch=arm64]  "${INTEL64_SRCS} ${INTEL32_SRCS} ${intel_generic_sources} ${ARM_SRCS}"

    XCODE_ATTRIBUTE_GCC_PREPROCESSOR_DEFINITIONS[arch=armv7]  "$(GCC_PREPROCESSOR_DEFINITIONS) NEON_INTRINSICS=1"
    XCODE_ATTRIBUTE_GCC_PREPROCESSOR_DEFINITIONS[arch=armv7s] "$(GCC_PREPROCESSOR_DEFINITIONS) NEON_INTRINSICS=1"
    XCODE_ATTRIBUTE_GCC_PREPROCESSOR_DEFINITIONS[arch=arm64]  "$(GCC_PREPROCESSOR_DEFINITIONS) NEON_INTRINSICS=1"
)

This of course works only with the Xcode generator, and I really have no idea how to apply that to Ninja/Make with the CMAKE_OSX_ARCHITECTURES variable set.

The problem is not in Conan - it's usually in the packages and their build scripts.

The only thing I think Conan could help with here is the idea that was already mentioned above - add "built-in" support for obtaining both arm64 and x86_64 dependency graphs and generate a .xcconfig file in a way to reference header/library paths in both SDK-specific (i.e. iphoneos vs iphonesimulator) and arch-specific (e.g. x86_64 vs arm64) way. This can also be done using XCODE_ATTRIBUTE_* CMake variables if the Xcode generator is to be used.

But, on the other hand, it should also be possible to write a custom conan command that performs multiple arch installs and then post-processes the generated *.xcconfig (in case of XcodeDeps) or *.cmake (in case of CMakeDeps) files according to the description above.

An alternative is to use the custom deployer that smashes binaries together using lipo, as mentioned above.

You should choose which approach you should use depending on what suits you best. The deployer approach seems easier but AFAIK it won't work in an environment where some packages are already universal, and some are not. The postprocessor step in the custom command could possibly handle that (it only needs to check if a certain package has a universal arch or not).

[memsharded]

Moved this ticket to the conan-extensions repo, where the work is being done, like #58

[gmeeker]

Any thoughts about this limited approach in #52?
A custom command in #53 seemed to run into complications in the implementation while the deployer in #58 is straightforward. I have not tested either on iOS.
This is sufficient to obtain universal binaries out of conan and use a hand crafted Xcode project. This does not address building the final application through conan + cmake + Xcode.