PyTorch macosx-arm64 cross-build

[sbrunk]

TL;DR
Can we use the x86_64 Python interpreter for cross-compilation of PyTorch to macosx-arm64 or will it cause issues? If we can, what is a good way to configure it especially for CI builds? If not, what might be other options?

Background

We only have x86_64 machines in CI, so we currently have to cross-compile macosx-arm64 artifacts. With #1459 out of the way, I got cross-compilation PyTorch working locally for #1069. I can compile arm64 artifacts on x86_64 that are working fine on an arm64 machine.

What I'm not sure about though is the Python build dependency. One thing it's used for is to call the build:

javacpp-presets/pytorch/cppbuild.sh

Line 185 in ee1401d

"$PYTHON_BIN_PATH" setup.py build

Which means for cross-compilation we need an interpreter of the host arch (x86_64 here). It is also used for other things though, i.e. and for numpy stuff and in CFLAGS here:

javacpp-presets/pytorch/cppbuild.sh

Line 105 in ee1401d

export CFLAGS="-I$CPYTHON_PATH/include/ -I$PYTHON_LIB_PATH/include/python/ -L$CPYTHON_PATH/lib/ -L$CPYTHON_PATH/libs/"

The PyTorch build depends on cpython of its target arch though. Locally via

javacpp-presets/pytorch/cppbuild.sh

Line 59 in ee1401d

CPYTHON_PATH="$INSTALL_PATH/../../../cpython/cppbuild/$PLATFORM/"

For a maven build it seems BUILD_PATH is set and therefore CPYTHON_PATH is overwritten to use cpython cached from maven:

javacpp-presets/pytorch/cppbuild.sh

Lines 63 to 68 in ee1401d

	if [[ -n "${BUILD_PATH:-}" ]]; then
	PREVIFS="$IFS"
	IFS="$BUILD_PATH_SEPARATOR"
	for P in $BUILD_PATH; do
	if [[ $(find "$P" -name Python.h) ]]; then
	CPYTHON_PATH="$P"

I'm not sure how CPYTHON_PATH is set exactly (maven plugin?) but I think it is coming in through the transitive maven-compiler-plugin dependency on cpython via numpy-platform in pom.xml:

javacpp-presets/pytorch/pom.xml

Lines 52 to 56 in ee1401d

	<dependency>
	<groupId>org.bytedeco</groupId>
	<artifactId>numpy-platform</artifactId>
	<version>1.26.3-${project.parent.version}</version>
	</dependency>

To get it working locally, I did a little hack: Build a native x86_64 cpython but put it into the cppbuild/macosx-arm64 directory so CPYTHON_PATH is set to it and we can actually run that Python interpreter for kicking off the PyTorch cross-compilation.

So the open question is: Can we use the x86_64 Python interpreter for cross-compilation of PyTorch or will it cause issues? If we can, what is a good way to configure it especially for CI builds? If not, what might be other options?