PyArrow builds aren't correctly pinning to the locally produced arrow-cpp build

[kkraus14]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

For example, see here: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=538009&view=logs&j=3b077b59-19b3-5c3c-d9ea-4f2d359714ca&t=4427250c-9566-5227-2be0-fd328c273206

This produced 4 packages:

    /home/conda/feedstock_root/build_artifacts/linux-64/arrow-cpp-8.0.1-py39h2531139_0_cpu.tar.bz2
    /home/conda/feedstock_root/build_artifacts/linux-64/arrow-cpp-proc-3.0.0-cpu.tar.bz2
    /home/conda/feedstock_root/build_artifacts/linux-64/pyarrow-8.0.1-py39h42d110c_0_cpu.tar.bz2
    /home/conda/feedstock_root/build_artifacts/linux-64/pyarrow-tests-8.0.1-py39h42d110c_0_cpu.tar.bz2

If we look at the pyarrow-8.0.1-py39h42d110c_0_cpu build we can see that it pins to the incorrect arrow-cpp build:

pyarrow 8.0.1 py39h42d110c_0_cpu
--------------------------------
file name   : pyarrow-8.0.1-py39h42d110c_0_cpu.tar.bz2
name        : pyarrow
version     : 8.0.1
build       : py39h42d110c_0_cpu
build number: 0
size        : 3.3 MB
license     : Apache-2.0
subdir      : linux-64
url         : https://conda.anaconda.org/conda-forge/linux-64/pyarrow-8.0.1-py39h42d110c_0_cpu.tar.bz2
md5         : f8e42cd6b5bee024305502104efba504
timestamp   : 2022-07-29 21:55:46 UTC
constraints :
  - arrow-cpp-proc * cpu
dependencies:
  - arrow-cpp 8.0.1 py39h811ffd7_0_cpu
  - libgcc-ng >=12
  - libstdcxx-ng >=12
  - numpy >=1.16,<2.0a0
  - numpy >=1.19.5,<2.0a0
  - parquet-cpp 1.5.1.*
  - python >=3.9,<3.10.0a0
  - python_abi 3.9.* *_cp39

This is causing issues where for example we're building arrow-cpp with two abseil versions, but the pyarrow builds are only depending on arrow-cpp packages built against one of those abseil versions.

Installed packages

n/a

Environment info

n/a

[kkraus14]

Actually, could be that we're clobbering pyarrow packages because arrow-cpp isn't used in producing the hash for the pyarrow build?

  "pyarrow-8.0.1-py39h42d110c_0_cpu": {
    "recipe": {
      "c_compiler": "gcc",
      "c_compiler_version": "10",
      "channel_targets": "conda-forge main",
      "cuda_compiler_version": "None",
      "cxx_compiler": "gxx",
      "cxx_compiler_version": "10",
      "numpy": "1.19",
      "openssl": "1.1.1",
      "python": "3.9.* *_cpython",
      "target_platform": "linux-64"
    }

[jakirkham]

Sounds kind of like issue ( conda/conda-build#4142 )

[kkraus14]

Yea that sounds exactly like what we're hitting here.

[jakirkham]

Should we be adding abseil-cpp to pyarrow as a dependency too? Maybe this would help alleviate this issue

[kkraus14]

pyarrow doesn't actually depend on abseil-cpp and arrow-cpp doesn't leak it into its interfaces.

I think we're just in a funky place because we're using split outputs, otherwise we could build much fewer pyarrow packages since the ABI of arrow-cpp is stable.

[jakirkham]

Yeah that makes sense. Just thinking through our options to address this.

The other option would be to drop the exact pinning and replace with the pinning we'd like to have with * to replace any bits we don't want to constrain (like hashes).

[kkraus14]

I think adding abseil-cpp as a dependency to pyarrow probably makes the most sense for now to avoid this thrashing. Will open a PR.