GH-42006: [CI][Python] Use pip install -e instead of setup.py build_ext --inplace for installing pyarrow on verification script

[raulcd]

Rationale for this change

Due to #37929 we require a higher version of setuptools and setuptools_scm to be installed otherwise the job fails with setuptools_scm failing with TypeError: Configuration.__init__() got an unexpected keyword argument 'version_file'

What changes are included in this PR?

Remove the dependencies for the environment and let installation handle those using pip install -e instead of setup.py build_ext --inplace for installing pyarrow on verification script

Are these changes tested?

Via Archery

Are there any user-facing changes?

No

• GitHub Issue: [Python][CI] verify-release for Python is currently failing due to old version of setuptools_scm being installed #42006

[raulcd]

@github-actions crossbow submit verify-rc-source-python-*

[raulcd]

I was missing dev tags on my fork that's why the jobs failed to generate the correct version, I've pushed them and will re-run

[raulcd]

@github-actions crossbow submit verify-rc-source-python-*

[github-actions]

Revision: ec8ce7e

Submitted crossbow builds: ursacomputing/crossbow @ actions-d9b2b774f4

Task	Status
verify-rc-source-python-linux-almalinux-8-amd64
verify-rc-source-python-linux-conda-latest-amd64
verify-rc-source-python-linux-ubuntu-20.04-amd64
verify-rc-source-python-linux-ubuntu-22.04-amd64
verify-rc-source-python-macos-amd64
verify-rc-source-python-macos-arm64
verify-rc-source-python-macos-conda-amd64

[jorisvandenbossche]

FWIW, for release verification I think we should actually remove this line entirely, and rely on the fact that we specify those dependencies in pyproject.toml (i.e. replacing the python setup.py build_ext --inplace below with python -m pip install (-e) .

[raulcd]

The verify-rc-source-python-linux-almalinux-8-amd64 are unrelated and are failing on some python gandiva tests.

[raulcd]

@github-actions crossbow submit verify-rc-source-python-linux-ubuntu-20.04-amd64

[github-actions]

Revision: f098baf

Submitted crossbow builds: ursacomputing/crossbow @ actions-75636dfc2c

Task	Status
verify-rc-source-python-linux-ubuntu-20.04-amd64

[raulcd]

@github-actions crossbow submit verify-rc-source-python-linux-ubuntu-20.04-amd64

[github-actions]

Revision: f39efac

Submitted crossbow builds: ursacomputing/crossbow @ actions-ac4eb35fba

Task	Status
verify-rc-source-python-linux-ubuntu-20.04-amd64

[raulcd]

I don't think running the other extra CI is required

[kou]

+1

The verify-rc-source-python-linux-almalinux-8-amd64 failure is #39695.
Workaround:

• [Python][Gandiva] Could not create LLJIT instance: Symbols not found: [ llvm_orc_registerEHFrameSectionWrapper ] #39695 (comment)
• Use LLVM < 17

[raulcd]

👍 Updated title + description with final fix!

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit a045770.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them.