Add tests and CI for optional pyarrow module

[wjones127]

Which issue does this PR close?

Closes #1635.

Rationale for this change

The build was reported as broken, so we should test it.

In addition, datafusion-contrib/datafusion-python#21 will benefit from the completed implementation of roundtrip conversion.

What changes are included in this PR?

• Adds a new CI job to tests pyarrow module.
• Completes to_pyarrow implementation for ScalarValue
• Adds unit tests to pyarrow conversion logic.

Are there any user-facing changes?

ScalarValue.to_pyarrow will no longer throw a not implemented error. Does that count as an API change?

[wjones127]

cc @alamb @kszucs

[houqp]

@wjones127 looks like the newly added CI job is failing.

[wjones127]

@wjones127 looks like the newly added CI job is failing.

@houqp Thanks. I got it passing now in my branch: https://github.com/wjones127/arrow-datafusion/runs/5008841849?check_suite_focus=true

[alamb]

https://github.com/apache/arrow-datafusion/runs/5009634913?check_suite_focus=true

I tried to run this locally and I got an error (even after following the suggestion of setting PYO3_PYTHON=$(which python)).

PYO3_PYTHON=$(which python) cargo test --features=pyarrow -p datafusion -- pyarrow
    Finished test [unoptimized + debuginfo] target(s) in 0.16s
     Running unittests (target/debug/deps/datafusion-59322809a250018b)

running 2 tests
test pyarrow::tests::test_py_scalar ... FAILED
test pyarrow::tests::test_roundtrip ... FAILED

failures:

---- pyarrow::tests::test_py_scalar stdout ----
thread 'pyarrow::tests::test_py_scalar' panicked at 'pyarrow not found
Executable: /Users/alamb/Software/arrow-datafusion/target/debug/deps/datafusion-59322809a250018b
Python path: ["/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python39.zip", "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9", "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload", "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages"]
HINT: try `pip install pyarrow`
                         HINT: if wrong Python path, try `PYO3_PYTHON=$(which python)`: PyErr { type: <class 'ModuleNotFoundError'>, value: ModuleNotFoundError("No module named 'pyarrow'"), traceback: Some(<traceback object at 0x110e69740>) }', datafusion/src/pyarrow.rs:97:26
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- pyarrow::tests::test_roundtrip stdout ----
thread 'pyarrow::tests::test_roundtrip' panicked at 'pyarrow not found
Executable: /Users/alamb/Software/arrow-datafusion/target/debug/deps/datafusion-59322809a250018b
Python path: ["/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python39.zip", "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9", "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload", "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages"]
HINT: try `pip install pyarrow`
                         HINT: if wrong Python path, try `PYO3_PYTHON=$(which python)`: PyErr { type: <class 'ModuleNotFoundError'>, value: ModuleNotFoundError("No module named 'pyarrow'"), traceback: Some(<traceback object at 0x110e69740>) }', datafusion/src/pyarrow.rs:97:26


failures:
    pyarrow::tests::test_py_scalar
    pyarrow::tests::test_roundtrip

test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 903 filtered out; finished in 0.02s

error: test failed, to rerun pass '-p datafusion --lib'

Any thoughts @wjones127 ?

[alamb]

👍

[alamb]

Thank you for the contribution @wjones127 ❤️

[wjones127]

Any thoughts @wjones127 ?

TBH, I get this too on my local. It seems to be using your homebrew Python 3.9 but not the version of Python that you've installed pyarrow in. (On my local, I was trying to use a pyenv-installed Python, but it didn't seem to want to use that.) To get it to pass, I set PYO3_PYTHON to the path of my Homebrew-installed Python (basically gave up on my Pyenv installs).

At the very least, it seems to work in CI without much configuration.

I assumed this is specific to my Mac OS + PyEnv setup, and that PYO3_PYTHON=$(which python) would cover other cases. But on second thought, maybe in all cases where that command would find the right Python, PyO3 will already do the right thing. So perhaps I need to change the instructions? Something like Try setting PYO3_PYTHON=<full-path-to-python>?

[alamb]

I also tried

PYO3_PRINT_CONFIG=1 PYO3_PYTHON=$(which python) cargo test --features=pyarrow -p datafusion -- pyarrow
error: failed to run custom build command for `pyo3 v0.15.1`

Caused by:
  process didn't exit successfully: `/Users/alamb/Software/arrow-datafusion/target/debug/build/pyo3-0d2dc5a0a08d5a5d/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=PYO3_CROSS
  cargo:rerun-if-env-changed=PYO3_CROSS_LIB_DIR
  cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_VERSION
  cargo:rerun-if-env-changed=PYO3_PRINT_CONFIG

  -- PYO3_PRINT_CONFIG=1 is set, printing configuration and halting compile --
  implementation=CPython
  version=3.9
  shared=true
  abi3=false
  lib_name=python3.9
  lib_dir=/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/lib
  executable=/Users/alamb/Software/virtual_envs/arrow_dev/bin/python
  pointer_width=64
  build_flags=WITH_THREAD
  suppress_build_script_link_lines=false

So it seems like maybe pyo3 is picking up the right executable (my virtual env) but not the correct libdir 🤔

[wjones127]

So it seems like maybe pyo3 is picking up the right executable (my virtual env) but not the correct libdir 🤔

Have you checked that your virtualenv has the python3.9.lib file? I think some installations just have the Python binary and no libraries.

Otherwise, I do think there is something on MacOS about shared libraries vs the Frameworks folder libraries that I don't yet comprehend, but might be key here.

[wjones127]

So it seems like maybe pyo3 is picking up the right executable (my virtual env) but not the correct libdir 🤔

@alamb Actually this seems to be a known issue: PyO3/pyo3#1741 (comment). Python doesn't seem to do the virtual environment path computation for embedded systems on Mac OS. A workaround is to extract that into PYTHON_PATH like so:

export PYTHONPATH=$(python -c "import sys; print(sys.path[-1])")

This is orthogonal to the original issue I had, which is that you must use a framework based Python install. That's default in Homebrew and Python.org installers, but not in pyenv.

I have updated the error help to reflect these two issues.

[alamb]

Sorry -- I missed this one -- I just started CI again. When that passes I'll plan to merge it in. Thanks again @wjones127