[Java][Integration][Release] verify-rc-source-integration-linux-conda-latest-amd64 fails with MemoryError

[raulcd]

Describe the bug, including details regarding any error messages, version, and platform.

The verify-rc-source-integration-linux-conda-latest-amd64 is failing consistently with:

##########################################################
  C Data Interface: C++ exporting, Java importing
  ##########################################################
Traceback (most recent call last):
  File "/tmp/arrow-HEAD.LSZ9o/miniforge/envs/conda-source/bin/archery", line 8, in <module>
    sys.exit(archery())
             ~~~~~~~^^
  File "/tmp/arrow-HEAD.LSZ9o/miniforge/envs/conda-source/lib/python3.13/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/tmp/arrow-HEAD.LSZ9o/miniforge/envs/conda-source/lib/python3.13/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/tmp/arrow-HEAD.LSZ9o/miniforge/envs/conda-source/lib/python3.13/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/tmp/arrow-HEAD.LSZ9o/miniforge/envs/conda-source/lib/python3.13/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/arrow-HEAD.LSZ9o/miniforge/envs/conda-source/lib/python3.13/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/arrow/dev/archery/archery/cli.py", line 872, in integration
    run_all_tests(**args)
    ~~~~~~~~~~~~~^^^^^^^^
  File "/arrow/dev/archery/archery/integration/runner.py", line 713, in run_all_tests
    runner.run_c_data()
    ~~~~~~~~~~~~~~~~~^^
  File "/arrow/dev/archery/archery/integration/runner.py", line 148, in run_c_data
    self._compare_c_data_implementations(producer, consumer)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/arrow/dev/archery/archery/integration/runner.py", line 461, in _compare_c_data_implementations
    with consumer.make_c_data_importer() as importer:
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/arrow/dev/archery/archery/integration/tester_java.py", line 356, in make_c_data_importer
    return JavaCDataImporter(self.debug, self.args)
  File "/arrow/dev/archery/archery/integration/tester_java.py", line 108, in __init__
    setup_jpype()
    ~~~~~~~~~~~^^
  File "/arrow/dev/archery/archery/integration/tester_java.py", line 92, in setup_jpype
    jpype.startJVM(jpype.getDefaultJVMPath(),
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
                   "-Djava.class.path=" + jar_path,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
                   "-Xrs",
                   ^^^^^^^
                   *_JAVA_OPTS)
                   ^^^^^^^^^^^^
  File "/tmp/arrow-HEAD.LSZ9o/miniforge/envs/conda-source/lib/python3.13/site-packages/jpype/_core.py", line 250, in startJVM
    initializeResources()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/tmp/arrow-HEAD.LSZ9o/miniforge/envs/conda-source/lib/python3.13/site-packages/jpype/_core.py", line 275, in initializeResources
    _jpype._java_lang_String = _jpype.JClass("java.lang.String")
                               ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/tmp/arrow-HEAD.LSZ9o/miniforge/envs/conda-source/lib/python3.13/site-packages/jpype/_jclass.py", line 99, in __new__
    return _jpype._getClass(jc)
           ~~~~~~~~~~~~~~~~^^^^
  File "/tmp/arrow-HEAD.LSZ9o/miniforge/envs/conda-source/lib/python3.13/site-packages/jpype/_jclass.py", line 147, in _jclassPost
    for cls in res.class_.getDeclaredClasses():
               ^^^^^^^^^^
MemoryError
Failed to verify release candidate. See /tmp/arrow-HEAD.LSZ9o for details.

This is also failing on the maintenance branch for 18.0.0

Component(s)

Integration, Java, Release

[lidavidm]

This may also be a Python 3.13-related issue. jpype-project/jpype#1204

[lidavidm]

While that issue is closed, I don't see a corresponding commit. It's possible this will be tied to Python <= 3.12 for the foreseeable future. Unfortunately I'm also not sure if we can feasibly use something other than jpype here...

[pitrou]

For now, JPype can't work on 3.13. There is a lengthy (and sometimes difficult to follow) explanation here: https://discuss.python.org/t/api-for-python-3-13-prevents-use-of-3rd-party-gc-allocators/62709/5

So, yes, we should pin Python to 3.12 in the integration builds.

[kou]

Should we keep this open for JPype update?
(We can remove this from the 18.0.0 milestone because we merged GH-44388.)

[raulcd]

Ooops, this wasn't solved by the other PR because this is on Linux not on macOS where the fix was done.
The job is still failing but knowing that this is just a matter of pinning Python to 3.13 I don't think we have to add a fix to the maintenance branch. See latest failure on the maintenance branch: https://github.com/ursacomputing/crossbow/actions/runs/11313587580/job/31462525580

[kou]

Oh, sorry. I missed it: GH-44395

[raulcd]

Issue resolved by pull request 44395
#44395