[Docs] Upgrade Sphinx from 6.2 to 7.x to support Python 3.12

[amoeba]

Describe the enhancement requested

I noticed the conda packages for the docs (https://github.com/apache/arrow/blob/main/ci/conda_env_sphinx.txt) currently can't be installed under Python 3.12 due to what I assume is sphinx 6.2 being pinned to a version of docutils that can't be installed on Python 3.12:

$ conda install -c conda-forge --file=./ci/conda_env_sphinx.txt
...
Could not solve for environment specs
The following packages are incompatible
├─ pin-1 is installable and it requires
│  └─ python 3.12.* , which can be installed;
└─ sphinx 6.2  is not installable because it requires
   └─ docutils >=0.18.1,<0.20  but there are no viable options
      ├─ docutils [0.18.1|0.19] would require
      │  └─ python >=3.10,<3.11.0a0 *_cpython, which conflicts with any installable versions previously reported;
      ├─ docutils [0.18.1|0.19] would require
      │  └─ python >=3.8,<3.9.0a0 *_cpython, which conflicts with any installable versions previously reported;
      ├─ docutils [0.18.1|0.19] would require
      │  └─ python >=3.9,<3.10.0a0 *_cpython, which conflicts with any installable versions previously reported;
      ├─ docutils 0.19 would require
      │  └─ python >=3.11,<3.12.0a0 *_cpython, which conflicts with any installable versions previously reported;
      └─ docutils 0.18.1 conflicts with any installable versions previously reported.

@jorisvandenbossche @AlenkaF have you already looked into how compatible Sphinx 7.x is with the docs? I think upgrading would be good for a variety of reasons, including making it easier for new contributors (who will likely have Python 3.12+ installed) to build the docs.

Edit: After assessment below, this work is currently blocked until Breathe supports the Sphinx 7.x series and the feedstock is updated. Breathe doesn't have a clear statement of support for Sphinx 7.x but I think its addition to their CI would be the best indicator, see https://github.com/breathe-doc/breathe/blob/main/.github/workflows/unit_tests.yml#L20.

Component(s)

Documentation

[amoeba]

I took a look at the changelog for the 7.x series and searched the docs source for deprecations and didn't find any. I then tried to build the docs with Sphinx 7.2.6 and ran into an error with breathe,

AttributeError: module 'sphinx.domains.python' has no attribute 'PyModulelevel'

This was fixed in breathe 4.35.0 but conda is only picking up breathe 4.15.0. It's unclear but it generally looks like breathe supports Sphinx >7 but when I ask conda to solve sphinx and breathe together I get sphinx 6.2.1 and breathe 4.35.0. pip happily installs sphinx 7.2.6 with breathe 4.35.0 so I think the blocker here is conda.

[amoeba]

The conda recipe for breathe is pinned to sphinx<7 and I don't think a PR to the feedstock bumping the pin is a good idea since it looks like breathe's support for Sphinx 7.x still WIP (based on their issue tracker).

I think we can leave this open and revisit it at some point in the future.

[AlenkaF]

Thank you @amoeba for looking into this! We have been planning to remove the pin for Sphinx for a while now but didn't get to it yet so this is much appreciated.

I do not remember why we haven't removed the pin on Sphinx altogether but I guess the work here will be similar to #36296 which is what you have already investigated. Waiting for breathe to support Sphinx 7.x is something we have to wait for.

[amoeba]

Thanks @AlenkaF. I did try building the docs with Sphinx 7.2.6 and it looks fine but a more thorough inspection would be warranted. I edited my original post here to make it clear what the blocker is on this work so we can revisit this in the future.

[lidavidm]

FWIW, it appears Breathe is considered effectively unmaintained now + Sphinx 8.0 will intentionally break compatibility with it: sphinx-doc/sphinx#12629 (comment)