[Doc] Use autodoc_mock_imports to mock deps for sphinx builds

[peytondmurray]

Why are these changes needed?

This PR is the first in a series needed to update Sphinx to the latest version. In this PR, we

• Remove a lot of external dependencies that aren't necessary for building the docs. The one exception to this is tune-sklearn, a project owned by ray-project which actually does have documentation hosted on the Ray docs site.
• External dependencies are now mocked out using Sphinx's autodoc_mock_imports mechanism, which is used by both autodoc and autosummary. The old module mocking mechanism has been removed in favor of this. The one exception to this is packaging.version.Version: Ray currently uses Version to modify some behaviors depending on the version of certain dependencies that the user has installed. For example, in python/ray/train/torch/train_loop_utils.py:

import torch
from packaging.version import Version

if Version(torch.__version__) < Version("1.11.0"):
    FullyShardedDataParallel = None
else:
    from torch.distributed.fsdp import FullyShardedDataParallel

During the Sphinx build, if torch is mocked out, torch.__version__ is not a string, it's a Mock object, which is not a valid argument for packaging.version.Version - the build will fail. Although packaging is an external dependency and therefore should be mocked out using autodoc_mock_modules, Sphinx itself depends on it, and during the documentation build step the module is already loaded by the time conf.py gets executed, and therefore can't be mocked out in the usual way. I've added a dummy class which correctly handles Mock objects that get passed to it during documentation builds. I've also added a check for any module specified in autodoc_mock_modules to ensure that mock targets aren't yet loaded when conf.py is executed, which hopefully saves the next person to touch this a lot of pain.

Related issue number

Partially addresses #37944.

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[peytondmurray]

There are some differences between this PR and the current master build. Normally this would be a bad thing because we aren't changing anything user-facing here, but so far I think this only solves bugs in the docs. For example, switching to use autodoc_mock_imports fixed these type names. Here's the master build:

And here's this branch:

I'm going to spend a bit longer scanning the differences to see if there's anything bad here. @angelinalg if you have time, I'd appreciate feedback as well.

[peytondmurray]

Here's another example of an issue on master that gets fixed here:

and on the branch:

[maxpumperla]

@peytondmurray this looks great, thanks so much! we should merge this asap, once we removed the rtd build issues.

[peytondmurray]

Okay, the RTD build is now working - I had to make sure not to import ray in conf.py because you can't autodoc_mock_imports on anything that is already initialized. The build now works!

[peytondmurray]

After a discussion with @maxpumperla I'm marking this as ready for review. We still are doing ad-hoc mocking of compiled ray modules, which still display as <MagicMock ...> in the docs, but all of the modules that have been moved into autodoc_mock_imports have improved the docs already.

Future effort: sphinx-doc/sphinx#4413 should allow us to put ray._raylet inside autodoc_mock_imports without mocking ray itself - need to investigate this going forward.

[maxpumperla]

LGTM, approving pending a green CI run and the comment below.

[bveeramani]

Nice!

[bveeramani]

We should remove this section or replace it with the information in How to write code snippets. We don't use make doctest anymore.