detect number of procs during sphinx build

[aaronmarkham]

Description

I ran into a build issue when making the python docs on an instance with 16 processors. The makefile for sphinx was hardcoded to 32 which I guess was fine for CI.

The error was like this:

/work/mxnet/docs/python_docs/python/build/tutorials/packages/gluon/training/fit_api_tutorial.ipynb:395: WARNING: File not found: 'tutorials/packages/gluon/blocks/save_load_params.html#saving-model-parameters-to-file'

Sphinx parallel build error:
IndexError: list index out of range

I turned off parallel builds with Sphinx and it worked, then switched to this new code that detects the number of processors in the Makefile and that worked.

Testing

1. Clone the repo.
2. Install Docker and Docker for python and make it so docker doesn't need sudo to run.
3. Run the following two commands to build the mxnet binary, then build the python docs.

ci/build.py --docker-registry mxnetci --platform ubuntu_cpu_lite /work/runtime_functions.sh build_ubuntu_cpu_docs
ci/build.py --docker-registry mxnetci --platform ubuntu_cpu_python /work/runtime_functions.sh build_python_docs

[larroy]

LGTM

[larroy]

Also if we have troubles with make we can use a python thing to build the sphinx docs, there are python alternatives. Make is just a dumb depgraph.

[aaronmarkham]

Also if we have troubles with make we can use a python thing to build the sphinx docs, there are python alternatives. Make is just a dumb depgraph.

This file is what's included in Sphinx so people expect it. But sure, we could use something different to invoke sphinx.

[aaronmarkham]

Kind of ridiculous how many times I've had to restart the tests on this PR.

Restarting centos-gpu now due to failing here: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fcentos-gpu/detail/PR-16512/4/pipeline

[larroy]

Can we collect the failures to understand what were the root causes?

[larroy]

In the one you linked S3 failed, so we could add a retry.