Website build on CI is down due to Julia error

[aaronmarkham]

Description

This is in the error log.

julia --color=yes --project=./ ./make.jl
┌ Warning: The environment variable `MXNET_HOME` has been renamed, please use `MXNET_ROOT` instead.
└ @ MXNet.mx /work/mxnet/docs/build_version_doc/apache-mxnet/master/julia/src/base.jl:63
ERROR: LoadError: LoadError: Cannot find or load libmxnet.so. Please see the document on how to build it.

[mxnet-label-bot]

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended label(s): CI, Build

[aaronmarkham]

http://jenkins.mxnet-ci.amazon-ml.com/job/restricted-website-build/

[aaronmarkham]

It's going to be this PR that triggered the problem:
#15568

[zachgk]

Since #16147 was raised that seems to address this issue and the website publishing has been successful for the last 15 runs, I will close this issue. Please reopen if I am closing in error.

[haojin2]

Seems to be failing again due to a different reason: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwebsite/detail/PR-17302/13/pipeline/

[2020-02-07T01:31:11.534Z] Traceback (most recent call last):

[2020-02-07T01:31:11.534Z]   File "/home/jenkins_slave/.local/bin/mkdocs", line 8, in <module>

[2020-02-07T01:31:11.534Z]     sys.exit(cli())

[2020-02-07T01:31:11.534Z]   File "/home/jenkins_slave/.local/lib/python3.5/site-packages/click/core.py", line 764, in __call__

[2020-02-07T01:31:11.534Z]     return self.main(*args, **kwargs)

[2020-02-07T01:31:11.534Z]   File "/home/jenkins_slave/.local/lib/python3.5/site-packages/click/core.py", line 717, in main

[2020-02-07T01:31:11.534Z]     rv = self.invoke(ctx)

[2020-02-07T01:31:11.534Z]   File "/home/jenkins_slave/.local/lib/python3.5/site-packages/click/core.py", line 1137, in invoke

[2020-02-07T01:31:11.534Z]     return _process_result(sub_ctx.command.invoke(sub_ctx))

[2020-02-07T01:31:11.534Z]   File "/home/jenkins_slave/.local/lib/python3.5/site-packages/click/core.py", line 956, in invoke

[2020-02-07T01:31:11.534Z]     return ctx.invoke(self.callback, **ctx.params)

[2020-02-07T01:31:11.534Z]   File "/home/jenkins_slave/.local/lib/python3.5/site-packages/click/core.py", line 555, in invoke

[2020-02-07T01:31:11.534Z]     return callback(*args, **kwargs)

[2020-02-07T01:31:11.534Z]   File "/home/jenkins_slave/.local/lib/python3.5/site-packages/mkdocs/__main__.py", line 162, in build_command

[2020-02-07T01:31:11.534Z]     site_dir=site_dir

[2020-02-07T01:31:11.534Z]   File "/home/jenkins_slave/.local/lib/python3.5/site-packages/mkdocs/config/base.py", line 197, in load_config

[2020-02-07T01:31:11.534Z]     errors, warnings = cfg.validate()

[2020-02-07T01:31:11.534Z]   File "/home/jenkins_slave/.local/lib/python3.5/site-packages/mkdocs/config/base.py", line 115, in validate

[2020-02-07T01:31:11.534Z]     post_failed, post_warnings = self._post_validate()

[2020-02-07T01:31:11.534Z]   File "/home/jenkins_slave/.local/lib/python3.5/site-packages/mkdocs/config/base.py", line 95, in _post_validate

[2020-02-07T01:31:11.534Z]     config_option.post_validation(self, key_name=key)

[2020-02-07T01:31:11.534Z]   File "/home/jenkins_slave/.local/lib/python3.5/site-packages/mkdocs/config/config_options.py", line 432, in post_validation

[2020-02-07T01:31:11.534Z]     config[key_name] = theme.Theme(**theme_config)

[2020-02-07T01:31:11.534Z]   File "/home/jenkins_slave/.local/lib/python3.5/site-packages/mkdocs/theme.py", line 48, in __init__

[2020-02-07T01:31:11.534Z]     self._load_theme_config(name)

[2020-02-07T01:31:11.534Z]   File "/home/jenkins_slave/.local/lib/python3.5/site-packages/mkdocs/theme.py", line 78, in _load_theme_config

[2020-02-07T01:31:11.534Z]     theme_dir = utils.get_theme_dir(name)

[2020-02-07T01:31:11.534Z]   File "/home/jenkins_slave/.local/lib/python3.5/site-packages/mkdocs/utils/__init__.py", line 278, in get_theme_dir

[2020-02-07T01:31:11.534Z]     return os.path.dirname(os.path.abspath(theme.load().__file__))

[2020-02-07T01:31:11.534Z]   File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 2443, in load

[2020-02-07T01:31:11.534Z]     self.require(*args, **kwargs)

[2020-02-07T01:31:11.534Z]   File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 2466, in require

[2020-02-07T01:31:11.534Z]     items = working_set.resolve(reqs, env, installer, extras=self.extras)

[2020-02-07T01:31:11.534Z]   File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 792, in resolve

[2020-02-07T01:31:11.534Z]     raise VersionConflict(dist, req).with_context(dependent_req)

[2020-02-07T01:31:11.534Z] pkg_resources.VersionConflict: (Markdown 3.2 (/home/jenkins_slave/.local/lib/python3.5/site-packages), Requirement.parse('markdown<3.2'))

[2020-02-07T01:31:11.534Z] Makefile:19: recipe for target 'all' failed

[ChaiBapchya]

Found here too - http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwebsite/detail/PR-17500/6/pipeline
#17500

[leezu]

This affects all PRs.

[samskalicky]

Looks like it was working 2 days ago:
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwebsite/detail/PR-17486/3/pipeline/86

What changed since then?

[aaronmarkham]

My guess is that the dependencies in the julia docs chain hit this error:

ERROR: mkdocs-material 4.6.0 has requirement markdown<3.2, but you'll have markdown 3.2 which is incompatible.

It does this:

pip install --user pygments mkdocs mkdocs-material python-markdown-math

So I'm thinking those all need to be version pinned and roll back mkdocs-material to a version that still works. Or pin markdown to a previous version...

That would be here:
https://github.com/apache/incubator-mxnet/blob/7caffa65e30f37e70796ba165ac5a4265e64974e/julia/docs/Makefile#L23

[aaronmarkham]

I'm about to head out to the airport and be traveling, so I don't know if I can test this and help with a patch in a reasonable amount of time... any takers?

[haojin2]

@aaronmarkham indeed it's a version problem:
pkg_resources.VersionConflict: (Markdown 3.2 (/home/jenkins_slave/.local/lib/python3.5/site-packages), Requirement.parse('markdown<3.2'))

[haojin2]

working on a fix, will have a PR out soon. Currently verifying my fix

[TaoLv]

#17549 is merged but looks like the issue is still there.

[aaronmarkham]

Oh boy:

04:34:06  ERROR: pymdown-extensions 6.3 has requirement Markdown>=3.2, but you'll have markdown 3.1 which is incompatible.
04:34:06  ERROR: mkdocs-material 4.6.2 has requirement markdown>=3.2, but you'll have markdown 3.1 which is incompatible.

So these dependencies are fighting over this markdown version. One says it must be 3.2 or greater the other says it must be less.

I have an idea:
If someone has an older setup building locally that works - try that and export the pip versions and we pin it like that. Or maybe there's some old docker image that has this setup?

Also, to get unblocked, we can disable Julia docs from the website build flow.

[aaronmarkham]

So it worked for me on my test. Not sure why exactly, but I ended up with slightly older versions of things. From my log:

mkdocs==1.0.4
mkdocs-material==4.6.0
pymdown-extensions==6.2.1 
python-markdown-math==0.6

I think maybe adding these will do the trick. Considering that my current setup isn't failing, I'm not entirely sure how to test this out, but it's worth a try.

Side note

Now, I tried to run pip freeze but the Docker image that I had left was (I guess) before these things ran. The logs state:

2020-02-08 00:43:31,273 - root - INFO - Waiting for status of container 73b619cf6eea for 600 s.
2020-02-08 00:43:31,467 - root - INFO - Container exit status: {'Error': None, 'StatusCode': 0}
2020-02-08 00:43:31,467 - root - INFO - Container exited with success 👍
2020-02-08 00:43:31,467 - root - INFO - Executed command for reproduction:

ci/build.py --docker-registry mxnetci --platform ubuntu_cpu_julia /work/runtime_functions.sh build_julia_docs

2020-02-08 00:43:31,468 - root - INFO - Stopping container: 73b619cf6eea
2020-02-08 00:43:31,469 - root - INFO - Removing container: 73b619cf6eea

So it removed what I had as a working state. How unfortunate!
Anyone know what flag (or whatever) I could have set when running build.py so that I'd get to keep the last version of the container around for testing?

[aaronmarkham]

Btw, this should have been a new issue, rather than re-opening an unrelated issue... sure, the title matches, but the errors were not the same. Anyhoo, I hope I fixed it with the above PR. If it doesn't work, then maybe the jenkins nodes are caching and they need to be restarted. If I can run the CI build and have it pass, then it I'd expect it to run in production! Otherwise, we're missing an important test capability.