Add scripts for doc versioning #2749

lilyminium · 2020-06-11T16:23:16Z

Fixes #2741 ... maybe

Changes made in this Pull Request:

added msmb_theme because they have a cool version sidebar that replicates the RTD one
added doc moving scripts to maintainer/
get travis to call them in place of the old deploy scripts
write stubs for index.html, dev.html and latest.html to redirect them to the relevant versions
~~add RTD config~~

This last one is so ~~people can choose to hook up ReadTheDocs to their own forks and see the docs built there~~ we can add an RTD check that builds the docs for you, which also makes it easier to review that kind of thing. This is currently in progress. It's quite hard to build MDAnalysis and have it accessible to conf.py, which hardcodes in looking three levels up, except RTD I'm pretty sure builds it in the same level as the root directory.

Examples:

I've kept the status quo for now (everything branches off mdanalysis.org/mdanalysis) but final URLs are still up for discussion at #2741

PR Checklist

Tests?
Docs? yes!
CHANGELOG updated?
Issue raised/referenced?

TO DO

Figure out what the following need to be and make them like so:

lilyminium · 2020-06-11T16:24:17Z

I guess one thing I haven't thought through is how to trigger this for old versions, i.e. 1.0.0 . The one I have at the address up there is just develop with version.py edited to say differently

codecov · 2020-06-11T17:45:15Z

Codecov Report

Merging #2749 into develop will decrease coverage by 0.11%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           develop    #2749      +/-   ##
===========================================
- Coverage    91.31%   91.19%   -0.12%     
===========================================
  Files          176      176              
  Lines        24018    23798     -220     
  Branches      3160     3134      -26     
===========================================
- Hits         21931    21702     -229     
- Misses        1459     1474      +15     
+ Partials       628      622       -6

Impacted Files	Coverage Δ
package/MDAnalysis/lib/log.py	`76.78% <0.00%> (-19.01%)`	⬇️
package/MDAnalysis/analysis/density.py	`80.00% <0.00%> (-9.07%)`	⬇️
package/MDAnalysis/core/universe.py	`95.88% <0.00%> (-0.46%)`	⬇️
package/MDAnalysis/lib/util.py	`86.98% <0.00%> (-0.30%)`	⬇️
util.py	`88.13% <0.00%> (-0.15%)`	⬇️
package/MDAnalysis/analysis/align.py	`95.32% <0.00%> (-0.04%)`	⬇️
package/MDAnalysis/topology/guessers.py	`100.00% <0.00%> (ø)`
package/MDAnalysis/topology/TOPParser.py	`100.00% <0.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3da643f...9dab4cf. Read the comment docs.

package/MDAnalysis/version.py

lilyminium · 2020-06-12T08:25:18Z

Ok it's weirdly hard to build MDAnalysis in the right place to be accessible to docs on ReadTheDocs; I give up for now.

I added in a function to get the absolute paths of files to setup.py, when trying to get RTD to work; I can remove this if unwanted. It doesn't really do any harm and might help the odd person wanting to go python package/setup.py install.

orbeckst · 2020-06-13T01:51:17Z

Very cool.

If we do a backport to 1.0.x then it should create the proper 1.0.x docs for correct linking, no? (If I am in your "1.0.0" docs then the version link to 2.0.0-dev is broken https://lilyminium.github.io/mdanalysis/1.0.0/lilyminium.github.io/mdanalysis/2.0.0-dev0).

(RTD seems a bit limited in how you can play with it, I remember being pretty frustrated as soon as stuff didn't work out of the box... when it works it's great, when not, it's pain.)

lilyminium · 2020-06-14T06:04:37Z

Oops. I hope that does it. (lilyminium.github.io/mdanalysis now redirects to mdadocs.minium.com.au, just temporarily)

I also added index.html, dev.html and latest.html stubs so that mdadocs.minium.com.au/ redirects to mdadocs.minium.com.au/1.0.0 , mdadocs.minium.com.au/latest redirects to mdadocs.minium.com.au/1.0.0, mdadocs.minium.com.au/dev redirects to mdadocs.minium.com.au/2.0.0-dev0 . This is written by the update_json_versions.py script every time it's updated.

MDTraj uses Javascript to accomplish the same redirect, but this seemed easier to me both in case people don't have JS turned on and also so we don't have to wait for the JS to figure out the dev URL.

<html><head>
<!-- redirect to /development after 3 seconds, in case something fails or
    the user has javascript disabled -->
<meta http-equiv="refresh" content="3;URL='/development'"/>

<script type="text/javascript">
// download the versions.json file and redirect to the URL associated
// with the entry that has latest=true
var xhr = new XMLHttpRequest();
xhr.open('GET', encodeURI('versions.json'));
xhr.onload = function() {
    if (xhr.status === 200) {
        var versions = JSON.parse(xhr.responseText);
        for (var key in versions) {
            if (versions[key]['latest']) {
                var latest = versions[key];
                window.location.replace(latest['url']);
            }
        }
    }
}
xhr.send();
</script>
</head></html>

orbeckst

Very nice, especially the demo at http://mdadocs.minium.com.au/ is convincing!

Just minor questions/comments.

Once this works, could you please update https://github.com/MDAnalysis/MDAnalysis.github.io/wiki/DNS (delete stuff gratuitously!) so that we have a record what CNAMEs we need for the doc magic to work. Perhaps add a link to the wiki page to the comments of your scripts, too, just so that other people or future us has some breadcrumbs to follow.

EDIT: You should now also be able to delete deploy_docs.sh and deploy_master_docs.sh. By the way, are the sitemap.xml files created correctly? This might require some more fiddling... but can be done as another PR if needed.

maintainer/deploy_docs_via_travis.sh

orbeckst · 2020-06-14T06:44:08Z

maintainer/update_versions_json.py

@@ -0,0 +1,93 @@
+# Adapted from mdtraj/devtools/travis-ci/update_versions_json.py by Robert T McGibbon


I assume the mdtraj script is under LGPL, in which case we can use it.

Yes it is (well the entire repo is), and 80% of the script is different, but it was adapted from his script originally

If you add "used under LGPL" then there's not doubt.

I assume @rmcgibbo will be ok with this attribution? (Thank you!)

maintainer/update_versions_json.py

.travis.yml

lilyminium · 2020-06-14T13:41:07Z

By the way, are the sitemap.xml files created correctly?

No, at present they're caught up in each $version directory. They're built by Sphinx before moving all the docs into each subdirectory, so I think what we would need to do is create a new one by concatenating them all into one big sitemap.xml at the root directory. edit: and edit all the URLs, the important part

edit 2: I will do this tomorrow...

lilyminium · 2020-06-15T02:50:46Z

Thanks for pointing out that sitemap.xml and other meta files might need some special attention, @orbeckst!

So, each subdirectory should function as its own little website for the most part. Each of the following uses relative links, so they should be fine in the subdirectories instead of the root directory without further editing:

search.html
py-modindex.html
searchindex.js
genindex.html

I guess people intersphinxing with objects.inv will have to link to the right URL? Unfortunately, mdadocs.minium.com.au/latest redirects to the relevant release, but mdadocs.minium.com.au/latest/objects.inv will error. We could copy the relevant objects.inv into latest/, but I think intersphinx uses the given URL in conf.py as the base URL, so the same problem remains. Two suggestions:

copy the entire set of docs into a latest/ and a dev/ directory. This is redundant but easy and robust. Or,
add a CloudFlare pagerule converting *.mdanalysis.org/latest/* to $1.mdanalysis.org/$version/$2. We would need another for /dev/. This is doable if we remove the current rule for docs.mdanalysis.org (currently off) and the 2018 workshop. It would require manual updating for each release.

The new root sitemap_index.xml just links to the other sitemaps for reasons explained in the script (the file must be a max of 50 MB or 50k URLs) but it can easily be one very big sitemap.xml. By the way, I think the original sitemaps used incorrect URLs with shortened versions:

docs.mdanalysis.org/sitemap.xml

<?xml version='1.0' encoding='utf-8'?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"><url><loc>https://www.mdanalysis.org/mdanalysis/2.0/documentation_pages/analysis/align.html</loc>
</url>
<url>
    <loc>https://www.mdanalysis.org/mdanalysis/2.0/documentation_pages/analysis/base.html</loc>
</url>
<url>
    <loc>https://www.mdanalysis.org/mdanalysis/2.0/documentation_pages/analysis/contacts.html</loc>
</url>

orbeckst

Still lgtm, see minor comments.

package/doc/sphinx/source/conf.py

lilyminium · 2020-06-15T11:39:08Z

Well, this is interesting. I think until this PR is merged, every other merge will reset the docs URL back to mdanalysis.org/mdanalysis by wiping the CNAME record from gh-pages -- specifically due to this line that removes all entries not built in the docs by Travis:

mdanalysis/maintainer/deploy_docs.sh

Line 49 in 46e505c

git add -A .

Those commits down there are me setting the URL back to docs.mdanalysis.org in settings. 👇

lilyminium · 2020-06-15T12:35:07Z

https://docs.mdanalysis.org/2.0.0-dev0/

✅ versions.json looks fine
✅ sitemap_index.xml looks fine
✅ URLs in 2.0.0-dev0/sitemap.xml look sensible
✅ CNAME not overwritten
✅ Internal links are working
✅ docs.mdanalysis.org/ , docs.mdanalysis.org/dev and docs.mdanalysis.org/latest all redirecting to 2.0.0-dev0 (if no proper release is found, it just goes to the most recent URL)
😕 forgot that we need to remove existing files like objects.inv and sitemap.xml from gh_pages, but that doesn't necessarily mess anything up

Next: revert back to old 1.0.0 docs and do those...

- Scripts move docs into a version subfolder - Write versions.json at root directory - Write sitemap_index.xml at root directory - Write html stubs for redirection - Change URL to CNAME one of docs.mdanalysis.org/

add doc stuff

feaf8a0

lilyminium force-pushed the docversions branch from 472d520 to 1bfc3f4 Compare June 11, 2020 16:25

lilyminium force-pushed the docversions branch 4 times, most recently from 46bb32b to af673dd Compare June 12, 2020 01:15

added rtd stuff

eaabe4a

lilyminium force-pushed the docversions branch 6 times, most recently from daddd30 to bc9d959 Compare June 12, 2020 03:08

lilyminium marked this pull request as draft June 12, 2020 06:26

lilyminium force-pushed the docversions branch from bc9d959 to e641601 Compare June 12, 2020 06:27

lilyminium mentioned this pull request Jun 12, 2020

Dev docs build MDAnalysis/UserGuide#67

Closed

abs path ins setup.py

3899446

lilyminium force-pushed the docversions branch from e641601 to 3899446 Compare June 12, 2020 07:18

richardjgowers reviewed Jun 12, 2020

View reviewed changes

package/MDAnalysis/version.py Outdated Show resolved Hide resolved

lilyminium added 3 commits June 12, 2020 18:11

will pip work

2526f7c

give up on rtd

10ff210

rename relpath to abspath

8861ad7

lilyminium marked this pull request as ready for review June 12, 2020 08:26

lilyminium marked this pull request as draft June 14, 2020 04:19

added stubs and updated url request

d33b966

lilyminium force-pushed the docversions branch from c05165c to d33b966 Compare June 14, 2020 05:55

give credit to mdtraj

f29218b

lilyminium marked this pull request as ready for review June 14, 2020 06:12

orbeckst approved these changes Jun 14, 2020

View reviewed changes

orbeckst reviewed Jun 14, 2020

View reviewed changes

.travis.yml Show resolved Hide resolved

fixed url and other small things

e095939

lilyminium marked this pull request as draft June 15, 2020 01:20

lilyminium added 2 commits June 15, 2020 12:48

added sitemap writing

529ff80

update version in conf

779906b

lilyminium marked this pull request as ready for review June 15, 2020 02:52

orbeckst approved these changes Jun 15, 2020

View reviewed changes

package/doc/sphinx/source/conf.py Outdated Show resolved Hide resolved

package/doc/sphinx/source/conf.py Show resolved Hide resolved

package/doc/sphinx/source/conf.py Outdated Show resolved Hide resolved

added comments and urls

9dab4cf

lilyminium force-pushed the docversions branch from 1e6358f to 9dab4cf Compare June 15, 2020 08:53

lilyminium changed the title ~~Add stuff for doc versions~~ Add scripts for doc versioning Jun 15, 2020

lilyminium merged commit c761ba3 into MDAnalysis:develop Jun 15, 2020

lilyminium deleted the docversions branch June 15, 2020 11:49

orbeckst mentioned this pull request Jun 17, 2020

Release 1.0 #2443

Closed

6 tasks

richardjgowers mentioned this pull request Feb 21, 2021

abspaths slipping into SOURCES and breaking windows conda build #3129

Closed

fiona-naughton added the Component-Docs label Sep 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add scripts for doc versioning #2749

Add scripts for doc versioning #2749

lilyminium commented Jun 11, 2020 •

edited

Loading

lilyminium commented Jun 11, 2020

codecov bot commented Jun 11, 2020 •

edited

Loading

lilyminium commented Jun 12, 2020

orbeckst commented Jun 13, 2020

lilyminium commented Jun 14, 2020

orbeckst left a comment •

edited

Loading

orbeckst Jun 14, 2020

lilyminium Jun 14, 2020

orbeckst Jun 14, 2020

orbeckst Jun 14, 2020

lilyminium commented Jun 14, 2020 •

edited

Loading

lilyminium commented Jun 15, 2020 •

edited

Loading

orbeckst left a comment

lilyminium commented Jun 15, 2020 •

edited

Loading

lilyminium commented Jun 15, 2020 •

edited

Loading

		@@ -0,0 +1,93 @@
		# Adapted from mdtraj/devtools/travis-ci/update_versions_json.py by Robert T McGibbon

Add scripts for doc versioning #2749

Add scripts for doc versioning #2749

Conversation

lilyminium commented Jun 11, 2020 • edited Loading

PR Checklist

TO DO

lilyminium commented Jun 11, 2020

codecov bot commented Jun 11, 2020 • edited Loading

Codecov Report

lilyminium commented Jun 12, 2020

orbeckst commented Jun 13, 2020

lilyminium commented Jun 14, 2020

orbeckst left a comment • edited Loading

Choose a reason for hiding this comment

orbeckst Jun 14, 2020

Choose a reason for hiding this comment

lilyminium Jun 14, 2020

Choose a reason for hiding this comment

orbeckst Jun 14, 2020

Choose a reason for hiding this comment

orbeckst Jun 14, 2020

Choose a reason for hiding this comment

lilyminium commented Jun 14, 2020 • edited Loading

lilyminium commented Jun 15, 2020 • edited Loading

orbeckst left a comment

Choose a reason for hiding this comment

lilyminium commented Jun 15, 2020 • edited Loading

lilyminium commented Jun 15, 2020 • edited Loading

lilyminium commented Jun 11, 2020 •

edited

Loading

codecov bot commented Jun 11, 2020 •

edited

Loading

orbeckst left a comment •

edited

Loading

lilyminium commented Jun 14, 2020 •

edited

Loading

lilyminium commented Jun 15, 2020 •

edited

Loading

lilyminium commented Jun 15, 2020 •

edited

Loading

lilyminium commented Jun 15, 2020 •

edited

Loading