Documenting WorkChain and CalcJob plugins

[espenfl]

I notice that upon including https://aiida.readthedocs.io/projects/aiida-core/en/latest/developer_guide/aiida_sphinxext.html for workchains and similar for calculations (after #3299) that it might be convenient to try to change the formatting slightly. Here is my suggestions:

1. Since we use a lot of inline literals for variables etc. it would be nice if we could get the spec.inputs and spec.outputs names to be rendered as inline literals. Now they are rendered with lineral_strong function from the addnode from Sphinx. I have tried to see it it is possible to render reStructuredText in any of the docutils.nodes or sphinx.addnodes but gave up due to lack of examples that I considered to be relevant. If anyone could chime in on this, that would be great.

2. Similarly, I would like to have the data type following the name to be projected as :py:class:'aiida.orm.nodes.data.Dict or similar. Then they are clickable etc. and fits with the documentation otherwise. Again, tried to search for solutions, but did not find any similar requests.

3. It seems when using expose_inputs etc. we pull along everything from the bottom, meaning that for instance metadata is shown for all WorkChain. I think ideally we would like to offer some granularity to this as we would like to restrict what is projected to the users.

4. Also, when the docstring is taken from the class definition, reStructuredText is not supported, making it a bit difficult to support a good API documentation and at the same time leveraging the functionality to get the workchain and calculation documentation into a more general documentation on the workchains and calculation functionality.

Comments, suggestions, limitations etc?

[espenfl]

Discovered now that the reStructuredText, at least the inline literals are preserved for the port.help parameters. Investigating if we can apply this to the rest.

[espenfl]

Point 1 is not almost fixed. It is still projected using a large font and I am not sure why. But it is better. Now at least these things are projected in red and it seems easier to follow if you compliment with some reStructuredText in the documentation itself. Following the same principles for point 2 did not work and I got this:

Exception occurred:
  File "/home/efl/env/aiida/lib/python3.7/site-packages/sphinx/util/docutils.py", line 406, in env
    return self.inliner.document.settings.env
AttributeError: 'Values' object has no attribute 'env'

which is caused by the py:class: entry.

[sphuber]

Realize PR #3314 only addressed this issue partially in generalizing it from WorkChain classes to all Process classes. The styling improvements have not yet been dealt with it I think

[espenfl]

That is correct @sphuber. Thanks for leaving it open. I believe we should try to fix these issues. Maybe I will have a look at this next week if it is still open. Should be possible.

[ltalirz]

Yes - I originally planned to address the points Espen mentioned here but then didn't manage to figure it out in time either.

After looking at the built documentation with suggestion 1. implement, I think I prefer it without this modification (showing just the port names in red to me looks inconsistent, also compared to the rest of the API docs), but I would definitely support points 2., 3. and 4.

[greschd]

From a UI perspective, I think making namespaces collapsible would be really neat. In HTML this could be done with the <details> tag. I don't know how to do that in docutils / sphinx though, maybe it would be possible to create a custom node like in https://github.com/sphinx-doc/sphinx/blob/master/sphinx/addnodes.py, and a HTML renderer for that.

If that's reasonably possible, it would make point 3 better. Personally I'd like to keep all inputs because I keep forgetting the names of the different metadata flags, but I also agree the list is too long.

[greschd]

FYI, this is a reasonable intro to creating sphinx extensions (which I should probably also watch again...) https://www.youtube.com/watch?v=8vwtgMkqE9o

[espenfl]

@greschd Thanks, I will watch it as well. I have rather limited experience with this, but seems like it would be time well spent. Probably not the first time we want to create something a bit outside the box.

[chrisjsewell]

Oo interesting, I hadn't noticed the aiida.sphinxext before. Randomly, I've actually recently made some sphinx/docutils directives for collapsible notebook cells (see here). But I'm guessing this is what you are looking for: sphinxcontrib-details-directive :)

[greschd]

What do think about this? The "Namespace Ports" is foldable (see lowest namespace), and only appears if there are explicit ports defined in the namespace.

[greschd]

It seems sphinxcontrib-details-directive requires Sphinx 2.0, which doesn't support Python2. We could wait with this feature until we drop py2, implement the necessary parts for the detail directive ourselves, or just code it such that the collapsing only works when the doc is built in py3.

[ltalirz]

Thanks @greschd , looks nice!
One minor suggestion: Could one simply replace the bullet point for the input namespace with the "triangledown" directly? That would save one line. Just in case it's not too difficult to implement.

We could wait with this feature until we drop py2, implement the necessary parts for the detail directive ourselves, or just code it such that the collapsing only works when the doc is built in py3.

Since you're working on it now and we are going to drop python2 support early next year, I would vote for option 2; otherwise we risk it gets hanging for some time.

[ltalirz]

One other point that isn't quite clear to me is where the extra spacing around the "namespace" port comes from - in particular there also seems to be extra space before it (e.g. namespace y)

[greschd]

WIP branch: https://github.com/greschd/aiida_core/tree/details_directive_for_namespace

It was surprisingly easy to just have the feature conditional on Py3 (did it before I saw your comment).

Not sure if simply making the top part clickable is easily doable: It might mess with the indentation, because those are different levels of "bullet list".

Spacing definitely needs a bit of clean-up.

[chrisjsewell]

I believe this was closed by #3441, but feel free to re-open if not

[ltalirz]

There are still some formatting issues with the current extension - see, e.g. here

Space missing after workchain :

It would be super useful if the docstring of the workchain could itself be rendered using sphinx - currently it's ending up in the docs as plain text, which makes it hard to read:

[chrisjsewell]

That's easy to fix, you just add a nested parse. But actually there's bigger problems than that; it doesn't work correctly at all with autodocs. In #4470, I fixed the fact that autodoc was just being ignored. But now it's not, it overrides the normal class/method documenting, and the directive does not correctly store the documented objects for referencing. So now all the references are broken.

There's no easy fix I see for that, I think I just need to re-write the whole thing

[espenfl]

I support this issue not being fully resolved. But then there is of course the question of what we can do. It seems @chrisjsewell has a pretty good overview of what is needed to make this fly properly. Can we open an issue at sphinx or the autodoc module? An alternative is to close this issue and make a couple which are more to the point. Okey for me.

[chrisjsewell]

Note this issue is already being tracked in #3689, we can discuss more there