Markdown support

[renefritze]

I'm currently moving our tutorial notebooks/documentation from RST with jupyter-sphinx to Markdown with myst-nb. We'd eventually like to also move docstrings to be written in myst-markdown. To that end I'm now looking at what I actually need from sphinx-autoapi to make that possible. I came up with this:

1. Cast the existing templates into Markdown. Includes using a (potential) sphinx.util.docstrings.prepare_docstring analog from myst. This is to allow two sets of default templates to exist.
2. Make the hardcoded '*.rst' instances in template loading/rendering (mappers/base.py) use some config value instead. The actual file output suffix is already (not really explicitly) configurable.

I'd be happy to contribute this. If I was to do this just for our project, I'd still have to do step 1., but keep the .rst filenames. And step 2. seems pretty trivial.

What do you think?

[AWhetter]

I wouldn't want to have to maintain two sets of templates (one in rst and one in markdown) so I think it makes sense to let users do that themselves if they want to.
The way that AutoAPI selects an output file extension isn't great. I think making it more explicit by adding a new configuration option is a good idea. Adding a second option to allow configuration of the file extension of the templates also makes sense to me.

[renefritze]

I wouldn't want to have to maintain two sets of templates (one in rst and one in markdown)

Yeah, I can understand that.

so I think it makes sense to let users do that themselves if they want to.

Do you have an idea on how best to make those converted-to-markdown templates available to other projects than mine? For an end user shipping them via pypi package would be simplest I guess, so it could be dependent upon. That could install them and export the location via package.template_dir and the user could then set autoapi_template_dir to that in conf.py. Exporting that install location could potentially be problematic in some cases though, IIRC.
The other option would be just a repo where people can copy the templates from into their own source tree.

The way that AutoAPI selects an output file extension isn't great. I think making it more explicit by adding a new configuration option is a good idea. Adding a second option to allow configuration of the file extension of the templates also makes sense to me.

Great. I'll make a PR for that sometime then.

[chrisjsewell]

Heya, cross-posting this update: executablebooks/MyST-Parser#228 (comment)

[chrisjsewell]

Heya, I've now created https://sphinx-autodoc2.readthedocs.io/en/latest/quickstart.html#using-markdown-myst-docstrings 😅

Its based on the great work here but, I feel with some improvements: https://github.com/chrisjsewell/sphinx-autodoc2#design-and-comparison-to-sphinx-autoapi

Happy to collaborate and even upstream/consolidate the two packages if you are interested!

[AWhetter]

I've started implementing this. You can see what I have so far here: https://github.com/readthedocs/sphinx-autoapi/tree/markdown
I haven't settled on exactly how to implement this yet. There's a few things that I don't like about my current implementation.

1. While I'm less bothered about needing to maintain two sets of templates than I was, if we don't need a second set of templates though then that would be great.

   • The need for a second set of templates is only necessary if we support markdown output. We could support markdown docstrings without writing out markdown output by using a new directive to parse the body of the directive as myst. So similar to .. include::, but we'd parse the body of the directive instead of reading from a file:

     .. myst_enable_extensions = ["fieldlist"]
     
     .. py:function:: my_function()
     
        .. autoapi-parse::
           :parser: myst_parser.sphinx_
     
           This is the docstring for `my_function`.
     
           It contains some [CommonMark](https://spec.commonmark.org/).

   • I can only see the output of Markdown being important in two situations. I don't feel particularly strongly about supporting them but I could be convinced otherwise.

     1. You're using AutoAPI only to generate the documentation once, and then you transition to manual documentation.
     2. You want to customise the templates and you aren't familiar with restructuredtext.

2. Figuring out which fence type to use is a bit awkward.

   sphinx-autoapi/autoapi/mappers/base.py

   Lines 21 to 23 in 4568f5a

   	def _md_fence(config):
   	colon_fence = "colon_fence" in getattr(config, "myst_enable_extensions", [])
   	return ":::" if colon_fence else "```"

   At the moment I'm only looking at the global option. I don't think that it would ever be possible for AutoAPI to respect a change in the fence if it were configured in the document by an overridden template (https://myst-parser.readthedocs.io/en/latest/intro.html#configuring-myst-parser).

3. Nested directives are supposed to be nested by making the outer directive use more fence characters than the child directives (https://myst-parser.readthedocs.io/en/latest/syntax/roles-and-directives.html#nesting-directives). However it's difficult for AutoAPI to figure out what an appropriate fence length is. The current implementation is nested by indentation but I don't know if that's actually valid. @chrisjsewell do you know?

So far I've taken the approach of using two sets of templates. But I'm thinking I may pivot and allow templates to be provided via plugins. Essentially what the original description suggested. So AutoAPI wouldn't provide the markdown templates necessary, but I would support everything required for a plugin to provide them (including the new autoapi_output_suffix option to make the output file extension configurable).

@chrisjsewell I have another question. My implementation isn't quite working properly. Possibly due to nesting the directive by indentation. AutoAPI is outputting this:

sphinx-autoapi/tests/python/pymarkdownexample/autoapi/example/index.md

Lines 137 to 149 in 4568f5a

	:::{py:method} method_sphinx_docs(foo, bar=0)
	This method is documented with sphinx style docstrings.
	:param foo: The first argument.
	:type foo: int
	:param int bar: The second argument.
	:returns: The sum of foo and bar.
	:rtype: int
	:::

)

Sphinx renders that like this:

Do you know why this wouldn't be working?

[h-vetinari]

Happy new year! Just though I'd check in on the status here, would be great to have native markdown support :)

[AWhetter]

This will be the next big thing I work on after getting #399 over the finish line.
I think I've figured out how to get around the problems that I described in my comments above, but I'm a little worried that the generation won't be (and can't be) very fast. The fact that nested fences have longer fences in the outer block, rather than the inner block, means that parent objects need their children's docstrings rendered first so it has knowledge of how long its fences need to be. So each object will need to render its docstring, scan the docstring for the longest fence, then replace the fence placeholders with the correct fence. Perhaps this won't matter too much in the grand scheme of things, but it's a necessary consequence of the design decision of how nested blocks are implemented in myst.

[michaelweinold]

@AWhetter, we would love to write our docstrings in Markdown - is there any updated on this feature? Many thanks!

[LecrisUT]

Tangent, can't an approach here be back-ported to sphinx.ext.autodoc? From what I understand there are 2 components to this:

• A template for filling in the metadata variables such as :param foo:.
  stumbled upon: docstring-parser, which has a neat way of extracting (part of) the metadata for all backends
• Using self.state.nested_parse to parse the docstring content independent-ish of the format

If sphinx.ext.autodoc can extend itself a bit to use jinja templates (preferably with jinja blocks so that it can be more easily extended), then maybe these changes can be upstreamed and make the extension more minimal.