Create documentation on best-practices for Sphinx extensions

[choldgraf]

Details

In a similar spirit to @astrojuanlu's Jupyter notebook docs (#8283), I think it could be really useful if the ReadTheDocs website provided a short tutorial on "best practices in creating Sphinx extensions".

One of the best parts about Sphinx is its extension mechanisms / community, but I find the documentation for how to build extensions to be extremely under-complete and hard to navigate. I think many would benefit from tutorial-level material and how-to information about creating Sphinx extensions. Ideally this could come with a simple template repository that others could copy and modify, and that would be kept up-to-date with the latest APIs etc that Sphinx uses to define extensions.

I imagine that it could be more opinionated than the Sphinx docs themselves would provide, and could be used to cover a bit more depth about how people generate extensions to do different things in Sphinx, for example:

• Plugging into the events chain
• Creating new roles / directives
• Modifying page documents
• Adding new JS/CSS/etc behavior

Longer term, perhaps parts of this could be merged into the Sphinx documentation itself. But I feel like RTD would have the benefit of moving a bit more quickly, and being able to cover more topics in one guide than the Sphinx docs currently follow.

Curious what others think about this - perhaps @ericholscher and others have thought about this before? If so, feel free to close! I just wanted to throw out the question in case it creates sparks :-)

[astrojuanlu]

Just my quick 2 cents: (1) I think this is a great idea, (2) @humitos can probably teach me a lot about this as author of sphinx-hoverxref, sphinx-notfound-page and others and (3) it would be awesome if we could have this on the Sphinx documentation itself (and then if for whatever reason they think it's out of scope, move it outside). The experience with the tutorial is being super positive.

[choldgraf]

I am all for this being in the sphinx docs if they are up for it!

[humitos]

I agree with @choldgraf that the documentation for writing extensions is not extensive and I found myself having trouble understanding how all of this works. Honestly, once I learned the basics for it from the official documentation, I just started reading Sphinx's code and other well-known Sphinx extensions to move forward because there are no docs at all for medium to advanced tasks.

I remember that we have talked about this in the past with @agjohnson because we realized that we were repeating too much code in our Sphinx extensions. By that time, I thought that it could be useful to have a cookiecutter template. It would give developers an initial bootstrap for a new Sphinx extension and it could include an initial wizard with some questions to generate different structures depending on the type of the extension easily.

IMO, a combination of a guide describing the commons plus a well-structured template with the basics already implemented (that you can select to include or not from the wizard) would be super powerful and would allow more people to create their own extensions.

and could be used to cover a bit more depth about how people generate extensions to do different things in Sphinx, for example:

I would add these topics to this list as well at first sight:

• Writing test cases
• Debugging your extension
• Sphinx's specific nodes (e.g. pending_xref)
• Overriding Sphinx's default behavior (advanced)
  • override an already registered event
  • override an already registered translator
  • override functions from HTML context
  • manipulate original toctree
• Properly deprecate an extension config

[ericholscher]

+1 on this work. There is a lot of benefit in general to the ecosystem here. I'm imagining something like this:

• Finish Sphinx tutorial
• Start working on RTD tutorial
• Iterate on a few other tutorials in the meantime

In my mind the RTD tutorial is probably still higher priority for us & the grant, but this is definitely one that should be on the list in the near future, or if Juanlu has bandwidth to do multiple concurrently.

Wish list

I'd also love to do a bit more coding as well to codify very common extensions (eg. new roles, directives, etc.). Having that be much more declarative (eg. Django-style overridable classes) would make them way easier to implement and reduce the documentation overhead. We've taken small steps in this direction over time, but never got all the way there.

I'm thinking something like this:

class ExpandDirective(Directive):

  def handle_args(self, args):
    if `expand_all` in args:
      return `Text('Awesome')`

Or something along those lines, where you have a super small and well-defined interface for common extension patterns.

[choldgraf]

Yeah something I've wanted for a long time is a GitHub repo with a bunch of boilerplate patterns that are common in sphinx extensions that I can just copy paste from, or copy and then delete the code I don't need. This could include common "helper" functionality like testing infrastructure etc

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[humitos]

We have been moving away from creating specific Sphinx extensions and we have been trying to make features that are documentation tool agnostic lately. Taking into account the amount of work this issue conveys, I think that we won't be able to do this work by ourselves, unfortunately. I'm proceeding to close this issue for now but I'm open to receive a PR for our documentation that explains this in a clear way.

[ericholscher]

It should likely go into the Sphinx docs, if anyone wants to write it, as well. I agree though we're not likely to be the ones to write it.