Markdown display output has missing header

[jtf621]

If I have a cell with the following code::

from IPython.display import Markdown
xx = f'''
## this is a test of returning stuff

it is time to **see** if this works

### a list follows

- one
- two
- three
'''
Markdown(xx)

jupyter will render the output normally,

<div class="output_subarea output_markdown rendered_html output_result"><h2 id="this-is-a-test-of-returning-stuff">this is a test of returning stuff</h2>
<p>it is time to <strong>see</strong> if this works</p>
<h3 id="a-list-follows">a list follows</h3>
<ul>
<li>one</li>
<li>two</li>
<li>three</li>
</ul>
</div>

but when I compile to html with sphinx, the headers do not appear in the output

<div class="output_area docutils container">
<p>it is time to <strong>see</strong> if this works</p>
<ul class="simple">
<li>one</li>
<li>two</li>
<li>three</li>
</ul>

I am using nbsphinx==0.2.14 installed with pip.

[mgeier]

Thanks for the report!

It turns out there are two ways to parse nested RST content:
http://www.sphinx-doc.org/en/stable/extdev/markupapi.html#parsing-directive-content-as-rest

I switched to nested_parse_with_titles() in #153.
Can you please try if that works for you?

Please note the comment I've added over there.

[jtf621]

Thanks for the very quick response and patch. Sorry about my earlier comment, something in my path was shadowing the newly installed version.

The change does now insert the headers in the output.

The remaining question is that these headers do not show up in the Table of Contents. If I put the same string in an input in markdown mode cell, the h1, h2, h3 levels show up in the index table of contents. However the rendered Markdown output cell headers do not show up in the TOC.

Do you think this is related to keep this issue open?

[mgeier]

Good to know that #153 works and solves the problem to some extent!

TBH, I don't know how the TOC is generated in Sphinx.

To some extent I'd say it makes sense that the headings are not shown in the TOC, since they appear in a separate "text box" of the document, namely in an output cell, and are therefore not part of the main document flow.
But it seems that you want them to be, right?
In that case we probably have to do something similar to what is done in the only directive, see the links in #153.

Just out of curiosity, how are you using code cells to create Markdown content?
Do you have a real-life example?

[jtf621]

I had hoped that the output heading would appear in the TOC.

I use sphinx to keep a programming/data analysis notebook where I can use jupyter notebooks for a lot of exploratory coding and analysis in several languages as well as reference material that I type up in rst directly. When I encountered the issue, I was trying out returning data cleaning and merging issues summaries in markdown. With the inclusion in the TOC I can return a large amount of info and jump to it easily.

Up to this point, my notes on the issues that I need to research and fix are manually entered into markdown cells after finding them with code cells. I had hoped to make the summaries programmatically. If it helps you understand, I can make a minimal real-life sample for you.

Can you point me any resources to help me understand the tool chain data flow in more detail so I can try to look into this myself?

Thank you very much for this package. The combination of jupyter notebooks "pages" in a sphinx document has been a very powerful way of keeping an electronic version of an old fashioned inventor/scientists notebook.

[mgeier]

There isn't a lot of documentation about the internals of Sphinx, I think the best entry point is the extension tutorial: http://www.sphinx-doc.org/en/stable/extdev/index.html.

Other than that, the most reliable source of information is the Sphinx source code itself. But it's huge and complex ...
If you need help, don't hesitate to ask at the "sphinx-dev" or "sphinx-users" mailing lists, the people there are normally very responsive and helpful.
It may also be interesting to study the source code of some built-in or third-party extensions. You should have a look at nbsphinx.py!

If you want to try to get your headings into the TOC, you could try to copy some code from Sphinx' only directive (see the links in #153) to nbsphinx' NbOutput class (look for "nested_parse").

If this works, you could ask the Sphinx people if they are willing to turn this into a re-usable function: https://github.com/sphinx-doc/sphinx/blob/2e04c2a0588d1b9e7b1cf0d1b67e1a5496704960/sphinx/directives/other.py#L369-L407. This would be just a variation on util.nested_parse_with_titles(), probably this could be even implemented as a keyword argument to this function?

BTW, are you also using LaTeX output?
All this might be causing even more problems there ...

[mgeier]

As an alternative to the whole thing, you could try to programmatically create and save an RST file in a code cell. After that cell you can insert a raw reST cell with an include directive that includes that same file.

This is a bit clunky, but it should immediately work and the headings should automatically be part of the TOC (I didn't try it though).

[mgeier]

@jtf621 I've closed this issue because the missing headers are not missing anymore.

Please create a new issue for those headers not being part of the TOC.

[jtf621]

Ok, I will review your suggestions and submit an issue. Thanks for getting the headers included in the output.

[jtf621]

@mgeier I tried the create a RST file and include it. The headers to show up in the output (I use html) but the content is not visible in the jupyter notebook which defeats my workflow. I am looking into your prior suggestions and will open an issue once I review the internals of sphinx a bit.

[rddaz2013]

if the toc ist build from a RST file (normal way in sphinx)

• If the Markdown is a source .. then this will be recognized and linked as a link in the TOC (from sphinx)
• If the Markdown is a output...then the link within the document is build but it does not occur in the TOC.

if the toc ist build from a ipynb file (http://nbsphinx.readthedocs.io/en/0.3.1/subdir/toctree.html)

• If the fixed Markdown is a source then this will be recognized and linked as a link in the TOC (from sphinx)
• If the Markdown is a dynamic with

from IPython.display import display,Markdown,Image,HTML,Latex 
Markdown('[Test_dyn](Untitled.ipynb)')

and within Untitled.ipynb

from IPython.display import display,Markdown,Image,HTML,Latex 
Markdown('# Nachmal Test der Überschrift')

then the link within the document is build but it does not occur in the TOC. And a warning emerges 'source/Untitled.ipynb: WARNING: document isn't included in any toctree'