Maths environments converted as markdown --> rendering issues

[jfbercher]

I have noticed the following with nbconvert 4.1.0:

• Problem with double subscripts: try something like this in a markdown cell:

\begin{align}
[x*\delta_{n_0}] (n) & =  \sum_m x(m) \delta_{n_0}(n-m)  \\
 & =  \sum_m x(m) \delta(n-m-n_0) \\
  & =    x(n-n_0) \\
 \end{align}

The conversion via jupyter nbconvert FILE.ipynb leads to an html file with this excerpt

\begin{align}
[x<em>\delta<em>{n_0}] (n) &amp; =  \sum_m x(m) \delta</em>{n_0}(n-m)  \
 &amp; =  \sum_m x(m) \delta(n-m-n_0) \
  &amp; =    x(n-n_0) \
 \end{align}

The pair of subscripts have been converted to em tags, and of course MathJax then refuses to render the equation. A similar issue occurs with an equation like

\begin{equation}
[x*h](n) = \sum_l h(l)x(n-l)
\end{equation}

where the notation at the beginning is interpreted as a link (and of course not rendered)

\begin{equation}
<a href="n">x*h</a> = \sum_l h(l)x(n-l)
\end{equation}</p>

In the past, equations/align/etc (perhaps all environments) were not converted. It might certainly be a good idea to do so, but this causes this kind of problems. Also this happens only in the nbconvert path. The same equations are rendered correctly in the notebook.

[takluyver]

I don't see this problem when I convert it with master.

Can you check what versions of nbconvert and mistune you have installed?

[jfbercher]

nbconvert.__version__
Out[2]: '4.1.0'
In [5]: mistune.__version__
Out[5]: '0.7.1'

The issue described above was under debian with vanilla python/jupyter. I have reproduced the same behaviour under windows with an anaconda distribution. I still hope that this is not a side effect of something I may have installed elsewhere... I took the precaution to disable nbextensions and renamed ~/.jupyter into something else before attempting again.

I also observed that the conversion issue happens only if the markdown cell contains something before the equation. That is

An equation with a "link" in it
\begin{equation}
[h*x](n)=z
\end{equation}

produces an issue like I reported before, while a cell with the equation alone yields the correct conversion. Another example is

A simple sum
\begin{equation}
a
+ b
+ c
\end{equation}

which yields

A simple sum
\begin{equation}
a</p>
<ul>
<li>b</li>
<li>c
\end{equation}</li>
</ul>

[jfbercher]

Output of debug mode:

[NbConvertApp] Converting notebook tst_equation.ipynb to html
[NbConvertApp] Notebook name is 'tst_equation'
[NbConvertApp] Applying preprocessor: coalesce_streams
[NbConvertApp] Applying preprocessor: CSSHTMLHeaderPreprocessor
[NbConvertApp] Applying preprocessor: HighlightMagicsPreprocessor
[NbConvertApp] Attempting to load template full.tpl
[NbConvertApp] Loaded template full.tpl
[NbConvertApp] Attempting to load template full.tpl
[NbConvertApp] Loaded template full.tpl
[NbConvertApp] Writing 188507 bytes to tst_equation.html

[takluyver]

Aha, the 'something before' was the key. Now I can reproduce it.

Mistune tries to match a series of block level rules, to which we add a rule for latex environments, which passes the contents through without parsing inline markup. However, in this case, the paragraph rule is matching the whole block, so our latex_environment rule never sees it. You can work around it by leaving a blank line before your \begin tag:

A simple sum

\begin{equation}
a
+ b
+ c
\end{equation}

The paragraph rule has a series of clauses that will cause it to stop matching at other block-level elements, e.g. > block quoting. But when we extend BlockGrammar, there's no easy way to add our block-level elements to that to stop the paragraph rule from consuming them.

@lepture, would you be interested in making this more robustly extensible? I can spend some time working on it, if you like.

[lepture]

@takluyver I am going to rewrite mistune with suggestions by http://lucumr.pocoo.org/2015/11/18/pythons-hidden-re-gems/ some days (that would be next year).

Currently, I don't have much time working on it. If you could help me, I'd be happy to review it.

[hmeine]

@lepture Are you aware of pyparsing? The more regexp magic one needs, the more likely pyparsing would be more elegant, and maybe even faster.
Just a heads up, who codes decides (and I can't help in the foreseeable future).