Display math highlighting

[postylem]

Issue: to use latex environments (like align or equation) in a Quarto document, the best practice (iiuc) is to use them raw (\begin{align} ... \end{align}), not to enclose them in dollar signs (like $$\begin{align} ... \end{align}$$).

However, the VSCode extension implicitly encourages the latter (see image below; enclosing in dollar signs makes the align environment be nicely highlighted and preview-able when editing, whereas without the dollar signs it just looks like plain text). This is not good, because one shouldn't enclose environments in dollar signs, since this will lead to LaTeX compiling error when rendering (as described below).

Possible solution: Make it so highlight/preview-ing works for such latex environments without enclosing them in dollar signs.

---

For an example, here is a quarto document example.md:

---
title: Display math highlighting example
format:
  html:
    html-math-method: katex
  pdf:
    documentclass: article
    keep-tex: true
---
Here is some display math using bare `\begin{align} ... \end{align}`:
\begin{align}
C &= \max_{p_X}{I(X;Y)}\\
  &= \max_{p_X}{H(X) - H(X\mid Y)}
\end{align}
It is _not_ highlighted.

Here is some display math using `$$\begin{align} ... \end{align}$$`:

$$
\begin{align}
C &= \max_{p_X}{I(X;Y)}\\
  &= \max_{p_X}{H(X) - H(X\mid Y)}
\end{align}
$$
It _is_ highlighted, but causes issues in pdf/latex output.

This looks like the following, in with quarto-vscode:

---

Enclosing the align environment in double dollar signs, like $$\begin{align} ... \end{align}$$ makes the highlighting/previewing work in VSCode, but makes the following (bad) TeX, if converted to PDF or TeX formats. E.g., with quarto render example.md --to pdf, the second example above results in the following tex code:

\[
\begin{align}
 ...
\end{align}
\]

which causes (xe)latex to give the error

compilation failed- error
Package amsmath Error: Erroneous nesting of equation structures;
(amsmath)                trying to recover with `aligned'.

Removing the double dollar signs fixes this issue, but then you can't get nice highlighting in VSCode.

[jjallaire]

It would be great to get some clarify what the use cases for "raw" equations are. I definitely don't think that raw LaTeX environments are a best practice if you are concerned about portability, as equations without $$ are stripped entirely from many output formats including docx, pptx, and gfm. So for the sake of portability I'd like to continue to encourage $$. i.e. the following equation is rendered in pdf and html, but stripped in pptx, docx, and gfm:

\begin{equation}
  E=mc^2
\end{equation}

I do however know that the bookdown documentation uses a lot of raw equation environments, and I'm sure there is some motivation for this (and awareness that the tradeoff of having them stripped out of some formats is worth it -- perhaps there is a special workaround for docx in bookdown?)

@yihui What do you know about the tradeoffs here and when its important/preferred/required to escape out of constraints imposed by $$?

If it turns out that raw equation environments are indeed required in some cases because of limitations in $$ I will certainly give them the requisite treatment in the VS Code extension. In that case though it does seem like some improvements in pandoc might be warranted as well (cc @tarleb)

[tarleb]

One reason to use raw input is to prevent pandoc from converting math environments. The syntax $$\begin{align}...\end{align}$$ is not valid LaTeX, so pandoc converts the align environment into an aligned environment. Of course, this is not without issues, see, e.g., jgm/pandoc#4104 and jgm/pandoc#7968.

[tarleb]

Oh, I missed that the conversion only happens when parsing LaTeX, not Markdown. This means it could possibly be fixed in the LaTeX writer.

[postylem]

Okay, I see, when converting .tex -> .md, pandoc -f latex -t markdown converts \begin{align}...\end{align} into $$\begin{aligned}...\end{aligned}$$.

But if you use $$\begin{align}...\end{align}$$ in markdown, pandoc -f markdown -t latex will output the invalid \[\begin{align} ... \end{align}\].

Unless this is something that is easy to address, perhaps as things are, the suggested style for general aligned equations in a .(q)md file should be $$\begin{aligned}...\end{aligned}$$? At least, this seems better than the alternatives

1. $$\begin{align}...\end{align}$$ in the input markdown document works for converting to HTML, but makes invalid latex.
2. raw \begin{align}...\end{align} in the input markdown document works for latex or html output, but is ignored by other formats (like docx, etc).

[yihui]

In bookdown, I had to apply a hack to preserve LaTeX math environments for output formats that are not PDF or HTML (e.g., EPUB): rstudio/bookdown@8a4d8c0 That is, I don't require $$ for these environments, but will add $$ automatically before calling Pandoc to convert the .md file to a non-PDF/HTML output format.

[jjallaire]

@tarleb Do you think there is something else/more that Pandoc should be doing here? I confess that I'm not excited about the idea of math blocks having two syntaxes (one of which does not show up in the AST as math) so if there is a way to make $$....$$ behave as desired at the lowest level is preferable to special escapes by various tools.

[tarleb]

I'm not sure, but it does seem reasonable to me to change pandoc to do the equivalent of

function Math (mth)
  local is_align = mth.text:match [[^%s*\begin%{align%}.*\end%{align%}%s*$]]
  if FORMAT:match('tex') and is_align then
    return pandoc.RawInline('tex', mth.text)
  end
end

I think this would should be doable with minor changes to the LaTeX and ConTeXt writers, but I might be missing something. I'll dig a bit deeper and bring it up with John.