R Markdown styled directives

[fkohrt]

Hi,

I discovered Jupyter Book in early 2019, but didn't really had time to further look into it. In the meantime I've been doing a lot with R Markdown and Pandoc and now, rediscovering MyST, am happy that this ecosystem is growing.

I wondered, however, why you went for the following directive-argument syntax

```{directivename} arguments
---
key1: val1
key2: val2
---
This is
directive content
```

...instead of doing it knitr-like...

```{directivename arguments}
---
key1: val1
key2: val2
---
This is
directive content
```

...given that one of your principles is

Use pre-existing standards where possible. In the event that we must create new patterns of content creation or tooling, utilize prior art in the open source community as much as possible, especially when it comes to markup languages.

One of the reasons why Markdown is so great is its ubiquity: If I decide to use a certain toolset, I can be pretty sure that my content will also work with different toolsets. It makes my content future proof. Sadly, the MyST directive syntax doesn't degrade nicely in R Markdown documents nor Pandoc's markdown. At executablebooks/MyST-Parser#63 it is mentioned that the knitr style syntax breaks an existing regex, which seems to be the reason why you proposed a different one. I did not dig deeper into it, but wonder why you ultimately decided as you did? And, subsequently, whether the knitr style could be introduced again at a later point in time?

[chrisjsewell]

Well I guess essentially the directive syntax is not looking to imitate knitr, it is looking to imitate rST directives/Jupyter Notebook cell metadata. The arguments here are the not same as the arguments in knitr, aka the chunk metadata, in fact the key: val are what map to the knitr arguments.
You can translate between the two though using jupytext: see https://jupyterbook.org/file-types/jupytext.html#file-types-custom and https://jupytext.readthedocs.io/en/latest/formats.html

my content will also work with different toolsets

Maybe you can explain more here what you mean?
From my perspective, R Markdown is a proprietary format; there is no other tool that I know of that can directly "run" RMarkdown files, apart from using R Studio?
By contrast, we are looking for a 1:1 mapping with Jupyter Notebooks, which is more of an open standard.
Similarly the knitr arguments are a proprietary format (which IMO is not particularly great for large amounts of metadata).
Instead, we specify metadata using the common YAML standard, which can be directly mapped to Jupyter Notebook cell metadata.

the MyST directive syntax doesn't degrade nicely in R Markdown documents nor Pandoc's markdown

Again I would ask for clarification on what you mean by degrade?
You won't be able to directly run MyST in the (proprietary) knitr engine, in the same way that you can't run RMarkdown files directly with Jupyter kernels (but jupytext will help with this).
Pandoc has no directly associated "engine", and so the directives just degrade to standard CommonMark code fences.

[chrisjsewell]

that being said, obviously if there are ways to help integration, then I'm certainly open to suggestions 😄

[fkohrt]

Thank you for your quick response!

The arguments here are the not same as the arguments in knitr, aka the chunk metadata, in fact the key: val are what map to the knitr arguments.

I see, I missed that.

You can translate between the two though using jupytext

Cool, I'll look into that!

Maybe you can explain more here what you mean?

Well I thought that the ```{directive} syntax means something similar to the ```{language} syntax that emerges quite naturally (in my opinion) from Pandoc's backtick code blocks: ```python indicates printing code and ```{python} indicates running code. But as they apparently mean something very different (I don't know too much about rST), there's of course no point in using this existing syntax for arguing that the arguments should be inside brackets. Still, among the standardization efforts for markup languages, I see proposals that put parameters into brackets (like :: name [content] {key=val}), but none like :: name [content] key=val, that write parameters without anything. That's why I (maybe wrongly) regarded this as an unnecessary divergence from standardization efforts.

From my perspective, R Markdown is a proprietary format

Do you mean R Markdown is proprietary because it emerged from a company's initiative (as opposed to grass roots) and was (is?) in that sense not participatory? (Out of curiosity and not directly related: Would you consider PDF as standardized by ISO proprietary as well? Sure, calling it an open format sounds strange, but previously I had different things associated with the term "proprietary".)

there is no other tool that I know of that can directly "run" RMarkdown files, apart from using R Studio?

Strictly speaking, RStudio is not necessary for "running" R Markdown files, only if you want the interface. And, while RStudio is the company's flagship product, it's at the same time AGPL-licensed. But that's not what you're hinting at, right?

Similarly the knitr arguments are a proprietary format

Again, could you elaborate on that?

(which IMO is not particularly great for large amounts of metadata).

Yeah, you're right about that, although one could imagine something like

```{python,
    echo = FALSE,
    message = FALSE,
    ...}
print("Hello")
```

Again I would ask for clarification on what you mean by degrade?

Sure, I meant that writing something like

```{.python key1=val1}
print("Hello")
```

LaTeX

\begin{Shaded}
\begin{Highlighting}[]
\BuiltInTok{print}\NormalTok{(}\StringTok{"Hello"}\NormalTok{)}
\end{Highlighting}
\end{Shaded}

is converted to something closer to what was originally intended than

```{.python} key1=val1
print("Hello")
```

LaTeX

\texttt{\{.python\}\ key1=val1\ print("Hello")}

But, to be fair, writing

```{python key1=val1}
print("Hello")
```

LaTeX

\texttt{\{python\ key1=val1\}\ print("Hello")}

also becomes ugly.

Pandoc has no directly associated "engine"

What I meant was Pandoc's markdown.

[chrisjsewell]

Pandoc's backtick code blocks

It note here that, although Pandoc uses these, it didn't invent it 😉
Markdown was originated from John Gruber, however, he's specification was not unambiguous, which is why commonmark was developed.
markdown-it-py, and by extension MyST, is strictly tested against the commonmark specification.

Still, among the standardization efforts for markup languages, I see proposals that put parameters into brackets

The de facto standardisation for metadata is JSON/YAML, thats what we use

```{directive} syntax means something similar to the ```{language}

No, the directive name, like the name of RMarkdown chunks, has no relation to a syntax highlighting lexer; they map to some "callable" which knows how to interpret the text.
Using ```{python} is not IMO "natural" for a Jupyter Notebook, and in fact misleading, because it implies that the this relates to the kernel used to run the code. Which it does not, since that is defined at the top-level for the whole notebook. Instead we use ```{code-cell}, which means we don't have to arbitrarily state every possible kernel name up front.
For example what happens in Pandoc if I were to use the coconut kernel: {.coconut}?
(Also BTW, you can just write pandoc filters to get out what you want anyway)

Would you consider PDF as standardized by ISO proprietary as well?

In your analogy, I would not relate RMarkdown to PDF, which has many different parser/renderer implementations. Instead, I would say it is more like a Microsoft Word document, which is only really useable with one piece of software, in the same way that RMarkdown is only really useable via knitr/RStudio