knitr::read_chunks and knitr::purl for other languages

[baptiste]

Description

I'm a big fan of knitr's code externalisation: it lets me tinker with the code in a self-contained document, reuse it in other files (e.g. source('funs.R')) etc. while focusing more on the text/narrative in the Rmd/qmd file, but still benefit from the reproducibility of literate programming (namely: the code is the actual code, and all artefacts have resulted from it).

Unfortunately, I believe this only exists for the knitr+R combination, not for other languages or engines?

As a workaround, I've resorted to a reverse strategy, where all the code is in a qmd file, and I use knitr::purl to extract it to a standalone source file.

---
title: "testpurl"
engine: knitr
---

```{julia, eval=FALSE}
1 + 1
```


```{julia, eval=FALSE}
plot(1,1)
```

`r invisible(knitr::purl(xfun::with_ext(knitr::current_input(), "qmd"),output=xfun::with_ext(knitr::current_input(), "jl")))`

Note the R code at the end is not in a chunk, lest it be included in the julia script (and would cause errors). It's a bit awkward, I think it'd be nice if quarto had such an option to "dump" all code into a script at the user's request? (and ideally this script could be listed as "other formats" in a quarto webpage). (So I guess this is 3 feature requests into one)

[allefeld]

I would like to have this feature, too.

My use-case are qmd files published as a website, including Matlab code executed via a custom Jupyter kernel. I didn't think to use knitr::purl, but engine: knitr is not an option for me anyway.

What I ended up doing is to implement a conversion from Jupyter notebook to Matlab script in Matlab, called from a code cell with echo: false. That notebook exists (with extension .quarto_ipynb) during execution, because that is the way Quarto executes Jupyter-kernel based qmd files. Implementation is relatively easy, Jupyter notebooks are quite simple JSON structures.

But such a conversion could be implemented in Quarto itself, activated e.g. via a YAML header

format:
  html: default
  script: default

and offered for download as an "other format".

Somewhat related: #7411.

[allefeld]

@baptiste, if it's just about the conversion from qmd to a script, that's not too hard. I wrote a bit about existing tools here. There is nbconvert, but much better is jupytext which now understands the qmd format and supports 32 different programming languages.

For me, the reason it would be neat to have this built into quarto as a format is that then the script could be included in HTML output as an "other format", see my #7411.

Such a feature could rely on an external tool, preferably jupytext, which has to be separately installed.

(An alternative proposed to me by @mcanouil would be to use code-tools / source for this purpose, but preferably only after it is integrated with "Other formats", see his #7436.)

---

But also @cscheid, I don't think it would be that complicated to implement the conversion in Quarto itself – but maybe I'm missing something? For knitr it can be based on purl, and for Jupyter all the code and all the Markdown text can be accessed from the JSON data in the .quarto-ipynb file which is created anyway. There are three things this functionality needs, and they could be provided as options to the script format.

format:
  script:
    language: python  # code blocks of which language should be included?
    extension: py     # what should the file extension for the script be?
    comment: "#"      # which comment character should be used?

This way Quarto doesn't need to know anything about specific languages.

(As far as I can see, a script conversion which includes code blocks of more than one language doesn't make sense.)

As for quarto inspect, it provides a lot of useful information, but it does not include the Markdown part which imho should be included in the script as comments.

[cscheid]

Those are the options you want. But if we're going to do this as a feature, Quarto needs to define and implement the options that anyone could possibly want, and I don't currently believe we can do this better than teaching people about quarto inspect so they can write their own downstream tooling.

[allefeld]

I don't understand. Of course many more features are conceivable and may be nice, but those are the bare-minimum options to create a script file which includes Markdown text as comments. And that seems to be what people typically want, judging by what nbconvert and jupytext do by default.

Anyway, what about the idea to use jupytext as an external tool? It's very good at figuring out the language, and knows extension and comment style for many languages. And there could be a one-size-fits-all option for jupytext command line flags, externalizing all the complexity (and follow-up feature requests) there.

teaching people about quarto inspect so they can write their own downstream tooling

That's definitely a good idea – I only found about it recently and wish I had earlier.
(Maybe a page documenting all Quarto CLI commands would be good? quarto --help is a bit sparse.)
But I still think for conversion the JSON-format ipynb is more useful.

[mcanouil]

Maybe a page documenting all Quarto CLI commands would be good? quarto --help is a bit sparse.

There is an open issue about that.

[cderv]

Nice thread and ideas !

FWIW I opened a FR in quarto R package for a wrapper function to extract script

• Add a function to extract cell code from a .qmd document quarto-r#208

I will be for R user main, but can help prototype and be useful while we think if / how / when this could be a script format.

Personally, I think more of this as a quarto convert feature than a rendered output format

[mcanouil]

For reference:

• Command line tool to extract code #389