Citations in tables do not render correctly outside of raw markdown and kable

[andrewheiss]

Bug description

There is inconsistent behavior when including citation keys inside tables, but it's a really tricky issue and I'm not sure if it's a Quarto issue or an issue with other table-making packages.

When making a table with regular Markdown or with knitr::kable(), citation keys that are included in table rows are rendered correctly by pandoc.

However, when making a table with kableExtra::kbl() or gt::gt(), pandoc doesn't see the citation keys.

Here's a reproducible example:

---
title: "Citations in tables"
format: html

references:
- type: article-journal
  id: Lovelace1842
  author:
  - family: Lovelace
    given: Augusta Ada
  issued:
    date-parts:
    - - 1842
  title: >-
    Sketch of the analytical engine invented by Charles Babbage, by LF Menabrea, 
    officer of the military engineers, with notes upon the memoir by the translator
  title-short: Molecular structure of nucleic acids
  container-title: Taylor’s Scientific Memoirs
  volume: 3
  page: 666-731
  language: en-GB
---

## Raw Markdown

This works.

| Thing | Citation      |
|-------|---------------|
| 1234  | @Lovelace1842 |


## `knitr::kable()`

This works.

```{r}
library(knitr)
tibble::tribble(
  ~Thing, ~Citation,
  1234, "@Lovelace1842"
) |>
  kable()
```


## `kableExtra::kbl()`

This doesn't work.

```{r}
library(kableExtra)
tibble::tribble(
  ~Thing, ~Citation,
  1234, "@Lovelace1842"
) |>
  kbl() |>
  kable_styling()
```


## `gt::gt()`

This doesn't work.

```{r}
library(gt)
tibble::tribble(
  ~Thing, ~Citation,
  1234, "@Lovelace1842"
) |>
  gt()
```

Raw Markdown and knitr::kable() work correctly:

kableExtra::kbl() and gt::gt() do not render the citation:

---

This is a known issue in both kableExtra (haozhu233/kableExtra#214 (comment)) and gt (rstudio/gt#112 (comment)), and the existing solution for both packages is to use bookdown's text reference feature to render the citation outside of the table and then automatically include the rendered citation inside the table when knitting, like this:

(ref:lovelace) @Lovelace1842

```{r}
library(kableExtra)
tibble::tribble(
  ~Thing, ~Citation,
  1234, "(ref:lovelace)"
) |>
  kbl() |>
  kable_styling()
```

This doesn't work in Quarto, though. #1959 and #1785 are about adding bookdown-esque text references to Quarto, which would potentially solve this issue.

---

This is with RStudio 2022.07.2+576 on macOS Monterey 12.6, but it happens across platforms

Checklist

• Please include a minimal, fully reproducible example in a single .qmd file? Please provide the whole file rather than the snippet you believe is causing the issue.
• Please format your issue so it is easier for us to read the bug report.
• Please document the RStudio IDE version you're running (if applicable), by providing the value displayed in the "About RStudio" main menu dialog?
• Please document the operating system you're running. If on Linux, please provide the specific distribution.

[andrewheiss]

Ooh, @debruine's issue/solution in #1710 also works here! This can be fixed by removing the class with unclass(), printing the result with cat(), and including results="asis" in the chunk options

## `kableExtra::kbl()` with `unclass()`

This works!

```{r results="asis"}
library(kableExtra)
tibble::tribble(
  ~Thing, ~Citation,
  1234, "@Lovelace1842"
) |>
  kbl() |>
  kable_styling() |>
  unclass() |> cat()
```

[MohdAzmiSuliman]

i was looking for this too. however if i use result = "asis", the rendered html files unable to recognise it as table for cross reference.

---
title: "kable extra citation"
format: html
---

refer @tbl-table

```{r}
#| label: tbl-table
#| tab-cap: "table with citation"
#| results: asis

library(kableExtra)
tibble::tribble(
  ~Thing, ~Citation,
  1234, "@Lovelace1842"
) |>
  kbl() |>
  kable_styling() |>
  unclass() |> cat()
```

is there any way to caption the table and crossref it?

[cscheid]

I think you need tbl-cap instead of tab-cap?

[andrewheiss]

Using tbl-cap adds the caption to the table itself, but it is still not cross-referencable:

See @tbl-table

```{r}
#| echo: fenced
#| label: tbl-table
#| tbl-cap: Table with citation
#| results: asis

library(kableExtra)
tibble::tribble(
  ~Thing, ~Citation,
  1234, "@Lovelace1842"
) |>
  kbl() |>
  kable_styling() |>
  unclass() |> cat()
```

[cscheid]

This is now fixed, although kableExtra will need to use a slightly different method (it's currently emitting plain HTML without enclosing it in a RawBlock, which is not HTML syntax that we officially support).

For a preview of how this is going to work in the future, consider this snippet using an in-development branch of gt:

```{r}
library(gt)
tibble::tribble(
  ~Thing, ~Citation,
  1234, "@Lovelace1842"
) |>
  gt() |> fmt_markdown()
```

If you tell gt that you're providing it with markdown input (through fmt_markdown, which can also be scoped to cells, columns, rows, etc), then you get this:

If you don't call fmt_markdown(), you get this:

[cderv]

This is now fixed, although kableExtra will need to use a slightly different method (it's currently emitting plain HTML without enclosing it in a RawBlock, which is not HTML syntax that we officially support).

@cscheid about this, Aren't we enclosing ourself the result in Raw Block ? I believe we do it here

What more would be needed ? I think we need to make sure what is planned for gt will work for other framework.

Also, some comment on why in R Markdown ecosystem the raw block is not used. This is because that way it can benefit with Pandoc from the markdown_in_html_blocks extension.
With Quarto we loose this feature.

This means that with rmarkdown,

Rmd file

---
title: "Citations in tables"
output: 
  bookdown::html_document2:
    keep_md: true
references:
- type: article-journal
  id: Lovelace1842
  author:
  - family: Lovelace
    given: Augusta Ada
  issued:
    date-parts:
    - - 1842
  title: >-
    Sketch of the analytical engine invented by Charles Babbage, by LF Menabrea, 
    officer of the military engineers, with notes upon the memoir by the translator
  title-short: Molecular structure of nucleic acids
  container-title: Taylor’s Scientific Memoirs
  volume: 3
  page: 666-731
  language: en-GB
---

## Raw Markdown

This works.

| Thing | Citation      |
|-------|---------------|
| 1234  | @Lovelace1842 |


## `knitr::kable()`

This works.

```{r}
library(knitr)
tibble::tribble(
  ~Thing, ~Citation,
  1234, "@Lovelace1842"
) |>
  kable()
```


## `kableExtra::kbl()`

This doesn't work.

```{r}
library(kableExtra)
tibble::tribble(
  ~Thing, ~Citation,
  1234, "@Lovelace1842"
) |>
  kbl() |>
  kable_styling()
```


## `gt::gt()`

This doesn't work.

```{r}
library(gt)
tibble::tribble(
  ~Thing, ~Citation,
  1234, "@Lovelace1842"
) |>
  gt()
```

• We get the reference in kable() because it outputs markdown
• We get the reference in kableExtra() because the inside markdown is process thanks to markdown_in_html_blocks
• We don't get it working in gt because it outputs its code in inside raw block

Putting that here in case R Markdown user are wondering why the difference.

[andrewheiss]

@cscheid Do you know which branch of gt has support for the preview version here (#3340 (comment))? Using the gt() > fmt_markdown() approach with today's GitHub version still leads to non-working output:

[cscheid]

@rich-iannone ?

[jameshowison]

Anyone know if this should be working now? I don't think it is for me.

---
title: "trial"
format: html
---

## Quarto {#sec-testing}

Quarto enables you to weave together content and executable code into a finished document. To learn more about Quarto see <https://quarto.org>.

Inside a table cross-refs don't render:

```{r}
library(gt)
tibble::tribble(
  ~Thing, ~Link,
  1234, "@sec-testing"
) |>
  gt() |> fmt_markdown()
```

But outside a table it works fine @sec-testing

Produces this output:

I'm expecting a Linked "Section 1" to appear inside the table? This also doesn't work with kable for me either.

Current versions:
gt (0.9.0)
quarto (cmdline) 1.3.242

[cscheid]

Actually, now that I say that, I wonder. @rich-iannone, shouldn't that fmt_markdown be wrapping the markdown content in a <span data-qmd="..."></span> element? It doesn't seem to be right now. This is the HTML being emitted:

<table class="gt_table" data-quarto-disable-processing="false" data-quarto-bootstrap="false">
  <thead>
    
    <tr class="gt_col_headings">
      <th class="gt_col_heading gt_columns_bottom_border gt_right" rowspan="1" colspan="1" scope="col" id="Thing">Thing</th>
      <th class="gt_col_heading gt_columns_bottom_border gt_left" rowspan="1" colspan="1" scope="col" id="Link">Link</th>
    </tr>
  </thead>
  <tbody class="gt_table_body">
    <tr><td headers="Thing" class="gt_row gt_right"><div class='gt_from_md'><p>1234</p>
</div></td>
<td headers="Link" class="gt_row gt_left"><div class='gt_from_md'><p>@sec-testing</p>
</div></td></tr>
  </tbody>
  
  
</table>

[cscheid]

We do have a second issue, though, which is that gt emits data-quarto-disable-processing="false", and quarto checks for

              if table.attributes[constants.kDisableProcessing] ~= nil then

This one is a quarto bug and I'll fix that one. In my testing, if <div class='gt_from_md'><p>...</p></div> is replaced with <span data-qmd="..."></span>, then quarto (after this bugfix) actually renders this crossref correctly.

@jameshowison : you'll need to install the latest 1.4 prerelease (and wait for the confirmation and fix of the gt issue) but this should all be working very soon. Thanks for the patience!

[rich-iannone]

Okay, there was a change required in gt to ensure that Quarto is detected during the render. I've made that change just now as rstudio/gt@e1feb6b and the interface mentioned above by Carlos now works. You'll need to get the latest dev version of gt with devtools::install_github("rstudio/gt").

I tested with this .qmd:

---
format: html
references:
- type: article-journal
  id: Lovelace1842
  author:
  - family: Lovelace
    given: Augusta Ada
  issued:
    date-parts:
    - - 1842
  title: >-
    Sketch of the analytical engine invented by Charles Babbage, by LF Menabrea, 
    officer of the military engineers, with notes upon the memoir by the translator
  title-short: Molecular structure of nucleic acids
  container-title: Taylor’s Scientific Memoirs
  volume: 3
  page: 666-731
  language: en-GB
---

```{r}
library(gt)

tibble::tribble(
  ~Thing, ~Citation,
  1234, "@Lovelace1842"
) |>
  gt() |> fmt_markdown()
```

and got this document, where hovering on the formatted citation yields the full reference.

Really sorry it took so long to discover the underlying gt issue. Try it out and let me know if it doesn't work on your system.

[cscheid]

I can confirm this works on main(and the version of gt on github) as well. kableExtra will need to emit its output slightly differently, so I'm going to go ahead and close this one.

[cderv]

kableExtra will need to emit its output slightly differently,

I believe this could be in knitr directly that this should be done. So we could try sync an update with 1.4 release for this. I opened an issue there.

[andrewheiss]

This issue persists when rendering to PDF (using gt 0.10.0 and Quarto 1.4.424):

---
title: "Citations in tables"
format: pdf

references:
- type: article-journal
  id: Lovelace1842
  author:
  - family: Lovelace
    given: Augusta Ada
  issued:
    date-parts:
    - - 1842
  title: >-
    Sketch of the analytical engine invented by Charles Babbage, by LF Menabrea, 
    officer of the military engineers, with notes upon the memoir by the translator
  title-short: Sketch of the analytical engine
  container-title: Taylor’s Scientific Memoirs
  volume: 3
  page: 666-731
  language: en-GB
---

```{r}
library(gt)

tibble::tribble(
  ~Thing, ~Citation,
  1234, "@Lovelace1842"
) |>
  gt() |> fmt_markdown()
```

[andrewheiss]

knitr::kable() works with PDF:

```{r}
tibble::tribble(
  ~Thing, ~Citation,
  1234, "@Lovelace1842"
) |>
  knitr::kable()
```

 

kableExtra::kbl() doesn't work with PDF:

```{r}
library(kableExtra)

tibble::tribble(
  ~Thing, ~Citation,
  1234, "@Lovelace1842"
) |>
  kbl() |> kable_styling()
```

 

That might be related to yihui/knitr#2289, though that issue is specifically about HTML output

[cscheid]

```{r}
library(gt)

tibble::tribble(
  ~Thing, ~Citation,
  1234, "@Lovelace1842"
) |>
  gt() |> fmt_markdown()
```

Somehow, gt is not formatting this with markdown, but is emitting LaTeX, cf keep-md: true

::: {.cell}

```{.r .cell-code}
library(gt)

tibble::tribble(
  ~Thing, ~Citation,
  1234, "@Lovelace1842"
) |>
  gt() |> fmt_markdown()
```

::: {.cell-output-display}
\begin{longtable}{rl}
\toprule
Thing & Citation \\ 
\midrule\addlinespace[2.5pt]
1234 & @Lovelace1842 \\ 
\bottomrule
\end{longtable}

:::
:::

@rich-iannone, do you know what's going on here?

[cderv]

The document is using format: pdf here, this means that PDF output is expected. gt will then output as_latex() because this is used in a format: pdf rendering.

I don't think we have yet an equivalent of data-qmd trick on Span that works with LaTeX tables. (probably because we don't parse tables).

If you want HTML tables, even in format: pdf document, because you known Quarto will parse the HTML table anyway, you can ask for it

```{r}
library(gt)

tibble::tribble(
  ~Thing, ~Citation,
  1234, "@Lovelace1842"
) |>
  gt() |> fmt_markdown() |> as_raw_html()
```

This will correctly process the citation in the table, even for PDF outputs.

So at the end, this is a gt choice: when inside quarto document rendering, if output format is LaTeX, should it generate a HTML table anyway ? At least when fmt_markdown() is used maybe ?

Otherwise, it will be to the user to explicitly ask for HTML version of the table.

[cscheid]

I don't think we have yet an equivalent of data-qmd trick on Span that works with LaTeX tables. (probably because we don't parse tables).

But if you provide an HTML quarto-style table in LaTeX, we will do the right thing. That was the whole point of doing a uniform parser for tables in quarto. See:

--- format: pdf --- ```{r} #| label: fig-1 #| fig-cap: A caption plot(1:100) ``` ```{r} #| label: fig-2 #| fig-cap: Another caption plot(20:30) ``` ```{=html} <table> <thead> <tr><th>What</th><th>Where</th></tr> </thead> <tbody><tr> <td><span data-qmd="@fig-1"></span></td> <td><span data-qmd="@fig-2"></span></td> </tr></tbody> </table> ```

[cscheid]

And, more specifically, I thought that gt had been changed to have a way to send HTML tables to quarto, even in non-HTML formats.

[andrewheiss]

Ok cool, yeah, I also thought that {gt} had changed for non-HTML formats and emitted HTML. Adding as_raw_html() works for PDF as expected, but leaving it on messes with regular HTML output, so some conditional work has to be done to not use it with HTML

Normal HTML output:

 

With as_raw_html() so that LaTeX is happy, but now the HTML output is messed up because something changed with the gt CSS or something:

But that's definitely a {gt} issue, not a Quarto issue

[cderv]

But if you provide an HTML quarto-style table in LaTeX, we will do the right thing. That was the whole point of doing a uniform parser for tables in quarto.

yeah I know that. We all agree here ! The code I shared was about specifically outputting HTML in the PDF format for it to work.

And, more specifically, I thought that gt had been changed to have a way to send HTML tables to quarto, even in non-HTML formats.

I don't think it has. @rich-iannone is this a plan to output HTML table for LaTeX document when rendered in Quarto ?

Only drawback of this approach is possibly loosing all the custom LaTeX styling for gt table if any, that can't be set through Quarto/Pandoc writing the raw LaTeX instead of gt directly.

The example share by @andrewheiss is a hint about this I believe. Possibly some quarto improvement on this not gt as this is Quarto/Pandoc that is generating the raw LaTeX inserted, and no more gt.

@rich-iannone I let you open a ticket on your repo to track potential improvement and more testing.

[andrewheiss]

Yeah, using as_raw_html() strips away all custom LaTeX stuff like column widths when outputting to HTML

With HTML:

With LaTeX:

[rich-iannone]

@andrewheiss me and Carlos discussed this and we have a good plan in place that will preserve the LaTeX table code that gt generates while also allowing Quarto to properly handle citations as well as performing the Markdown conversion. More to come.

[cscheid]

@rich-iannone the PR on the quarto side is here #7451

[andrewheiss]

Following up on this (and related to #9342), @rich-iannone is there a way to use \QuartoMarkdownBase64{} from #7451 in {gt}?

For example, this qmd has an equation and a citation:

---
title: "Citations in tables"
format: 
  html: default
  pdf: 
    keep-tex: true

references:
- type: article-journal
  id: Lovelace1842
  author:
  - family: Lovelace
    given: Augusta Ada
  issued:
    date-parts:
    - - 1842
  title: >-
    Sketch of the analytical engine invented by Charles Babbage, by LF Menabrea, 
    officer of the military engineers, with notes upon the memoir by the translator
  title-short: Molecular structure of nucleic acids
  container-title: Taylor’s Scientific Memoirs
  volume: 3
  page: 666-731
  language: en-GB
---

$$
a^2 + b^2 = c^2
$${#eq-math}

```{r}
library(gt)
tibble::tribble(
  ~Thing, ~Citation,
  1234, "@Lovelace1842",
  5678, "@eq-math"
) |>
  gt() |> 
  fmt_markdown(Citation)
```

When rendering to PDF, those @ things don't get processed, but doing it manually with LaTex and \QuartoMarkdownBase64{} works (kind of; in #9342 this doesn't work with citation keys yet)

```{=latex}
% QExvdmVsYWNlMTg0Mg==} is "@Lovelace1842" in base-64 encoding
% QGVxLW1hdGg= is "@eq-math" in base-64 encoding
\begin{tabular}{cc}
Thing & Citation \\
1234 & \QuartoMarkdownBase64{QExvdmVsYWNlMTg0Mg==} \\
5678 & \QuartoMarkdownBase64{QGVxLW1hdGg=} \\
\end{tabular}
```

I tried adding the base64 content myself in gt():

tibble::tribble(
  ~Thing, ~Citation,
  1234, "\\QuartoMarkdownBase64{QGVxLW1hdGg=}",
  5678, "\\QuartoMarkdownBase64{QExvdmVsYWNlMTg0Mg==}"
) |>
  gt()

…but {gt} automatically converts \\ to \textbackslash{}, so we end up with this LaTeX:

\begin{longtable*}{rl}
\toprule
Thing & Citation \\ 
\midrule\addlinespace[2.5pt]
1234 & \textbackslash{}QuartoMarkdownBase64\{QExvdmVsYWNlMTg0Mg==\} \\ 
5678 & \textbackslash{}QuartoMarkdownBase64\{QGVxLW1hdGg=\} \\ 
\bottomrule
\end{longtable*}

and this PDF:

Is there a way to get {gt} to emit \QuartoMarkdownBase64{} without automatically escaping the LaTeX?

[cderv]

@andrewheiss I believe this is somewhat related to this issue I opened in GT for the case for HTML

• [quarto] data-qmd content should be escaped at least on double quotes rstudio/gt#1487

But same workaround applies

library(gt)
tibble::tribble(
  ~Thing, ~Citation,
  1234, "\\QuartoMarkdownBase64{QGVxLW1hdGg=}",
  5678, "\\QuartoMarkdownBase64{QExvdmVsYWNlMTg0Mg==}"
) |>
  gt() |> 
  fmt_passthrough(
    columns = Citation,
    escape = FALSE
  )

Using fmt_passthrough with escape = FALSE should help here.

It should write this table

\begin{longtable}{rl}
\toprule
Thing & Citation \\ 
\midrule\addlinespace[2.5pt]
1234 & \QuartoMarkdownBase64{QGVxLW1hdGg=} \\ 
5678 & \QuartoMarkdownBase64{QExvdmVsYWNlMTg0Mg==} \\ 
\bottomrule
\end{longtable}

Hope it helps