Review compatibility of LaTeX dependencies with PDF tagging

[Witiko]

During their TUG talk, @u-fischer mentioned a list of LaTeX packages (and classes) that are (in)compatible with the PDF tagging project. We should review the list, compare it with the LaTeX dependencies of the Markdown package (see technical documentation, Section 1.1.3), and see if we can remove or replace any packages that are not listed as "compatible".

In the example markdown documents for the PDF tagging workshops at TUG 2023 and 2024, there is the following code:

\usepackage[tightLists=false,
            fencedCode %%don't load paralist ...
            ]{markdown}

This indicates that the paralist package is a part of the problem.

[Witiko]

In the list, the markdown package is listed as "unknown" and "needs tests". I wonder if we can do anything about that?!

[u-fischer]

I can change that to "partially-compatible" with a comment and add the workshop file as a first test file. But as there are lots of options it really needs more tests and examples. (And you can write "in her TUG talk ...", it is not a secret that I'm a woman.)

[Witiko]

But as there are lots of options it really needs more tests and examples.

Most options enable syntax extensions. These produce renderer commands that are handed over to various LaTeX packages unless the commands are redefined by users to do something else. All dependencies are listed in Section 1.1 of the technical documentation. The default renderer commands are defined in Section 3.3.4 of the same document. Most incompatibilities should be visible at a glance and due to our use of an outdated LaTeX package. We do little low-level TeX stuff or patching on our own.

I am happy to write tests and examples. As I understand it, this was one of the topics of this year's LaTeX developers' workshop. Sadly, I could not attend it but I am happy to go through the materials on my own.

And you can write "in her TUG talk ...", it is not a secret that I'm a woman.

I am sorry. I have been using the singular they for possessives in my technical writing for the past ~15 years, long before it was the politically correct thing to do. I realize that it may feel somewhat impersonal as technical writing often does.

[Witiko]

This indicates that the paralist package is a part of the problem.

Earlier today, I happened to glance on my notes from the TUG 2023 workshop, which indicate that our use of packages enumitem and fancyvrb for the default renderer prototypes was also a problem, not just paralist.

Looking at https://latex3.github.io/tagging-project/tagging-status/, I can see that enumitem is already partially-compatible, which may indicate that it will eventually be fully compatible. In that case, we should simply migrate our uses of paralist to enumitem.

By contrast, fancyvrb is listed as incompatible but we don't have a suitable replacement for the minted package at the moment, which uses fancyvrb as its backend; therefore, it's unclear what should be done about our use of the fancyvrb package, if anything.

[Witiko]

I took a systematic look at all LaTeX requirements of the Markdown package for TeX:

• ✅ url: A package that provides the \url macro for the typesetting of links.
  • ✅ On 2024-10-08, the package was marked as compatible at [tagging-status].
• ✅ graphicx: A package that provides the \includegraphics macro for the typesetting of images. Furthermore, it also provides a key-value interface that is used in the default renderer prototypes for image attribute contexts.
  • ✅ On 2024-10-08, the package was marked as compatible at [tagging-status].
  • ✅ Since commit 3.0.0-alpha-8-gd28ea76e from 2023-07-13, the default image renderer prototype for LaTeX has used the alt parameter to provide the alt text for images.
• ❌ paralist: A package that provides the compactitem, compactenum, and compactdesc macros for the typesetting of tight bulleted lists, ordered lists, and definition lists as well as the rendering of fancy lists.
  • ❌ On 2024-10-08, the package was marked as currently-incompatible at [tagging-status].
• ✅ ifthen: A package that provides a concise syntax for the inspection of macro values. It is used in the witiko/dot LaTeX theme (see Section 2.3.3).
  • ✅ On 2024-10-08, the package was marked as compatible at [tagging-status].
• 🤔 fancyvrb: A package that provides the \VerbatimInput macros for the verbatim inclusion of files containing code.
  • ❌ On 2024-10-08, the package was marked as currently-incompatible at [tagging-status].
  • ✅ The article [automated-tagging] lists the package as an "important package" whose "replacements or adaptations will be provided at a later stage".
• ❌ csvsimple: A package that provides the \csvautotabular macro for typesetting csv files in the default renderer prototypes for iA Writer content blocks.
  • 🤔 On 2024-10-08, the package was marked as partially-compatible at [tagging-status].
  • ❌ On 2024-10-08, the dependent package tabularray was marked as incompatible at [tagging-status].
• ✅ gobble: A package that provides the \@gobblethree TeX command that is used in the default renderer prototype for citations. The package is included in TeXLive ≥ 2016.
  • 🤔 On 2024-10-08, the package was not listed at [tagging-status].
  • ✅ The package provides simple macros for the expansion processor that should not affect PDF tagging.
• ✅ amsmath and amssymb: Packages that provide symbols used for drawing ticked and unticked boxes.
  • 🤔 On 2024-10-08, the package amsmath was marked as partially-compatible at [tagging-status].
  • ✅ The article [automated-tagging] lists the PDF tagging code as "compatible with amsmath".
  • ✅ On 2024-10-08, the package amssymb was marked as compatible at [tagging-status].
• ✅ catchfile: A package that catches the contents of a file and puts it in a macro. It is used in the witiko/graphicx/http LaTeX theme, see Section 2.3.3.
  • ✅ On 2024-10-08, the package was marked as compatible at [tagging-status].
• ✅ grffile: A package that extends the name processing of the graphics package to support a larger range of file names in 2006 ≥ TeX Live ≤ 2019. Since TeX Live ≥ 2020, the functionality of the package has been integrated in the LaTeX 2ε kernel. It is used in the witiko/dot and witiko/graphicx/http LaTeX themes, see Section 2.3.3.
  • 🤔 On 2024-10-08, the package was not listed at [tagging-status].
  • ✅ The package is a stub that loads the package graphics, which is compatible, see above.
• ✅ etoolbox: A package that is used to polyfill the general hook management system in the default renderer prototypes for YAML metadata, see Section 3.3.4.8, and also in the default renderer prototype for identifier attributes.
  • ✅ On 2024-10-08, the package was marked as compatible at [tagging-status].
• ❌ soulutf8: A package that is used in the default renderer prototype for strike-throughs and marked text.
  • ❌ On 2024-10-08, the package was marked as incompatible at [tagging-status].
• ✅ ltxcmds: A package that is used to detect whether the minted and listings packages are loaded in the default renderer prototype for fenced code blocks.
  • ✅ On 2024-10-08, the package was marked as compatible at [tagging-status].
• ✅ verse: A package that is used in the default renderer prototypes for line blocks.
  • ✅ On 2024-10-08, the package was marked as compatible at [tagging-status].

Here are the actionable steps that we can take before the next release:

• ❌ paralist: A package that provides the compactitem, compactenum, and compactdesc macros for the typesetting of tight bulleted lists, ordered lists, and definition lists as well as the rendering of fancy lists.
  • ❌ On 2024-10-08, the package was marked as currently-incompatible at [tagging-status].

• 🤔 We should replace the package paralist with the package enumitem, as discussed in #466 (comment).
  • 🤔 On 2024-10-08, the package was marked as partially-incompatible at [tagging-status].
  • 🤔 I recall beamer having trouble displaying lists with the package enumitem, whereas it had no trouble with the package paralist.
  • ✅ The article [automated-tagging] lists the package enumitem as an "important package" whose "replacements or adaptations will be provided at a later stage".

• ❌ csvsimple: A package that provides the \csvautotabular macro for typesetting CSV files in the default renderer prototypes for iA Writer content blocks.
  • 🤔 On 2024-10-08, the package was marked as partially-compatible at [tagging-status].
  • ❌ On 2024-10-08, the dependent package tabularray was marked as incompatible at [tagging-status].

• ✅ We should remove the \csvautotabular macro from the default renderer prototypes for iA Writer content blocks.
• 🤔 I wonder if there are any suitable replacements compatible with PDF tagging that we might use instead?

• ❌ soulutf8: A package that is used in the default renderer prototype for strike-throughs and marked text.
  • ❌ On 2024-10-08, the package was marked as incompatible at [tagging-status].

• ✅ We have already scheduled the replacement of the package soulutf8 with soul in the code after we have ceased to support TeX Live ≤ 2022.
• 🤔 We should load the package soul instead of the package soulutf8 in TeX Live ≥ 2023.
  • ✅ On 2024-10-08, the package soul was suggested as a replacement for the package soulutf8 at [tagging-status].
  • ❌ On 2024-10-08, the package soul was marked as currently-incompatible at [tagging-status].

@u-fischer: I will appreciate your feedback regarding items marked with the thinking face emoji (🤔) as well as any information about the expected tagging support for the abovelisted packages. 🙏

[u-fischer]

• I just added a first support for fancyvrb to our develop branch. I also made a test with minted and it seemed to work (but I did not test all commands, mainly the environment).

• we are working on the enumitem replacement but I didn't look at paralist yet.

• regarding tabularray you should ask the maintainer, e.g. by commenting to this issue tagpdf support lvjr/tabularray#27. That is not something we can tag from the outside.

• there is not difference between soul and soututf8. We merged the packages. I haven't looked yet why it errors (probably some box nesting). With the (tagging-wise recommended) lualatex I wouldn't use it anyway but lua-ul instead.

[Witiko]

I just added a first support for fancyvrb to our develop branch. I also made a test with minted and it seemed to work (but I did not test all commands, mainly the environment).

That's good news! I won't worry about it for the moment, then.

we are working on the enumitem replacement but I didn't look at paralist yet.

When you say "replacement", do you mean external patches to the existing packages or new packages?

regarding tabularray you should ask the maintainer, e.g. by commenting to this issue lvjr/tabularray#27. That is not something we can tag from the outside.

It seems to me that the maintainer has made some progress. In lvjr/tabularray#27 (comment), you asked them for "places to inject tagging code at the begin and end of a cell and of a row", which they added in lvjr/tabularray#197 and demonstrated in lvjr/tabularray#27 (comment). I am not sure about the other two requirements from lvjr/tabularray#27 (comment) (ways to identify header rows, header columns, and lines).

I submitted a question to the maintainer in lvjr/tabularray#27 (comment).

there is not difference between soul and soututf8. We merged the packages. I haven't looked yet why it errors (probably some box nesting).

Then this should be a non-issue.

With the (tagging-wise recommended) lualatex I wouldn't use it anyway but lua-ul instead.

Thanks for the tip. We can use the package lua-ul with LuaLaTeX instead of the package soul. However, the package lua-ul is also marked as currently-incompatible at [tagging-status]; should that bother me?

[u-fischer]

lua-ul doesn't error. It only doesn't tag anything.

And regarind soul: Regardless how easy or complicated it is to tag it, someone has too look into the code and create test files etc. That costs time and it is currently not a high priority problem.

[Witiko]

@u-fischer: In lvjr/tabularray#27 (comment), @lvjr wrote:

@Witiko I always keep tabularray minimal and extendable. Therefore real tex4ht/lwarp/tagpdf code should and could be maintained by any other people. Code in trial folder will get them started. And they can make some pull requests if more hooks or other code adjustment is needed.

[Witiko]

• ✅ We should remove the \csvautotabular macro from the default renderer prototypes for iA Writer content blocks.
• 🤔 I wonder if there are any suitable replacements compatible with PDF tagging that we might use instead?

Let's table this for the moment (pun intended), since there is potential for the package tabularray moving in the direction of PDF tagging support as discussed in lvjr/tabularray#27 (comment).

[Witiko]

To wrap up, here are the changes that I am comfortable bringing to the October release of the Markdown package for TeX:

1. In LuaLaTeX, use the package lua-ul for strike-through/mark renderer prototypes instead of the package soul(utf8).
2. In pdfLaTeX, use the package soul instead of the package soulutf8 in TeX Live ≥ 2023.
3. Use the package enumitem for tight and fancy lists instead of the package paralist.

The last change is major. We should still keep definitions for the package paralist for users who explicitly load it and highlight the change in the release notes / changelog.

Thanks again, @u-fischer, and let's revisit this later.

[Witiko]

The changes from #512 still seem significant and while the Markdown package makes no guarantees about the default renderer prototype definitions, it seems beneficial to provide a controlled way for users to delay the update of these definitions without delaying the entire package update.

Therefore, we may want to only use these new definitions for the experimental version or when any document property with the prefix testphase/ is set. See also Section 4.3 of the document documentmetadata-support-code.pdf and the discussion with @TeXhackse on our Matrix.org server.

This way, we won't rock the boat as much for users who are more interested in stable defaults and less in PDF tagging. Furthermore, loading new definitions for any test phase also exposes (some) experimental definitions to more users, which means more user feedback, which is good.