Typesetting Pandoc JSON files directly with ConTeXt; an XML equivalent of JSON and native

[massifrg]

Hello,
I wrote a module for ConTeXt that lets you typeset Pandoc JSON files directly with ConTeXt.

The goal is to have all the information of a JSON file available in ConTeXt.

In particular, the module converts Pandoc JSON into an equivalent XML, something like this:

<Pandoc api-version="1,23,1">
<meta>
...
</meta>
<blocks>
<Para><Quoted quote-type="DoubleQuote">Hello, <Emph>world</Emph>!</Quoted></Para>
</blocks>
</Pandoc>

The conversion happens in memory, and it is pretty straightforward, because the tags have the same names of Pandoc items (see the project README).

The project has also a script to convert JSON files to XML files.

ConTeXt has great tools to typeset XML, that's why I chose it, instead of directly parsing JSON or markdown with Lua+LPEGs.

I also started writing a Haskell Writer for Pandoc for that XML format, but I'm a newbie in Haskell, so don't expect anything soon.

Also, before delving into that, I'd like to see if there's any interest in such a format, that would be a third form of Pandoc AST, beside native and JSON.

[jgm]

Well, it's much more readable than pandoc's JSON serialization, and unlike the native serialization, it's readable by standard tools. So I like the idea, personally. Drawback: adds complexity, possibly more things to change when the AST changes. We would need both a reader and a writer. If they could be implemented in a way that just piggybacked off the FromJSON / ToJSON instances for the types, then that would be ideal, because changing these instances would automatically affect both the JSON and XML serializations, and presumably the code could be quite short.

[jgm]

FromJSON/ToJSON use an intermediate Value structure.
https://hackage.haskell.org/package/aeson-2.2.3.0/docs/Data-Aeson.html#t:Value
So code that converts a Value to XML would suffice for the writer, and code that converts XML to a Value would suffice for the reader.

[jgm]

We use xml-conduit for XML parsing (and it also supports rendering).
https://hackage.haskell.org/package/xml-conduit-1.9.1.4/docs/Text-XML.html
So two way conversion between an xml-conduit Node and an aeson Value would probably be sufficient for this.

[massifrg]

Thank you for the feedback and the advice on how to do it.

Apart from processing Pandoc files with ConTeXt, my main concern is about long term storage and diff/merge.

XML has many tools to work with; it's not easy to diff/merge, but in that respect it's better than (prettified) Pandoc JSON, which is the format I'm currently using to store my files.

Recently I realized that markdown with pandoc extensions is good for those goals (at least for my documents), as long as you don't have complex tables.
My editor (pundok-editor) works with Pandoc AST, so a format (or a subset of it) that translates 1:1 with the AST is better, because open and save-again operations are guaranteed to be lossless.

[massifrg]

FromJSON/ToJSON use an intermediate Value structure. https://hackage.haskell.org/package/aeson-2.2.3.0/docs/Data-Aeson.html#t:Value So code that converts a Value to XML would suffice for the writer, and code that converts XML to a Value would suffice for the reader.

I understand your point, but after trying to write down some haskell code to make a "rote" conversion from JSON to XML, I'm finding it very difficult.
Most of it is due to my very limited knowledge of Haskell, but perhaps there's still a problem that is independent from that: data structures that have no explicit name, like list items, DefinitionList terms and defs, or colspecs in tables.

I'm afraid the resulting XML would be less meaningful; unnamed values would likely result in XML elements with generic names like "array", "list", "object", etc.
Unless you discriminate on the "t" field of the parent node, making it less automatic.
Discriminating on the "t" field is more like a traditional fromXML/toXML approach, and this would pull back the complexity you wanted to keep out.

[jgm]

Agreed, you'd need to look at the t field and come up with meaningful names. Going via Aeson Value might still reduce some complexity, though.