Change 'loose'/'tight' list parsing to be more like CommonMark

[gwern]

Initial discussion on pandoc-discussion.

Pandoc currently parses ordered/unordered lists with block elements in a surprising & inconsistent way, which differs from CommonMark, by not putting paragraph level items into paragraphs except for the final item (leading to subtle rendering problems in HTML). CommonMark puts all paragraph elements in paragraphs.

Simple example:

$ pandoc --version
pandoc 2.6
Compiled with pandoc-types 1.17.5.4, texmath 0.11.2, skylighting 0.7.5
Default user data directory: /home/gwern/.pandoc
Copyright (C) 2006-2019 John MacFarlane
Web:  http://pandoc.org
This is free software; see the source for copying conditions.
There is no warranty, not even for merchantability or fitness
for a particular purpose.

$ cat foo.page
1. Foo

    bar
2. foo

    bar
3. foo

    Bar
4. Foo

    Bar

$ pandoc foo.page
<ol type="1">
<li><p>Foo</p>
bar</li>
<li><p>foo</p>
bar</li>
<li><p>foo</p>
Bar</li>
<li><p>Foo</p>
<p>Bar</p></li>
</ol>

$ cmark foo.page 
<ol>
<li>
<p>Foo</p>
<p>bar</p>
</li>
<li>
<p>foo</p>
<p>bar</p>
</li>
<li>
<p>foo</p>
<p>Bar</p>
</li>
<li>
<p>Foo</p>
<p>Bar</p>
</li>
</ol>

Possibly related issues:

• When I wrote the list in consecutive elements of the first end of the list wrapped in p tags. #17
• Unexpected behaviour with two adjacent lists in markdown #2939
• html output treated as loose list item if blank line left #1292
• mix of compact and loose lists #479

[gwern]

Looks good, thanks!

[gwern]

Actually, on further examination, it appears this bug may not be quite fixed, although it's possible this is an entirely separate issue which should be in a different bug report.

It seems that if the final list (unordered or ordered) item contains a blockquote block item (but not merely an indented paragraph/continuation), the entire list gets switched back to the <p>-less version. We noticed that the fix wasn't working in my tea reviews list even though it was working in the test case, and I finally minimized it down to a 2-3 item list example:
Lists with final items using a blockquote:

- foo

    foo
- foo

    > foo


1. Foo

    bar
2. foo

    bar
4. Foo

    > Bar

yields the broken behavior:

<ul>
<li><p>foo</p>
foo</li>
<li><p>foo</p>
<blockquote>
<p>foo</p>
</blockquote></li>
</ul>
<ol type="1">
<li><p>Foo</p>
bar</li>
<li><p>foo</p>
bar</li>
<li><p>Foo</p>
<blockquote>
<p>Bar</p>
</blockquote></li>
</ol>

However, adding an additional paragraph after the blockquote, or adding an additional list item at all to the end, fixes it:

- foo

    foo
- foo

    > foo

    bar

---

- foo

    foo
- foo

    > foo
- bar

yields:

<ul>
<li><p>foo</p>
<p>foo</p></li>
<li><p>foo</p>
<blockquote>
<p>foo</p>
</blockquote>
<p>bar</p></li>
</ul>
<hr />
<ul>
<li><p>foo</p>
<p>foo</p></li>
<li><p>foo</p>
<blockquote>
<p>foo</p>
</blockquote></li>
<li><p>bar</p></li>
</ul>

Pandoc version:

$ ls -al /home/gwern/bin/bin/pandoc
-rwxr-xr-x 1 gwern gwern 102022976 Feb  9 09:27 /home/gwern/bin/bin/pandoc
$ pandoc --version
pandoc 2.6
Compiled with pandoc-types 1.17.5.4, texmath 0.11.2, skylighting 0.7.5