Bump pandoc upper bound to 2.5

[tk3369]

Let's keep it up to date unless there's any breaking issue.

Reference: https://stackoverflow.com/questions/53966433/jupyter-nbconvert-cannot-work-with-latest-version-of-pandoc/53967931#53967931

[mgeier]

Related: #928

[choldgraf]

the error in travis on 3.4 ends with this:

  s.bind("tcp://%s:%i" % (self.ip, port))
  File "zmq/backend/cython/socket.pyx", line 547, in zmq.backend.cython.socket.Socket.bind
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc
zmq.error.ZMQError: Address already in use
==================== 1 failed, 253 passed in 150.85 seconds ====================
The command "py.test --cov nbconvert -v --pyargs nbconvert" exited with 1.

perhaps that's unrelated to the pandoc bump? In which case somebody could try restarting to see if that fixes it. I think a good way to answer the question "should newer versions of pandoc be supported" is to see if the tests all pass w/ newer pandoc

[choldgraf]

I know that @ivanov has been looking into pandoc/notebook/etc stuff, maybe he has experience w/ pandoc 2.X and knows if there would be compatibility issues

[tk3369]

The ZMQ error feels like a system issue rather than a code problem... Can anyone trigger the build again?

The nightly also failed but it seems to have a more tangible reason:

/home/travis/virtualenv/python3.7-dev/lib/python3.7/site-packages/nbconvert/preprocessors/tests/test_execute.py:83: in assert_notebooks_equal
    self.assertEqual(normalized_expected_outputs, normalized_actual_outputs)
E   AssertionError: Lists differ: [{'data': {'text/plain': "'Code 1'"}, 'exe[46 chars]lt'}] != []
E   
E   First list contains 1 additional elements.
E   First extra element 0:
E   {'data': {'text/plain': "'Code 1'"}, 'execution_count': 1, 'output_type': 'execute_result'}
E   
E   + []
E   - [{'data': {'text/plain': "'Code 1'"},
E   -   'execution_count': 1,
E   -   'output_type': 'execute_result'}]

[takluyver]

FWIW, you can usually rerun tests by closing and reopening the PR. At the moment, though, there's an unrelated problem with incompatible testing packages, so it will fail if we re-run it now. I'm trying to fix that in #941.

The most fragile bit for pandoc versions is this filter which converts links within a notebook to Latex references. It's used here:

nbconvert/nbconvert/templates/latex/report.tplx

Line 25 in 8b47cf1

((( cell.source | citation2latex | strip_files_prefix | convert_pandoc('markdown+tex_math_double_backslash', 'json',extra_args=[]) | resolve_references | convert_pandoc('json','latex', extra_args=["--top-level-division=chapter"]) )))

That converts the markdown to pandoc's JSON document format, applies the link converter to it, and converts the updated JSON to Latex. So it depends on the structure of pandoc's document format.

So before we expand the constraint, please double check that the latest version of Latex pandoc you want to be supported still performs that conversion correctly.

[chrisjsewell]

Yes restricting the pandoc version is particularly a pain if you are using conda, which currently prohibits installing the latest versions of nbconvert and pandoc together.

Personally I would suggest you swap out pandocfilters for panflute, which is more 'pythonic' and will deal with changes in the pandoc AST.

Somemthing along the lines of:

import io
import panflute as pf

def resolve_references(source):
   input_stream = io.StringIO(source)
   with io.StringIO() as output_stream:
       pf.run_filter(resolve_one_reference, 
                          input_stream=input_stream, 
                          output_stream=outut_stream)
       return output_stream..getvalue()

def resolve_one_reference(link, doc):
    # type: (Link, Doc) -> Element
    """extract links that point to internal items, e.g. [text](#label)"""
    if not isinstance(link, pf.Link):
        return None
    match = re.match(r'#(.+)$', link.url)
    if not match:
        return None
    # pandoc automatically makes labels for headings.
    label = m.group(1).lower()
    label = re.sub(r'[^\w-]+', '', label) # Strip HTML entities
    return pf.RawInline(r'Section \ref{%s}' % label, format="tex")

You could also basically replace nbconvert/nbconvert/utils/pandoc.py with panflute functions

[choldgraf]

Just a note that another reason to bump pandoc at some point might be to support 2.6, which supports ipynb files and might make it possible to remove some code in nbconvert after a few depreciation cycles