nbconvert 6.0.0 causing build failures due to TemplateNotFound error #1369

vEpiphyte · 2020-09-10T19:08:49Z

When executing nbconvert to convert a jupyter notebook into RST format with a custom template that extends rst.tpl, I receive an exception for jinja2.exceptions.TemplateNotFound: display_priority.j2. We are using a custom template to extend the rst.tpl file in order to surpress output blocks in the RST output which have been marked with the hide_code jupyter extension. Since we run notebooks as part of our unittests to ensure our documentation is up-to-date, this is causing build failures for our project (vertexproject/synapse). This can be seen in vertexproject/synapse#1876 and https://app.circleci.com/pipelines/github/vertexproject/synapse/4660/workflows/dcb1e54f-fc51-49fd-9134-e88ef80c5c7d/jobs/19312

This is the custom template file

{%- extends 'rst.tpl' -%}

{% block input_group -%}
{%- if cell.metadata.hideCode or nb.metadata.hide_input -%}
{%- else -%}
    {{ super() }}
{%- endif -%}
{% endblock input_group %}

{% block output_group -%}
{%- if cell.metadata.hideOutput -%}
{%- else -%}
    {{ super() }}
{%- endif -%}
{% endblock output_group %}

{#{% block output_area_prompt %}#}
{#{%- if cell.metadata.hide_input or nb.metadata.hide_input -%}#}
{#    <div class="prompt"> </div>#}
{#{%- else -%}#}
{#    {{ super() }}#}
{#{%- endif -%}#}
{#{% endblock output_area_prompt %}#}

We're not currently constraining the version of the nbconvert library installed, instead relying on the packages installed by installing the jupyter library.

I do not have a minimal test case example, but it is reproducible.

# Make a temporary directory to test with
cd /tmp
mkdir jtest
cd jtest
# Make and activate a venv (using pyenv in this example
pyenv virtualenv --copies 3.7.7 jtest377
pyenv shell jtest377
python -m pip install -U pip wheel
# Check out the affected project and install the build requirements
git clone https://github.com/vertexproject/synapse.git
cd synapse
python -m pip install -U -r requirements_doc.txt 
# Execute nbconvert directly. normally this is done in bulk with scripts/doctests.py
jupyter nbconvert --debug --execute --stdout --to rst --template /home/epiphyte/git/synapse/docs/vertex.tpl /home/epiphyte/git/synapse/docs/synapse/httpapi.ipynb

This eventually fails with the following error

[NbConvertApp] content: {'execution_state': 'idle'}                                                                              
[NbConvertApp] Skipping non-executing cell 4        
[NbConvertApp] Skipping non-executing cell 5                                                                       
[NbConvertApp] Applying preprocessor: coalesce_streams
[NbConvertApp] Applying preprocessor: HighlightMagicsPreprocessor                      
[NbConvertApp] Applying preprocessor: ExtractOutputPreprocessor
[NbConvertApp] Attempting to load template vertex.tpl                                                                                                                                                              
[NbConvertApp]     template_paths: /home/epiphyte/.local/share/jupyter/nbconvert/templates:/home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates:/usr/local/share/jupyter/nbconvert/templates:/
usr/share/jupyter/nbconvert/templates:/home/epiphyte/git/synapse/docs:/home/epiphyte/.local/share/jupyter:/home/epiphyte/.local/share/jupyter/nbconvert/templates:/home/epiphyte/.local/share/jupyter/nbconvert/tem
plates/compatibility:/home/epiphyte/.pyenv/versions/jtest377/share/jupyter:/home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates:/home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconver
t/templates/compatibility:/usr/local/share/jupyter:/usr/local/share/jupyter/nbconvert/templates:/usr/local/share/jupyter/nbconvert/templates/compatibility:/usr/share/jupyter:/usr/share/jupyter/nbconvert/template
s:/usr/share/jupyter/nbconvert/templates/compatibility      
[NbConvertApp] Template paths:
        /home/epiphyte/.local/share/jupyter/nbconvert/templates
        /home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates
        /usr/local/share/jupyter/nbconvert/templates
        /usr/share/jupyter/nbconvert/templates
        /home/epiphyte/git/synapse/docs
        /home/epiphyte/.local/share/jupyter
        /home/epiphyte/.local/share/jupyter/nbconvert/templates
        /home/epiphyte/.local/share/jupyter/nbconvert/templates/compatibility
        /home/epiphyte/.pyenv/versions/jtest377/share/jupyter
        /home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates
        /home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates/compatibility
        /usr/local/share/jupyter
        /usr/local/share/jupyter/nbconvert/templates
        /usr/local/share/jupyter/nbconvert/templates/compatibility
        /usr/share/jupyter
        /usr/share/jupyter/nbconvert/templates
        /usr/share/jupyter/nbconvert/templates/compatibility
Traceback (most recent call last):
  File "/home/epiphyte/.pyenv/versions/jtest377/bin/jupyter-nbconvert", line 8, in <module>
    sys.exit(main())
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/jupyter_core/application.py", line 270, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/traitlets/config/application.py", line 837, in launch_instance
    app.start()
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 345, in start
    self.convert_notebooks()
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 519, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 484, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 413, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 182, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 200, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/exporters/templateexporter.py", line 382, in from_notebook_node
    output = self.template.render(nb=nb_copy, resources=resources)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/home/epiphyte/git/synapse/docs/vertex.tpl", line 5, in top-level template code
    {%- extends 'rst.tpl' -%}
  File "/home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates/compatibility/rst.tpl", line 2, in top-level template code
    {%- extends 'rst/index.rst.j2' -%}
  File "/home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates/rst/index.rst.j2", line 1, in top-level template code
    {%- extends 'display_priority.j2' -%}
jinja2.exceptions.TemplateNotFound: display_priority.j2

Executing the conversion without the --template argument provided does work to convert the notebook, but includes the information I want to drop.

I did try modifying the index.rst.j2 file to load 'base/display_priotiy.j2 and that did successfully execute my builds, but I don't know if that change has other ramifications for nbconvert at a larger scale or not.

(jtest377) epiphyte@vertex05:/tmp/jtest/synapse$ head -n 2 /home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates/rst/index.rst.j2 
{%- extends 'base/display_priority.j2' -%}

This is using nbconvert 6.0.1. I have also confirmed that 6.0.0 also fails. 5.6.1 worked fine.

The following is a dump of libraries from python -m pip freeze the venv used above

aiohttp==3.6.2
alabaster==0.7.12
apipkg==1.5
argon2-cffi==20.1.0
async-generator==1.10
async-timeout==3.0.1
attrs==20.2.0
autopep8==1.5.4
Babel==2.8.0
backcall==0.2.0
bleach==3.1.5
bump2version==1.0.0
certifi==2020.6.20
cffi==1.14.2
chardet==3.0.4
coverage==5.2.1
cryptography==3.1
decorator==4.4.2
defusedxml==0.6.0
docutils==0.16
entrypoints==0.3
execnet==1.7.1
fastjsonschema==2.14.5
hide-code==0.5.2
idna==2.10
imagesize==1.2.0
importlib-metadata==1.7.0
ipykernel==5.3.4
ipython==7.18.1
ipython-genutils==0.2.0
ipywidgets==7.5.1
jedi==0.17.2
Jinja2==2.11.2
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.1.7
jupyter-console==6.2.0
jupyter-core==4.6.3
jupyterlab-pygments==0.1.1
lark-parser==0.9.0
lmdb==0.99
MarkupSafe==1.1.1
mistune==0.8.4
more-itertools==8.5.0
msgpack==0.6.2
multidict==4.7.6
nbclient==0.5.0
nbconvert==6.0.1
nbformat==5.0.7
nbstripout==0.3.9
nest-asyncio==1.4.0
notebook==6.1.4
packaging==20.4
pandocfilters==1.4.2
parso==0.7.1
pdfkit==0.6.1
pexpect==4.8.0
pickleshare==0.7.5
pluggy==0.13.1
prometheus-client==0.8.0
prompt-toolkit==3.0.7
ptyprocess==0.6.0
py==1.9.0
pycodestyle==2.6.0
pycparser==2.20
Pygments==2.6.1
pyOpenSSL==19.1.0
pyparsing==2.4.7
pyrsistent==0.16.0
pytest==5.4.3
pytest-cov==2.10.1
pytest-forked==1.3.0
pytest-xdist==1.34.0
python-dateutil==2.8.1
pytz==2020.1
PyYAML==5.3.1
pyzmq==19.0.2
qtconsole==4.7.7
QtPy==1.9.0
regex==2020.7.14
requests==2.24.0
Send2Trash==1.5.0
six==1.15.0
snowballstemmer==2.0.0
Sphinx==1.8.5
sphinx-rtd-theme==0.5.0
sphinxcontrib-serializinghtml==1.1.4
sphinxcontrib-websupport==1.2.4
terminado==0.8.3
testpath==0.4.4
toml==0.10.1
tornado==6.0.4
traitlets==5.0.4
typing-extensions==3.7.4.3
urllib3==1.25.10
wcwidth==0.2.5
webencodings==0.5.1
widgetsnbextension==3.5.1
xxhash==1.4.4
yarl==1.5.1
zipp==3.1.0

The text was updated successfully, but these errors were encountered:

* For service deletion, spin the del event storm commmand off in a schedcoro so it is out of line with the bottom half of the nexus function. This changes the del event semantics by executing a del handler without the presence of the supporting package, but breaks a nested nexus loop action. * Ensure that cull is passed in the coreQueueGets call, since it was missed * Temporarily pin nbconvert until jupyter/nbconvert#1369 is resolved.

SylvainCorlay · 2020-09-10T20:16:22Z

Hey @vEpiphyte thanks for reporting!

The way custom templates are done is one of the main changes in nbconvert 6.0, and the main reason for the major version bump indicating backward incompatibility. You can check out the documentation here: https://nbconvert.readthedocs.io/en/latest/customizing.html There is also a tutorial in the works on how to create more sophisticated templates.

anpaz · 2020-09-13T07:12:29Z

I'm seeing the same problem on a Mac.
After installing transitively nbconvert 6.0.1 from installing jupyter with:
pip install --user pytest jupyter
when I try to use jupyter nbconvert Notebook.ipynb --execute --stdout --to html
I get a ValueError: No template sub-directory with name 'lab' found in the following paths:
looking into the templates paths, like /usr/local/share/jupyter/nbconvert/templates/html/; they are empty. From the documentation seems to me there should be some directories in it with files for the templates.
How are the templates supposed to get there? Are they part of another package that is missing, or are they not getting copied during installation?

SylvainCorlay · 2020-09-13T08:22:40Z

@anpaz-msft thanks for reporting. Your issue is unrelated to the original one (which was expected behavior). Yours appear to be a legitimate bug with --user installes of nbconvert. I am closing this one and opening another related to user installs.

SylvainCorlay · 2020-09-13T08:32:35Z

@anpaz-msft actually, I am not reproducing your issue.

Could you please report the output to

which jupyter
jupyter --paths

and the content of your ~/.local/share/jupyter/nbconvert/templates directory?

vEpiphyte · 2020-09-16T17:37:52Z

@SylvainCorlay Thanks for following up. I look forward to seeing the tutorial about migrating old templates.

LuisVictoria · 2020-09-17T06:56:38Z

I'm seeing the same problem on a Mac.
After installing transitively nbconvert 6.0.1 from installing jupyter with:
pip install --user pytest jupyter
when I try to use jupyter nbconvert Notebook.ipynb --execute --stdout --to html
I get a ValueError: No template sub-directory with name 'lab' found in the following paths:
looking into the templates paths, like /usr/local/share/jupyter/nbconvert/templates/html/; they are empty. From the documentation seems to me there should be some directories in it with files for the templates.
How are the templates supposed to get there? Are they part of another package that is missing, or are they not getting copied during installation?

I had the same problem on Windows 10, and after reading @SylvainCorlay reply, I unistall nbconvert and then reinstalled it without the flag --user and it worked. Every template appers and I was able to convert my notebook.

I think @SylvainCorlay is correct, something goes wrong when using the --user.

My directory ...\share\jupyter\nbconvert\templateswas empty before my ''fix''.

MSeal · 2020-10-11T18:34:48Z

Tutorials, docs updates, and some fixes for a few issues related to this have all been resolved. I'm going to close this issue as I believe it's addressed

…nbconvert 6.0 (see jupyter/nbconvert#1369 )

jupyter/nbconvert#1369

…nbconvert 6.0 (see jupyter/nbconvert#1369 )

ArefAz · 2021-04-30T15:59:00Z

I faced the same issue using nbconvert version 6.0.7.

ValueError: No template sub-directory with name 'report' found in the following paths:

I used the command suggested in docs:

jupyter-nbconvert --to pdf --no-input --no-prompt --template report nb.ipynb

However, rolling back to version 5.6.1 fixed the error.

AerLiberum · 2021-08-25T04:39:03Z

However, rolling back to version 5.6.1 fixed the error.

The same on Windows 10 (without admin rights). 5.6.1 fixed the issue, while 6+ versions are still bagged.

vEpiphyte added a commit to vertexproject/synapse that referenced this issue Sep 10, 2020

Temporarily pin nbconvert until jupyter/nbconvert#1369 is resolved.

8216e17

tjcrone mentioned this issue Sep 11, 2020

CI Fail earth-env-data-science/earth-env-data-science-book#13

Closed

txoof mentioned this issue Sep 28, 2020

nbconvert 6.0+ handles templates in a new way txoof/develtools#1

Open

IanMatthewHuff mentioned this issue Sep 30, 2020

NB Convert 6.0 support for export microsoft/vscode-python#14177

Merged

10 tasks

MSeal closed this as completed Oct 11, 2020

adundovi added a commit to adundovi/CRPropa3 that referenced this issue Oct 13, 2020

Trying to fix make doc by forcing nbconvert==5.6.1 due to changes in …

605bb2c

…nbconvert 6.0 (see jupyter/nbconvert#1369 )

adundovi mentioned this issue Oct 13, 2020

Trying to fix make doc by forcing nbconvert==5.6.1 due to changes in … CRPropa/CRPropa3#314

Merged

Tam-Pham added a commit to neuropsychology/NeuroKit that referenced this issue Oct 14, 2020

Force nbconvert==5.6.1

c810012

jupyter/nbconvert#1369

ageorgou mentioned this issue Oct 16, 2020

Build breaks for newest nbconvert UCL/rsd-engineeringcourse#186

Closed

tfitoussi pushed a commit to tfitoussi/CRPropa3 that referenced this issue Oct 24, 2020

Trying to fix make doc by forcing nbconvert==5.6.1 due to changes in …

24e7fc6

…nbconvert 6.0 (see jupyter/nbconvert#1369 )

ttimbers mentioned this issue Jan 18, 2021

use nbconvert 5.6.1 until nbgrader templates are updated UBC-DSCI/docker-stacks#7

Open

banderlog mentioned this issue Feb 24, 2021

Used nbconverter template is deprecated now and throws an error aaren/notedown#95

Open

tillahoffmann mentioned this issue Sep 4, 2021

No run button tillahoffmann/obsidian-jupyter#4

Closed

kirienko mentioned this issue Sep 9, 2021

tidy up pelican call oscovida/oscovida#257

Closed

IanMatthewHuff mentioned this issue Apr 28, 2022

Export for HTML and PDF fails microsoft/vscode-jupyter#9816

Closed

astonzhang mentioned this issue Aug 3, 2022

Unexpected error while saving file: xxx display_priority.j2 d2l-ai/d2l-book#59

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nbconvert 6.0.0 causing build failures due to TemplateNotFound error #1369

nbconvert 6.0.0 causing build failures due to TemplateNotFound error #1369

vEpiphyte commented Sep 10, 2020

SylvainCorlay commented Sep 10, 2020

anpaz commented Sep 13, 2020

SylvainCorlay commented Sep 13, 2020

SylvainCorlay commented Sep 13, 2020 •

edited

Loading

vEpiphyte commented Sep 16, 2020

LuisVictoria commented Sep 17, 2020

MSeal commented Oct 11, 2020

ArefAz commented Apr 30, 2021

AerLiberum commented Aug 25, 2021

nbconvert 6.0.0 causing build failures due to TemplateNotFound error #1369

nbconvert 6.0.0 causing build failures due to TemplateNotFound error #1369

Comments

vEpiphyte commented Sep 10, 2020

SylvainCorlay commented Sep 10, 2020

anpaz commented Sep 13, 2020

SylvainCorlay commented Sep 13, 2020

SylvainCorlay commented Sep 13, 2020 • edited Loading

vEpiphyte commented Sep 16, 2020

LuisVictoria commented Sep 17, 2020

MSeal commented Oct 11, 2020

ArefAz commented Apr 30, 2021

AerLiberum commented Aug 25, 2021

SylvainCorlay commented Sep 13, 2020 •

edited

Loading