Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nbconvert 6.0.0 causing build failures due to TemplateNotFound error #1369

Closed
vEpiphyte opened this issue Sep 10, 2020 · 9 comments
Closed

Comments

@vEpiphyte
Copy link

When executing nbconvert to convert a jupyter notebook into RST format with a custom template that extends rst.tpl, I receive an exception for jinja2.exceptions.TemplateNotFound: display_priority.j2. We are using a custom template to extend the rst.tpl file in order to surpress output blocks in the RST output which have been marked with the hide_code jupyter extension. Since we run notebooks as part of our unittests to ensure our documentation is up-to-date, this is causing build failures for our project (vertexproject/synapse). This can be seen in vertexproject/synapse#1876 and https://app.circleci.com/pipelines/github/vertexproject/synapse/4660/workflows/dcb1e54f-fc51-49fd-9134-e88ef80c5c7d/jobs/19312

This is the custom template file

{%- extends 'rst.tpl' -%}

{% block input_group -%}
{%- if cell.metadata.hideCode or nb.metadata.hide_input -%}
{%- else -%}
    {{ super() }}
{%- endif -%}
{% endblock input_group %}

{% block output_group -%}
{%- if cell.metadata.hideOutput -%}
{%- else -%}
    {{ super() }}
{%- endif -%}
{% endblock output_group %}

{#{% block output_area_prompt %}#}
{#{%- if cell.metadata.hide_input or nb.metadata.hide_input -%}#}
{#    <div class="prompt"> </div>#}
{#{%- else -%}#}
{#    {{ super() }}#}
{#{%- endif -%}#}
{#{% endblock output_area_prompt %}#}

We're not currently constraining the version of the nbconvert library installed, instead relying on the packages installed by installing the jupyter library.

I do not have a minimal test case example, but it is reproducible.

# Make a temporary directory to test with
cd /tmp
mkdir jtest
cd jtest
# Make and activate a venv (using pyenv in this example
pyenv virtualenv --copies 3.7.7 jtest377
pyenv shell jtest377
python -m pip install -U pip wheel
# Check out the affected project and install the build requirements
git clone https://github.com/vertexproject/synapse.git
cd synapse
python -m pip install -U -r requirements_doc.txt 
# Execute nbconvert directly. normally this is done in bulk with scripts/doctests.py
jupyter nbconvert --debug --execute --stdout --to rst --template /home/epiphyte/git/synapse/docs/vertex.tpl /home/epiphyte/git/synapse/docs/synapse/httpapi.ipynb

This eventually fails with the following error

[NbConvertApp] content: {'execution_state': 'idle'}                                                                              
[NbConvertApp] Skipping non-executing cell 4        
[NbConvertApp] Skipping non-executing cell 5                                                                       
[NbConvertApp] Applying preprocessor: coalesce_streams
[NbConvertApp] Applying preprocessor: HighlightMagicsPreprocessor                      
[NbConvertApp] Applying preprocessor: ExtractOutputPreprocessor
[NbConvertApp] Attempting to load template vertex.tpl                                                                                                                                                              
[NbConvertApp]     template_paths: /home/epiphyte/.local/share/jupyter/nbconvert/templates:/home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates:/usr/local/share/jupyter/nbconvert/templates:/
usr/share/jupyter/nbconvert/templates:/home/epiphyte/git/synapse/docs:/home/epiphyte/.local/share/jupyter:/home/epiphyte/.local/share/jupyter/nbconvert/templates:/home/epiphyte/.local/share/jupyter/nbconvert/tem
plates/compatibility:/home/epiphyte/.pyenv/versions/jtest377/share/jupyter:/home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates:/home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconver
t/templates/compatibility:/usr/local/share/jupyter:/usr/local/share/jupyter/nbconvert/templates:/usr/local/share/jupyter/nbconvert/templates/compatibility:/usr/share/jupyter:/usr/share/jupyter/nbconvert/template
s:/usr/share/jupyter/nbconvert/templates/compatibility      
[NbConvertApp] Template paths:
        /home/epiphyte/.local/share/jupyter/nbconvert/templates
        /home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates
        /usr/local/share/jupyter/nbconvert/templates
        /usr/share/jupyter/nbconvert/templates
        /home/epiphyte/git/synapse/docs
        /home/epiphyte/.local/share/jupyter
        /home/epiphyte/.local/share/jupyter/nbconvert/templates
        /home/epiphyte/.local/share/jupyter/nbconvert/templates/compatibility
        /home/epiphyte/.pyenv/versions/jtest377/share/jupyter
        /home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates
        /home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates/compatibility
        /usr/local/share/jupyter
        /usr/local/share/jupyter/nbconvert/templates
        /usr/local/share/jupyter/nbconvert/templates/compatibility
        /usr/share/jupyter
        /usr/share/jupyter/nbconvert/templates
        /usr/share/jupyter/nbconvert/templates/compatibility
Traceback (most recent call last):
  File "/home/epiphyte/.pyenv/versions/jtest377/bin/jupyter-nbconvert", line 8, in <module>
    sys.exit(main())
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/jupyter_core/application.py", line 270, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/traitlets/config/application.py", line 837, in launch_instance
    app.start()
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 345, in start
    self.convert_notebooks()
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 519, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 484, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 413, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 182, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 200, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/nbconvert/exporters/templateexporter.py", line 382, in from_notebook_node
    output = self.template.render(nb=nb_copy, resources=resources)
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/home/epiphyte/.pyenv/versions/jtest377/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/home/epiphyte/git/synapse/docs/vertex.tpl", line 5, in top-level template code
    {%- extends 'rst.tpl' -%}
  File "/home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates/compatibility/rst.tpl", line 2, in top-level template code
    {%- extends 'rst/index.rst.j2' -%}
  File "/home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates/rst/index.rst.j2", line 1, in top-level template code
    {%- extends 'display_priority.j2' -%}
jinja2.exceptions.TemplateNotFound: display_priority.j2

Executing the conversion without the --template argument provided does work to convert the notebook, but includes the information I want to drop.

I did try modifying the index.rst.j2 file to load 'base/display_priotiy.j2 and that did successfully execute my builds, but I don't know if that change has other ramifications for nbconvert at a larger scale or not.

(jtest377) epiphyte@vertex05:/tmp/jtest/synapse$ head -n 2 /home/epiphyte/.pyenv/versions/jtest377/share/jupyter/nbconvert/templates/rst/index.rst.j2 
{%- extends 'base/display_priority.j2' -%}

This is using nbconvert 6.0.1. I have also confirmed that 6.0.0 also fails. 5.6.1 worked fine.

The following is a dump of libraries from python -m pip freeze the venv used above

aiohttp==3.6.2
alabaster==0.7.12
apipkg==1.5
argon2-cffi==20.1.0
async-generator==1.10
async-timeout==3.0.1
attrs==20.2.0
autopep8==1.5.4
Babel==2.8.0
backcall==0.2.0
bleach==3.1.5
bump2version==1.0.0
certifi==2020.6.20
cffi==1.14.2
chardet==3.0.4
coverage==5.2.1
cryptography==3.1
decorator==4.4.2
defusedxml==0.6.0
docutils==0.16
entrypoints==0.3
execnet==1.7.1
fastjsonschema==2.14.5
hide-code==0.5.2
idna==2.10
imagesize==1.2.0
importlib-metadata==1.7.0
ipykernel==5.3.4
ipython==7.18.1
ipython-genutils==0.2.0
ipywidgets==7.5.1
jedi==0.17.2
Jinja2==2.11.2
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.1.7
jupyter-console==6.2.0
jupyter-core==4.6.3
jupyterlab-pygments==0.1.1
lark-parser==0.9.0
lmdb==0.99
MarkupSafe==1.1.1
mistune==0.8.4
more-itertools==8.5.0
msgpack==0.6.2
multidict==4.7.6
nbclient==0.5.0
nbconvert==6.0.1
nbformat==5.0.7
nbstripout==0.3.9
nest-asyncio==1.4.0
notebook==6.1.4
packaging==20.4
pandocfilters==1.4.2
parso==0.7.1
pdfkit==0.6.1
pexpect==4.8.0
pickleshare==0.7.5
pluggy==0.13.1
prometheus-client==0.8.0
prompt-toolkit==3.0.7
ptyprocess==0.6.0
py==1.9.0
pycodestyle==2.6.0
pycparser==2.20
Pygments==2.6.1
pyOpenSSL==19.1.0
pyparsing==2.4.7
pyrsistent==0.16.0
pytest==5.4.3
pytest-cov==2.10.1
pytest-forked==1.3.0
pytest-xdist==1.34.0
python-dateutil==2.8.1
pytz==2020.1
PyYAML==5.3.1
pyzmq==19.0.2
qtconsole==4.7.7
QtPy==1.9.0
regex==2020.7.14
requests==2.24.0
Send2Trash==1.5.0
six==1.15.0
snowballstemmer==2.0.0
Sphinx==1.8.5
sphinx-rtd-theme==0.5.0
sphinxcontrib-serializinghtml==1.1.4
sphinxcontrib-websupport==1.2.4
terminado==0.8.3
testpath==0.4.4
toml==0.10.1
tornado==6.0.4
traitlets==5.0.4
typing-extensions==3.7.4.3
urllib3==1.25.10
wcwidth==0.2.5
webencodings==0.5.1
widgetsnbextension==3.5.1
xxhash==1.4.4
yarl==1.5.1
zipp==3.1.0
vEpiphyte added a commit to vertexproject/synapse that referenced this issue Sep 10, 2020
vEpiphyte added a commit to vertexproject/synapse that referenced this issue Sep 10, 2020
* For service deletion, spin the del event storm commmand off in a schedcoro so it is out of line with the bottom half of the nexus function. This changes the del event semantics by executing a del handler without the presence of the supporting package, but breaks a nested nexus loop action.

* Ensure that cull is passed in the coreQueueGets call, since it was missed

* Temporarily pin nbconvert until jupyter/nbconvert#1369 is resolved.
@SylvainCorlay
Copy link
Member

Hey @vEpiphyte thanks for reporting!

The way custom templates are done is one of the main changes in nbconvert 6.0, and the main reason for the major version bump indicating backward incompatibility. You can check out the documentation here: https://nbconvert.readthedocs.io/en/latest/customizing.html There is also a tutorial in the works on how to create more sophisticated templates.

@anpaz
Copy link

anpaz commented Sep 13, 2020

I'm seeing the same problem on a Mac.
After installing transitively nbconvert 6.0.1 from installing jupyter with:
pip install --user pytest jupyter
when I try to use jupyter nbconvert Notebook.ipynb --execute --stdout --to html
I get a ValueError: No template sub-directory with name 'lab' found in the following paths:
looking into the templates paths, like /usr/local/share/jupyter/nbconvert/templates/html/; they are empty. From the documentation seems to me there should be some directories in it with files for the templates.
How are the templates supposed to get there? Are they part of another package that is missing, or are they not getting copied during installation?

@SylvainCorlay
Copy link
Member

@anpaz-msft thanks for reporting. Your issue is unrelated to the original one (which was expected behavior). Yours appear to be a legitimate bug with --user installes of nbconvert. I am closing this one and opening another related to user installs.

@SylvainCorlay
Copy link
Member

SylvainCorlay commented Sep 13, 2020

@anpaz-msft actually, I am not reproducing your issue.

Could you please report the output to

which jupyter
jupyter --paths

and the content of your ~/.local/share/jupyter/nbconvert/templates directory?

@vEpiphyte
Copy link
Author

@SylvainCorlay Thanks for following up. I look forward to seeing the tutorial about migrating old templates.

@LuisVictoria
Copy link

I'm seeing the same problem on a Mac.
After installing transitively nbconvert 6.0.1 from installing jupyter with:
pip install --user pytest jupyter
when I try to use jupyter nbconvert Notebook.ipynb --execute --stdout --to html
I get a ValueError: No template sub-directory with name 'lab' found in the following paths:
looking into the templates paths, like /usr/local/share/jupyter/nbconvert/templates/html/; they are empty. From the documentation seems to me there should be some directories in it with files for the templates.
How are the templates supposed to get there? Are they part of another package that is missing, or are they not getting copied during installation?

I had the same problem on Windows 10, and after reading @SylvainCorlay reply, I unistall nbconvert and then reinstalled it without the flag --user and it worked. Every template appers and I was able to convert my notebook.

I think @SylvainCorlay is correct, something goes wrong when using the --user.

My directory ...\share\jupyter\nbconvert\templateswas empty before my ''fix''.

@MSeal
Copy link
Contributor

MSeal commented Oct 11, 2020

Tutorials, docs updates, and some fixes for a few issues related to this have all been resolved. I'm going to close this issue as I believe it's addressed

@ArefAz
Copy link

ArefAz commented Apr 30, 2021

I faced the same issue using nbconvert version 6.0.7.

ValueError: No template sub-directory with name 'report' found in the following paths:

I used the command suggested in docs:

jupyter-nbconvert --to pdf --no-input --no-prompt --template report nb.ipynb

However, rolling back to version 5.6.1 fixed the error.

@AerLiberum
Copy link

However, rolling back to version 5.6.1 fixed the error.

The same on Windows 10 (without admin rights). 5.6.1 fixed the issue, while 6+ versions are still bagged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants