TemplateNotFound error when extending built-in template

[milliams]

I have the following code:

import nbformat
from nbconvert import HTMLExporter

html_exporter = HTMLExporter(template_file="foo.tpl")
with open("foo.ipynb") as infile:
    nb = nbformat.read(infile, as_version=4)
    html = html_exporter.from_notebook_node(nb)
    print(html)

and a template (which does nothing but still fails if it does do something) in the cwd, foo.tpl:

{% extends 'lab/index.html.j2' %}

I get the error (with the venv path replaced with <venv> for brevity):

Traceback (most recent call last):
  File "convert_test/__init__.py", line 7, in <module>
    html = html_exporter.from_notebook_node(nb)
  File "<venv>/lib64/python3.8/site-packages/nbconvert/exporters/html.py", line 122, in from_notebook_node
    return super().from_notebook_node(nb, resources, **kw)
  File "<venv>/lib64/python3.8/site-packages/nbconvert/exporters/templateexporter.py", line 382, in from_notebook_node
    output = self.template.render(nb=nb_copy, resources=resources)
  File "<venv>/lib64/python3.8/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "<venv>/lib64/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "<venv>/lib64/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/home/matt/temp/convert_test/foo.tpl", line 1, in top-level template code
    {% extends 'lab/index.html.j2'%}
  File "<venv>/share/jupyter/nbconvert/templates/lab/index.html.j2", line 1, in top-level template code
    {%- extends 'base.html.j2' -%}
jinja2.exceptions.TemplateNotFound: base.html.j2

It seems to be finding my foo.tpl file correctly, seeing the extends 'lab/index.html.j2' line and then correctly finding <venv>/share/jupyter/nbconvert/templates/lab/index.html.j2. This file, inside the install directory says it extends 'base.html.j2' but nbconvert fails to find that file.

If I do ls <venv>/share/jupyter/nbconvert/templates/lab/ then I see:

static  base.html.j2  conf.json  index.html.j2

so base.html.j2 is sitting there right next to index.html.j2.

This process works fine with nbconvert 5. I realise that templates have changed and I read https://nbconvert.readthedocs.io/en/latest/customizing.html but it doesn't seem to tell me how to create a custom template, only how the built-in templates are organised.

[desilinguist]

I am having the exact same issue with the following API code in my tool.

report_config = Config({'ExecutePreprocessor': {'enabled': True,
                                                'timeout': 3600},
                        'HTMLExporter': {'template_paths': [template_path],
                                         'template_file': 'report.tpl'}})

exportHtml = HTMLExporter(config=report_config)
output, _ = exportHtml.from_filename(notebook_file)

template_path is where my custom template lives and looks like this:

{%- extends "full.tpl" -%}

{% block input %}
{%- endblock input %}
{% block in_prompt %}
{%- endblock in_prompt %}

It used to work just perfectly with the previous versions of nbconvert but since 6.0.2 came out, I am seeing the following error:

Executing notebook with kernel: python3
Traceback (most recent call last):
  File "/Users/nmadnani/anaconda/envs/rsmdev/bin/rsmtool", line 33, in <module>
    sys.exit(load_entry_point('rsmtool', 'console_scripts', 'rsmtool')())
  File "/Users/nmadnani/work/rsmtool/rsmtool/rsmtool.py", line 374, in main
    overwrite_output=args.force_write)
  File "/Users/nmadnani/work/rsmtool/rsmtool/rsmtool.py", line 321, in run_experiment
    figdir)
  File "/Users/nmadnani/work/rsmtool/rsmtool/reporter.py", line 688, in create_report
    join(reportdir, '{}.html'.format(report_name)))
  File "/Users/nmadnani/work/rsmtool/rsmtool/reporter.py", line 327, in convert_ipynb_to_html
    output, _ = exportHtml.from_filename(notebook_file)
  File "/Users/nmadnani/anaconda/envs/rsmdev/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 182, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/Users/nmadnani/anaconda/envs/rsmdev/lib/python3.6/site-packages/nbconvert/exporters/exporter.py", line 200, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/Users/nmadnani/anaconda/envs/rsmdev/lib/python3.6/site-packages/nbconvert/exporters/html.py", line 122, in from_notebook_node
    return super().from_notebook_node(nb, resources, **kw)
  File "/Users/nmadnani/anaconda/envs/rsmdev/lib/python3.6/site-packages/nbconvert/exporters/templateexporter.py", line 382, in from_notebook_node
    output = self.template.render(nb=nb_copy, resources=resources)
  File "/Users/nmadnani/anaconda/envs/rsmdev/lib/python3.6/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/Users/nmadnani/anaconda/envs/rsmdev/lib/python3.6/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/Users/nmadnani/anaconda/envs/rsmdev/lib/python3.6/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "/Users/nmadnani/work/rsmtool/rsmtool/notebooks/templates/report.tpl", line 1, in top-level template code
    {%- extends "full.tpl" -%}
jinja2.exceptions.TemplateNotFound: full.tpl

I also tried using {%- extends 'compatibility/full.tpl' -%} but that has the same issue.

[MSeal]

I see the issue. It's a bug in how the templates are referring to local files when being imported from a different template directory. @maartenbreddels @SylvainCorlay could use your eyes on this to brainstorm how we might remedy the transitive pathing issue here. For compatibility templates in #1387 I added the extra template paths explicitly to handle this but seeing these reports of issues I think that might be the wrong approach to that aspect of the problem.

[Sieboldianus]

In the meantime, until this is resolved, is there a possible workaround? I have the same issue, using a custom template that extends full.tpl.

  File "/opt/conda/envs/worker_env/share/jupyter/nbconvert/templates/compatibility/full.tpl", line 2, in top-level template code
    {%- extends 'lab/index.html.j2' -%}
  File "/opt/conda/envs/worker_env/share/jupyter/nbconvert/templates/lab/index.html.j2", line 1, in top-level template code
    {%- extends 'base.html.j2' -%}
jinja2.exceptions.TemplateNotFound: base.html.j2

I tried to copy all files from https://github.com/jupyter/nbconvert/tree/master/share/jupyter/nbconvert/templates/base to the top folder. This fixes all *.tpl issues, but I don't know how to proceed with this:

File "/opt/conda/envs/worker_env/lib/python3.8/site-packages/nbconvert/exporters/html.py", line 127, in resources_include_css
    code = """<style type="text/css">\n%s</style>""" % (env.loader.get_source(env, name)[0])
  File "/opt/conda/envs/worker_env/lib/python3.8/site-packages/jinja2/loaders.py", line 420, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: static/index.css

Found the css files: share/jupyter/nbconvert/templates/lab/static

If these are also present locally (from where nbconvert is run), then everything works. However, all code-cells have no syntax highlighted in the output HTML.

I finally used the workaround conda install -c conda-forge nbconvert=5.6.1.

[MSeal]

Until I get the PR in to fix transitive dependency pathing, you can add share/jupyter/nbconvert/templates/lab to the template_paths attribute of the TemplateExporter instance, but that's not exposed on the command line. I'll put effort into getting a resolved path on this for this week.

[MSeal]

This overall issue should now be resolved for backwards compatibility in 6.0.7 if you additionally set the template_name to be the base template you are using (lab I think in the cases above). Ideally though if you are using a 6.0 template you can follow https://nbconvert.readthedocs.io/en/latest/customizing.html#conf-json and set the base template name there, which will fix any template inheritance for importing those base template files as local files in jinja.

[jakubsatka]

I was able to help myself by reading this thread. Basically, I copied static folder content to where nbconvert was missing it.

[Sieboldianus]

This overall issue should now be resolved for backwards compatibility in 6.0.7 if you additionally set the template_name to be the base template you are using (lab I think in the cases above). Ideally though if you are using a 6.0 template you can follow https://nbconvert.readthedocs.io/en/latest/customizing.html#conf-json and set the base template name there, which will fix any template inheritance for importing those base template files as local files in jinja.

Many thanks, I can confirm this works now in 'nbconvert=6.5.*', installed from conda-forge, but as you said, I had to add a conf.json, otherwise I would still get jinja2.exceptions.TemplateNotFound: base.html.j2.

Here are two tested sample files.

conf.json

{
  "base_template": "classic",
  "mimetypes": {
    "text/html": true
  },
  "preprocessors": {
    "100-pygments": {
        "type": "nbconvert.preprocessors.CSSHTMLHeaderPreprocessor",
        "enabled": true
    },
    "500-reveal": {
      "type": "nbconvert.exporters.slides._RevealMetadataPreprocessor",
      "enabled": true
    }
  }
}

nbconvert.tpl

{%- extends 'classic/index.html.j2' -%}

{# This template will render cells with tags highlight,
highlight_red and hide_code differently 
- also fixes summary-details arrow not showing in exported HTML
- created from files in:
    https://github.com/ipython-contrib/jupyter_contrib_nbextensions/tree/master/src/jupyter_contrib_nbextensions/templates
    https://github.com/ipython-contrib/jupyter_contrib_nbextensions/tree/master/src/jupyter_contrib_nbextensions/nbextensions/toc2
#}

{% block header %}
    {{ super() }}
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/jqueryui/1.12.1/jquery-ui.css">
    
    <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/jqueryui/1.9.1/jquery-ui.min.js"></script>
    
    <link rel="stylesheet" type="text/css" href="https://ad.vgiscience.org/cdn/toc2/main.css">
    
    <link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">
    
    <script src="https://ad.vgiscience.org/cdn/toc2/toc2.js"></script>
    
    <script>
    $( document ).ready(function(){
            var cfg = {{ nb.get('metadata', {}).get('toc', {})|tojson|safe }};
            cfg.navigate_menu=false;
            // fire the main function with these parameters
            require(['nbextensions/toc2/toc2'], function (toc2) {
                toc2.table_of_contents(cfg);
            });
    });
    </script>
    <style type="text/css">
    /* Fix details summary arrow
       not shown in Firefox
       due to bootstrap
       display: block;
     */
    summary {
        display: list-item;
        outline: none;
    }
    </style>
{% endblock header%}

{% block input_group -%}
{%- if cell.metadata.hide_input or nb.metadata.hide_input -%}
{%- else -%}
    {% if 'highlight_red' in cell['metadata'].get('tags', []) %}
        <div style="background-color:#FFF0F2">
            {{ super() }}
        </div>
    {% elif 'highlight' in cell['metadata'].get('tags', []) %}
        <div style="background-color:#E0F0F5">
            {{ super() }}
        </div>
    {% else %}
        {% if 'hide_code' in cell['metadata'].get('tags', []) %}
            <div style="padding-left: 40px; font-size: 20px;">•••</div>
        {% else %}
            {{ super() }}
        {% endif %}
    {% endif %}
{%- endif -%}
{% endblock input_group %}
    
{% block output_group -%}
{%- if cell.metadata.hide_output -%}
{%- else -%}
    {{ super() }}
{%- endif -%}
{% endblock output_group %}

{% block output_area_prompt %}
{%- if cell.metadata.hide_input or nb.metadata.hide_input -%}
    <div class="prompt"> </div>
{%- else -%}
    {{ super() }}
{%- endif -%}
{% endblock output_area_prompt %}

{%- block html_head -%}

{{ super() }}

{% for css in resources.inliner.css -%}
    <style type="text/css">
    {{ css }}
    </style>
{% endfor %}

{% for js in resources.inliner.js -%}
    <script type="text/javascript">
    {{ js }}
    </script>
{% endfor %}

{%- endblock html_head -%}

Rendered with:

jupyter nbconvert --to html_toc     --output-dir=./html ./notebook.ipynb     --template=./nbconvert.tpl

[msat59]

Hasn't fixed the issue yet?
Still having issues with nbconvert==7.3.1 to run jupyter nbconvert notebook.ipynb --to html

I tried to downgrade to 5.6.1 but got these warnings from pip:

jupyter-server 2.4.0 requires nbconvert>=6.4.4, but you have nbconvert 5.6.1 which is incompatible.
jupyter-contrib-nbextensions 0.7.0 requires nbconvert>=6.0, but you have nbconvert 5.6.1 which is incompatible.

and while trying to run nbconvert:

ImportError: cannot import name 'contextfilter' from 'jinja2' (C:\Users\MehdiSattari\miniconda3\envs\py38\lib\site-packages\jinja2\__init__.py)

I think I should downgrade jinja2 as well, but I'm not sure what other packages may break.

[dboeckenhoff]

Same issue with 7.6.0

[dboeckenhoff]

@MSeal Could I ask for your attention here since you closde this?

[msat59]

If we have to downgrade to v5.6.x, what's the point for nbconvert development?

[Sieboldianus]

My workaround still works with the latest nbconvert and Jupyter versions. Does it work for you?

[dboeckenhoff]

Dann. My mistake. config file was misnamed🙈 Alexander Dunkel ***@***.***> schrieb am Di., 4. Juli 2023, 18:50:

…

My workaround <#1394 (comment)> still works with the latest nbconvert and Jupyter versions. Does it work for you? — Reply to this email directly, view it on GitHub <#1394 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHAWKXOXIUWV6DXYB6VCP6TXORCWRANCNFSM4RMZ73KA> . You are receiving this because you commented.Message ID: ***@***.***>