Question: thread safety of execute preprocessor

[akhmerov]

I'd like to execute a bunch of notebooks programmatically using nbconvert. For that I'd like to know if the execute preprocessor is thread-safe (since all the computation is done in an external process (kernel), a ThreadPoolExecutor seems to be a very reasonable solution).

Thanks in advance.

[mgeier]

It is currently not re-entrant (https://en.wikipedia.org/wiki/Reentrancy_(computing)), see my comment #886 (comment) (somewhere in the middle of the comment, look for "the real problem").

I guess re-entrancy is a requirement for "thread safety" (however you are defining that exactly), so the answer is "no".

Until the problem is fixed, you'll have to use a separate instance of ExecutePreprocessor for each potentially concurrent execution.

[akhmerov]

Until the problem is fixed, you'll have to use a separate instance of ExecutePreprocessor for each potentially concurrent execution.

That's my plan anyway: the preprocessors are stateful. Instead I'm wondering about kernel managers and their use of event loops/zmq/anything else that can cause trouble.

[MSeal]

Late response, but this is mainly blocked for ipython on the upstream issue ipython/ipython#11460

[akhmerov]

Would setting IPython's history to not write on disk be appropriate within nbconvert, or should this really be done on the side of IPython?

[MSeal]

We just did a deep dive on making this work at PyCon sprints. I think we have a working prototype. The history change makes ipython work, but there were additional issues on the kernel manager side that needed fixing. I think we'll have this fixed for the next nbconvert patch release.

[MSeal]

Turns out there's more upstream issues. There will need to be more fixes in ipykernel and jupyter_client (and possibly traitlets) to get this fully fixed. Working slowly on it in my spare time.

[MSeal]

Should now be resolved with nbconvert 5.6.0 -- see https://discourse.jupyter.org/t/nbconvert-5-6-0-release/1867 for details of the change.