-
Notifications
You must be signed in to change notification settings - Fork 567
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Question: thread safety of execute preprocessor #936
Comments
It is currently not re-entrant (https://en.wikipedia.org/wiki/Reentrancy_(computing)), see my comment #886 (comment) (somewhere in the middle of the comment, look for "the real problem"). I guess re-entrancy is a requirement for "thread safety" (however you are defining that exactly), so the answer is "no". Until the problem is fixed, you'll have to use a separate instance of |
That's my plan anyway: the preprocessors are stateful. Instead I'm wondering about kernel managers and their use of event loops/zmq/anything else that can cause trouble. |
Late response, but this is mainly blocked for ipython on the upstream issue ipython/ipython#11460 |
Would setting IPython's history to not write on disk be appropriate within nbconvert, or should this really be done on the side of IPython? |
We just did a deep dive on making this work at PyCon sprints. I think we have a working prototype. The history change makes ipython work, but there were additional issues on the kernel manager side that needed fixing. I think we'll have this fixed for the next nbconvert patch release. |
Turns out there's more upstream issues. There will need to be more fixes in ipykernel and jupyter_client (and possibly traitlets) to get this fully fixed. Working slowly on it in my spare time. |
Should now be resolved with nbconvert 5.6.0 -- see https://discourse.jupyter.org/t/nbconvert-5-6-0-release/1867 for details of the change. |
I'd like to execute a bunch of notebooks programmatically using nbconvert. For that I'd like to know if the execute preprocessor is thread-safe (since all the computation is done in an external process (kernel), a
ThreadPoolExecutor
seems to be a very reasonable solution).Thanks in advance.
The text was updated successfully, but these errors were encountered: