Output notebook while executing

[matthias-k]

I'm using nbconvert to execute notebooks from the command line. Some of these notebooks take a long time and I don't get any feedback on the progress (and e.g. potential convergence problems etc) unless I save some results to files in the notebook. Is there any way to e.g. e.g. write the cell and the output to the console (e.g. as markdown) while processing the notebook? This would make things a lot easier.

[Pita]

Agreed!

[AlJohri]

is there a way to force a print statement to go to the command line? I found that if I use a ProcessPoolExecutor the print statements end up the the terminal but I was hoping for a way to replicate this without a separate process

[RoyalTS]

related to #374?

[MSeal]

Also https://papermill.readthedocs.io/en/latest/ supports this as a wrapper to nbconvert for notebook-to-notebook transformation. I can look into pulling this functionality into the preprocessor when I get back to finishing #905 (or shortly there-after).

[laurentperrinet]

Somewhat related to #709 which suggests to save the notebook every T minutes.

Another option could be to add the possibility to save the notebook anytime a cell is executed.

[MSeal]

Generally at this point the advice is to use papermill for notebook execution when you want periodic saves and deeper reporting on execution. It's unlikely that this will be copied over to nbconvry, and probably not worth the library duplication, as papermill was made specifically to isolate complex execution concerns and execution ability in nbconvert is left for compatibility and ease of combining with template conversions in a singular call.

[laurentperrinet]

I did have a try at papermill and it works as expected:

papermill  --request-save-on-cell-execute --progress-bar input.ipynb output.ipynb

Thanks, Laurent