Papermill hangs if kernel is killed

[MSeal]

Today papermill is hanging when kernels get OOM killed instead of returning with an error status code within some reasonable timeframe.

Steps to reproduce:

• Make a notebook which sleeps indefinitely.
• Call papermill on the notebook
• Kill -9 the kernel process
• See papermill not exit (ever)

[betatim]

Can we add a thread that receives the kernels heartbeat and raises an exception when it stops receiving it?

An alternative is to have a timeout of N minutes of nothing happening. I don't like that option because you need to set it to (very) large numbers for notebooks doing "lots of stuff" and often I don't know what to set the timeout to. (For scikit-optimize we kept having this problem because travis nodes are a lot slower than laptops so tuning the timeout was tedious because you had to run it on travis...)

[rgbkrk]

I definitely don't want a timeout as we should be able to have very long running jobs. Is there something on the nbconvert side or jupyter client side that could be catching more process level events or if we need to track the pid of the kernel process more directly?

[MSeal]

I was thinking we want is to get the id of the kernel process and just monitor that the pid is still alive.

[betatim]

jupyter/nbconvert#791 a discussion about advantages and disadvantages of having the default timeout in nbconvert. Might be interesting to observe what people say.

[MSeal]

Solved in nbconvert 5.5 which is shipping tomorrow.