Explore reconnecting a running (remote kernel) cell after reload

[DonJayamanne]

For #3998

Ask:

• User connects to a remote Jupyter Server
• Runs a long running cell
• Shuts down the local computer or closes VS Code or Reloads VS Code,
• The expectation is
  • The notebook automatically connects to the previous kernel session
  • If the kernel is still busy executing the same code, then display the cell as busy and attach the output of the cells

[DonJayamanne]

Learnings

• We can reconnect to an executing cell after reload using existing API
• We can handle new outputs from the execution (for simple simple mime types, like text, streamed outupt, html, etc)
• For the most part this will work e.g. for progress bars and the like.
• Drawbacks:
  • Widgets will not work (requires more work, can be done later and only for IPyWidgets 8, we have a separate issue)
  • Display data will not work (similar to widgets, but dynamic outputs)
  • There will be loss in the outputs (e.g. streamed data or the like)
  • Need a way to inform the user of this (else they could get confused)

Other explorations

• We can run the cells in a special mode that would make the kernel capture all of the output
• Upon reconnecting to the cell after re-load, and once execution has completed we can then ask the kernel for all of the outupt
• This works very well for streamed data, simple mime types
• Drawbacks:
  • In current form this will not work if user has cell magics (need to find a way to do without a cell magic)
  • Needs more work (investigation) to get this working for Widgets
  • Outputs are buffered in memory in the kernel (if outupts are very large, then this could impact memory usage)
  • Cannot be enabled by default due to memory implications
  • User has to opt into this special execution mode

[DonJayamanne]

Suggestions:

• We could explore displaying a transient output message to the user that some outputs might be lost/partial
• Let users opt into this (from the above transient output or from user settings or both)
  • I.e. support reloading and display a message indicating there is some partial data and it can be displayed

[DonJayamanne]

TODO: Add screenshots along with the details of the final solution.

[DonJayamanne]

Solution:

• Only works for remote kernels started by the Jupyter extension
• Only works for streams and text outputs (widgets, display data is out of scope)
• Experience:
  • Assume user runs a long running cell,
  • Next user shutdown VS Code or the computer
  • Next user re-opens VS Code after a while
  • The notebook will be automatically connected to the same (previous kernel session) - this is existing behaviour
  • If the kernel is still busy running the cell, then the cell will continue to run
    • The timer for the cell will resume from when it started (i.e. it will reflect the amount of time the cell has been busy)
    • Any text outputs that continue to be sent by the kernel will be displayed in the notebook cell output
  • From this point the cell execution continue as though user ran it

See below for a sample experience

[aminfardi-CD]

@DonJayamanne stumbled on this as I have this exact use case on a remote dev server. Curious what release timelines are?