add preheated kernel for execution to speedup the computation

[pplonski]

It should be possible to add preheated kernels for executing notebook. Please check the following discussions:

• Allow the execute preprocessor to make use of an existing kernel jupyter/nbconvert#852
• How to share kernel with nbconvert Python API jupyter/nbconvert#1802
• Share running kernels between processes jupyter/jupyter_client#811
• How to reuse exsisting kernel? jupyter/nbclient#238

[pplonski]

Very simple demo:

process1.py

import nbformat
import json
from jupyter_client import MultiKernelManager
from nbconvert.preprocessors import ExecutePreprocessor

mkm = MultiKernelManager()
mkm.start_kernel(kernel_name="python3", kernel_id="some-kernel")
km = mkm.get_kernel("some-kernel") # the kernel connection file is not created

print(km.get_connection_info())

nb1 = nbformat.v4.new_notebook()

nb1["cells"] = [nbformat.v4.new_code_cell("a = 13")]

ep = ExecutePreprocessor()
ep.preprocess(nb1, km=km)

print(nb1.cells)

process2.py

from jupyter_client import BlockingKernelClient
client = BlockingKernelClient()
client.load_connection_file('./kernel-some-kernel.json')
client.start_channels()

print(client)
client.execute_interactive("a+=1")
client.execute_interactive("print(a)")

KernelManager cant accept existing running kernel. There are two ways:

• write customer KernelManager
• write custom provisioner

[pplonski]

This is fixed in Mercury V2.

We keep the active Python session in the worker. V2 ix ~ x20 faster than V1.