Add JEP for sub-shells

[davidbrochart]

This PR introduces kernel sub-shells to allow for concurrent code execution.
It is a JEP because it changes the kernel messaging protocol - these changes consist of (optional) additions.

[davidbrochart]

@minrk it would be great if you could be the shepherd for this JEP.

[minrk]

Do you envision sub-shells having any properties or inputs? Or are they all by definition identical for a given kernel (at least to start)?

[davidbrochart]

The only setting I can think of, since the difference between sub-shells and the main shell is that they run concurrently, would be to specify which concurrency "backend" should be used: a thread or a process or asynchronous programming.
But I think it would lead to too much complexity and threading will always be used anyway.

[minrk]

I think it's not yet specified exactly where exactly the shell id is passed on the shell messages. I imagine a shell_id field in the header (both request to route and response to identify) should suffice. I don't think it should be added to content.

[davidbrochart]

I agree that the shell ID should be passed in the header of request messages.
Should it be copied in the header of the reply or is it enough for it to be present in the parent header (since it's included in the reply)?

[jasongrout]

+1 to header, especially since many shell message types may use subshells, such as comm messages and autocomplete or inspect requests.

I think it's fine for it to be in the parent_header of the reply.

[jasongrout]

I think it's fine for it to be in the parent_header of the reply.

Actually, given that a frontend will likely have to route busy/idle messages, output messages, etc. based on their shell id, does it makes sense to have that shell id in the header of any messages coming from the subshell on the iopub channel, not just shell reply messages?

[davidbrochart]

It should already be in the parent header, which is also included in the messages coming from the subshell on the iopub channel, right?

[davidbrochart]

I started an implementation of sub-shells in ipykernel, that can be useful to validate the kernel protocol changes required by this JEP.

[davidbrochart]

Although tests fail in ipython/ipykernel#1062, it is functional (you can try it e.g. with JupyterLab) and the sub-shell test shows that sub-shells run in parallel.

[davidbrochart]

Maybe the user should be able to provide a shell ID in the content, in case they want to reuse it later, instead of getting the shell ID in the reply.
If the provided shell ID already exists, an error would be sent in the reply.

[jasongrout]

I think in general, the entity creating the resource should get to decide the id. Otherwise it turns into a guessing game picking an unused id.

So +1 to the kernel answering back with the subshell id.

[davidbrochart]

I opened a PR in JupyterLab that uses the ipykernel PR to implement "sub-consoles". Together, these PRs can serve as a proof-of-concept for this JEP.

[jasongrout]

Also, semantics of subshells for kernels that have not implemented them, i.e., using a subshell does not guarantee computations will run concurrently. Really, the only guarantee is that a specific subshell's computations will be run in order (and for a kernel not implementing subshells, this is done by just serializing all shell messages on to the main shell thread).

Also, what happens if you specify a subshell that does not exist on a kernel that supports subshells?

Is there any change to busy/idle message semantics?

[jasongrout]

What are the semantics of subshells around interrupt and restart messages, or more generally kernel restarts?

[davidbrochart]

Thanks @jasongrout for the questions, I have opinions but we should discuss these points.

Also, semantics of subshells for kernels that have not implemented them

Maybe the create_subshell_reply should return an error, and the shell_id field of any message should be ignored?

Also, what happens if you specify a subshell that does not exist on a kernel that supports subshells?

Again, I think the reply should be an error.

Is there any change to busy/idle message semantics?

I'm tempted to say that this is a front-end issue. It has all the information about which shell is busy/idle, and it should decide how to translate that information to the user: an OR of all the busy signals, or only select the main shell busy signal.

What are the semantics of subshells around interrupt and restart messages, or more generally kernel restarts?

This almost seems to be orthogonal, as it's about the control channel, which this JEP doesn't change.

[jasongrout]

This almost seems to be orthogonal, as it's about the control channel, which this JEP doesn't change.

I meant more: what is the lifecycle of subshells around kernel restarts. I think it makes sense for subshells to be terminated, but I also think that should be specified.

If I interrupt the kernel, does it interrupt all shells or just the main shell, or can I selectively interrupt a subshell?

[davidbrochart]

I also think that restarting a kernel should terminate subshells, and that interrupting the kernel should interrupt all shells, otherwise it could quickly become a mess. And yes, we should specify it 👍

[jasongrout]

A huge +1 to exploring this idea - thanks @davidbrochart!

Recently we've been having a number of cases where we'd like computations to run concurrently with normal execute requests (like autocomplete, some widget comm messages, etc.). Having some sort of way to run these would make a huge difference.

[jasongrout]

(Starting a new thread for separate conversations)

Also, semantics of subshells for kernels that have not implemented them

Maybe the create_subshell_reply should return an error, and the shell_id field of any message should be ignored?

Also, what happens if you specify a subshell that does not exist on a kernel that supports subshells?

Again, I think the reply should be an error.

Currently, in ipykernel, do you know what happens if you send an unrecognized control message? Does it reply with an error, or does it ignore the message?

I think if a shell message is sent with an unknown subshell id, a reasonable response is to ignore the subshell id and schedule the computation on the main shell. That would be backwards compatible with kernels that do not recognize subshell id info, and still guarantees that subshell requests are executed in order.

In other words, I think it is reasonable that giving subshell info does not guarantee concurrency with other messages. It only guarantees that subshell messages will be processed in order, and it is an optimization/implementation detail that messages on one subshell can be processed concurrently with another subshell's messages.

[davidbrochart]

Currently, in ipykernel, do you know what happens if you send an unrecognized control message? Does it reply with an error, or does it ignore the message?

ipykernel ignores the message, but I don't think the kernel protocol specifies this behavior.

I think if a shell message is sent with an unknown subshell id, a reasonable response is to ignore the subshell id and schedule the computation on the main shell.

I agree that a kernel that doesn't support subshells should ignore shell_ids, and process everything in the main shell, but should a kernel that supports subshells process messages with an unknown shell_id in the main shell, or reply with an error?

In other words, I think it is reasonable that giving subshell info does not guarantee concurrency with other messages. It only guarantees that subshell messages will be processed in order, and it is an optimization/implementation detail that messages on one subshell can be processed concurrently with another subshell's messages.

Agreed. Also, this JEP doesn't specify the concurrency backend. It will most likely use threads, but we can imagine that a kernel uses e.g. asyncio. In this case, concurrency would work only if cells are "collaborative", i.e. they await (a bit like in akernel).

[jasongrout]

ipykernel ignores the message, but I don't think the kernel protocol specifies this behavior.

Since the kernel_info reply carries with it the kernel protocol the kernel speaks, the client will also know what messages the kernel understands.

[davidbrochart]

Right. Then I'm wondering if we should even try to be backwards-compatible, since a client knowing that the kernel doesn't support sub-shells should not use them. Maybe we should remove this section?

[jasongrout]

I think we should try to be backwards compatible. It will be a long hard process to get many shells to support this subshell id (see what @krassowski mentioned in #91 (comment)). It will be much simpler from the client perspective if you can write one codebase that works with both the current protocol and the new protocol with reasonable degradations for the current protocol.

[jasongrout]

Though I do wonder if instead of incrementing the kernel protocol number, we should instead have the idea of kernel extensions, e.g., a kernel can advertise which messages it supports, and doesn't have to support other messages in previous kernel protocol versions. For example, I can see a kernel wanting to support subshells before it supports debugging, but there would be no way for a kernel to tell that to a client.

@minrk - what do you think about introducing a field in the kernel info reply for a kernel to advertise which messages it supports?

[minrk]

what do you think about introducing a field in the kernel info reply for a kernel to advertise which messages it supports?

I think if we're starting to increasingly implement additional features in the protocol that are optional for kernels, this makes sense. If we're doing that, is there a more general 'feature' list that can't be represented purely as support for a given message type (e.g. subset of capabilities of a given message type).

I think that should probably be a separate proposal, though.

I don't think the kernel protocol specifies this behavior.

I think we should probably define a response pattern for kernels to use for unrecognized/unsupported messages.

[jasongrout]

I think that should probably be a separate proposal, though.

+1

[JohanMabille]

what do you think about introducing a field in the kernel info reply for a kernel to advertise which messages it supports?

We actually already do it for the debugger (a field indicates whether the kernel supports the debugger). I agree that generalizing it as a list would be a nice improvement. Working on a JEP.

[krassowski]

This might be a first step to enable a highly-requested feature: built-in variable explorer (e.g. jupyterlab/jupyterlab#443) which would not depend on the debugger being active (or supported by the kernel for that matter, thinking about R here), do I see this right?

I wonder how to avoid the problem that we had/have with the debugger which was the slow adoption across kernels. Are there any plans to bootstrap this by providing pull requests to a few kernel with largest user base (e.g. Julia/R/pyodide kernels) - in addition to IPython/xeus stack - so that there are a few implementation examples available for other kernel maintainers to derive from?

[davidbrochart]

This might be a first step to enable a highly-requested feature: built-in variable explorer (e.g. jupyterlab/jupyterlab#443) which would not depend on the debugger being active (or supported by the kernel for that matter, thinking about R here), do I see this right?

Yes, it could allow a variable explorer to be responsive while the kernel is busy.

I wonder how to avoid the problem that we had/have with the debugger which was the slow adoption across kernels. Are there any plans to bootstrap this by providing pull requests to a few kernel with largest user base (e.g. Julia/R/pyodide kernels) - in addition to IPython/xeus stack - so that there are a few implementation examples available for other kernel maintainers to derive from?

I have not looked at kernels for other languages, but only in ipykernel this requires some architectural changes: reading the shell channel messages in a separate thread (usually this is done in the main thread), and processing the sub-shell execute requests in separate threads. For ipykernel, this also means getting rid of Tornado, and using asyncio, so quite a lot of work. So it depends on the code base of each kernel, I cannot say in advance if it would be easy or not. For pyodide, I'm afraid it won't be possible since threads are not supported in WASM, AFAIK.

[davidbrochart]

I made changes according to the comments in d389802. I removed the points of discussion because I think they don't really make sense: the operations I mentioned (create sub-shell, delete sub-shell, kernel shut-down, kernel restart) are on the control channel, which is orthogonal to sub-shells. It is not a sub-shell which deletes/creates another sub-shell, or restarts/shuts-down a kernel, it is the client/user.

[jasongrout-db]

I was thinking that perhaps an interrupt request could give a subshell id to interrupt only that subshell. However, if we want to be backwards compatible, we have to interrupt all shells: if all subshell requests are processed in the main shell, then interrupting the kernel will currently interrupt all shells.

[davidbrochart]

True. Maybe we could also say that a kernel should do its best at interrupting only the requested sub-shell, but that it may interrupts all shells?

[jasongrout]

True. Maybe we could also say that a kernel should do its best at interrupting only the requested sub-shell, but that it may interrupts all shells?

That sounds too unpredictable to me. I think if we want subshell-specific interrupt, we need another message so we can be backwards compatible and predictable.

[jasongrout-db]

I would say the kernel always generates the shell id and we don't support the client providing an id. Once you have clients providing ids, then it's always a guessing game if there is contention between clients, or you have clients generate UUIDs, at which point you might as well have the kernel generate a truly unique id.

[davidbrochart]

I was thinking that if in the future we allow per-cell sub-shells (through e.g. cell metadata), it could open up possibilities such that a cell creates a sub-shell, and other cells run in this sub-shell, so they would need the shell ID. We could build complex asynchronous systems like that.
akernel can do something similar but programmatically: __task__() returns a handle to the previous cell task, so the next cell can do whatever it wants with it (await it, etc.).

[jasongrout]

If the client specifies a subshell id, it will need to wait until it is confirmed in the reply to be sure it has reserved that name. In that case, why not just get the subshell id from the reply message, and be guaranteed it didn't fail because of a name conflict? What does having the client give the subshell id do for us?

[davidbrochart]

I thought that it allowed us to reuse it later, at least in the case of a self-contained notebook where we know there is no shell ID conflict.

[jasongrout]

A notebook might be opened with two copies, in which case each copy would want to start up a subshell with the same name? For example, either a notebook in real-time collaboration, or a notebook opened twice side by side in JLab?

Or perhaps if you try to create a subshell with an existing id, it just acknowledges that the subshell is already created, with no error? Multiple clients might send computations to the same subshell?

What if we treat it like we do kernel sessions right now, with a user-supplied name as a key? In other words, a client subshell creation request optionally gives a name (not an id). If a subshell with that name already exists, its id is returned. If it doesn't exist, a new subshell with that name is created and returned. And if a name is not given by the client, an unnamed subshell is created and returned. Thoughts? This gives you the ability to share subshells between clients addressable with some client-supplied string, but gives me always unique ids created by the resource manager.

[davidbrochart]

Or perhaps if you try to create a subshell with an existing id, it just acknowledges that the subshell is already created, with no error?

I like that. It seems that there is no distinction between a sub-shell name and a sub-shell ID in this case.

What if we treat it like we do kernel sessions right now, with a user-supplied name as a key?

In that case there seems to be an unnecessary mapping between sub-shell name and sub-shell ID, or am I missing something?

[jasongrout]

There is a technical difference: since the client name and shell id are different namespaces, the shell id (generated by the kernel) does not have to check for conflicts with the names already given, the client can request a shell that is guaranteed that no one else will ever request (i.e., a shell specific to itself, guaranteed to not collide with any other requested name).

For example, suppose the shell ids are generated by starting at 0 and incrementing for each new subshell. If the client asks for shell name 5, and the client name and shell id namespaces are conflated, the client won't know if it's getting some random subshell someone else created (i.e., shell id 5), or if it's getting a shell with specific meaning "5" (i.e., client name 5). Likewise, any time a new shell id is generated, the kernel would have to check to see if someone else had already claimed that number.

I think it's a much better design to keep the client-specific names in a separate namespace from the unique shell ids. With this design, any client asking for a shell named "autocomplete" gets the same autocomplete shell shared with every other client requesting the same subshell name. However, if you want to get your own subshell separate from any other client, you just request a subshell without a name.

[SylvainCorlay]

Thanks David for starting the discussion on this. I wanted to bring up the alternative design, which is the one I initially proposed for this.

In the proposal described here

There is still only one shell socket, and you address the subshell via a shell Id.

In my alternative design

Subshells open their own shell socket, and appear as a separate kernel in the UI on which you can connect any client.

I strongly prefer this design because:

• it is only additive (addition of a control message for creating a subshell).
• it does not require clients to change drastically (like to support shell ids on execution requests) or to specify a subshell id upon connexion.

[kevin-bates]

hi @SylvainCorlay.

Subshells open their own shell socket, and appear as a separate kernel in the UI on which you can connect any client.

Wouldn't the subshell socket need to be communicated back to the (potentially remote) launching server and does kernel lifecycle management get affected since there are now N additional ports to deal with?

How are subshells terminated? Is it simply a matter of (and only in the case this virtual kernel is a subshell) closing that socket?

I think applications will still need to apply an id (or name) to each subshell (for their own management) and feel an explicit approach (where that id/name is known by the kernel) would be easier to manage/troubleshoot, and less confusing to the end-user.

[jasongrout]

Subshells open their own shell socket, and appear as a separate kernel in the UI on which you can connect any client.

Thinking through how this would work in practice:

Currently we have control, iopub, input, heartbeat, and shell zmq sockets. You're proposing that we now have control, iopub, input, heartbeat, main shell (shell0?), shell1, shell2, ..., shellN sockets that the kernel manager must connect to? I imagine we then create a subshell with a rest api request to the kernel manager (similar to creating a new kernel connection), so the kernel manager knows to connect to the new shell zmq channel.

The current Jupyter web client multiplexes the kernel messages onto a single websocket. How would that multiplexing work in your scenario? There is a new websocket for each shell channel, and control, iopub, input messages are routed to the shared zmq sockets, but the shell messages are routed to the shellN channel for that particular websocket connection?

[SylvainCorlay]

Thinking through how this would work in practice:

Currently we have control, iopub, input, heartbeat, and shell zmq sockets. You're proposing that we now have control, iopub, input, heartbeat, main shell (shell0?), shell1, shell2, ..., shellN sockets that the kernel manager must connect to? I imagine we then create a subshell with a rest api request to the kernel manager (similar to creating a new kernel connection), so the kernel manager knows to connect to the new shell zmq channel.

The current Jupyter web client multiplexes the kernel messages onto a single websocket. How would that multiplexing work in your scenario? There is a new websocket for each shell channel, and control, iopub, input messages are routed to the shared zmq sockets, but the shell messages are routed to the shellN channel for that particular websocket connection?

The new subshell would appear as a separate running kernel, and frontends would connect to it in the same way as they connect to regular kernels. The fact that a "kernel" that happens to really be a subshell of another one would be mostly transparent to the frontend.

[SylvainCorlay]

Wouldn't the subshell socket need to be communicated back to the (potentially remote) launching server and does kernel lifecycle management get affected since there are now N additional ports to deal with?

Indeed, the subshell socket would be in the response to the control-channel request to create a subshell. Subshells would also be visible as regular running kernels in the kernels API.

I think applications will still need to apply an id (or name) to each subshell (for their own management) and feel an explicit approach (where that id/name is known by the kernel) would be easier to manage/troubleshoot, and less confusing to the end-user.

I don't think we would need a new ID for subshell in this new proposal (unlike the one presented here), because they would just have a separate kernel ID.

How are subshells terminated? Is it simply a matter of (and only in the case this virtual kernel is a subshell) closing that socket?

This is tan excellent idea. I think that there are two options:

• the first (bad IMO) option would be to use a regular shutdown request on a (sub)control, or through a shared control channel with the main shell. I am not in favor of this.
• the second option would be to have a special "control" message to kill a given subshell (referred to by its kernel id)

This second option is somewhat tied to something that we want to propose independently of subshells for kernels whose lifetime are tied to another process (like a desktop application having an embedded kernel), or a main shell. These kernels would be signalled to the frontend, so that it does not include a shutdown UI.

[kevin-bates]

ok. I think we need to understand and define how similar and different these kinds of "kernels" behave relative to standard kernels and feel they might deserve a different name so we can keep our sanity. It sounds like only the "parent"(?) kernel would be managed by a KernelManager (and KernelProvisioner) and the "others" are dependents of that kernel and essentially managed via messaging - is that a correct statement?

Are these "dependent kernels" independently interruptable and restartable?

Should this be a separate JEP since exposing these as separate kernels will have a significant impact (IMHO)?

[SylvainCorlay]

It sounds like only the "parent"(?) kernel would be managed by a KernelManager (and KernelProvisioner) and the "others" are dependents of that kernel and essentially managed via messaging - is that a correct statement?

I think that whether it is managed by KernelManager and KernelProvisioner is an implementation detail of jupyter-server.

But yes, subshells are "kernels" that are created by sending a control message to an existing kernel, nd their lifetime is controlled that way - but they have their own connection file and so on.

[kevin-bates]

I think that whether it is managed by KernelManager and KernelProvisioner is an implementation detail of jupyter-server.

This would be jupyter_client and believe these are details that need to be understood within the JEP. I'm sure this can be worked out but want to make sure we're all on the same page. After all, the devil is in the details. 😄

Is the original proposal that @davidbrochart submitted being abandoned for this alternative approach, or should a separate JEP be submitted? I think these two approaches vary enough to warrant separate discussions and don't want to continue on David's JEP unless this is the plan moving forward.

[minrk]

Thanks for summarizing the key differences!

I had a conversation with @jasongrout about this yesterday, summarizing thoughts before I forget with the deluge of JupyterCon information pushes it out of my brain.

First, I think it's hard to make a clear choice without seeing a sketch of both the kernel-side and client-side use case for these. That said, my inclination is to go for the "Kernel Sub-shell" proposal, which seems like a simpler change across the board. We don't have to solve the port-propagation problem in e.g. remote kernel provisioners,

I think whether that's the right choice really depends on how folks want to use this, though. If the primary use case is concurrent execution within one logical kernel, I think kernel sub-shell is the way to go. If it's a more lightweight spawning of a 'real' kernel wherever the original kernel was, that seems like a kernel provisioning problem, and doesn't really belong in kernels as I see it.

I don't think the additional shell message queue is a big challenge. It's a little more tedious, but I don't think it should be on the scale of implementing the concurrent shells in the first place. It should look something like:

async def process_msgs(q):
    while True:
        msg = await q.get()
        await process_msg(msg)


async def spawn_shell(name):
    q = asyncio.Queue()
    asyncio.create_task(process_msgs(q))
    return q


async def handle_shell():
    queues = {
        "": spawn_shell(""),
        "sub1": spawn_shell("sub1"),
    }
    while True:
        msg = await session.recv(shell_socket)
        shell = msg["header"].get("shell_id", "")
        q = queues[shell]
        await q.put(msg)

ipykernel already works this way with a queue for the shell channel, so adding more message queues should make very little difference (other than the fact that presumably it should be a threadsafe wrapper instead of a basic asyncio.Queue).

[ianthomas23]

Hello everyone, I am taking over this work. I will start with a summary of the proposal based on previous comments and it will be a little more verbose than is usual so that it is not necessary for anyone to read the previous discussion. There are two possible approaches; I am hoping that interested parties will be able to come to a consensus on the best approach before I update the wording of the JEP.

The proposal is to support extra threads within a kernel as a JEP 92 optional feature ( https://github.com/jupyter/enhancement-proposals/blob/master/92-jupyter-optional-features/jupyter-optional-features.md) so that whilst the main thread is doing a long blocking task it will be possible for other threads to do something useful within the same namespace. Example use cases:

• Read the value of one or more variables. This could be just to read a single variable that indicates the progress of another thread’s task, or could be a full variable explorer without the need to have a debugger available.
• Autocomplete and inspect requests whilst the kernel is busy.
• Visualize intermediate results before algorithm completion.
• Execute arbitrary code in parallel.

As a concrete example consider the screencast below. This runs a “long” calculation (only ~10 seconds here) that ray traces a simple scene. During the calculation there is no feedback other than the busy kernel status, and we don’t necessarily know in advance how long the calculation will take. When it finishes the ray traced scene is displayed. A better user experience is obtained in the second half using a new subshell thread in the same kernel, created here via a “New Sub-Console” option in the context-sensitive menu, and this thread reads the ray tracer variables and updates the image whilst the calculation is performed. We could of course combine the calculation and rendering code in a single thread without using subshells, but this would muddy the two separate areas of responsibility and it would be slower to run. Note that the UI experience here is TBD and not part of the proposal.

subshell_raytrace.webm

There are two possible approaches called kernel subshells and dependent kernels. In both approaches each of the (one or more) subshells has its own thread. When a kernel is started it has a single subshell and this is referred to as the parent subshell to distinguish it from the other optional subshells which are referred to as child subshells.

A new child subshell thread is started using a new create_subshell_request control message rather than via the REST API.

Kernel Subshells

Here there is a single new thread that manages all shell messages. Each subshell has a subshell_id which is a unique identifier within that kernel and is generated when the subshell is created. The parent subshell has a None subshell_id. Shell messages include the subshell_id as an optional parameter in the message header to indicate which subshell the message should be sent to; if this is not specified the parent subshell is targeted. Use of a subshell_id that is not recognised raises an error. Note a kernel that does not support subshell_id will just ignore it and run in the main thread.

Dependent Kernels

Here each subshell thread has its own shell socket and handles its own shell messages. The connection info for a child subshell is the same as that of the parent subshell but with the different shell_port. Once running, a child subshell appears as a separate kernel in the REST API and can be accessed by any client in the usual manner, hence there is no need for any extra message parameters unlike kernel subshells above. Child subshell kernels do not control the lifetime of the actual kernel, only the parent subshell can do this, hence the “dependent” in the name of this approach.

Essentially kernel subshells expose subshells as being part of the same kernel, and dependent kernels expose subshells as being different kernels.

Resources

The required resources for n subshells (1 parent subshell and n-1 child subshells):

	Kernel subshells	Dependent kernels
Shell sockets	1 shared socket	1 socket per subshell
Threads	1 thread per subshell 1 thread to receive shell messages	1 thread per subshell
Protocol changes	New create, delete and list subshell requests New subshell_id in requests	New create, delete and list subshell requests

New control messages

Note that all subshells within a kernel continue to share the same control port and thread.

Create subshell

Takes no arguments, creates a new subshell. For kernel subshells it returns a subshell ID that is unique within that kernel. This could just be the thread ID or an incrementing integer, but will probably be a UUID for global uniqueness.

For dependent kernels this will return the connection info dict for the subshell, which is the same as the connection info for the main shell but with a different shell_port. A separate subshell_id isn’t strictly speaking required as the shell_port is unique within a kernel, but should probably be generated and returned nevertheless to guarantee global uniqueness.

Delete subshell

Takes a subshell_id.

List subshells

Takes no arguments, returns a list of the subshell_ids of the kernel. The parent subshell, which has a subshell_id of None, could be explicitly included or not.

REST API changes

For kernel subshells, no changes are needed to the REST API. For dependent kernels, each subshell kernel should indicate if it is a child subshell and if so, the identity of its parent kernel. It may also be desirable for a parent subshell to list all of its child subshells, i.e. the equivalent of the list_subshells control message.

Lifetime management

The lifetime of an individual child subshell is controlled by the new create and delete subshell control messages. There are a number of options for handling existing interrupt and shutdown control and/or REST API messages:

1. They apply to the whole kernel (all subshells) rather than a single subshell.
2. Messages for a subshell only apply to that subshell, whether parent or child.
3. Support both options so that clients can choose whether to interrupt/shutdown a single subshell or the whole kernel.

I prefer option 3 so that clients have full control and can choose what to do.

Kernel status

There will need to be some clarification of what kernel status (busy, idle, etc) means in a multithreaded kernel. The simplest idea here is to have a separate status for each subshell thread, but this could lead to proliferation of status messages. I understand this issue is being discussed already with possibilities being to no report status of the control thread, as it should never take long to do anything control-related, and perhaps the busy/idle should be replaced with an integer count of the number of threads that are currently busy.

Implementation details

These shouldn’t really be part of this but may help understand the proposal.

ipykernel

In ipykernel we need to spawn new threads. Child subshell threads are almost identical to parent subshell threads (ignoring different lifetime management) but the parent subshell thread should always be the main thread as this is important for some GUI event loops.

Kernel subshells need a separate thread for all shell messages. This will communicate with the individual subshell threads via ZMQ inproc PAIR sockets (essentially shared memory). Messages need to be partially or wholly decoded by the shell message thread to extract the subshell_id to know which subshell thread to send the message to.

Dependent subshells each handle their own shell messages, so each subshell thread needs to create and bind to its own shell socket.

Jupyter server and client

There are no changes required for kernel subshells. For dependent subshells we need to instantiate a KernelManager for a child subshell so that it appears as a separate kernel. This can be achieved in a method similar to supporting external kernels (jupyter/jupyter_client#961) by passing in connection info. This connection info needs to be passed from the kernel to jupyter server/client in some way.

Implications for other projects

Kernel writers who wish to support subshells will need to write extra threading and socket management code.

Any client that wishes to create a subshell will have to issue a subshell request control message. Users of kernel subshells will have to pass the subshell_id in all shell messages. Users of dependent subshells will not need to modify any of their existing messaging, they will just need to use the subshell connection info, or subshell kernel in the REST API, to access a subshell.

There will need to be some sort of visual indicator for subshells in, for example, the JupyterLab UI, but this is not strictly speaking part of the JEP.

Demonstration implementations

I have branches of relevant ipykernel and jupyter-whatever repos containing minimal (incomplete) implementations of both approaches. The dependent kernels implementation is used in the ray tracing demo above. I can supply links and instructions for anyone who wishes to experiment with these.

Preferred approach

As the person who will (probably) implement anything decided here, I have been asked what my preference is. Currently I prefer the dependent kernels approach but I am conscious that many maintainers here have much longer and wider experience of the jupyter ecosystem than I and may have concerns about a child subshell thread appearing to be almost-but-not-quite a full kernel that I haven't thought of. If this occurs I consider the kernel subshells approach to be a suitable fallback position.

[krassowski]

Looking at the execute message, there are some less commonly discussed arguments like silent, store_history, allow_stdin and stop_on_error (cancelling queue on error). I understand that these will be passed transparently to the subshell, but I wonder if the semantics of these will remain the same in a case of multiple subshells. I guess toggling silent, store_history can be very useful for clients implementing some of the use cases discussed in the JEP. But I have some questions on the others:

• can two sub-shells request stdin at once? how does it work when stdin is written to from an external process? Will clients need to prevent subshell UI (if any) from closing to avoid a deadlock on pending input (this is what we do for cells which requested input)?
• would it be possible for the queues to interact in any way between sub-shells?

[JohanMabille]

I think the queues of subshells should be isolated and never interact (the only exception would be when stopping the kernels, but that's more the control channel interacting with the subshells than the subshells interacting together).

Regarding the stdin channel, this is a very interesting point. As specified in the protocol:

The stdin socket of the client is required to have the same zmq IDENTITY as the client’s shell socket. Because of this, the input_request must be sent with the same IDENTITY routing prefix as the execute_reply in order for the frontend to receive the message.

And from the kernel side, I think the stdin and the shell channel are strongly coupled; the fact that we have two distinct channels is more a technical constraint that a wanted isolation (think of stdin in a regular shell). So if we add subshells, we should probably add "substdins" channels too. Otherwise, asking for a input from a client may result to have the GUI component appearing in the other client, and that would be very confusing. Both appraoches can handle this scenario, with an implementation in the client similar to that in the kernel (either shell/stdin_id field in the message, or additional sockets).

[ianthomas23]

Update following Jupyter Server/Kernels meeting of 2024-02-01 (https://hackmd.io/Wmz_wjrLRHuUbgWphjwRWw?view#February-1st-2024).

1. stdin channels need to be considered alongside shell channels as discussed by Mike and Johan in the previous 2 comments.
2. Current demo of dependent kernels modifies Jupyter Server to create a kernel manager for an existing running kernel by wrapping a connection file. A real implementation would need to do this properly with some sort of DependentKernelManager that understands the relationships between the subshells. There is other current code in Server that is sub-optimal in that it bypasses kernel managers/provisioners causing problems in failing to free resources. This needs to be tidied/improved and a dependent kernels implementation would need to follow this approved route rather than just being a workaround.
3. The concerns about dependent kernels potentially causing problems by pretending to be full kernels when they are not means that the current preferred implementation is kernel subshells.
4. I will prepare some instructions for other maintainers to easily try out the demo implementations to better understand everything.

If anyone else has a strong opinion about item 3 (preferring kernel subshells over dependent kernels) it would be good to hear it.

[ianthomas23]

Here are some instructions for anyone who wants to try out the demo subshell code for themselves. It involves installing from a number of my own branches (2 for kernel subshells, 4 for dependent kernels). Install into a environment of your choice (virtualenv or conda or whatever) and do this in a directory where you don't mind files being placed/overwritten. If you try them both, put them in different directories.

Kernel subshells

pip install git+https://github.com/ianthomas23/ipykernel.git@kernel_subshells
git clone -b kernel_subshells git@github.com:ianthomas23/jupyterlab.git --single-branch --depth 1
cd jupyterlab && pip install -ve . && cd ..
jupyter lab --dev-mode

Dependent kernels

pip install git+https://github.com/ianthomas23/ipykernel.git@dependent_kernels git+https://github.com/ianthomas23/jupyter_client.git@dependent_kernels git+https://github.com/ianthomas23/jupyter_server.git@dependent_kernels
git clone -b dependent_kernels git@github.com:ianthomas23/jupyterlab.git --single-branch --depth 1
cd jupyterlab && pip install -ve . && cd ..
jupyter lab --dev-mode

Here is an example screencast of JupytyerLab, this uses kernel subshells but dependent kernels looks the same except for slightly different output for the new %subshell magic command:

subshell.webm

Walkthrough

1. Starts with a notebook in a console as usual.
2. Set some variable a.
3. Run the %subshell magic command to show various information about this parent subshell.
4. Right click and select the new item in the menu "New Sub-Console for Notebook". This opens a new console connected to a new subshell of the same kernel.
5. Check that a in the child subshell has the same value as in the parent subshell.
6. Run %subshell. Note the subshell id is a UUID rather than None, the PID is the same (so the same process), the thread id is different. The number of subshells is 2 now.
7. Rerun %subshell in the parent subshell and note this shows 2 subshells.
8. Run the loop in the parent subshell which takes 10 seconds to run and updates the variable b once a second.
9. Whilst this is running in the parent subshell, the child subshell can read the value of b without having to wait for the cell in the parent subshell to complete.

If you are using dependent kernels then look at http://localhost:8888/api/kernels (or whatever) and you will see that the each subshell appears as a fully-fledged kernel. With kernel subshells there is only the one kernel.

Disclaimer: these are bare-minimum implementations that work for simple tasks but are not complete. They don't handle stdout and stdin well and will break if you interrupt or shutdown the kernel.

You can create as many subshells as you wish in this way, and they all share the same namespace and run independently in separate threads.

This UI is unlikely to be available for most users but is really useful for testing. It is much more likely for subshells to be used by extension and widget developers who want to run extra code concurrently with the user's code in the kernel console.

[ianthomas23]

After the discussion last week about how dependent kernels shouldn't just use the simplistic approach in my demo code of wrapping connection_info but should instead exist in the server code as a DependentKernelManager which accurately represents the relationships between the sub-shells, I have a possible design for this.

Here is a pseudo-UML class diagram for it:

Currently an IPythonKernel, which has a single shell thread, shell socket and stdin socket, is represented in the server as a ServerKernelManager (SKM) in simple situations. If supporting dependent kernels an IPythonKernel has zero or more child subshells, each of which has an id, thread, and shell and stdin sockets. If we want to connect a child subshell via the server, it will be represented by a DependentKernelManager (DKM). Each DKM needs to know the SKM that is depends upon, and a SKM needs to know all of the DKMs that depend upon it. A DKM doesn't do any kernel provisioning, its kernel is already running. To connect to its kernel it uses the same connection info as its SKM (i.e. parent kernel) but with different shell and stdin ports. The key difference between an SKM (parent subshell) and DKM (child subshell) is that SKM completely controls the lifetime of its kernel whereas a DKM does not. A DKM can interrupt and shutdown its kernel (child subshell), but it is also shutdown when its SKM (parent subshell) shuts down.

We are now in a position to define what a DKM is, in general terms:

1. Corresponds to a kernel that is created by another kernel (its parent), so outside the control of the DKM.
2. The kernel lifetime is limited to that its parent kernel.
3. It connects to its kernel using 1 or more connection attributes, and the remainder are the same as the parent kernel.

This makes it different from the existing server code to wrap an external kernel as for the latter we have no idea how the lifetime of the kernel is controlled.

The above definition is purposefully as general as possible. The implementation we are currently talking about is for multiple threads running within the same process on the same machine, but we do not have to limit ourselves to just that. This generalised abstraction allows a number of other possibilities that might be useful in the future such as the one mentioned by Zach of a dependent kernel that is running on a different machine, so all of the connection info is different including the IP address.

Creating a DKM in the server needs two things:

1. The id of the parent kernel.
2. The connection info attributes that differ from the parent, which are returned in the create_subshell_reply message.

[ianthomas23]

I've updated the words in the JEP itself, in line with the latest kernel subshells thinking.

I have standardised on "subshells" rather than "sub-shells" throughout. I think the latter might be more correct but the former is used in chemistry and saves quite a lot of dashes.

[Zsailer]

Great stuff here Ian! Thank you for working on this!

I want mention that while there hasn’t been a lot of direct comment on this thread, there have been many discussion happening in the Jupyter Server Meeting and Jupyter SSC working call around this topic. (This highlights some of the cons of doing synchronous meetings. We don’t always collect the best notes.)

There are two areas I’d like to flesh out in the proposal:

1. How do we reconnect to a DK vs. subshelled kernel?
2. What experience is best for the “kernel client end user”?

Expanding on (1), how do we reconnect to each type of kernel?

In the current DK description, I don’t see how a kernel client could connect to a kernel with dependent kernels and know which subshell port is the parent. As I understand it so far, each DK would have a separate connection info that only differ by their shell port. First, how do we tell which connection_info includes the parent shell port? We can’t rely on this being stored in the client, since kernel clients can stop and start-again while the kernel stays running. How do we recover this information?

Typically, the connection info is stored in connection files on disk using the kernel ID. Maybe we could change these filenames to include subshell IDs too? I don’t love this either, because in my experience, it’s easy to leak these files and never clean them up (just look at your “runtime” path under jupyter --paths).

The subshell approach has a single connection file with just the parent subshell info—so reconnecting is easy. However, how would we know what subshells are available? We’d probably need a protocol-defined message type to list available subshell_ids to avoid leaking threads when a kernel client disconnects.

Expanding on (2).

I’m a little worried about the extra cognitive load on the kernel client end user in DKs.

DKs require an additional kernel management layer underneath the kernel client. This adds a lot of complexity to a simple client interface that most of the time, the end user won’t care about. But we can’t completely hide it, since (as mentioned) extra connection files will be created and e.g. in Jupyter Server, we have sessions attached to (dependent) kernels. We need somewhere to show/manage the dependent kernels and many possible UIs that connect to them. The “Running” tab in JupyterLab is already quite confusing for users in my experience. I think balancing an intuitive UX and API for DKs is far more challenging than sub-shells.

[JohanMabille]

I tend to agree with @Zsailer here. We should not try to "deceive" the user and claim that each client is connected to an independent kernel. The first use case for this JEP, opening a console to inspect a kernel that seems stucked, means that the user will explicitly open a subshell on an identified kernel. In the UI, the user will probably want to see the link between these two clients.

Regarding the reconnection issue with subshells, I think we can add a field to the kernel_info_response message, listing the available subshells.

[ianthomas23]

Thanks @Zsailer and @JohanMabille.

(1) Reconnecting

To obtain information about existing subshells, the proposal details include a new list_subshells_request control channel message that returns a list of the current subshell IDs (https://github.com/davidbrochart/enhancement-proposals/blob/sub-shells/kernel-subshells/kernel-subshells.md#list-subshells). For kernel subshells (KS) this is enough, you can open a new connection using the (common to all subshells) connection info, get the list of subshell IDs and address your messages to whichever you want.

But I agree that this is not enough for dependent kernels (DK). If you have disconnected from all of the subshells then the only foolproof re-connection you can make is to the parent subshell as the child subshells may have been shutdown by someone/something else. So a DK client does need to know the connection info of the parent subshell. This could be provided by widening the scope of the list_subshells message to include all necessary information that a client may require to connect to any subshell of their preference. The latest DK proposal has common connection info for all subshells except for unique shell and stdin ports. This information could be in the kernel_info_response as well, or instead. It could also/instead be in the create_subshell_response if only the parent ports are required, not the other child subshell ports.

I think this means that a client that only accesses a DK child subshell (not the parent) will store the connection info for the child subshell as normal, but must (if it ever wants to reconnect) obtain the parent subshell's connection info and store that too, and also store the mapping from child to parent subshell, either in RAM (is that sufficient?) or as a "parent kernel ID" in the child subshell connection info. This seems to be getting messy, but I am not particularly familiar with this part of the codebase.

(2) UX

If we want an accurate representation of subshells in the JupyterLab UI and we don't want to completely redesign it (which is out of scope here) then I think the only sensible option is (apologies for my ascii/unicode art):

🐍 Python 3 (ipykernel)
├─ 🗎 notebook.ipynb
├─ 🗎 Console 1
├─ 🐍 Dependent kernel A
├─ 🐍 Dependent kernel B
│  └─ 🗎 Console 2
└─ 🐍 Dependent kernel C

This represents a single running kernel where the parent subshell is accessed using a notebook and a separate console, and there are three subshells running, one of which we are accessing via console and the other two we are not accessing. The top 3 lines are what we would see currently. The dependency between the child and parent subshells is included intrinsically, as if you shutdown the parent subshell you expect that to disappear from the tree control and so do all the child subshells as they are underneath the parent in the displayed hierarchy. This also allows an intuitive way to open a console to a subshell via right-clicking on the appropriate subshell or some equivalent keystrokes.

This would become unwieldy if there were many child subshells. Possible variations are to group all the dependent kernels together under a common "Dependent kernels" parent in case the subshells are a distraction and you want to be able to minimise them in the tree control, and sometimes it might be desirable to not show the subshells if their use is hidden from the user, but maybe it then becomes too complicated and there should only be the one option here. I would be inclined to use the same display for either approach (KS or DK), and what the "Dependent kernel A" should actually show is TBD.

[JohanMabille]

the proposal details include a new list_subshells_request control channel message that returns a list of the current subshell IDs

Ah indeed, I totally missed this one. I find this solution actually better than returning the shubshell lists in the kernel_info_response: the available subshells list is a dynamic property, while the data in kernel_info_response is static info about the kernel.

[jasongrout]

@blois and @DonJayamanne - this proposal exposes multithreaded execution in Jupyter kernels via the kernel protocol (see the text of this PR, which @ianthomas23 says is up to date with the current proposal). It may be useful to Colab or VS Code to be able to send messages to the kernel in parallel to user computations (for example, autocomplete and inspect requests, or even allowing the user to send messages to separate threads for execution). We'd love any feedback from people that may want to use this capability.

[Zsailer]

Also, pinging @alexarchambault—who created the Almond kernel—too. It would be great to get someone who works on an alternative Jupyter kernel (other than IPython) to take a look at this proposal

[alexarchambault]

Author of Almond here, which is a Scala kernel for Jupyter (and which should be the most maintained / up-to-date Scala kernel).

Implementation-wise, to me, the kernel sub-shell path seems to require slightly less work than the dependent kernel path (and also seems somehow more natural as a user). But none of the two choices' implementations should be a problem…

One point might lead to discrepancies or be tricky to implement though: the per sub-shell / dependent kernel history and execution count. For unnamed variables, Almond generates variables like res0, res1, res2, etc. to hold their values. Also, even though users don't see it, these values are put in classes named cmd0, cmd1, cmd2, etc., before being compiled. The number in those variable and class names are currently in sync with the execution count (res0 and cell0 for the first cell, etc.). IIUC, sub-shells should at least share history and execution count before they're being created. Then, when a new sub-shell is created, it increases its own execution count, and adds entries to its own copy of the history. This is probably going to be quite tricky to implement, as we should come up with naming like subshell1cmd2 say, for the classes of sub-shells not to conflict, and keep referring to earlier classes with a different subshell number. Also, if two subshells define res2 say, some kind of shadowing should allow each to only see its own res2. Yet, if only a single subshell defines res5 say, other sub-shell will be able to access it as is, which might be surprising. I foresee difficulties in implementing that properly, or working around the second point, to the point that I'd probably go for a naive solution first: keeping a single global history and execution count in a first time, not respecting the specification on that point.

I don't know if that can be an issue in other kernels.

[DonJayamanne]

This is truly a great proposal, and cannot wait for this to go mainstream.
I will need to go through most of comments to ensure I have a good understanding of the proposal/discussions.

Got a few questions

• When debugging kernels, I'm assuming there are no changes to the current behaviour.
  At least from what i know about debugpy, there shouldn't be.
• How will this impact the existing requets auto complete, inspect requests?
  • If using subshell approach, would inspect and auto complete requests automatically end up getting handled by subshells? I'm assuming not.
  • I.e. callers would explicitly create subshells and send the request to the appropriate subshell.
  • Is this the current thinking?
• Would adding a name (description/purpose) of Subshell be useful
  • This information could be displayed when using the %subshell magic, giving users an idea of why it was created
  • E.g. if a client (jupyter extension or VS Code extension) ends up creating a subshell for the purpose of analysing python code in the kernel (for better intellisense,. etc), then users know why such a subshell has been created.
  • Later, optionally users can manage these subshells, e.g. shut them down via some UI (this would work very well with the Kernel Management UI when using the dependant kernel solution).

[ianthomas23]

IIUC, sub-shells should at least share history and execution count before they're being created.

That is not my intention. Each subshell would independently store its own history and execution count, so when you create a new subshell it starts with an empty history and execution count.

I need to clarify this in the JEP wording. In the current messaging docs (https://jupyter-client.readthedocs.io/en/stable/messaging.html) there are a number of statements along the lines of "For clients to explicitly request history from a kernel". For kernels consisting of only a single subshell this is equivalent to "For clients to request history from a subshell". But with the existence of more than one subshell per kernel we'll need to be explicit about what is handled and/or stored on a subshell or kernel basis.

[ianthomas23]

Got a few questions

• When debugging kernels, I'm assuming there are no changes to the current behaviour.
  At least from what i know about debugpy, there shouldn't be.

Correct.

• How will this impact the existing requets auto complete, inspect requests?

  • If using subshell approach, would inspect and auto complete requests automatically end up getting handled by subshells? I'm assuming not.
  • I.e. callers would explicitly create subshells and send the request to the appropriate subshell.
  • Is this the current thinking?

Correct. A client could choose to do inspect, etc, this way but would have to do so explicitly just like it would have to explicitly use create_subshell_request to create a subshell in the first place.

• Would adding a name (description/purpose) of Subshell be useful

  • This information could be displayed when using the %subshell magic, giving users an idea of why it was created
  • E.g. if a client (jupyter extension or VS Code extension) ends up creating a subshell for the purpose of analysing python code in the kernel (for better intellisense,. etc), then users know why such a subshell has been created.
  • Later, optionally users can manage these subshells, e.g. shut them down via some UI (this would work very well with the Kernel Management UI when using the dependant kernel solution).

Yes, I can see that this would be useful but I am not sure if this should be the responsibility of the kernel or the client. I'll call it name from now on, but I mean name/description/purpose/TBD. Options:

1. Kernel stores name:

   • name is an optional string argument to create_subshell_request.
   • Each subshell stores its name along with its subshell ID, thread ID, execution count, etc.
   • There needs to be a way of getting the name back to a client. Options:
     • list_subshell_request could include it for all current subshells.
     • %subshell magic could include it for the current subshell. Note the %subshell magic is not yet mentioned in the JEP wording as it was just something I wrote to help with debugging.
     • Some other new message such as list_subshell_name_request.

2. Client stores name:

   • Client has to explicitly call create_subshell_request to create a subshell, so it could simply store its own mapping of subshell ID to name.
   • Kernel knows nothing about subshell name.

Both sound reasonable but I lean towards option 2. With 1 am concerned about requirements creep here (and in all other details of subshells!) pushing down the the kernel level. For example, if a client has set a name then it is reasonable to ask for the functionality to change it. So then we need a change_subshell_name message or equivalent. Then we think as we are displaying the name in a UI it would be nice to support HTML rather than text. Or an animated SVG file. Or a set of different renderable/MIME types to show depending on what the UI supports. And does a name have to be unique within a kernel? With option 2 these are all the responsibility of the client.

[jasongrout-db]

I just noticed there are some assumptions in python around the main thread and signals:

Python signal handlers are always executed in the main Python thread of the main interpreter, even if the signal was received in another thread. This means that signals can’t be used as a means of inter-thread communication. You can use the synchronization primitives from the threading module instead.

Besides, only the main thread of the main interpreter is allowed to set a new signal handler.

I was thinking that the main thread would be routing shell channel messages to subshell threads, one of which would be started by default, but it sounds like instead the main thread should be executing user code, and a separate thread should be routing shell channel messages, so that user code that sets signal handlers still works.

I think this also has implications about interrupting subshells, which currently uses sigint to trigger a KeyboardInterrupt exception.

[jasongrout-db]

Also, I mentioned recently in our kernels dev meeting that we've been dealing with some issues in the ipython post_execute hooks executing on the control thread. I think we'll need to clarify how those execution hooks (pre_execute, post_execute, pre/post_run_cell, etc.) behave for subshells. I bet there are a lot of ipython execute hook callbacks out there that currently assume single-threaded execution (like the matplotlib display hooks I mentioned in last week's meeting).

[ianthomas23]

I was thinking that the main thread would be routing shell channel messages to subshell threads, one of which would be started by default, but it sounds like instead the main thread should be executing user code, and a separate thread should be routing shell channel messages, so that user code that sets signal handlers still works.

I agree. I made the same original assumption in the demo code that the main thread would handle shell messages and each subshell (parent or child) would run in a new thread, as that was the easiest way to implement the demo. But as you say the parent subshell needs to run in the main thread (for signal handling, GUI event loop support, etc, etc) so the message handler and each child subshell will be in new threads.

[ianthomas23]

Also, I mentioned recently in our kernels dev meeting that we've been dealing with some issues in the ipython post_execute hooks executing on the control thread. I think we'll need to clarify how those execution hooks (pre_execute, post_execute, pre/post_run_cell, etc.) behave for subshells. I bet there are a lot of ipython execute hook callbacks out there that currently assume single-threaded execution (like the matplotlib display hooks I mentioned in last week's meeting).

For the favoured kernel subshells implementation this is an example of how the subshell_id will necessarily appear in messages and hook callbacks, and clients will have to choose if and/or how to use that information. In the same way that an execute_reply_message will contain a subshell_id, so will the info passed to a post_run_cell hook. If a client needs different behaviour based on the subshell triggering the hook, it will have to to do itself. If I understand post_run_cell hooks there is no such information passed so we cannot add a subshell_id, which may be problematic.

[jasongrout]

If I understand post_run_cell hooks there is no such information passed so we cannot add a subshell_id, which may be problematic.

I think you're right - I think we'd need a different hook to run on non-main subshell executions. We shouldn't change the single-threaded semantics of the existing hooks.

[jasongrout-db]

@ianthomas23 - are the instructions above at #91 (comment) still the most current instructions for trying out subshells?

CC @ryan-wan

[ianthomas23]

@ianthomas23 - are the instructions above at #91 (comment) still the most current instructions for trying out subshells?

Yes. I have not rebased any of the branches since then.