Parameterized kernel specs proposal

[hbcarlos]

Parameterized kernel specs

In this JEP we propose to parameterize the kernel specs, simplifying the way some kernels are installed reducing the amount of kernel specs files, and at the same time improving the UI of some of the Jupyter front-ends.

Co-author: @SylvainCorlay

Contributors that may be interested in this topic from past conversations:

• @ellisonbg
• @jasongrout
• @rgbkrk

[dhirschfeld]

I imagine @kevin-bates would be interested in this too.

[blink1073]

@echarles has been exploring similar ideas in https://github.com/deshaw/ksmm

[echarles]

There also is a issue opened in jupyter-server team-compass jupyter-server/team-compass#16 to give connect to people interested in this feature.

So basically, you are adding the following stanza to the kernelspec?

  "metadata": {
      "parameters": {
        "cpp_version": {
          "type": "string",
          "default": 'C++14',
          "enum": ['C++11', 'C++14', 'C++17'],
          "save": true
        }
      }
    },
}

[echarles]

... and that stanza is a json-schema.

For KSMM, we give the option to define env var (see https://raw.githubusercontent.com/deshaw/ksmm/main/screenshots/general_settings_ss.png). I read you say ...lists (respectively the command-line arguments and environment variables)... but I don't see example for such env var definitions.

[kevin-bates]

Thanks for the ping @dhirschfeld and reference to the team-compass issue @echarles.

@hbcarlos - this is great - thank you for opening this issue!

I'm really hoping this proposal can be expanded a bit to also include provisioner_parameters (and perhaps rename the parameters stanza to kernel_parameters) since this proposal appears to focus only on kernel-based parameters. For reference, here's the originally proposed JEP that I closed due to dormancy: https://github.com/kevin-bates/enhancement-proposals/blob/4c0727fbe03653aba789de67c388e2d28cda4cb9/parameterized-launch/parameterized-launch.md. It includes both schemas.

I would love to participate in any way possible. Thank you for opening this.

[SylvainCorlay]

For KSMM, we give the option to define env var (see https://raw.githubusercontent.com/deshaw/ksmm/main/screenshots/general_settings_ss.png). I read you say ...lists (respectively the command-line arguments and environment variables)... but I don't see example for such env var definitions.

Exactly, the env list of the kernelspec can also have reference to parameters as said in the proposal, although the example that we included does not make use of it.

E.g.

{
  "display_name": "C++",
  "argv": [
      "/home/user/micromamba/envs/kernel_spec/bin/xcpp",
      "-f",
      "{connection_file}",
      "-std={parameters.cpp_version}"
  ],
  env: [
    "XEUS_LOGLEVEL={parameters.xeus_log_level}"
  ],
  "language": "C++"
  "metadata": {
      "parameters": {
        "cpp_version": {
          "type": "string",
          "default": "C++14",
          "enum": ["C++11", "C++14", "C++17"],
          "save": true
        },
        "xeus_log_level": {
          "type": "string",
          "default": "ERROR",
          "enum": ["TRACE", "DEBUG", "INFO", "WARN", "ERROR", "FATAL"],
          "save": true
        }
      }
    },
}

[hbcarlos]

I would love to participate in any way possible. Thank you for opening this.

Thanks, @kevin-bates! Help is always welcome!

I'm really hoping this proposal can be expanded a bit to also include provisioner_parameters (and perhaps rename the parameters stanza to kernel_parameters) since this proposal appears to focus only on kernel-based parameters.

This proposal allows specifying parameters independently of how they may be used (as environment variables, command-line arguments). What we are trying to achieve on this JEP, is to include and formalize the concept of parameters in the kernel specs but not the semantics, so that developers can use these parameters for any purpose (provisioner parameters, environment variables, etc.). I'll update the proposal to make it more clear.

[kevin-bates]

This is a great start - thank you.

I'm not sure if this is necessary here but should we discuss where/how these parameters will be applied when, and when not, provided in the start request payload.

I've also been thinking about how the JSON schemas get into existence in the first place. Yes, we definitely need to support their presence in the kernel.json file (and I know @SylvainCorlay has brought up having a separate "sibling file" for the schema), but I'd also like to see a "query capability" whereby the KernelSpecManager asks the configured kernel provisioner "give me your parameter schema". This would allow the provisioner's current environment to perhaps be utilized to come up with the schema. For example, if memory were a parameter that the provisioner applies, then the provisioner could (hypothetically) detect available memory and use some heuristic to determine the range of memory values applicable to that configuration. Even if the "query" returns static information, it would go a long way to ease the kernelspec deployment burden on operators.

Of course, what is returned by the KernelSpecManager is the (possibly merged) result of these interactions, so perhaps we could drive toward that later - we should just try to make sure we don't preclude us from having this option.

(The KernelSpecManager could do the same kind of query against the underlying kernel, although that may be tougher because I'd like to see a given kernel spec have language as a parameter, where the kernel provisioner (or certain provisioners at least) support multiple kernel types. This may be pie-in-the-sky kind of stuff, but dynamic kernelspecs is where we're heading.)

[kevin-bates]

I think, at a minimum, we need to require default values for all required parameters and the "source" (i.e., kernel or kernel provisioner) should have a reasonable default for others. Of course, having a default for each parameter would be a "best practice".

Where the default values probably fall down are for things like credentials, but I would argue those kinds of things are probably best retained in a backing store (e.g, configuration file or database), where any inputted values would override the persisted values.

I guess the point is that we don't want to abandon all the jupyter_client-based applications that don't have the ability to present a JSON schema for parameter selection - so some form of reasonable defaulting is necessary.

[rgbkrk]

As much as possible we need to be backwards compatible with existing kernel spec users so I'm in agreement with @kevin-bates on default values. Maybe it's fine to let the kernel fail if it's missing a parameter (or if the parameter is invalid).

[kevin-bates]

Hi @hbcarlos,

I'm not sure where this JEP stands but I'd like to try to revive it a bit. I think we should discuss some of the issues I raised in the previous comment, namely parameter discovery (which, at the time, I had referred to as "query capability").

While I think having static parameters reside in the kernel.json file is sufficient, the KernelSpecManager should be capable of asking the configured kernel provisioner (including the default LocalProvisionser when one is not specified directly in kernel.json) to provide its parameter schema.

Moreover, this interaction should pass the kernel.json to the provisioner and, using a kernel hint located in the kernel.json, the provisioner would then, similarly, ask the kernel "parameter provider" for its parameter schema. The provisioner would then return both parameter schemas which the KernelSpecManager would reconcile with the existing static parameters to produce an "in-memory" kernel.json file that is then returned to the application asking for this information. In essence, any static parameters would override those returned from the provisioner parameter provider.

By introducing this discovery mechanism, existing kernel.json instances would not have to be overwritten when new versions of kernels are deployed that contain an updated set of parameters. (We'd need to work out what this would mean if a given kernel/provisioner parameter provider were to remove support for a parameter between versions. This makes me wonder if we should allow static parameter schema at all.) Likewise, until the discovered parameters are persisted as static parameters into the kernel.json file, no parameter schema would even need to necessarily exist.

Because a specific kernel package may not be necessarily installed locally, I was thinking of exposing "parameter providers" via entry points. The previously referenced kernel hint would reflect the name of the kernel parameter provider that the provisioner calls to fetch the corresponding kernel parameter schema. This would allow kernel implementations to be decoupled from their parameter provider if that was advantageous. Similarly, we could have provisioner parameter providers, although since the provisioner must be locally available, the configured provisioner package could expose itself as a provisioner parameter provider.

For applications using KernelSpecManager for discovering available kernel options, we'd also want to institute a kernelspec cache so that this parameter discovery is minimized.

The KernelSpecManager that performs this discovery and reconciliation, could be a subclass of the existing manager, if we only wanted this capability in, say, Jupyter Server due to additional dependency issues, or introduce an optional installation dependency on jupyter_client with something like jupyter_client[caching] that includes the extra dependency (probably something akin to watchfiles).

I think by introducing discoverable parameter schemas we would essentially have "dynamic kernelspecs" which would be easier to maintain and be more backward compatible between releases (including those releases that predate parameterized kernel specs).

[davidbrochart]

@kevin-bates Things like "entry points", KernelSpecManager and "subclass" sound like implementation details to me, and I think they are out of the scope of this JEP, or JEPs in general, which should remain language-agnostic.

[hbcarlos]

Hi @kevin-bates,

Yes, I would like to revive this JEP as well. Give me a couple of days to get up to speed, and I'll get back to you.

[kevin-bates]

Things like "entry points", KernelSpecManager and "subclass" sound like implementation details to me, and I think they are out of the scope of this JEP, or JEPs in general, which should remain language-agnostic.

I understand. I just wanted to help paint a picture of how a discovery mechanism might be conceptualized. Somehow a link from the discoverer to the discoverable must be made and my terminology was something concrete that others can understand. I'll try to be more abstract in future responses.

[hbcarlos]

Thanks for the comments and the interest you have shown in this, @kevin-bates.

After looking into the Jupyter Client, Jupyter Protocol, the Jupyter Kernel management, and discussions and JEPs about the kernel discovery framework, parameterized kernel launch, kernel providers, etc. In my opinion, your comments in #87 (comment) are out of the scope of this JEP. They might take advantage of the parameterization of the Kernel Specs we propose here. In addition, Kernel Provisioners could be an excellent example to show the necessity of parameterized Kernel Specs.

I just wanted to clarify with you. In this JEP, we want to formalize the introduction of parameters to the Kernel Specs. We are trying to introduce parameters for everyone to take advantage of it, from provisioners to kernels. The idea is to make it as generic as possible to include every possible use case.

[kevin-bates]

Thanks for your response @hbcarlos.

I just wanted to clarify with you. In this JEP, we want to formalize the introduction of parameters to the Kernel Specs. We are trying to introduce parameters for everyone to take advantage of it, from provisioners to kernels. The idea is to make it as generic as possible to include every possible use case.

By this comment, it sounds like there will be pending updates to take into account parameters that are kernel-provisioner-relative - which is great. Thanks.

Will these be conveyed in separate stanzas so that things like discovery (and dynamic kernelspecs) could be implemented?

[kevin-bates]

Would it make sense to have a format_string meta-property so software that launches the kernel can simply build the rest of the argv stanza on the fly? Otherwise, the baked-in argv in the kernel.json dictates what parameters will be used, yet users may want to include others that are defined in the schema. For example, in this example, parameters xeus_log_level will never be included despite the user wanting to enable TRACE output.

Suggested change

	"save": true
	"save": true,
	"format_string": "-std={cpp_version}"

Then, only those parameters that have been provided by the user (or are required) would be included in the finalized argv list.

[hbcarlos]

The xeus_log_level parameter is used in the environment variable:

env: [
    "XEUS_LOGLEVEL={parameters.xeus_log_level}"
  ],

That's why we are trying to make it as generic as possible so we can use these parameters anywhere.

Maybe the format_string is helpful for flags or optional parameters. I don't have an example right now but imagine a kernel with an optional flag to activate or deactivate LSP (Language Server Protocol) features.

{
  "display_name": "C++",
  "argv": [
      "/home/user/micromamba/envs/kernel_spec/bin/xcpp",
      "-f",
      "{connection_file}",
      "{parameters.lsp}"
  ],
  env: [
    "XEUS_LOGLEVEL={parameters.xeus_log_level}"
  ],
  "language": "C++"
  "metadata": {
    "parameters": {
      "lsp": {
        "type": "string",
        "default": "True",
        "format_string": "--lsp",
        "save": true
      }
    }
  },
}

To be honest, I don't have a strong opinion on this. More people could chime in.

[kevin-bates]

The xeus_log_level parameter is used in the environment variable:

Ah - I'm sorry, I missed that. I figured environment variables would be classified in a different manner. In that vein,
I think there should be a means of adding environmental variables, free form, and those that are specified in the schema should be classified as environment variables to assist in UX. These are the kinds of things that integrations typically need and we should enable the ability to add any environment variable to the env of the kernel.

[kevin-bates]

Since these will include provisioner-relative parameters, I think it would be good to have separate kernel_parameters and provisioner_parameters stanzas - both for end-user applications and programmatic processing.

[westurner]

• Cdsdashboards does provisioning atop JupyterHub (~BinderHub but with parameters)

https://github.com/ideonate/cdsdashboards/blob/3d6cb7d4f60f82a7d366a8f439a57ab6f2479070/cdsdashboards/builder/processbuilder.py#L46-L91

README:

• User sees a safe user-friendly version of the original notebook - served by Voilà, Streamlit, Dash, Bokeh, Panel, R Shiny etc.
  All of this works through a new Dashboards menu item added to JupyterHub's header.

[westurner]

• jupyter-repodocker looks for REES config files like {requirements.txt, environment.yml, Dockerfile} in / and /.binder/ ; but there are also command line argument parameters:
  https://repo2docker.readthedocs.io/en/latest/usage.html#command-line-api

[hbcarlos]

Since these will include provisioner-relative parameters, I think it would be good to have separate kernel_parameters and provisioner_parameters stanzas - both for end-user applications and programmatic processing.

It is okay for me to distinguish between kernel and provisioner. Nevertheless, I believe we should not distinguish them because provisioner parameters will get a form for free once we implement the form for choosing kernel parameters.

For example, in JupyterLab, when a user selects the kernel, if there is a custom provisioned, they will see and be able to configure parameters like (CPU, GPU, and memory) from the same form.

[kevin-bates]

Nevertheless, I believe we should not distinguish them because provisioner parameters will get a form for free once we implement the form for choosing kernel parameters.

For example, in JupyterLab, when a user selects the kernel, if there is a custom provisioned, they will see and be able to configure parameters like (CPU, GPU, and memory) from the same form.

I understand, but users should know that certain parameters are relative to the provisioned environment while others are relative to the kernel. In addition, their metadata specifications in the schemas will be separated, so their values should remain separated as well.

If they don't separate them, then users will see completely different sets of parameters for the "same kernel" depending on the provisioner and (I believe) it makes sense logically to separate the two. That said, the UX can choose how these should be organized but at least they'd have that option if the two are separated in their schemas and submission values.

[SylvainCorlay]

I don't think that we need to give a specific semantics to the new kernel parameters, since they are completely generic and can have very different semantics depending on the use case (language version, connexion parameters, options, or anything else).

[westurner]

Specifically which things need to be parameterized at the kernel level?

"Reproducibility" is jeopardized if all parameters are not persisted for repeatability. Which additional files would then also be necessary to archive and distribute in order to reproduce the notebook output?

Variance in parameters like PYTHONHASHSEED and PYTHOPTIMIZE should also be isolated (for Python notebooks, for example). Should you also with this measure specify how users should document such non-kernel simulation parameters?

E.g cookiecutter has a config schema with default values IIRC, but it doesn't specify e.g. jsonschema or SHACL to validate runtime parameter datatypes or constraints, and I don't think autoescape=on is on for the templates because we trust developers who understand the code to not insecurely pass unescaped parameters to templates. Here, the potential vulnerabilities of additional parameters do include OS command injection (especially if the os commands are string-parametrized without something like sarge or subprocess.Popen(shell=False).

Users will want to pass urlargs to the kernel from URL arguments and HTML form data, but that runs ipykernel --$thing and thing=a;cp -Rv /etc /usr/local/etc

How do I determine whether or not an experimental outcome is sensitive to these new unspecified parameters?

What additional risks to reproducible science and users is posed by adding parametrization to os commands?

[westurner]

"Reproducibility" is jeopardized if all parameters are not persisted for repeatability. Which additional files would then also be necessary to archive and distribute in order to reproduce the notebook output

If defaults could change over time, the reproducible ScholarlyArticle author must persist and archive at time t and publish the default values that were specified at that time, too

[hbcarlos]

Specifically which things need to be parameterized at the kernel level?

It is up to the kernel. We just offer the possibility of having parameters.

"Reproducibility" is jeopardized if all parameters are not persisted for repeatability. Which additional files would then also be necessary to archive and distribute in order to reproduce the notebook output?

That is addressed in this JEP. We talk about adding a particular attribute "save": true, to indicate whether that parameter should be saved in the notebook's metadata.

Here, the potential vulnerabilities of additional parameters do include OS command injection (especially if the os commands are string-parametrized without something like sarge or subprocess.Popen(shell=False).

We could do a sanity check before launching the kernel.

Users will want to pass urlargs to the kernel from URL arguments and HTML form data, but that runs ipykernel --$thing and thing=a;cp -Rv /etc /usr/local/etc

The user can now open a terminal and run cp -Rv /etc /usr/local/etc.