Layout for co-existing SEA resources

[RaisinTen]

Now that we have an initial implementation of SEA in Node.js, we should decide how we're gonna embed other resources for the SEA like CLI arguments, snapshot blob (PR - nodejs/node#46824), etc.

I really like the idea that @joyeecheung suggested in nodejs/node#46824 (comment):

Maybe a better workflow would be to add some tooling to process the inputs (JS bundle, snapshots, any other resources we might want to support in the future, like VFS) and generate a blob whose format is controlled by us (and entirely internal so we don't need to worry about layout breakages). Then users can inject that blob into our specified section. Maybe something like this (pardon my naming suggestions):

# User users can configure JS bundle, snapshots, VFS etc. in config.json and we copy/bake whatever we need into the blob
$ node --sea-config config.json --sea-prep-blob sea-prep.blob
$ postject sea NODE_SEA_RESOURCE sea-prep.blob --sentinel-fuse NODE_SEA_FUSE_fce680ab2cc467b6e072b8b5df1996b2 --macho-segment-name NODE_SEA
$ ./sea

I wonder if anyone else has any better ideas.

If we're all good with the suggested approach, we should decide on the file format of the config.json file and the sea-prep.blob file.

cc @nodejs/single-executable

[joyeecheung]

Another thing this unblocks is compression - I've doing some work (slowly) to port JS2C to C++ in order to compress the sources using internal libraries (instead of relying on whatever python offers). When we have decent compression support for internal resources (like source code, snapshot and code cache) in core we can easily reuse that for SEA too.

[RaisinTen]

@joyeecheung any thoughts on where the tooling should exist? Do you think it should be inside node or are you okay with having it as an external npm package (which might potentially be maintained in the nodejs org)?

[joyeecheung]

I think it would be better to keep the part that emits the blob internal during development at least - otherwise whenever we have a breaking change of the layout the breakage needs to be reflected in the npm package, and I suspect breakages would be frequent while SEA is under development and we probably would not want to rush into specifying the layout too early.

[mhdawson]

My 2 cents.

I do think that defining a generic format that we can use to embed different things while using the same tools to find/read etc. make sense. My initial thought is still that the tool to embed it is better outside of the node binary itself for the same reasons we talked about for SEA. The part which finds the blob, figures out what type it is etc. can likely be common code in Node.js itself and probably not too much different to start from what we have today, except that it would filter based on type. It might also makes sense to have a single pass where it finds/stores info for all injected segments and provide an API where Node.js code can ask for a segment of type X. For example the existing SEA code could ask for the application segment if there is one.

In terms of generating the content for the blob, if that is using snapshots then it makes sense for the generation to be done by the node binary, but injection into the binary, signing etc I think could still be in an external tool. In terms of changes and syncing the two I think we should have a versioning mechanism in place so that support for a new format can be added without breaking existing workflows and make it easier to sync those changes between Node.js and whatever tool is used for the injection.

[joyeecheung]

Another idea is we can have some internal/undocumented CLI arguments (e.g. the badly named --sea-config and --sea-prep-blob I mentioned in the other thread - if they are undocumented we don't have to feel so bad about bad names :P) in the Node.js binary for generating the blob, and on top of that we can have a one-line command that copies a Node.js binary, generates the blob using a configuration file, and injects the blob to produce the SEA, something like this:

$ node-sea --config sea-config.json --binary /path/to/node --output /path/to/sea

The node-sea command can invoke node (with appropriate flags to generate the blob) and postject (or any other similar tools, to inject the blob) under the hood. It may also attempt to perform the code signing for macos, etc. Then this can be an npm package itself.

[RaisinTen]

If the --sea-config and --sea-prep-blob CLI args are not documented anywhere, it wouldn't be possible to describe the steps for creating an SEA in https://nodejs.org/api/single-executable-applications.html without making the usage of the node-sea NPM package and postject (via node-sea) mandatory. This is something people wanted to avoid.

Maybe we can start off with names that are appropriate to the best of our knowledge and change them when the need arises? I think it should be possible to change the names relatively easily given how experimental SEAs are right now.

We could also prefix the CLI options with experimental (--experimental-sea-config and --experimental-sea-prep-blob) for now to make sure that users are aware of the chances of the options being renamed later on?

[joyeecheung]

experimental flag STGM. Actually I wonder if we need the --experimental-sea-prep-blob flag at all - I think this could also just be a field in the configuration, then we don't have to add another obscure flag into the CLI soup ;)

[RaisinTen]

Yea, good idea!

[joyeecheung]

I did nodejs/node#47125 to test out the ideas discussed so far. I think we can start by changing the interface of SEA (JSON config + blob content), and then tweak the blob layout and add more things to the blob. We might have to do some internal refactoring in the process e.g. better file reading and writing utils in C++, and make the snapshot serializer/deserializer more generic, etc. I also notice that we might want to trim the snapshot format for SEA a bit e.g. the metadata checks can be done together with the SEA checks, so there is no need to duplicate the metadata in the snapshot, though that's not a lot of overhead so it can be done as a low-priority cleanup.

[joyeecheung]

Another question I have while rebasing nodejs/node#46824: do we want the snapshot in SEA to be built specifically for SEA? For now it's possible to load snapshots in SEA even if the snapshot is not specifically built for SEA, but it might be useful in the future - I imagine this might be helpful if we happen to have to fix up the application state/code cache specifically for SEA.

For example do we want the config file to look like this:

{
  "snapshot_blob": "/path/to/snapshot.blob"
}

where the blob needs to be built by a node --build-snapshot ... command first, or do we want

{
  "snapshot_script": "/path/to/snapshot/entry.js"
}

where we will just take entry.js and build the blob ourselves. Personally I think the latter is just more user-friendly, and the chance of needing the former while the latter is already available might be very thin. And if we only have the latter, starting from requiring the snapshot to be built for SEA seems safer and we could relax the restrictions if that turns out to be unnecessary.

[RaisinTen]

Is it going to be accessible through another build flag made specifically for SEAs?

Do you mean we are adding another build option for SEA?

When you said "lighter-weight version of the --node-snapshot-main build option", I thought you were talking about introducing another option that could be passed to ./configure which would be similar to --node-snapshot-main but would have some SEA specific attributes.

I think in that case having a configurable script would be better than having a configurable blob

I fully agree.

Although I am not sure if "compiling Node.js from source" falls in the scope of SEA...

If you see https://github.com/nodejs/node/blob/main/doc/contributing/maintaining-single-executable-application-support.md#compile-with-nodejs-into-executable, the previous understanding was that this approach wouldn't require any additional code changes but if someone is willing to compile node from source using --node-snapshot-main to generate the binary with the replaced snapshot, I think injecting the source into this new binary too should work out of the box, right?

In terms of snapshots specifically, compiling the binary with --node-snapshot-main means the binary is replacing the Node.js default snapshot with a customized one.

Would it be possible to place the Node.js default snapshot in a separate section / segment / resource? If so, I think it might be possible to replace that with the runtime snapshot using postject or any such tool. If it's possible, would both the runtime snapshot and the build time one be equivalent to each other?

Is that going to change once nodejs/node#44014 gets completed?

I don't think so, that's something separate.

Okay, I thought the objective of the runtime snapshots was to do the same thing that --node-snapshot-main does but at runtime instead of compile time, so that building from source isn't necessary any more if someone wants to use snapshots.

[joyeecheung]

I thought you were talking about introducing another option that could be passed to ./configure which would be similar to --node-snapshot-main but would have some SEA specific attributes.

By "lighter-weight version of the --node-snapshot-main build option" I mean that with SEA + snapshot support, the user can build the snapshot into a single binary without having to build Node.js from source using --node-snapshot-main. When using --node-snapshot-main the user does not have to know anything about the node_snapshot.cc (which contains the snapshot data), they only need to care about the snapshot script, so similarly I think with the SEA toolchain the user should not need to know about the existence of a blob (that contains the snapshot data) either, it should "just work" with the tools we provide.

but if someone is willing to compile node from source using --node-snapshot-main to generate the binary with the replaced snapshot, I think injecting the source into this new binary too should work out of the box, right?

It depends on what's in that snapshot - if the user already configures an main function (with v8.startupSnapshot.setDeserializeMainFunction()) in the snapshot script then I think we should throw an error (either at build time or at runtime), since obviously an application is not supposed to have two main functions. If there's no configured main function in the snapshot then yes it should just work as usual (like how currently the SEA is already running on top of the default Node.js snapshot).

Would it be possible to place the Node.js default snapshot in a separate section / segment / resource? If so, I think it might be possible to replace that with the runtime snapshot using postject or any such tool. If it's possible, would both the runtime snapshot and the build time one be equivalent to each other?

It is possible for some platforms, but I'm not sure if we can do that for all the platforms that snapshot is currently available. In addition the blob format needs to be deserialized (with copying) and checked before we can use it to create a Node.js instance, which comes at a cost, whereas the current build-time option simply generates C++ code that initializes SnapshotData with static data, which is going to be faster and incurs less copying. I think a 3.7MB size increase in SEA is probably worth it if we can avoid a penalty on the default binary...?

[RaisinTen]

I think a 3.7MB size increase in SEA is probably worth it if we can avoid a penalty on the default binary...?

Yea, I too agree.

---

Okay. So IIUC, if we go ahead with the approach where the user would just specify the entry.js path in the SEA config, this is what the workflow would look like:

• Node.js is going to parse the path of the entrypoint js file from the config and internally build the snapshot blob (could you explain how this part would happen? does it require compiling any C++ sources, like an intermediate node_snapshot.cc file made for SEAs?) and create the final blob, which would contain the contents of the entrypoint js file and the snapshot blob.
• Then users are going to create the SEA by using postject (or some other tool) to inject the blob created in the previous step into the binary.
• When the SEA is executed, node would parse out both the entrypoint JS file contents and the snapshot blob from the injected resource during startup, initialize SnapshotData with the snapshot blob and then run the entrypoint JS file.

Does this look correct?

[joyeecheung]

does it require compiling any C++ sources, like an intermediate node_snapshot.cc file made for SEAs?

No, for runtime-generated SEAs we can reuse the code for the --build-snapshot CLI flag to generate a snapshot and serialize it as a binary blob, which Node.js already knows how to deserialize at runtime. The C++ source route is only used for binaries compiled from source.

which would contain the contents of the entrypoint js file and the snapshot blob

Yes. Although specifically for the entrypoint JS file I think we can even just make it part of the snapshot (pre-compiled). But the general idea is that all the resources would go into one final blob, and it gets injected into the SEA. Whether the entrypoint JS is stored inside the snapshot or alongside the snapshot is a separate optimization (maybe configurable, even).

Then users are going to create the SEA by using postject (or some other tool) to inject the blob created in the previous step into the binary.

Yes, sort of - I think with the node-sea command idea above the user probably don't even need to know about postject. They only need to do:

1. Write the json configuration config.json, which for example contains the path to the snapshot script.
2. Do node-sea --config config.json --binary /path/to/node --output /path/to/sea
3. And voila, /path/to/sea contains the SEA

And in step 2, the node-sea command (maintained by us as a separate npm package) essentially does all these under the hood:

$ cp /path/to/node /path/to/sea
$ /path/to/node --sea-config config.json --sea-prep-blob /tmp/sea-prep.blob
$ postject /path/to/sea NODE_SEA_RESOURCE sea-prep.blob --sentinel-fuse NODE_SEA_FUSE_fce680ab2cc467b6e072b8b5df1996b2 --macho-segment-name NODE_SEA
$ codesign --sign - /path/to/sea # for macOS
$ rm /tmp/sea-prep.blob

The user does not need to know about the existence of the blob, or sections, or the fuse, or postject. That would only be useful information when they run into issues, but generally they shouldn't need to concern themselves with these details.

When the SEA is executed, node would parse out both the entrypoint JS file contents and the snapshot blob from the injected resource during startup, initialize SnapshotData with the snapshot blob and then run the entrypoint JS file.

Yes.

[RaisinTen]

I think this approach makes sense.

[RaisinTen]

Any thoughts on what the file format of the injected SEA resource blob should be like? Should we create our own custom format or is anyone aware of an existing file format that might work for our use case?

Our use case for now is to be able to store some kind of a map but it doesn't have to be a map really because of these reasons:

• there is no nesting
• it has purely string values
• the list of possible keys are fixed and known in advance (so these can be translated to numbers (or indices) under the hood)
• parsing the strings from the file should be very efficient because that's going to happen during the application start up phase, so probably no compression
• random access isn't needed because the entire file contents would be parsed into memory anyways during start up
• the serializer and deserializer implementations shouldn't have a notable impact on the Node.js binary size

Let me know if I missed something.

I think this is what an appropriate file format would look like:

unique magic bytes
size of value at index 0
value corresponding to index 0
size of value at index 1
value corresponding to index 1
size of value at index 2
value corresponding to index 2
...

Here, all the size of value at index x values will have a fixed size (4 bytes?).

[joyeecheung]

I think we can probably just refactor the SnapshotDeserializer and SnapshotSerializer to serialize/deserialize the blob for SEA (most of the code there doesn't have much to do with snapshot anyway, they are mostly generic reader/writer of strings/numeric types/vectors). Currently the snapshot serializer and deserializer simply make assumptions about the blob layout and isn't very different from the format mentioned above (e.g. when reading a vector it reads the size first and then read the elements). I think the overall layout could be:

[ 4 bytes ] sea::kMagic
[ ? bytes ] metadata header of SEA
[ ? bytes ] entry point JS code, as UTF-8 string (the metadata will indicate its existence)
[ ? bytes ] snapshot blob (the metadata will indicate its existence)
[ ? bytes ] ? other resources we might want to add later

We already have a specific layout for the snapshot blob here: https://github.com/nodejs/node/blob/8713c83462df192dbf319e4ac0c2640b3a8cfff7/src/node_snapshotable.cc#L834

[joyeecheung]

I opened nodejs/node#47458 for splitting the blob serializer/deserializer, should be easy to build a SEA blob serializer/deserializer on top of it that supports the layout described above. Will rebase nodejs/node#46824 on top of that.