Access content of individual files in the CAR stream fetched via Lassie

[bajtos]

This is a follow-up for #243, see also filecoin-station/roadmap#19

Lassie supports only one response format: CAR file. We should enable modules to work with the data stored inside this CAR file.

For example, if the CAR file contains UnixFS with a single file (e.g. https://bafybeib36krhffuh3cupjml4re2wfxldredkir5wti3dttulyemre7xkni.ipfs.dweb.link), then a module should have a way how to obtain the content of that file:

My most famous drawing, and one of the first I did for the site

There are different ways how to approach this.

1. We can leverage IPFS packages for the browser to load the CAR file into an in-memory block store, interpret it as UnixFS and export file content.

   • https://github.com/ipld/js-car
   • https://github.com/web3-storage/ipfs-car
   • https://github.com/web3-storage/ipfs-car/blob/main/cmd/unpack.js
   • https://github.com/ipld/js-unixfs

2. Loading large CAR files into an in-memory block store is inefficient when the content is large. Zinnia should provide a built-in block store backed by the filesystem (persisted in a file stored in the CACHE DIR), including APIs to load CAR streams into this block store in a streaming way. Related: Storage API #192

3. Since Zinnia does not support 3rd party modules yet (see Import 3rd-party dependencies #209), I think we should either work on that story to allow modules to leverage existing JS packages for handling UnixFS content or else add a built-in version of UnixFS. This built-in version can be a vendored bundle of an existing JS package or a think JS API backed by Rust UnixFS implementation.

   Unfortunately, it seems that all Rust UnixFS crates have been abandoned by now. 😢

Based on the above, I think we should start with the first option - find a way to leverage existing JS browser-compatible tooling for both UnixFS and CAR loader/store. Then we can replace the JS version of CAR loader/store with a built-in Zinnia block store & CAR loader. Finally we can look into improving DX of using JS UnixFS.

[juliangruber]

For the first step, would that be to vendor the packages into zinnia, write a tutorial on how to vendor them into a module, or implement 3rd party dependency import?

[bajtos]

For the first step, would that be to vendor the packages into zinnia, write a tutorial on how to vendor them into a module, or implement 3rd party dependency import?

I prefer to write a tutorial or proper documentation on how to vendor 3rd-party packages into a module.

There is a limit on how many packages we can bundle into zinnia and still keep well-maintained.

I had a good experience with deno bundle (we are using it for zinnia:assert), but that command has been deprecated by Deno. We can look into other options out there, e.g. esbuild, swc or even rollup.