Forest executables structure

[LesnyRumcajs]

Let's discuss how we want to see the Forest structure in the long run. The main goal is to have a solid architecture that wouldn't require many breaking changes once it is set; we don't like breaking existing workflows or constantly memorising "new ways" of doing the same thing.

Other things to highlight:

• clear separation of responsibilities,
• some degree of backward compatibility,
• facilitation of testing,
• potential "specialisations" across the team, which could give a pleasant feeling of ownership,

Current proposal sketch that we can improve together:

In this proposal, we have four executables:

• forest / forestd - the primary node.
  • no subcommands; execution means spawning a long-running process.
• forest-cli / forest-client - the program that connects to the node and communicates with it via RPC.
  • not allowed to have any direct operations on the database or filesystem. Every subcommand there must communicate via RPC.
• forest-tool - aggregate for all other commands we consider helpful but don't require having a running node.
  • current candidates in terms of functionality: snapshot, db.
• forest-wallet / forest-account - per @q9f suggestion, all operations related to signing.
  • currently sitting in forest-cli wallet
  • The only binary that communicates with the keystore and can modify it.
  • this one needs more care and an additional design process, plus at least a rough security audit.
  • the keystore is separated from the database directory and lives elsewhere.

To address the unresolved configuration discussion, here are my proposals regarding this:

• get_configuration is implemented in forest and exposed via RPC,
• forest-cli is not aware of the specific structure behind the node configuration - this ensures backward compatibility in case of a single changed parameter,
• forest-cli config --node dump asks via RPC the forest to send its serialized configuration.
• forest-cli config --client dump dumps the client configuration file.
• we could think of having some parameters modifiable in runtime, for example, dynamically setting log level via forest-cli, but that's something for the future.
• binaries should be free to have their configurations (and the ability to load them from files). They should be separated, though. For forest, it's a no-brainer; for forest-cli I believe it may be helpful (i.e. for JWT, node address and logging params).

Let's discuss this, reach a consensus and then start implementing it!

[lerajk]

@LesnyRumcajs this look great. Really awesome.

Would be good to see a sketch or explanation of our current architecture. This way we can scope and estimate the tasks involved for the more leaner proposed model. We can also magnify / point out the bottlenecks of the current architecture.

[LesnyRumcajs]

My understanding of the current architecture is as follows:

The pain points we would like to address:

• tight coupling between the node, node client and the key management (including a shared directory tree). Ideally, they should be separated. On top of security concerns, this would facilitate testing a lot,
• responsibilities sprawl - forest-cli is doing too many things,
• code mixup - because of a lack of clear boundaries, we're getting issues like Incorrect CLI usage flags for forest and forest-cli #2076, where we incorrectly use the same code. And it is not trivial to untangle.

[lerajk]

Thanks for theses diagrams @LesnyRumcajs, clarifies a lot of pain points. Ya, definitely a lot of coupling at the moment which we need to address.

• it seems like the CLI is overburdened because a lot is going through it
• the data layer definitely needs to be a separate component, it is currently mixed up with keystore (which is a major pain point)
• the current architecture is not very lean or modular, that's why we run into debugging fiascos

Your proposed architecture is a very good start, we should discuss that more. I'll post some questions after some thinking.

[lerajk]

ok, gonna be asking a few not so smart questions as follows:

• How heavy is the RPC API at the moment. What are the core commands? From the current architecture, it seems like there is only one bridge between forest-cli and forest-node operations. Thinking if we can spilt RPC API operations.
• I'm guessing the forest-node is the one communicating with the underlying libp2p and Filecoin network? Trying to understand this a bit more. Some more explanations would be good.
• Why the forest-node doesn't handle both rocks-db and snapshots operations? This way the entire data layer is the node component's responsibility. Currently. forest-cli does this for convenience I suppose?

Some problems that I see with the current architecture is:

• forest-node is responsible for keystore
• RPC commands may be over overburdened through CLI
• snapshots are isolated from the data directory. May not be important to group it with the data directory though

[LesnyRumcajs]

How heavy is the RPC API at the moment. What are the core commands? From the current architecture, it seems like there is only one bridge between forest-cli and forest-node operations. Thinking if we can spilt RPC API operations.

Not heavy enough. Ideally (at least in my mind), we should have 100% in https://github.com/ChainSafe/forest/blob/main/API_IMPLEMENTATION.md

This would potentially allow some forest and lotus interoperability. I'm not convinced it's a valuable case, though. Perhaps it is.

What do you mean by splitting RPC API operations?

I'm guessing the forest-node is the one communicating with the underlying libp2p and Filecoin network? Trying to understand this a bit more. Some more explanations would be good.

Yes, only the node communicates with the Filecoin network. Other components don't talk with any peers.

Why the forest-node doesn't handle both rocks-db and snapshots operations? This way the entire data layer is the node component's responsibility. Currently. forest-cli does this for convenience I suppose?

For the forest-cli, the only DB operation we currently have is purging it. We could do it via an RPC call, but it would introduce non-trivial logic (stopping all tasks, clearing the DB, and exiting cleanly). Plus, it's helpful to purge the DB before starting the node. All in all, it's just a wrapper for a crude rm -rf DB ;). So definitely a candidate for forest-tool.

For the snapshots, the diagram is a bit misleading, sorry. The node can download to the snapshots directory. It's a valuable feature recently introduced to automatically download snapshots (and not do it manually in a previous step). But it's mainly the forest-cli that does more sophisticated tasks there:

    clean     Clean all local snapshots, use with care
    dir       Shows default snapshot dir
    export    Export a snapshot of the chain to `<output_path>`
    fetch     Fetches the most recent snapshot from a trusted, pre-defined location
    list      List local snapshots
    prune     Prune local snapshot, keeps the latest only. Note that file names that do not match
              forest_snapshot_{chain}_{year}-{month}-{day}_height_{height}.car pattern will be ignored
    remove    Remove local snapshot

[lerajk]

Not heavy enough. Ideally (at least in my mind), we should have 100% in https://github.com/ChainSafe/forest/blob/main/API_IMPLEMENTATION.md

Ya ok 72 is not that heavy, even 173 is not that bad (?). I imagined there were like over a few hundred commands. We may not need to complete all 173 because our use case will be different compared to Lotus. But, I understand the temptation from compatibility reasons.

What do you mean by splitting RPC API operations?

Part of not so smart question, so ignore it 😂

For the forest-cli, the only DB operation we currently have is purging it.

Yes ok, understood now.

The node can download to the snapshots directory. It's a valuable feature recently introduced to automatically download snapshots (and not do it manually in a previous step). But it's mainly the forest-cli that does more sophisticated tasks

Ya, this one is pretty great added feature.

[LesnyRumcajs]

Old sketches

[LesnyRumcajs]

For now, it'd be just a dump for other tasks that do not meet the requirements for being forest-cli (communication via RPC with forest, node online required), forest-wallet (authorization, keys management) and forest (being a part of the node). Perhaps in time, it will not be enough and forest-tool will have to split, but I also would like to avoid over-engineering and putting abstractions that we will most likely not use. What do you think @lerajk?

[lerajk]

Ya got it, make sense for now. I think forest-wallet and forest-cli separation is the next big step. Let's talk a bit more about the forest-cli actually.

• How is the CLI behaving now, what / how many commands do we have for it.
• What do we want the CLI to be aka how do we make it more lean? thoughts & ideas?

Also, can you elaborate more on what the forest-node component?

[LesnyRumcajs]

The commands categories that we currently have for forest-cli are

SUBCOMMANDS:
    auth            Manage RPC permissions
    chain           Interact with Filecoin blockchain
    config          Manage node configuration
#    db              Database management
?    fetch-params    Download parameters for generating and verifying proofs for given size
?    genesis         Work with blockchain genesis
    mpool           Interact with the message pool
    net             Manage P2P network
    send            Send funds between accounts
#    snapshot        Manage snapshots
    state           Interact with and query Filecoin chain state
    sync            Inspect or interact with the chain synchronizer
$    wallet          Manage wallet

where # are the commands that should be potentially moved to either forest-tool, $ forest-wallet and ? probably forest-tool, but I am not entirely sure.

I see forest-cli as a client for forest. Perhaps forest-client is a better name in this case. In my mind, the commands and binaries should be limited so that pairs forest-node+forest-tool and forest-client+forest-wallet could live on entirely separate machines, or you could even have multiple clients+wallets.

Does it answer your question @lerajk ?

[lerajk]

So, it is not clear to me on what basis we are separating the forest-client and forest-tool commands.

• forest-node - spans the process, manages congif, communicates with the Filecoin network
• forest wallet - manages keystore, FILs
• forest-client - authentication, account management, interact with the Filecoin network and node operations management?
• forest-tool - data management ? What else this is for?

so that pairs forest-node+forest-tool and forest-client+forest-wallet could live on entirely separate machines, or you could even have multiple clients+wallets.

definitely lost me here 😂 Elaborate with an example?

[LesnyRumcajs]

@lerajk I'm so sorry. I missed the question amidst the crazy amount of notifications in December. I owe you a cookie.

I see forest-cli as a CLI frontend for forest-node. So whatever forest-node exposes as an RPC interface should be available in forest-cli. In ideal world, forest-cli could be automatically generated from a YAML definition, like OpenAPI, but it will most likely stay in the world of dreams. In other words, if a feature needs a running node it should be in forest-cli. Otherwise, it can be in forest-tool. Does it make sense?

so that pairs forest-node+forest-tool and forest-client+forest-wallet could live on entirely separate machines, or you could even have multiple clients+wallets.

definitely lost me here joy Elaborate with an example?

Say we have a solid Forest instance in production. That would be our backend. We could have multiple frontends calling that one instance, each with a different set of keys. If it is not separated, each frontend would require running a massive Forest instance. I believe this approach is a step towards the light client. What do you think?

[q9f]

forest-wallet / forest-account - per @q9f suggestion, all operations related to signing.

might be clear but just to be safe: keys should not be saved in the same directory tree as the database

[LesnyRumcajs]

I tried to highlight this with solid lines, but it's a good point; I will write it down as well, so there's that this is clear. Thanks.

[lerajk]

Let's discuss how we want to see the Forest structure in the long run. The main goal is to have a solid architecture that wouldn't require many breaking changes once it is set; we don't like breaking existing workflows or constantly memorising "new ways" of doing the same thing.

Other things to highlight:

• clear separation of responsibilities,
• some degree of backward compatibility,
• facilitation of testing,
• potential "specialisations" across the team, which could give a pleasant feeling of ownership,

Current proposal sketch that we can improve together:

In this proposal, we have four executables:

• forest / forestd - the primary node.

  • no subcommands; execution means spawning a long-running process.

• forest-cli / forest-client - the program that connects to the node and communicates with it via RPC.

  • not allowed to have any direct operations on the database or filesystem. Every subcommand there must communicate via RPC.

• forest-tool - aggregate for all other commands we consider helpful but don't require having a running node.

  • current candidates in terms of functionality: snapshot, db.

• forest-wallet / forest-account - per @q9f suggestion, all operations related to signing.

  • currently sitting in forest-cli wallet
  • The only binary that communicates with the keystore and can modify it.
  • this one needs more care and an additional design process, plus at least a rough security audit.
  • the keystore is separated from the database directory and lives elsewhere.

To address the unresolved configuration discussion, here are my proposals regarding this:

• get_configuration is implemented in forest and exposed via RPC,
• forest-cli is not aware of the specific structure behind the node configuration - this ensures backward compatibility in case of a single changed parameter,
• forest-cli config --node dump asks via RPC the forest to send its serialized configuration.
• forest-cli config --client dump dumps the client configuration file.
• we could think of having some parameters modifiable in runtime, for example, dynamically setting log level via forest-cli, but that's something for the future.
• binaries should be free to have their configurations (and the ability to load them from files). They should be separated, though. For forest, it's a no-brainer; for forest-cli I believe it may be helpful (i.e. for JWT, node address and logging params).

Let's discuss this, reach a consensus and then start implementing it!

Few questions on this diagram:

• What are you implying with the RPC line at the bottom that goes to forest-cli and forest-wallet
• forest-tool has access to rocksDB as well as forest-node. Why both?

[LesnyRumcajs]

What are you implying with the RPC line at the bottom that goes to forest-cli and forest-wallet

This is for some commands that require authorisation/signing the message, like sending FIL. Currently, it's done on the node level. The forest-cli would communicate via JSON-RPC (or something else, it's a details) with forest-wallet to get the signed message.

forest-tool has access to rocksDB as well as forest-node. Why both?

It doesn't warrant creating a separate binary just for the snapshots. We can, of course, but this may be too fine-grained.

forest-tool should be able to read from the database and/or snapshots without the need to run a node. imo, a brilliant idea. (as long as the tool cannot write.)

The forest-tool would have some write permissions to DB, but only to purge it (in case of data corruption, for example). Other than that (plus downloading snapshots), I would see it as a maintenance tool for some comprehensive inspections.

What do you think? @q9f @lerajk

[q9f]

Sounds good.

However, since forrest-wallet is managing the keys, it should also sign (unless I miss something).

[LesnyRumcajs]

@q9f Yes, that's what I had in mind. When forest-cli would need something signed it would delegate it to the forest-wallet

[lerajk]

This is for some commands that require authorisation/signing the message, like sending FIL. Currently, it's done on the node level. The forest-cli would communicate via JSON-RPC (or something else, it's a details) with forest-wallet to get the signed message.

got it. Ya, make sense.

The forest-tool would have some write permissions to DB, but only to purge it (in case of data corruption, for example). Other than that (plus downloading snapshots), I would see it as a maintenance tool for some comprehensive inspections.

Do you have list of commands that belong to forest-tool Feels like we need to discuss this area more.

forest-tool should be able to read from the database and/or snapshots without the need to run a node. imo, a brilliant idea. (as long as the tool cannot write.)

@q9f why is this a brilliant idea 😂

[LesnyRumcajs]

Do you have list of commands that belong to forest-tool Feels like we need to discuss this area more.

I may be missing some but from a quick glimpse:

db clean/stats
fetch-params

and all snapshot methods

    clean       Clean all local snapshots, use with care
    dir         Shows default snapshot dir
    export      Export a snapshot of the chain to `<output_path>`
    fetch       Fetches the most recent snapshot from a trusted, pre-defined location
    help        Prints this message or the help of the given subcommand(s)
    list        List local snapshots
    prune       Prune local snapshot, keeps the latest only. Note that file names that do not match
                forest_snapshot_{chain}_{year}-{month}-{day}_height_{height}.car pattern will be ignored
    remove      Remove local snapshot
    validate    Validates the snapshot

[lemmih]

Great work, @LesnyRumcajs.

We are likely to want Forest to be a drop-in replacement of Lotus for people who are running miners (aka storage providers). As things stand right now, the miner requires the node to be responsible for wallet handling. This makes it a bit more complicated to move the wallet out of the Forest node. Maybe we remove the keystore from Forest and then add a separate compatibility layer specifically for the Lotus miner?

[LesnyRumcajs]

In which way the node needs to be responsible required for wallet handling? I mean, I imagine it would still be possible to send sign commands to a local wallet agent living in a completely different process. Or am I missing something?

[lemmih]

The lotus miner needs a wallet to operate because it can both gain and lose FIL (at least that's my understanding). The lotus miner talks directly (I believe) to the daemon via the RPC and it requires the daemon to have access to a wallet. Maybe not the best architecture but that's how they wrote it.

My understanding of the lotus miner is shaky at best, though.

[LesnyRumcajs]

Maybe it's a potential FIP to introduce a standard interface between the node/miner and the wallet? Just a wild idea; I know close to nothing about miner architecture and workflow.

[lerajk]

I'm not sure about an FIP between miner architecture and wallet. But an FIP between wallet interface and node sounds interesting lol or 'more doable'

[jdjaustin]

Reviving this discussion to see if there is enough of a consensus to move forward with issue #2076.