Calculate a storage root and proof of some changes without permanently altering the underlying backend

[liamaharon]

Kian had this idea while we were discussing #14076.

Obviously when using a disk-backed DB in the try-runtime-cli we don't want to alter the underlying DB. This means that all changes must remain in the OverlayedChanges, never being flushed to the backend. Unfortunately this is that we cannot generate the trie root or any proofs.

It may be a useful abstraction to be able to

1. Temporarily alter the DB with some new trie nodes
2. Calculate the new root and/or generate a proof
3. Revert the DB back to its original state

This would be quite involved and probably not high enough priority to work on in the near future, but we wanted to kick off discussion now.

Some things to think about:

• We'd need to be careful about accidentally corrupting the underlying DB. e.g. ideally it would need to automatically recover from this logic crashing before it's able to revert the DB back to its original state.
• Aside from try-runtime-cli, are there other contexts that this would this be useful?

[crystalin]

That would be awesome to get this :)

[bkchr]

All the things, besides generating a proof, already work. We always generate the new storage root from the Overlay. This also doesn't try to write anything to the db.

[kianenigma]

Aside from try-runtime-cli, are there other contexts that this would this be useful?

I presume some existing substrate cli commands can also use this. We have a check-block that supposedly re-executes the block. How does this work? does this alter the backend? If yes, how is it reverted? can this also know the proof size of the block once it is over?

Once we integrate try_state into the main node's syncing process, I imagine it could use the same type of backend interaction. For each block:

1. first execute the block with Core_execute_block
2. then re-execute it with TryRuntime_execute_block, but drop all changes.

Although, in neither of these two cases we need proof, so this is a slightly different scenario.

All the things, besides generating a proof, already work. We always generate the new storage root from the Overlay. This also doesn't try to write anything to the db.

Interesting. I presume since generating the root using an "Overlay + ReadOnlyBackend" is possible, in principle, generating proof from it should also be possible, because both of them just require the entire data to be merklized.

Some pointers on where to look into for this would be useful.

[bkchr]

Interesting. I presume since generating the root using an "Overlay + ReadOnlyBackend" is possible, in principle, generating proof from it should also be possible, because both of them just require the entire data to be merklized.

Proof is not that easy. We currently record all the accesses to the trie and not to the overlay. As when you write something, we do not need to record it as it is an output of the block execution. If you want to record over multiple blocks, you will need to keep the trie around from the old block (which is generated when calculating the storage root) and then record these nodes. However, there is currently nothing present that does this as we normally don't need it.