Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support upgradability of the consensus parameters and state transition bytecode in genesis #1826

Merged

Conversation

xgreenx
Copy link
Collaborator

@xgreenx xgreenx commented Apr 13, 2024

The changes make the state transition bytecode part of the ChainConfig. It guarantees the state transition's availability for the network's first blocks.

The change has many minor improvements in different areas related to the state transition bytecode:

  • The state transition bytecode lies in its own file(state_transition_bytecode.wasm) along with the chain config file. The ChainConfig loads it automatically when ChainConfig::load is called and pushes it back when ChainConfig::write is called.

  • The fuel-core release bundle also contains the fuel-core-wasm-executor.wasm file of the corresponding executor version.

  • The regenesis process now considers the last block produced by the previous network. When we create a (re)genesis block of a new network, it has the height = last_block_of_old_netowkr + 1. It continues the old network and doesn't overlap blocks(before, we had old_block.height == new_genesis_block.hegiht).

  • Along with the new block height, the regenesis process also increases the state transition bytecode and consensus parameters versions. It guarantees that a new network doesn't use values from the previous network and allows us not to migrate StateTransitionBytecodeVersions and ConsensusParametersVersions tables.

  • Added a new CLI argument, native-executor-version, that allows overriding of the default version of the native executor. It can be useful for side rollups that have their own history of executor upgrades.

  • Replaced:

             let file = std::fs::File::open(path)?;
             let mut snapshot: Self = serde_json::from_reader(&file)?;

    with a:

             let mut json = String::new();
             std::fs::File::open(&path)
                 .with_context(|| format!("Could not open snapshot file: {path:?}"))?
                 .read_to_string(&mut json)?;
             let mut snapshot: Self = serde_json::from_str(json.as_str())?;

    because it is faster in 100 times for big JSON files.

  • Updated all tests to use Config::local_node_* instead of working with the SnapshotReader directly. It is the preparation of the tests for the futures bumps of the Executor::VERSION. When we increase the version, all tests continue to use GenesisBlock.state_transition_btyecode = 0 while the version is different, which forces the usage of the WASM executor, while for tests, we still prefer to test native execution. The Config::local_node_* handles it and forces the executor to use the native version.

  • Reworked the build.rs file of the upgradable executor. The script now has a cache used between builds to avoid recompilation of the WASM bytecode. Also, fixed the issue with outdated WASM bytecode. The script reacts on any modifications of the fuel-core-wasm-executor and forces recompilation(it is why we need cache), so WASM bytecode always is actual now.

Checklist

  • Breaking changes are clearly marked as such in the PR description and changelog

Before requesting review

  • I have reviewed the code myself

xgreenx and others added 30 commits April 6, 2024 15:35
Updated CHANGELOG.md
…ers-from-database' into feature/featch-consensus-parameters-from-database
…atabase

# Conflicts:
#	CHANGELOG.md
#	crates/fuel-core/src/service/adapters.rs
#	crates/fuel-core/src/service/adapters/graphql_api.rs
#	crates/fuel-core/src/service/adapters/producer.rs
#	crates/fuel-core/src/service/adapters/txpool.rs
#	crates/fuel-core/src/service/sub_services.rs
#	crates/services/producer/src/block_producer.rs
#	crates/services/producer/src/block_producer/gas_price.rs
#	crates/services/producer/src/block_producer/tests.rs
#	crates/services/txpool/src/service.rs
Fixed the compilation of the debug image.
Use WASM executor in some tests.
…ion-bytecode-part-chain-config

# Conflicts:
#	crates/chain-config/src/config/state/reader.rs
#	crates/fuel-core/src/p2p_test_helpers.rs
#	crates/fuel-core/src/service/genesis.rs
#	crates/fuel-core/src/service/genesis/importer.rs
…tecode-part-chain-config' into feature/make-state-transition-bytecode-part-chain-config
@xgreenx xgreenx added the breaking A breaking api change label Apr 15, 2024
@xgreenx xgreenx removed the no changelog Skip the CI check of the changelog modification label Apr 15, 2024
@xgreenx xgreenx requested a review from a team April 15, 2024 16:02
@xgreenx xgreenx marked this pull request as ready for review April 15, 2024 16:02
bin/fuel-core/src/cli/snapshot.rs Outdated Show resolved Hide resolved
@@ -13,6 +13,7 @@ description = "Fuel Chain config types"
[dependencies]
anyhow = { workspace = true }
bech32 = { version = "0.9.0", default-features = false, optional = true }
derivative = { workspace = true }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd never heard of derivative before. Just looked it up. Seems really nifty and helpful.

I've worked on projects before, that in the name of security have intentionally limited the number of dependencies we take. i.e. we were really picky about which dependencies we took on. I haven't really heard much about that here. I'm not particularly worried about derivative--it looks well maintained--but I'm guessing that there are deps in our project that were added for convenience but are actually a security risk.

Just a thought. I wonder if we should have some sort of vetting before we add new deps.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. I have the same feeling, but we are already using this dependency in the fuel-tx everywhere. An example.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For vetting I've heard good things about https://github.com/crev-dev/cargo-crev

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

crates/chain-config/src/config/state/writer.rs Outdated Show resolved Hide resolved
crates/chain-config/src/config/snapshot_metadata.rs Outdated Show resolved Hide resolved
crates/fuel-core/src/service/genesis.rs Show resolved Hide resolved
crates/fuel-core/src/service/sub_services.rs Outdated Show resolved Hide resolved
tests/tests/blocks.rs Outdated Show resolved Hide resolved
tests/tests/tx/tx_pointer.rs Outdated Show resolved Hide resolved
@xgreenx xgreenx requested review from MitchTurner and a team April 16, 2024 10:50
Base automatically changed from feature/fuel-vm-0.49.0 to master April 16, 2024 14:05
…t-chain-config

# Conflicts:
#	CHANGELOG.md
#	crates/fuel-core/src/service/genesis.rs
#	crates/services/upgradable-executor/build.rs
#	tests/test-helpers/src/builder.rs
CHANGELOG.md Outdated Show resolved Hide resolved
CHANGELOG.md Outdated Show resolved Hide resolved
CHANGELOG.md Outdated Show resolved Hide resolved
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we actually want to commit wasm files?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep=) Because this state transition function is used in some tests

@@ -13,6 +13,7 @@ description = "Fuel Chain config types"
[dependencies]
anyhow = { workspace = true }
bech32 = { version = "0.9.0", default-features = false, optional = true }
derivative = { workspace = true }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For vetting I've heard good things about https://github.com/crev-dev/cargo-crev

Co-authored-by: Hannes Karppila <hannes.karppila@gmail.com>
@xgreenx xgreenx requested a review from a team April 17, 2024 19:04
@xgreenx xgreenx merged commit bdabd84 into master Apr 17, 2024
33 checks passed
@xgreenx xgreenx deleted the feature/make-state-transition-bytecode-part-chain-config branch April 17, 2024 19:28
@xgreenx xgreenx mentioned this pull request Apr 18, 2024
4 tasks
xgreenx added a commit that referenced this pull request Apr 18, 2024
## Version v0.25.0

### Fixed

- [#1821](#1821): Can handle
missing tables in snapshot.
- [#1814](#1814): Bugfix: the
`iter_all_by_prefix` was not working for all tables. The change adds a
`Rust` level filtering.

### Added

- [#1831](#1831): Included the
total gas and fee used by transaction into `TransactionStatus`.
- [#1821](#1821): Propagate
shutdown signal to (re)genesis. Also add progress bar for (re)genesis.
- [#1813](#1813): Added back
support for `/health` endpoint.
- [#1799](#1799): Snapshot
creation is now concurrent.
- [#1811](#1811): Regenesis
now preserves old blocks and transactions for GraphQL API.

### Changed

- [#1833](#1833): Regenesis of
`SpentMessages` and `ProcessedTransactions`.
- [#1830](#1830): Use
versioning enum for WASM executor input and output.
- [#1816](#1816): Updated the
upgradable executor to fetch the state transition bytecode from the
database when the version doesn't match a native one. This change
enables the WASM executor in the "production" build and requires a
`wasm32-unknown-unknown` target.
- [#1812](#1812): Follow-up PR
to simplify the logic around parallel snapshot creation.
- [#1809](#1809): Fetch
`ConsensusParameters` from the database
- [#1808](#1808): Fetch
consensus parameters from the provider.

#### Breaking

- [#1826](#1826): The changes
make the state transition bytecode part of the `ChainConfig`. It
guarantees the state transition's availability for the network's first
blocks.
The change has many minor improvements in different areas related to the
state transition bytecode:
- The state transition bytecode lies in its own
file(`state_transition_bytecode.wasm`) along with the chain config file.
The `ChainConfig` loads it automatically when `ChainConfig::load` is
called and pushes it back when `ChainConfig::write` is called.
- The `fuel-core` release bundle also contains the
`fuel-core-wasm-executor.wasm` file of the corresponding executor
version.
- The regenesis process now considers the last block produced by the
previous network. When we create a (re)genesis block of a new network,
it has the `height = last_block_of_old_netowkr + 1`. It continues the
old network and doesn't overlap blocks(before, we had `old_block.height
== new_genesis_block.hegiht`).
- Along with the new block height, the regenesis process also increases
the state transition bytecode and consensus parameters versions. It
guarantees that a new network doesn't use values from the previous
network and allows us not to migrate `StateTransitionBytecodeVersions`
and `ConsensusParametersVersions` tables.
- Added a new CLI argument, `native-executor-version,` that allows
overriding of the default version of the native executor. It can be
useful for side rollups that have their own history of executor
upgrades.
    - Replaced:
      
      ```rust
               let file = std::fs::File::open(path)?;
               let mut snapshot: Self = serde_json::from_reader(&file)?;
      ```
      
      with a:
      
      ```rust
               let mut json = String::new();
               std::fs::File::open(&path)
.with_context(|| format!("Could not open snapshot file: {path:?}"))?
                   .read_to_string(&mut json)?;
let mut snapshot: Self = serde_json::from_str(json.as_str())?;
      ```
      because it is 100 times faster for big JSON files.
- Updated all tests to use `Config::local_node_*` instead of working
with the `SnapshotReader` directly. It is the preparation of the tests
for the futures bumps of the `Executor::VERSION`. When we increase the
version, all tests continue to use
`GenesisBlock.state_transition_bytecode = 0` while the version is
different, which forces the usage of the WASM executor, while for tests,
we still prefer to test native execution. The `Config::local_node_*`
handles it and forces the executor to use the native version.
- Reworked the `build.rs` file of the upgradable executor. The script
now caches WASM bytecode to avoid recompilation. Also, fixed the issue
with outdated WASM bytecode. The script reacts on any modifications of
the `fuel-core-wasm-executor` and forces recompilation (it is why we
need the cache), so WASM bytecode always is actual now.
- [#1822](#1822): Removed
support of `Create` transaction from debugger since it doesn't have any
script to execute.
- [#1822](#1822): Use `fuel-vm
0.49.0` with new transactions types - `Upgrade` and `Upload`. Also added
`max_bytecode_subsections` field to the `ConsensusParameters` to limit
the number of bytecode subsections in the state transition bytecode.
- [#1816](#1816): Updated the
upgradable executor to fetch the state transition bytecode from the
database when the version doesn't match a native one. This change
enables the WASM executor in the "production" build and requires a
`wasm32-unknown-unknown` target.

### Before requesting review
- [x] I have reviewed the code myself

### After merging, notify other teams

- [x] [Rust SDK](https://github.com/FuelLabs/fuels-rs/)
- [x] [Sway compiler](https://github.com/FuelLabs/sway/)
- [x] DevOps

## What's Changed
* Add PR template by @Dentosal in
#1806
* feat: Parallellize snapshot creation by @segfault-magnet in
#1799
* Follow-up PR to simplify the logic around parallel snapshot creation
by @xgreenx in #1812
* Bugfix: the `iter_all_by_prefix` was not working for all tables by
@xgreenx in #1814
* Added back support for `/health` endpoint by @xgreenx in
#1813
* Fetch consensus parameters from the provider by @xgreenx in
#1808
* Fetch `ConsensusParameters` from the database by @xgreenx in
#1809
* Handle FTI messages in executor by @Voxelot in
#1787
* Use state transition bytecode from the database by @xgreenx in
#1816
* feat: (re)genesis graceful shutdown by @segfault-magnet in
#1821
* Use `fuel-vm 0.49.0` with new transactions types by @xgreenx in
#1822
* Included the total gas and fee into `TransactionStatus` by @xgreenx in
#1831
* Use versioning enum for WASM executor input and output by @xgreenx in
#1830
* Support upgradability of the consensus parameters and state transition
bytecode in genesis by @xgreenx in
#1826
* Store old blocks and txs after regenesis by @Dentosal in
#1811
* Regenesis of `SpentMessages` and `ProcessedTransactions` by @xgreenx
in #1833


**Full Changelog**:
v0.24.2...v0.25.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking A breaking api change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants