Use state transition bytecode from the database #1816

xgreenx · 2024-04-10T01:02:56Z

Related issue #1754

Updated the upgradable executor to fetch the state transition bytecode from the database when the version doesn't match a native one. This change enables the WASM executor in the "production" build and requires a wasm32-unknown-unknown target.

The executor has a VERSION constant that must be updated when the behavior of the state transition function is changed. That constant is used to decide what executor to use(WASM or native).

Checklist

New behavior is reflected in tests

Before requesting review

I have reviewed the code myself

Updated CHANGELOG.md

…ch-consensus-parameters-from-database

…ers-from-database' into feature/featch-consensus-parameters-from-database

…ch-consensus-parameters-from-database

… feature/use-bytecode-from-database

…ilation time of 1000 modules

…se' into feature/use-bytecode-from-database

Dentosal · 2024-04-10T07:33:57Z

crates/services/upgradable-executor/src/executor.rs

+        // The test verifies that `Executor::get_module` method caches the compiled WASM module.
+        // If it doesn't cache the modules, the test will fail with a timeout.
+        #[test]
+        #[ntest::timeout(40_000)]


nit: Potentially flaky, especially in the CI. Cannot thing of an easy way to do this better, though.

Dentosal · 2024-04-10T07:38:18Z

crates/services/upgradable-executor/src/executor.rs

+                .map(|_| ());
+
+            // Then
+            assert!(result.is_err());


nit: use .expect_err for better error output

Dentosal

The logic itself looks good. I left a couple of nits on tests.

…atabase # Conflicts: # CHANGELOG.md # crates/fuel-core/src/service/adapters.rs # crates/fuel-core/src/service/adapters/graphql_api.rs # crates/fuel-core/src/service/adapters/producer.rs # crates/fuel-core/src/service/adapters/txpool.rs # crates/fuel-core/src/service/sub_services.rs # crates/services/producer/src/block_producer.rs # crates/services/producer/src/block_producer/gas_price.rs # crates/services/producer/src/block_producer/tests.rs # crates/services/txpool/src/service.rs

… feature/use-bytecode-from-database

Fixed the compilation of the debug image. Use WASM executor in some tests.

…se' into feature/use-bytecode-from-database

Voxelot · 2024-04-11T01:06:44Z

crates/services/upgradable-executor/src/executor.rs

+                }
+            }
+        } else {
+            let module = self.get_module(block_version)?;


Do we have a way to ensure that the initial version of the executor is available in wasm? For example, if there have been several upgrades to the network and a node on the latest native version attempts to sync the network from genesis, will the bytecode for the initial executor version (before any upgrades occurred) be available?

No, we don't have a way. It should already be part of the snapshot/genesis block.

Automated the process of insertion of the Wasm bytecode into the ChainConfig #1826

The base branch was changed.

Dentosal

LGTM. @Voxelot still has one open comment, so you might want to wait for him to review as well. Also needs conflict resolution.

The merge-base changed after approval.

@Dentosal

## Version v0.25.0 ### Fixed - [#1821](#1821): Can handle missing tables in snapshot. - [#1814](#1814): Bugfix: the `iter_all_by_prefix` was not working for all tables. The change adds a `Rust` level filtering. ### Added - [#1831](#1831): Included the total gas and fee used by transaction into `TransactionStatus`. - [#1821](#1821): Propagate shutdown signal to (re)genesis. Also add progress bar for (re)genesis. - [#1813](#1813): Added back support for `/health` endpoint. - [#1799](#1799): Snapshot creation is now concurrent. - [#1811](#1811): Regenesis now preserves old blocks and transactions for GraphQL API. ### Changed - [#1833](#1833): Regenesis of `SpentMessages` and `ProcessedTransactions`. - [#1830](#1830): Use versioning enum for WASM executor input and output. - [#1816](#1816): Updated the upgradable executor to fetch the state transition bytecode from the database when the version doesn't match a native one. This change enables the WASM executor in the "production" build and requires a `wasm32-unknown-unknown` target. - [#1812](#1812): Follow-up PR to simplify the logic around parallel snapshot creation. - [#1809](#1809): Fetch `ConsensusParameters` from the database - [#1808](#1808): Fetch consensus parameters from the provider. #### Breaking - [#1826](#1826): The changes make the state transition bytecode part of the `ChainConfig`. It guarantees the state transition's availability for the network's first blocks. The change has many minor improvements in different areas related to the state transition bytecode: - The state transition bytecode lies in its own file(`state_transition_bytecode.wasm`) along with the chain config file. The `ChainConfig` loads it automatically when `ChainConfig::load` is called and pushes it back when `ChainConfig::write` is called. - The `fuel-core` release bundle also contains the `fuel-core-wasm-executor.wasm` file of the corresponding executor version. - The regenesis process now considers the last block produced by the previous network. When we create a (re)genesis block of a new network, it has the `height = last_block_of_old_netowkr + 1`. It continues the old network and doesn't overlap blocks(before, we had `old_block.height == new_genesis_block.hegiht`). - Along with the new block height, the regenesis process also increases the state transition bytecode and consensus parameters versions. It guarantees that a new network doesn't use values from the previous network and allows us not to migrate `StateTransitionBytecodeVersions` and `ConsensusParametersVersions` tables. - Added a new CLI argument, `native-executor-version,` that allows overriding of the default version of the native executor. It can be useful for side rollups that have their own history of executor upgrades. - Replaced: ```rust let file = std::fs::File::open(path)?; let mut snapshot: Self = serde_json::from_reader(&file)?; ``` with a: ```rust let mut json = String::new(); std::fs::File::open(&path) .with_context(|| format!("Could not open snapshot file: {path:?}"))? .read_to_string(&mut json)?; let mut snapshot: Self = serde_json::from_str(json.as_str())?; ``` because it is 100 times faster for big JSON files. - Updated all tests to use `Config::local_node_*` instead of working with the `SnapshotReader` directly. It is the preparation of the tests for the futures bumps of the `Executor::VERSION`. When we increase the version, all tests continue to use `GenesisBlock.state_transition_bytecode = 0` while the version is different, which forces the usage of the WASM executor, while for tests, we still prefer to test native execution. The `Config::local_node_*` handles it and forces the executor to use the native version. - Reworked the `build.rs` file of the upgradable executor. The script now caches WASM bytecode to avoid recompilation. Also, fixed the issue with outdated WASM bytecode. The script reacts on any modifications of the `fuel-core-wasm-executor` and forces recompilation (it is why we need the cache), so WASM bytecode always is actual now. - [#1822](#1822): Removed support of `Create` transaction from debugger since it doesn't have any script to execute. - [#1822](#1822): Use `fuel-vm 0.49.0` with new transactions types - `Upgrade` and `Upload`. Also added `max_bytecode_subsections` field to the `ConsensusParameters` to limit the number of bytecode subsections in the state transition bytecode. - [#1816](#1816): Updated the upgradable executor to fetch the state transition bytecode from the database when the version doesn't match a native one. This change enables the WASM executor in the "production" build and requires a `wasm32-unknown-unknown` target. ### Before requesting review - [x] I have reviewed the code myself ### After merging, notify other teams - [x] [Rust SDK](https://github.com/FuelLabs/fuels-rs/) - [x] [Sway compiler](https://github.com/FuelLabs/sway/) - [x] DevOps ## What's Changed * Add PR template by @Dentosal in #1806 * feat: Parallellize snapshot creation by @segfault-magnet in #1799 * Follow-up PR to simplify the logic around parallel snapshot creation by @xgreenx in #1812 * Bugfix: the `iter_all_by_prefix` was not working for all tables by @xgreenx in #1814 * Added back support for `/health` endpoint by @xgreenx in #1813 * Fetch consensus parameters from the provider by @xgreenx in #1808 * Fetch `ConsensusParameters` from the database by @xgreenx in #1809 * Handle FTI messages in executor by @Voxelot in #1787 * Use state transition bytecode from the database by @xgreenx in #1816 * feat: (re)genesis graceful shutdown by @segfault-magnet in #1821 * Use `fuel-vm 0.49.0` with new transactions types by @xgreenx in #1822 * Included the total gas and fee into `TransactionStatus` by @xgreenx in #1831 * Use versioning enum for WASM executor input and output by @xgreenx in #1830 * Support upgradability of the consensus parameters and state transition bytecode in genesis by @xgreenx in #1826 * Store old blocks and txs after regenesis by @Dentosal in #1811 * Regenesis of `SpentMessages` and `ProcessedTransactions` by @xgreenx in #1833 **Full Changelog**: v0.24.2...v0.25.0

@Dentosal

## Version v0.25.0 ### Fixed - [#1821](FuelLabs/fuel-core#1821): Can handle missing tables in snapshot. - [#1814](FuelLabs/fuel-core#1814): Bugfix: the `iter_all_by_prefix` was not working for all tables. The change adds a `Rust` level filtering. ### Added - [#1831](FuelLabs/fuel-core#1831): Included the total gas and fee used by transaction into `TransactionStatus`. - [#1821](FuelLabs/fuel-core#1821): Propagate shutdown signal to (re)genesis. Also add progress bar for (re)genesis. - [#1813](FuelLabs/fuel-core#1813): Added back support for `/health` endpoint. - [#1799](FuelLabs/fuel-core#1799): Snapshot creation is now concurrent. - [#1811](FuelLabs/fuel-core#1811): Regenesis now preserves old blocks and transactions for GraphQL API. ### Changed - [#1833](FuelLabs/fuel-core#1833): Regenesis of `SpentMessages` and `ProcessedTransactions`. - [#1830](FuelLabs/fuel-core#1830): Use versioning enum for WASM executor input and output. - [#1816](FuelLabs/fuel-core#1816): Updated the upgradable executor to fetch the state transition bytecode from the database when the version doesn't match a native one. This change enables the WASM executor in the "production" build and requires a `wasm32-unknown-unknown` target. - [#1812](FuelLabs/fuel-core#1812): Follow-up PR to simplify the logic around parallel snapshot creation. - [#1809](FuelLabs/fuel-core#1809): Fetch `ConsensusParameters` from the database - [#1808](FuelLabs/fuel-core#1808): Fetch consensus parameters from the provider. #### Breaking - [#1826](FuelLabs/fuel-core#1826): The changes make the state transition bytecode part of the `ChainConfig`. It guarantees the state transition's availability for the network's first blocks. The change has many minor improvements in different areas related to the state transition bytecode: - The state transition bytecode lies in its own file(`state_transition_bytecode.wasm`) along with the chain config file. The `ChainConfig` loads it automatically when `ChainConfig::load` is called and pushes it back when `ChainConfig::write` is called. - The `fuel-core` release bundle also contains the `fuel-core-wasm-executor.wasm` file of the corresponding executor version. - The regenesis process now considers the last block produced by the previous network. When we create a (re)genesis block of a new network, it has the `height = last_block_of_old_netowkr + 1`. It continues the old network and doesn't overlap blocks(before, we had `old_block.height == new_genesis_block.hegiht`). - Along with the new block height, the regenesis process also increases the state transition bytecode and consensus parameters versions. It guarantees that a new network doesn't use values from the previous network and allows us not to migrate `StateTransitionBytecodeVersions` and `ConsensusParametersVersions` tables. - Added a new CLI argument, `native-executor-version,` that allows overriding of the default version of the native executor. It can be useful for side rollups that have their own history of executor upgrades. - Replaced: ```rust let file = std::fs::File::open(path)?; let mut snapshot: Self = serde_json::from_reader(&file)?; ``` with a: ```rust let mut json = String::new(); std::fs::File::open(&path) .with_context(|| format!("Could not open snapshot file: {path:?}"))? .read_to_string(&mut json)?; let mut snapshot: Self = serde_json::from_str(json.as_str())?; ``` because it is 100 times faster for big JSON files. - Updated all tests to use `Config::local_node_*` instead of working with the `SnapshotReader` directly. It is the preparation of the tests for the futures bumps of the `Executor::VERSION`. When we increase the version, all tests continue to use `GenesisBlock.state_transition_bytecode = 0` while the version is different, which forces the usage of the WASM executor, while for tests, we still prefer to test native execution. The `Config::local_node_*` handles it and forces the executor to use the native version. - Reworked the `build.rs` file of the upgradable executor. The script now caches WASM bytecode to avoid recompilation. Also, fixed the issue with outdated WASM bytecode. The script reacts on any modifications of the `fuel-core-wasm-executor` and forces recompilation (it is why we need the cache), so WASM bytecode always is actual now. - [#1822](FuelLabs/fuel-core#1822): Removed support of `Create` transaction from debugger since it doesn't have any script to execute. - [#1822](FuelLabs/fuel-core#1822): Use `fuel-vm 0.49.0` with new transactions types - `Upgrade` and `Upload`. Also added `max_bytecode_subsections` field to the `ConsensusParameters` to limit the number of bytecode subsections in the state transition bytecode. - [#1816](FuelLabs/fuel-core#1816): Updated the upgradable executor to fetch the state transition bytecode from the database when the version doesn't match a native one. This change enables the WASM executor in the "production" build and requires a `wasm32-unknown-unknown` target. ### Before requesting review - [x] I have reviewed the code myself ### After merging, notify other teams - [x] [Rust SDK](https://github.com/FuelLabs/fuels-rs/) - [x] [Sway compiler](https://github.com/FuelLabs/sway/) - [x] DevOps ## What's Changed * Add PR template by @Dentosal in FuelLabs/fuel-core#1806 * feat: Parallellize snapshot creation by @segfault-magnet in FuelLabs/fuel-core#1799 * Follow-up PR to simplify the logic around parallel snapshot creation by @xgreenx in FuelLabs/fuel-core#1812 * Bugfix: the `iter_all_by_prefix` was not working for all tables by @xgreenx in FuelLabs/fuel-core#1814 * Added back support for `/health` endpoint by @xgreenx in FuelLabs/fuel-core#1813 * Fetch consensus parameters from the provider by @xgreenx in FuelLabs/fuel-core#1808 * Fetch `ConsensusParameters` from the database by @xgreenx in FuelLabs/fuel-core#1809 * Handle FTI messages in executor by @Voxelot in FuelLabs/fuel-core#1787 * Use state transition bytecode from the database by @xgreenx in FuelLabs/fuel-core#1816 * feat: (re)genesis graceful shutdown by @segfault-magnet in FuelLabs/fuel-core#1821 * Use `fuel-vm 0.49.0` with new transactions types by @xgreenx in FuelLabs/fuel-core#1822 * Included the total gas and fee into `TransactionStatus` by @xgreenx in FuelLabs/fuel-core#1831 * Use versioning enum for WASM executor input and output by @xgreenx in FuelLabs/fuel-core#1830 * Support upgradability of the consensus parameters and state transition bytecode in genesis by @xgreenx in FuelLabs/fuel-core#1826 * Store old blocks and txs after regenesis by @Dentosal in FuelLabs/fuel-core#1811 * Regenesis of `SpentMessages` and `ProcessedTransactions` by @xgreenx in FuelLabs/fuel-core#1833 **Full Changelog**: FuelLabs/fuel-core@v0.24.2...v0.25.0

xgreenx and others added 15 commits April 6, 2024 15:35

Fetch consensus parameters from the provider

310a6dc

Self review

d8ee459

Updated CHANGELOG.md

Merge branch 'master' into feature/use-conensus-params-provider

48d373b

Fetch ConsensusParameters from the database

14ca7c8

Updated CHANGELOG.md

28a3045

Merge branch 'feature/use-conensus-params-provider' into feature/feat…

bc19087

…ch-consensus-parameters-from-database

Removed redundant Arc

4e3cf61

Merge remote-tracking branch 'origin/feature/featch-consensus-paramet…

08de09d

…ers-from-database' into feature/featch-consensus-parameters-from-database

Merge branch 'master' into feature/use-conensus-params-provider

b4fba35

Merge branch 'feature/use-conensus-params-provider' into feature/feat…

e4a9a62

…ch-consensus-parameters-from-database

Merge branch 'master' into feature/use-conensus-params-provider

9ede6dd

Merge branch 'feature/use-conensus-params-provider' into feature/feat…

38ea324

…ch-consensus-parameters-from-database

Merge branch 'master' into feature/use-conensus-params-provider

dd15952

Merge branch 'feature/use-conensus-params-provider' into feature/feat…

e1bf37e

…ch-consensus-parameters-from-database

Use state transition bytecode from the database

1d5dbd8

xgreenx requested a review from a team April 10, 2024 01:02

xgreenx self-assigned this Apr 10, 2024

xgreenx and others added 3 commits April 10, 2024 03:06

Updated CHANGELOG.md

d4164e4

Fixed a comment

e12cca6

Merge branch 'feature/featch-consensus-parameters-from-database' into…

316de9a

… feature/use-bytecode-from-database

xgreenx marked this pull request as ready for review April 10, 2024 01:17

xgreenx added 2 commits April 10, 2024 03:36

Increased timeout for the CI. 40 seconds not even closer to real comp…

eaffb17

…ilation time of 1000 modules

Merge remote-tracking branch 'origin/feature/use-bytecode-from-databa…

f7cb1dd

…se' into feature/use-bytecode-from-database

Dentosal reviewed Apr 10, 2024

View reviewed changes

Dentosal previously approved these changes Apr 10, 2024

View reviewed changes

xgreenx and others added 4 commits April 10, 2024 10:37

Added tests for the service

6ec1a6b

Merge branch 'feature/featch-consensus-parameters-from-database' into…

9e20a2c

… feature/use-bytecode-from-database

Applied comment from PR.

3f06311

Fixed the compilation of the debug image. Use WASM executor in some tests.

xgreenx added 4 commits April 10, 2024 13:24

Merge remote-tracking branch 'origin/feature/use-bytecode-from-databa…

41ca96d

…se' into feature/use-bytecode-from-database

Increase the memory for the docker

874b119

Increase memory to 48g, if it is possible=D

3217787

Use 64 GB builder

a6df714

Voxelot reviewed Apr 11, 2024

View reviewed changes

Base automatically changed from feature/featch-consensus-parameters-from-database to master April 11, 2024 17:53

Dentosal previously approved these changes Apr 11, 2024

View reviewed changes

Merge branch 'master' into feature/use-bytecode-from-database

003cba2

xgreenx requested review from Voxelot, Dentosal, a team and MitchTurner April 11, 2024 18:57

Merge branch 'master' into feature/use-bytecode-from-database

b14be52

Dentosal approved these changes Apr 15, 2024

View reviewed changes

xgreenx merged commit 2e71605 into master Apr 15, 2024
33 checks passed

xgreenx deleted the feature/use-bytecode-from-database branch April 15, 2024 10:51

xgreenx mentioned this pull request Apr 18, 2024

Release v0.25.0 #1839

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use state transition bytecode from the database #1816

Use state transition bytecode from the database #1816

xgreenx commented Apr 10, 2024 •

edited

Loading

Dentosal Apr 10, 2024

Dentosal Apr 10, 2024

Dentosal left a comment

Voxelot Apr 11, 2024 •

edited

Loading

xgreenx Apr 11, 2024

xgreenx Apr 13, 2024

Dentosal left a comment •

edited

Loading

Use state transition bytecode from the database #1816

Use state transition bytecode from the database #1816

Conversation

xgreenx commented Apr 10, 2024 • edited Loading

Checklist

Before requesting review

Dentosal Apr 10, 2024

Choose a reason for hiding this comment

Dentosal Apr 10, 2024

Choose a reason for hiding this comment

Dentosal left a comment

Choose a reason for hiding this comment

Voxelot Apr 11, 2024 • edited Loading

Choose a reason for hiding this comment

xgreenx Apr 11, 2024

Choose a reason for hiding this comment

xgreenx Apr 13, 2024

Choose a reason for hiding this comment

Dentosal left a comment • edited Loading

Choose a reason for hiding this comment

xgreenx commented Apr 10, 2024 •

edited

Loading

Voxelot Apr 11, 2024 •

edited

Loading

Dentosal left a comment •

edited

Loading