feat: `pop bench pallet` logic implementation #407

chungquantin · 2025-02-11T08:40:47Z

Features

Add a new pop-cli command bench to benchmark pallets & parachain.
Integrates with frame-benchmarking-cli as a subcommand.
Run benchmarking on pallets without node binary required. Only runtime needed, inspired by FRAME Omni Bencher.
Locate the runtime WASM binary in the project and trigger runtime build - PR: feat: benchmark existing <runtime-name>.wasm or trigger build if needed #411
Nice to have: Support both benchmarking with chain spec and runtime. FRAME Omni Bencher only support benchmarking with runtime wasm. This addition enhances the user experience by allowing the developers to provide either the chain spec file path or the runtime wasm file.

How to test?

Use the below command in your pop-cli project to benchmark the base_parachain.wasm.

cargo run -- bench pallet --runtime=./crates/pop-cli/test-resources/base_parachain.wasm --pallet=pallet_timestamp --extrinsic= --output=./weights.rs

Or run a command pop bench pallet

Benchmarking completed successfully

Benchmark a pallet or parachain

Usage: pop bench
       pop bench <COMMAND>

Commands:
  pallet  Benchmark the extrinsics of FRAME Pallets
  help    Print this message or the help of the given subcommand(s)

To run benchmarking, builds the parachain runtime and specify the runtime path with --runtime

┌   Pop CLI : Benchmarking your pallets
│
▲  NOTE: the `pop bench pallet` is not yet battle tested - double check the results.
│
▲  NOTE: this may take some time...
│
[2025-02-12T17:27:41Z INFO  pallet_collator_selection::pallet] assembling new collators for new session 0 at #0
[2025-02-12T17:27:41Z INFO  pallet_collator_selection::pallet] assembling new collators for new session 1 at #0
[2025-02-12T17:27:42Z INFO  polkadot_sdk_frame::benchmark::pallet] Loading WASM from file
[2025-02-12T17:27:57Z INFO  polkadot_sdk_frame::benchmark::pallet] [  0 % ] Starting benchmark: pallet_timestamp::set
[2025-02-12T17:27:57Z INFO  polkadot_sdk_frame::benchmark::pallet] [ 50 % ] Starting benchmark: pallet_timestamp::on_finalize
Pallet: "pallet_timestamp", Extrinsic: "set", Lowest values: [], Highest values: [], Steps: 50, Repeat: 20
Raw Storage Info
========
Storage: `Timestamp::Now` (r:1 w:1)
Proof: `Timestamp::Now` (`max_values`: Some(1), `max_size`: Some(8), added: 503, mode: `MaxEncodedLen`)
Storage: `Aura::CurrentSlot` (r:1 w:0)
Proof: `Aura::CurrentSlot` (`max_values`: Some(1), `max_size`: Some(8), added: 503, mode: `MaxEncodedLen`)

Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=       97
              µs

Reads = 2
Writes = 1
Recorded proof Size = 86

Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=       97
              µs

Reads = 2
Writes = 1
Recorded proof Size = 86

Pallet: "pallet_timestamp", Extrinsic: "on_finalize", Lowest values: [], Highest values: [], Steps: 50, Repeat: 20
Raw Storage Info
========

Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=       46
              µs

Reads = 0
Writes = 0
Recorded proof Size = 57

Min Squares Analysis
========
-- Extrinsic Time --

Model:
Time ~=       46
              µs

Reads = 0
Writes = 0
Recorded proof Size = 57

⚙  Weight file is generated to ./weights.rs
│
└  Benchmark completed successfully!

Failed with error

┌   Pop CLI : Benchmarking your pallets
│
▲  NOTE: the `pop bench pallet` is not yet battle tested - double check the results.
│
▲  NOTE: this may take some time...
│
└  Failed to run benchmarking: No such file or directory (os error 2)
└  Benchmark completed successfully!

[sc-2881]

chungquantin · 2025-02-11T08:47:57Z

crates/pop-cli/src/commands/bench/mod.rs

+use frame_benchmarking_cli::PalletCmd;
+use sp_runtime::traits::BlakeTwo256;
+
+type HostFunctions = (


These are set based on the implementation of FRAME Omni Bencher. These are necessary to benchmark parachain runtime.

chungquantin · 2025-02-11T09:07:51Z

crates/pop-cli/src/main.rs

@@ -23,6 +23,11 @@ mod wallet_integration;

 #[tokio::main]
 async fn main() -> Result<()> {
+	// Set environment for logging configuration, requires for `pop bench`.
+	if std::env::var("RUST_LOG").is_err() {


Requires for the log messages of frame-benchmarking-cli

Can we implement this when using pop bench pallet? Maybe we could also add a cli.info message to inform the user that we are changing the variable.

The CLI is initialized with env_logger.init() due to telemetry feature. The environment can only be set before the initialization of env_logger

codecov · 2025-02-11T09:26:53Z

Codecov Report

Attention: Patch coverage is 70.45455% with 39 lines in your changes missing coverage. Please review.

Project coverage is 75.09%. Comparing base (2e03233) to head (667e997).

Files with missing lines	Patch %	Lines
crates/pop-cli/src/commands/bench/mod.rs	74.07%	15 Missing and 13 partials ⚠️
crates/pop-cli/src/common/prompt.rs	64.28%	0 Missing and 5 partials ⚠️
crates/pop-cli/src/main.rs	0.00%	5 Missing ⚠️
crates/pop-cli/src/commands/mod.rs	0.00%	1 Missing ⚠️

@@                           Coverage Diff                           @@
##           chungquantin/feat-pop_bench_command     #407      +/-   ##
=======================================================================
- Coverage                                75.11%   75.09%   -0.03%     
=======================================================================
  Files                                       63       66       +3     
  Lines                                    13842    13946     +104     
  Branches                                 13842    13946     +104     
=======================================================================
+ Hits                                     10398    10473      +75     
- Misses                                    2118     2139      +21     
- Partials                                  1326     1334       +8

Files with missing lines	Coverage Δ
crates/pop-cli/src/commands/call/chain.rs	`74.82% <ø> (+0.17%)`	⬆️
crates/pop-cli/src/commands/call/contract.rs	`80.53% <ø> (+0.23%)`	⬆️
crates/pop-parachains/src/bench.rs	`100.00% <100.00%> (ø)`
crates/pop-cli/src/commands/mod.rs	`16.00% <0.00%> (-0.33%)`	⬇️
crates/pop-cli/src/common/prompt.rs	`64.28% <64.28%> (ø)`
crates/pop-cli/src/main.rs	`40.50% <0.00%> (-2.74%)`	⬇️
crates/pop-cli/src/commands/bench/mod.rs	`74.07% <74.07%> (ø)`

chungquantin · 2025-02-11T16:42:21Z

@AlexD10S CI fails but the PR is good for a review.

AlexD10S

Great work! The implementation is clean, and I love seeing how you're already handling the UI testing.

I’ve left some comments on areas for improvement, along with a couple of additional tests that could help improve coverage.

One refactor I want to see: In the rest of the commands, we typically separate the core logic (handled in pop_contracts or pop_parachain) from the "frontend logic" (pop-cli crate). In this PR, the logic is just cmd.run_with_spec, but it might be worth following the same approach by adding a generate_benchmarks(cmd) function in pop_parachain crate to handle this logic.

crates/pop-cli/test-resources/base_parachain.wasm

crates/pop-cli/src/commands/bench/mod.rs

AlexD10S · 2025-02-12T10:47:07Z

crates/pop-cli/src/main.rs

@@ -23,6 +23,11 @@ mod wallet_integration;

 #[tokio::main]
 async fn main() -> Result<()> {
+	// Set environment for logging configuration, requires for `pop bench`.
+	if std::env::var("RUST_LOG").is_err() {


Can we implement this when using pop bench pallet? Maybe we could also add a cli.info message to inform the user that we are changing the variable.

crates/pop-cli/src/commands/bench/mod.rs

AlexD10S · 2025-02-12T10:51:46Z

crates/pop-cli/src/commands/bench/mod.rs

+		cli.warning("NOTE: this may take some time...")?;
+
+		cmd.run_with_spec::<BlakeTwo256, HostFunctions>(None).map_err(|e| {
+			anyhow::anyhow!(format!(


Same as above. We can also add a test that checks the error is throw for better coverage.

Resolved by adding test.

AlexD10S · 2025-02-12T10:58:13Z

crates/pop-cli/src/commands/bench/mod.rs

+		})?;
+
+		if let Some(output_path) = cmd.output {
+			cli.info(format!("Weight file is generated to {:?}", output_path.to_str()))?;


The ouput with the line coming from the wrapped tool is:

Created file: "./weights.rs" ⚙ Weight file is generated to Some("./weights.rs")

If we want to have our own maybe is worth cleaning the latest line and removing Some:

console::Term::stderr().clear_last_lines(1)?; cli.info(format!("Weight file is generated to {}", output_path.as_path().display().to_string()))?; ``

Resolved by following the suggestion

chungquantin added 8 commits February 10, 2025 15:51

feat: add bench subcommand

051d5ed

Merge branch 'r0gue-io:main' into chungquantin/feat-pop_bench

23ba99f

feat: integrates frame-benchmarking-cli

5987931

refactor: PalletCmd run with spec

0d5f5ef

feat: set RUST_LOG=info for benchmarking display

f6a8f06

feat: add CLI messages and tests

a3139e4

feat: add benchmark runtime wasm test

45c6ade

chore: fix comment

53258ac

chungquantin commented Feb 11, 2025

View reviewed changes

fix: operator cannot be applied to type

8f82fde

chungquantin force-pushed the chungquantin/feat-pop_bench branch from 5c891b5 to 8f82fde Compare February 11, 2025 09:13

chungquantin requested a review from AlexD10S February 11, 2025 13:06

AlexD10S requested changes Feb 12, 2025

View reviewed changes

chungquantin added 2 commits February 13, 2025 00:33

chore: display error and refactor test file

fa2db46

chore: rename bench file

667e997

chungquantin self-assigned this Feb 13, 2025

chungquantin added the ready-for-final-review The PR is ready for final review label Feb 13, 2025

chungquantin requested a review from AlexD10S February 13, 2025 13:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: `pop bench pallet` logic implementation #407

feat: `pop bench pallet` logic implementation #407

chungquantin commented Feb 11, 2025 •

edited

Loading

chungquantin Feb 11, 2025

chungquantin Feb 11, 2025

AlexD10S Feb 12, 2025

chungquantin Feb 12, 2025

codecov bot commented Feb 11, 2025 •

edited

Loading

chungquantin commented Feb 11, 2025

AlexD10S left a comment

AlexD10S Feb 12, 2025

AlexD10S Feb 12, 2025

chungquantin Feb 12, 2025

AlexD10S Feb 12, 2025

chungquantin Feb 13, 2025

feat: pop bench pallet logic implementation #407

Are you sure you want to change the base?

feat: pop bench pallet logic implementation #407

Conversation

chungquantin commented Feb 11, 2025 • edited Loading

Features

How to test?

Benchmarking completed successfully

Failed with error

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Feb 11, 2025 • edited Loading

Codecov Report

chungquantin commented Feb 11, 2025

AlexD10S left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feat: `pop bench pallet` logic implementation #407

feat: `pop bench pallet` logic implementation #407

chungquantin commented Feb 11, 2025 •

edited

Loading

codecov bot commented Feb 11, 2025 •

edited

Loading