subsystem-bench: add regression tests for availability read and write (#3311)
### What's been done
- `subsystem-bench` has been split into two parts: a cli benchmark
runner and a library.
- The cli runner is quite simple. It just allows us to run `.yaml` based
test sequences. Now it should only be used to run benchmarks during
development.
- The library is used in the cli runner and in regression tests. Some
code is changed to make the library independent of the runner.
- Added first regression tests for availability read and write that
replicate existing test sequences.
### How we run regression tests
- Regression tests are simply rust integration tests without the
harnesses.
- They should only be compiled under the `subsystem-benchmarks` feature
to prevent them from running with other tests.
- This doesn't work when running tests with `nextest` in CI, so
additional filters have been added to the `nextest` runs.
- Each benchmark run takes a different time in the beginning, so we
"warm up" the tests until their CPU usage differs by only 1%.
- After the warm-up, we run the benchmarks a few more times and compare
the average with the exception using a precision.
### What is still wrong?
- I haven't managed to set up approval voting tests. The spread of their
results is too large and can't be narrowed down in a reasonable amount
of time in the warm-up phase.
- The tests start an unconfigurable prometheus endpoint inside, which
causes errors because they use the same 9999 port. I disable it with a
flag, but I think it's better to extract the endpoint launching outside
the test, as we already do with `valgrind` and `pyroscope`. But we still
use `prometheus` inside the tests.
### Future work
* https://github.com/paritytech/polkadot-sdk/issues/3528
* https://github.com/paritytech/polkadot-sdk/issues/3529
* https://github.com/paritytech/polkadot-sdk/issues/3530
* https://github.com/paritytech/polkadot-sdk/issues/3531
---------
Co-authored-by:
Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
Showing
- .gitlab/pipeline/test.yml 2 additions, 1 deletion.gitlab/pipeline/test.yml
- Cargo.lock 2 additions, 0 deletionsCargo.lock
- polkadot/node/network/availability-distribution/Cargo.toml 11 additions, 0 deletionspolkadot/node/network/availability-distribution/Cargo.toml
- polkadot/node/network/availability-distribution/tests/availability-distribution-regression-bench.rs 113 additions, 0 deletions...ution/tests/availability-distribution-regression-bench.rs
- polkadot/node/network/availability-recovery/Cargo.toml 7 additions, 0 deletionspolkadot/node/network/availability-recovery/Cargo.toml
- polkadot/node/network/availability-recovery/tests/availability-recovery-regression-bench.rs 103 additions, 0 deletions...-recovery/tests/availability-recovery-regression-bench.rs
- polkadot/node/subsystem-bench/Cargo.toml 5 additions, 1 deletionpolkadot/node/subsystem-bench/Cargo.toml
- polkadot/node/subsystem-bench/README.md 72 additions, 74 deletionspolkadot/node/subsystem-bench/README.md
- polkadot/node/subsystem-bench/examples/approvals_no_shows.yaml 3 additions, 3 deletions...dot/node/subsystem-bench/examples/approvals_no_shows.yaml
- polkadot/node/subsystem-bench/examples/approvals_throughput.yaml 1 addition, 2 deletions...t/node/subsystem-bench/examples/approvals_throughput.yaml
- polkadot/node/subsystem-bench/examples/approvals_throughput_best_case.yaml 1 addition, 1 deletion...system-bench/examples/approvals_throughput_best_case.yaml
- polkadot/node/subsystem-bench/examples/approvals_throughput_no_optimisations_enabled.yaml 1 addition, 1 deletion...amples/approvals_throughput_no_optimisations_enabled.yaml
- polkadot/node/subsystem-bench/src/availability/cli.rs 0 additions, 37 deletionspolkadot/node/subsystem-bench/src/availability/cli.rs
- polkadot/node/subsystem-bench/src/cli/subsystem-bench.rs 66 additions, 83 deletionspolkadot/node/subsystem-bench/src/cli/subsystem-bench.rs
- polkadot/node/subsystem-bench/src/cli/valgrind.rs 0 additions, 0 deletionspolkadot/node/subsystem-bench/src/cli/valgrind.rs
- polkadot/node/subsystem-bench/src/lib/approval/helpers.rs 1 addition, 1 deletionpolkadot/node/subsystem-bench/src/lib/approval/helpers.rs
- polkadot/node/subsystem-bench/src/lib/approval/message_generator.rs 6 additions, 13 deletions...ode/subsystem-bench/src/lib/approval/message_generator.rs
- polkadot/node/subsystem-bench/src/lib/approval/mock_chain_selection.rs 0 additions, 0 deletions.../subsystem-bench/src/lib/approval/mock_chain_selection.rs
- polkadot/node/subsystem-bench/src/lib/approval/mod.rs 26 additions, 24 deletionspolkadot/node/subsystem-bench/src/lib/approval/mod.rs
- polkadot/node/subsystem-bench/src/lib/approval/test_message.rs 2 additions, 3 deletions...dot/node/subsystem-bench/src/lib/approval/test_message.rs
Please register or sign in to comment