Skip to content
  • Andrei Eres's avatar
    subsystem-bench: add regression tests for availability read and write (#3311) · f0e589d7
    Andrei Eres authored
    
    
    ### What's been done
    - `subsystem-bench` has been split into two parts: a cli benchmark
    runner and a library.
    - The cli runner is quite simple. It just allows us to run `.yaml` based
    test sequences. Now it should only be used to run benchmarks during
    development.
    - The library is used in the cli runner and in regression tests. Some
    code is changed to make the library independent of the runner.
    - Added first regression tests for availability read and write that
    replicate existing test sequences.
    
    ### How we run regression tests
    - Regression tests are simply rust integration tests without the
    harnesses.
    - They should only be compiled under the `subsystem-benchmarks` feature
    to prevent them from running with other tests.
    - This doesn't work when running tests with `nextest` in CI, so
    additional filters have been added to the `nextest` runs.
    - Each benchmark run takes a different time in the beginning, so we
    "warm up" the tests until their CPU usage differs by only 1%.
    - After the warm-up, we run the benchmarks a few more times and compare
    the average with the exception using a precision.
    
    ### What is still wrong?
    - I haven't managed to set up approval voting tests. The spread of their
    results is too large and can't be narrowed down in a reasonable amount
    of time in the warm-up phase.
    - The tests start an unconfigurable prometheus endpoint inside, which
    causes errors because they use the same 9999 port. I disable it with a
    flag, but I think it's better to extract the endpoint launching outside
    the test, as we already do with `valgrind` and `pyroscope`. But we still
    use `prometheus` inside the tests.
    
    ### Future work
    * https://github.com/paritytech/polkadot-sdk/issues/3528
    * https://github.com/paritytech/polkadot-sdk/issues/3529
    * https://github.com/paritytech/polkadot-sdk/issues/3530
    * https://github.com/paritytech/polkadot-sdk/issues/3531
    
    ---------
    
    Co-authored-by: default avatarAlexander Samusev <[email protected]>
    f0e589d7