Co-authored-by: alvicsam <[email protected]>

Implements the idea from
https://github.com/paritytech/polkadot-sdk/pull/3899
- Removed latencies
- Number of runs reduced from 50 to 5, according to local runs it's
quite enough
- Network message is always sent in a spawned task, even if latency is
zero. Without it, CPU time sometimes spikes.
- Removed the `testnet` profile because we probably don't need that
debug additions.

After the local tests I can't say that it brings a significant
improvement in the stability of the results. However, I belive it is
worth trying and looking at the results over time.

Currently `subsystem-regression-tests` job fails if the first benchmarks
fail and there is no result for the second benchmark. Also dividing the
job makes the pipeline faster (currently it's a longest job)

cc https://github.com/paritytech/ci_cd/issues/969
cc @AndreiEres



---------

Co-authored-by: Andrei Eres <[email protected]>

This MR contains two major changes and some maintenance cleanup.  

## 1. Free Standing Pallet Benchmark Runner

Closes https://github.com/paritytech/polkadot-sdk/issues/3045, depends
on your runtime exposing the `GenesisBuilderApi` (like
https://github.com/paritytech/polkadot-sdk/pull/1492).

Introduces a new binary crate: `frame-omni-bencher`.  
It allows to directly benchmark a WASM blob - without needing a node or
chain spec.

This makes it much easier to generate pallet weights and should allow us
to remove bloaty code from the node.
It should work for all FRAME runtimes that dont use 3rd party host calls
or non `BlakeTwo256` block hashing (basically all polkadot parachains
should work).

It is 100% backwards compatible with the old CLI args, when the `v1`
compatibility command is used. This is done to allow for forwards
compatible addition of new commands.

### Example (full example in the Rust docs)

Installing the CLI:
```sh
cargo install --locked --path substrate/utils/frame/omni-bencher
frame-omni-bencher --help
```

Building the Westend runtime:
```sh
cargo build -p westend-runtime --release --features runtime-benchmarks
```

Benchmarking the runtime:
```sh
frame-omni-bencher v1 benchmark pallet --runtime target/release/wbuild/westend-runtime/westend_runtime.compact.compressed.wasm --all
```

## 2. Building the Benchmark Genesis State in the Runtime

Closes https://github.com/paritytech/polkadot-sdk/issues/2664

This adds `--runtime` and `--genesis-builder=none|runtime|spec`
arguments to the `benchmark pallet` command to make it possible to
generate the genesis storage by the runtime. This can be used with both
the node and the freestanding benchmark runners. It utilizes the new
`GenesisBuilder` RA and depends on having
https://github.com/paritytech/polkadot-sdk/pull/3412 deployed.

## 3. Simpler args for `PalletCmd::run`

You can do three things here to integrate the changes into your node:
- nothing: old code keeps working as before but emits a deprecated
warning
- delete: remove the pallet benchmarking code from your node and use the
omni-bencher instead
- patch: apply the patch below and keep using as currently. This emits a
deprecated warning at runtime, since it uses the old way to generate a
genesis state, but is the smallest change.

```patch
runner.sync_run(|config| cmd
-    .run::<HashingFor<Block>, ReclaimHostFunctions>(config)
+    .run_with_spec::<HashingFor<Block>, ReclaimHostFunctions>(Some(config.chain_spec))
)
```

## 4. Maintenance Change
- `pallet-nis` get a `BenchmarkSetup` config item to prepare its
counterparty asset.
- Add percent progress print when running benchmarks.
- Dont immediately exit on benchmark error but try to run as many as
possible and print errors last.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>

Use of `- >` instead of `- |` workarounds GitLab quirk when it crops
collapsed multiline `script:` section commands in its CI job logs.
This PR also fixes `- |` based `script:` steps to behave properly after
`- >` conversion.

Resolves https://github.com/paritytech/ci_cd/issues/972.

Fixes
https://github.com/paritytech/polkadot-sdk/issues/3884#issuecomment-2026058687

After moving regression tests to benchmarks
(https://github.com/paritytech/polkadot-sdk/pull/3741) we don't need to
filter tests anymore.

---------

Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>
Co-authored-by: Alin Dima <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Javier Viola <[email protected]>
Co-authored-by: Serban Iorga <[email protected]>
Co-authored-by: Adrian Catangiu <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Alexandru Vasile <[email protected]>
Co-authored-by: Niklas Adolfsson <[email protected]>
Co-authored-by: Dastan <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Clara van Staden <[email protected]>
Co-authored-by: Ron <[email protected]>
Co-authored-by: Vincent Geddes <[email protected]>
Co-authored-by: Svyatoslav Nikolsky <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Few fixes in CI to publish benchmarks jobs.

cc https://github.com/paritytech/ci_cd/issues/934

PR adds CI jobs that collect subsystem-benchmarks results and publishes
them to gh-pages.

cc https://github.com/paritytech/ci_cd/issues/934
cc @AndreiEres

Adds availability-write regression tests.
The results for the `availability-distribution` subsystem are volatile,
so I had to reduce the precision of the test.

Closes: https://github.com/paritytech/polkadot-sdk/issues/3704

Fixes https://github.com/paritytech/polkadot-sdk/issues/3530

### What's been done
- `subsystem-bench` has been split into two parts: a cli benchmark
runner and a library.
- The cli runner is quite simple. It just allows us to run `.yaml` based
test sequences. Now it should only be used to run benchmarks during
development.
- The library is used in the cli runner and in regression tests. Some
code is changed to make the library independent of the runner.
- Added first regression tests for availability read and write that
replicate existing test sequences.

### How we run regression tests
- Regression tests are simply rust integration tests without the
harnesses.
- They should only be compiled under the `subsystem-benchmarks` feature
to prevent them from running with other tests.
- This doesn't work when running tests with `nextest` in CI, so
additional filters have been added to the `nextest` runs.
- Each benchmark run takes a different time in the beginning, so we
"warm up" the tests until their CPU usage differs by only 1%.
- After the warm-up, we run the benchmarks a few more times and compare
the average with the exception using a precision.

### What is still wrong?
- I haven't managed to set up approval voting tests. The spread of their
results is too large and can't be narrowed down in a reasonable amount
of time in the warm-up phase.
- The tests start an unconfigurable prometheus endpoint inside, which
causes errors because they use the same 9999 port. I disable it with a
flag, but I think it's better to extract the endpoint launching outside
the test, as we already do with `valgrind` and `pyroscope`. But we still
use `prometheus` inside the tests.

### Future work
* https://github.com/paritytech/polkadot-sdk/issues/3528
* https://github.com/paritytech/polkadot-sdk/issues/3529
* https://github.com/paritytech/polkadot-sdk/issues/3530
* https://github.com/paritytech/polkadot-sdk/issues/3531

---------

Co-authored-by: Alexander Samusev <[email protected]>

Tests run as part of the `test-linux-stable` jobs could hang as shown
[here](https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5341393).
This PR adds a failsafe for such cases.

Add [forklift
caching](https://gitlab.parity.io/parity/infrastructure/ci_cd/forklift/forklift)
to remainig jobs

by .sh and .py scripts:
- cargo-check-each-crate x6 (`.gitlab/check-each-crate.py`)
- build-linux-stable (`polkadot/scripts/build-only-wasm.sh`)

by before_script:
- build-linux-substrate
- build-subkey-linux (with `.build-subkey` job)
- cargo-check-benches x2

**To disable feature set FORKLIFT_BYPASS variable to true in [project
settings in
gitlab](https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/settings/ci_cd)**
(forklift now handles FORKLIFT_BYPASS by itself)

<del>PR custom review is deprecated. Now `review-bot` performs these
functions.</del>
PR removes unused ci jobs and adjusts zombienet jobs for merge queues

cc @Bullrich

- Keep feature name short s/riscv-experimental/riscv
- Add to feature to CI build

---------

Co-authored-by: Bastian Köcher <[email protected]>

After GitLab update it started to remove artifacts from master if PR has
more than 20 commits. PR fixes this for `node-bench-regression-guard`
job

cc https://github.com/paritytech/ci_cd/issues/915

## Summary

This PR turns on `-D warnings` for `cargo-check-each-crate job` job to
fail on warnings e.g. like this:
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4673130

Before this PR, there was a warning and `cargo-check-each-crate` job did
not fail:
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4641444
```
warning: unused import: `ToTokens`
  --> substrate/primitives/api/proc-macro/src/utils.rs:22:34
   |
22 | use quote::{format_ident, quote, ToTokens};
   |                                  ^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` on by default
warning: `sp-api-proc-macro` (lib) generated 1 warning (run `cargo fix --lib -p sp-api-proc-macro` to apply 1 suggestion)
```

Fixes on the way:
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4641444
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4673265
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4673410
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4673681
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4673836
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4673941
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4674256
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4679328



## Questions
- [ ] why does this check triggers only `cargo check --locked`?
`--all-features` or `--all-targets` are not needed? Or aren't they
avoided intentionally?

---------

Co-authored-by: command-bot <>

This PR is a continuation of
https://github.com/paritytech/polkadot-sdk/pull/2102 and part of an
initiative started here https://hackmd.io/@romanp/rJ318ZCEp
What has been done:
- The content under `docs/*` (with the exception of `docs/mermaid`) has
been moved to `docs/contributor/`
- Developer Hub has been renamed to Polkadot SDK Docs, and the crate has
been renamed from `developer-hub` to `polkadot-sdk-docs`
- The content under `developer-hub/*` has been moved to `docs/sdk`

---
Original PR https://github.com/paritytech/polkadot-sdk/pull/2565, it has
been close due to too many rebase conflicts

---------

Co-authored-by: Serban Iorga <[email protected]>
Co-authored-by: Chevdor <[email protected]>
Co-authored-by: Egor_P <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Sergejs Kostjucenko <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Mira Ressel <[email protected]>

These two jobs are now required by github branch protection settings.

Co-authored-by: command-bot <>

This pull request is removing the Kusama and Polkadot runtime crates. As
still some crates dependent on the runtime crates, this pull request is
doing some more changes.

- It removes the `hostperfcheck` CLI command. This CLI command could
compare the current node against the standard hardware by doing some
checks. Later we added the hardware benchmark feature to Substrate. This
hardware benchmark is running on every node startup and prints a warning
if the current node is too slow. This makes this CLI command a duplicate
that was also depending on the kusama runtime.

- The pull request is removing the emulated integration tests that were
requiring the Kusama or Polkadot runtime crates.

- Updated ./docs/CONTRIBUTION.md to clarify how to use command bot and
update ui-tests locally or in PR
- Moved update-ui-tests.sh to a root of project to run them along for
substrate and polkadot together

---------

Co-authored-by: Mira Ressel <[email protected]>

* [ci] Move additional tests back to test-linux-stable

* run slow tests with all tests

* check if test upgrade_version_checks_should_work works

* rm comment

* [WIP][ci] Return publish-rustdoc

* rm files

* fix ref

* add build doc

* add comment

* move test-deterministic-wasm to test stage

* rm test-deterministic-wasm from root

* test publish

* enable pipeline

* add test_deterministic_wasm.sh for resolving conflicts

* rm unused bash script

* [ci] Remove runtime-benchmarks from tests

* Update .gitlab/pipeline/test.yml

Co-authored-by: Oliver Tale-Yazdi <[email protected]>

* remove ing

* move benchmark tests to additional tests

* rm -q option

* try release profile

* use testnet profile

* move to a separate job

* rm dup

---------

Co-authored-by: Oliver Tale-Yazdi <[email protected]>

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>

* [ci] Add DAG

* add dag

* add more dag and disable deny

* test cancel pipeline

* fix clippy

* [ci] Add missing components to CI

* add timestamp

* add pipeline-stopper-artifacts to jobs

* break clippy

* mv definition

* fix clippy

* comment timestamp, rm verbose

* commen cancel

* rm --verbose from build jobs

* disable rusty-cachier before_script

* enable timestamp back

* rollback .gitignore

* rollback .gitignore

* rollback .gitignore