PR prepares CI to the GitHub Merge Queues. All github actions that were
running in PR adjusted so they can run in the merge queues. Zombienet
jobs will do nothing during PRs but they will run during merge queues.

Jobs that will be skipped during PR:
 - all zombienet jobs
 - all publish docker jobs

Jobs that will be skipped during merge queue:
 - check-labels
 - check-prdoc
 - pr-custom-review
 - review trigger

cc https://github.com/paritytech/ci_cd/issues/862

**_PR migrated from https://github.com/paritytech/polkadot/pull/6782_** 

This PR will upgrade the network protocol to version 3 -> VStaging which
will later be renamed to V3. This version introduces a new kind of
assignment certificate that will be used for tranche0 assignments.
Instead of issuing/importing one tranche0 assignment per candidate,
there will be just one certificate per relay chain block per validator.
However, we will not be sending out the new assignment certificates,
yet. So everything should work exactly as before. Once the majority of
the validators have been upgraded to the new protocol version we will
enable the new certificates (starting at a specific relay chain block)
with a new client update.

There are still a few things that need to be done:

- [x] Use bitfield instead of Vec<CandidateIndex>:
https://github.com/paritytech/polkadot/pull/6802


  - [x] Fix existing approval-distribution and approval-voting tests
  - [x] Fix bitfield-distribution and statement-distribution tests
  - [x] Fix network bridge tests
  - [x] Implement todos in the code
  - [x] Add tests to cover new code
  - [x] Update metrics
  - [x] Remove the approval distribution aggression levels: TBD PR
  - [x] Parachains DB migration 
  - [x] Test network protocol upgrade on Versi
  - [x] Versi Load test
  - [x] Add Zombienet test
  - [x] Documentation updates
- [x] Fix for sending DistributeAssignment for each candidate claimed by
a v2 assignment (warning: Importing locally an already known assignment)
 - [x]  Fix AcceptedDuplicate
 - [x] Fix DB migration so that we can still keep old data.
 - [x] Final Versi burn in

---------

Signed-off-by: Andrei Sandu <[email protected]>
Signed-off-by: Alexandru Gheorghe <[email protected]>
Co-authored-by: Alexandru Gheorghe <[email protected]>

Otherwise the return code is not correctly propagated (ref
https://github.com/ggwpez/zepter/pull/48

).

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Co-authored-by: Sergejs Kostjucenko <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Runtime migration CI is currently failing
(https://gitlab.parity.io/parity/mirrors/polkadot-sdk/builds/4122083)
for the contracts testnet due to unnecessary panicing in a `pre_upgrade`
hook.

Soon idempotency will be enforced
https://github.com/paritytech/try-runtime-cli/issues/42, in the mean
time we need to manually fix these issues as they arise.

---

also removes backticks from the string in `echo`, which caused a
'command not found' error in ci output

Need to run this job in master to make it `Required`

Adds a config file that allows to run `zepter` without any arguments in
the workspace to address all issues.
A secondary workflow for the CI is provided as `zepter run check`. Both
the formatting and linting are now in one check for efficiancy.

The latest version also detects some more things that `featalign` was
already showing.

Error message [in the
CI](https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/3916205)
now looks like this:
```pre
...
crate 'test-parachains' (/Users/vados/Documents/work/polkadot-sdk/polkadot/parachain/test-parachains/Cargo.toml)
  feature 'std'
    must propagate to:
      parity-scale-codec
Found 55 issues (run with --fix to fix).
Error: Command 'lint propagate-feature' failed with exit code 1

Polkadot-SDK uses the Zepter CLI to detect abnormalities in the feature configuration.
It looks like one more more checks failed; please check the console output. You can try to automatically address them by running `zepter`.
Otherwise please ask directly in the Merge Request, GitHub Discussions or on Matrix Chat, thank you.

For more information, see:
  - https://github.com/paritytech/polkadot-sdk/issues/1831
  - https://github.com/ggwpez/zepter


```

TODO:
- [x] Check that CI fails correctly

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

`cargo install` takes a long time in CI. We want to run it relatively
frequently without chewing through so much compute (see
https://github.com/paritytech/ci_cd/issues/771) so I added automatic
binary releases to the try-runtime-cli repo.

A small added benefit is we can use it in our existing
`on-runtime-upgrade` checks, which should cut their execution time by
about half.

cumulus: add asset-hub-rococo runtime based on asset-hub-kusama and add asset-bridging support to it (#1215)

This commit adds Rococo Asset Hub dedicated runtime so we can test new
features here, before merging them in Kusama Asset Hub.
Also adds one such feature: asset transfer over bridge (Rococo AssetHub
<> Wococo AssetHub)

- clone `asset-hub-kusama-runtime` -> `asset-hub-rococo-runtime`
- make it use Rococo primitives, names, assets, constants, etc
- add asset-transfer-over-bridge support to Rococo AssetHub <> Wococo
AssetHub

Fixes #1128

---------

Co-authored-by: Branislav Kontur <[email protected]>
Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: Francisco Aguirre <[email protected]>

Co-authored-by: Mira Ressel <[email protected]>

These two jobs are now required by github branch protection settings.

Co-authored-by: command-bot <>

Makes SPs first class citizens along with the relay chains in the
context of our CI runtime upgrade checks.

## Code changes

- Sets missing current storage version in `uniques` pallet
- Adds multisig V1 migration to run where it was missing
- Removes executed migration whos pre/post hooks were failing from
collectives runtime
- Initializes storage versions for SP pallets added after genesis
- Originally I was going to wait for
https://github.com/paritytech/polkadot-sdk/pull/1297 to be merged so
this wouldn't need to be done manually, but it doesn't seem like it'll
be merged any time soon so I've decided to set them manually to unblock
this

## CI changes

- Removed dependency of `westend` runtime upgrades being complete prior
to other ones running. I assume it is supposed to cache the
`try-runtime` build for a performance benefit, but it seems it wasn't
working. Maybe someone from the CI team can look into this or explain
why it needs to be there?

- Adds check-runtime-migration jobs for Parity asset-hubs, bridge-hubs
and contract chains

- Updated VARIABLES to accomodate the `kusama-runtime` package being
renamed to `staging-kusama-runtime` in
https://github.com/paritytech/polkadot-sdk/pull/1241

- Added `EXTRA_ARGS` variable to `check-runtime-migration`, and set
`--no-weight-warnings` to the relay chain runtime upgrade checks (relay
chains don't have weight restrictions).

This pull request is removing the Kusama and Polkadot runtime crates. As
still some crates dependent on the runtime crates, this pull request is
doing some more changes.

- It removes the `hostperfcheck` CLI command. This CLI command could
compare the current node against the standard hardware by doing some
checks. Later we added the hardware benchmark feature to Substrate. This
hardware benchmark is running on every node startup and prints a warning
if the current node is too slow. This makes this CLI command a duplicate
that was also depending on the kusama runtime.

- The pull request is removing the emulated integration tests that were
requiring the Kusama or Polkadot runtime crates.

[`RococoGenesisExt`](https://github.com/paritytech/polkadot-sdk/blob/a414ea7515c9cdc81f1d12410e646afc148250e8/polkadot/node/service/src/chain_spec.rs#L152-L171

)
is removed. It was the hack to allow overwriting
`EpochDurationInBlocks`. Removal of `RococGenesisExt` prevents from
manipulating the state to change the runtime constants.

Changes:
- Environment variable which controls the `time::EpochDurationInBlocks`
value was added: `ROCOCO_EPOCH_DURATION` (epoch duration will be set to
the value of env),
- `10,100,600` versions of rococo-runtime are built in CI and put into `polkadot-debug` docker image.

`rococo-runtime` building examples:
- to build runtime for `versi_staging_testnet` which had
EpochDurationInBlocks set to 100:
  ```
ROCOCO_EPOCH_DURATION=100 cargo build --features=fast-runtime -p
rococo-runtime
  ```
- to build runtime for `wococo_development`
  ```
ROCOCO_EPOCH_DURATION=10 cargo build --features=fast-runtime -p
rococo-runtime
  ```
- to build `versi-staging` chain spec:
  ```
ROCOCO_EPOCH_DURATION=100 cargo run -p polkadot --features=fast-runtime
-- build-spec --chain versi-staging --raw
  ```
- to build `wococo-dev` chain spec:
  ```
ROCOCO_EPOCH_DURATION=10 cargo run -p polkadot --features=fast-runtime
-- build-spec --chain wococo-dev --raw
  ```

It is also possible to change the epoch duration by replacing the `code` field in the chain spec with the hex dump of pre-built runtime wasm blob (because the epoch duration is hard-coded into wasm blob).

---------

Co-authored-by: Bastian Köcher <[email protected]>

- Async-backing related primitives are stable `primitives::v6`
- Async-backing API is now part of `api_version(7)`
- It's enabled on Rococo and Westend runtimes

---------

Signed-off-by: Andrei Sandu <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>

Add debug_assertions to WASM builds in the short-benchmarks CI job.

Disallow warnings, show a full backtrace and remove the WASM output
colour to improve information from this CI step in the event of a failure.

This is motivated by a
[case](https://github.com/paritytech/polkadot-sdk/pull/1676/commits/23d64918be3ca68413050383c5d8a73f63d451dd)
where a benchmark was misconfigured in master and failing to send
notifications, but passing CI. We don't want this to fail if the problem
happens in production, but ideally we'd know from the CI if it was
misconfigured and the messages weren't getting sent, as that could
(would?) affect the weights that the benchmark generates.

Enabling debug-assertions in WASM builds for the short-benchmarks would
catch "soft" failures like this in future.

- Updated ./docs/CONTRIBUTION.md to clarify how to use command bot and
update ui-tests locally or in PR
- Moved update-ui-tests.sh to a root of project to run them along for
substrate and polkadot together

---------

Co-authored-by: Mira Ressel <[email protected]>

Includes:
- fix for https://github.com/paritytech/zombienet/issues/1360 (root
cause of https://github.com/paritytech/polkadot-sdk/issues/1647)
- Improve default concurrency to spawn nodes.

Also, fix `var` name in CI file.

cc @skunert

This pull request removes the Polkadot and Kusama native runtime from
the polkadot node. This brings some implications with it:

There are no more kusama/polkadot-dev chain specs available. We will
need to write some tooling in the fellowship repo to provide them
easily.

The try-runtime job for polkadot & kusama is not available anymore as we
don't have the dev chain specs anymore.

Certain benchmarking commands will also not work until we migrate them
to use a runtime api.

Some crates in utils are still depending on the polkadot/kusama native
runtime that will also need to be fixed.

Port of: https://github.com/paritytech/polkadot/pull/7467

PR adds a job that publishes implementers guide

cc https://github.com/paritytech/polkadot-sdk/issues/1614
cc https://github.com/paritytech/ci_cd/issues/879

Add logic to check if the `BUILD_RELEASE_VERSION` tag is available in
docker registry, if not calculate the previous version to use as
secondary image. This fix the issue in test using the `secondary image`
and bumping the `NODE_VERSION`. (e.g #1495)

Changes:
- Disable runtime logging in benchmarks by building with a specific
profile

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

When `process_file` is run by `xargs`, it is executed inside a new shell
without access to variables defined outside of its scope.

This resulted in `script_content` being an empty string.

By exporting `script_content` prior to running `xargs` it is available
inside the new shells.

* Delete staking miner

New repo should be used instead https://github.com/paritytech/staking-miner-v2



Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Remove staking-miner CI jobs

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* [ci] Move additional tests back to test-linux-stable

* run slow tests with all tests

* check if test upgrade_version_checks_should_work works

* rm comment

* [WIP][ci] Return publish-rustdoc

* rm files

* fix ref

* add build doc

* add comment

* move test-deterministic-wasm to test stage

* rm test-deterministic-wasm from root

* test publish

* enable pipeline

* add test_deterministic_wasm.sh for resolving conflicts

* rm unused bash script

* zombienet: use test-parachain image for the slashing test

* use the right image

* try polkadot-parachain image

* try naming collator alice 🙈



* add needed job for the pipeline

* fix user id in polkadot-parachain-debug image

* small tweaks to the test

* another small tweak

* yet another small tweak

* bump zombienet version

---------

Co-authored-by: Javier Viola <[email protected]>

* add ghw and scripts for docker image deployment

* debug

* add permissions for content

* fix path to the bin folder

* add tags

* rename env

* fix path to docker file

* make polkadot-parachain executable

* fix typo

* fix more typos

* test

* revert back  use of  working directory

* mke bin executable in the artifacts folder

* use cd instead of working directory

* change path to cash

* fix path to cash

* change cache key

* delete old flows

* addressed PR comments

* fix path

* reorg docker files

* inject simple analytics

* comments

* fix chown

* comments

* doc features

* [ci] Remove runtime-benchmarks from tests

* Update .gitlab/pipeline/test.yml

Co-authored-by: Oliver Tale-Yazdi <[email protected]>

* remove ing

* move benchmark tests to additional tests

* rm -q option

* try release profile

* use testnet profile

* move to a separate job

* rm dup

---------

Co-authored-by: Oliver Tale-Yazdi <[email protected]>

* Added short-benchmarks for cumulus

* Added `--bin` flag for short-benchmarks

* fix dependency for short-benchmark-cumulus

* Fixed benchmark with new XCM::V3 `MAX_INSTRUCTIONS_TO_DECODE`

* Fixed benchmark for bridge messages pallets

---------

Co-authored-by: alvicsam <[email protected]>
Co-authored-by: Javier Viola <[email protected]>
Co-authored-by: Alexander Samusev <[email protected]>

* Fix deterministic WASM check

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix build-staking-miner

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Remove Kusama and Polkadot runtime-migration checks

Will be removed in https://github.com/paritytech/polkadot-sdk/pull/1304

 anyway.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* fix tests

Signed-off-by: Andrei Sandu <[email protected]>

* ZOMBIENET_INTEGRATION_TEST_SECONDARY_IMAGE

Signed-off-by: Andrei Sandu <[email protected]>

* deleted by mistake

Signed-off-by: Andrei Sandu <[email protected]>

* remove LOCAL_DIR override

Signed-off-by: Andrei Sandu <[email protected]>

* Fix secondary image

Signed-off-by: Andrei Sandu <[email protected]>

* fix get BUILD_RELEASE_VERSION in pipeline

---------

Signed-off-by: Andrei Sandu <[email protected]>
Co-authored-by: Javier Viola <[email protected]>

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>