PR changes forklift authentication to gcs

cc https://github.com/paritytech/ci_cd/issues/987

PR adds forklift settings and forklift to test-github-actions

cc https://github.com/paritytech/ci_cd/issues/939

close https://github.com/paritytech/ci_cd/issues/974

cc https://github.com/paritytech/ci_cd/issues/974

---------

Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <[email protected]>

Added check if forklift already exists in ci image as forklift binary is
now bundled with the ci-unified.
This is a temporary check for the transition period

Use of `- >` instead of `- |` workarounds GitLab quirk when it crops
collapsed multiline `script:` section commands in its CI job logs.
This PR also fixes `- |` based `script:` steps to behave properly after
`- >` conversion.

Resolves https://github.com/paritytech/ci_cd/issues/972.

PR adds CI jobs that collect subsystem-benchmarks results and publishes
them to gh-pages.

cc https://github.com/paritytech/ci_cd/issues/934
cc @AndreiEres

Add [forklift
caching](https://gitlab.parity.io/parity/infrastructure/ci_cd/forklift/forklift)
to remainig jobs

by .sh and .py scripts:
- cargo-check-each-crate x6 (`.gitlab/check-each-crate.py`)
- build-linux-stable (`polkadot/scripts/build-only-wasm.sh`)

by before_script:
- build-linux-substrate
- build-subkey-linux (with `.build-subkey` job)
- cargo-check-benches x2

**To disable feature set FORKLIFT_BYPASS variable to true in [project
settings in
gitlab](https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/settings/ci_cd)**
(forklift now handles FORKLIFT_BYPASS by itself)

Usual chore after bumping Rust toolchain version.

cc https://github.com/paritytech/ci_cd/issues/926

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Dónal Murray <[email protected]>
Co-authored-by: Vladimir Istyufeev <[email protected]>

In CI jobs I see calls to `forklift clean`, which apparently does not
exist.
From CI logs:
```
1m$ forklift clean[0;m
Cargo cache management utility

Usage:
  forklift [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  serve       Run forklift coordinator server for current location
  start       Start detached forklift coordinator server for current location
  stop        Stop forklift coordinator server for current location

Flags:
  -c, --compression string     Compression algorithm to use
                               Available: none, xz (default "zstd")
  -h, --help                   help for forklift
  -p, --param stringToString   map of additional parameters
                                ex: -p S3_BUCKET_NAME=my_bucket (default [])
  -s, --storage string         Storage driver
                               Available: s3, fs (default "s3")
  -v, --verbose string         Available: panic, fatal, error, warn, warning, info, debug, trace (default "info")
      --version                version for forklift

Use "forklift [command] --help" for more information about a command.
```

<del>PR custom review is deprecated. Now `review-bot` performs these
functions.</del>
PR removes unused ci jobs and adjusts zombienet jobs for merge queues

cc @Bullrich

bump polkavm-x deps to 0.4.0

close https://github.com/paritytech/ci_cd/issues/900

cc https://github.com/paritytech/ci_cd/issues/900

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Dónal Murray <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: command-bot <>

This PR adds cargo caching feature with custom ['forklift'
tool](https://gitlab.parity.io/parity/infrastructure/ci_cd/forklift/forklift)
Forklift acts as RUSTC_WRAPPER, intercepts rustc calls and stores
produced artifacts in S3 bucket (see [forklift
readme](https://gitlab.parity.io/parity/infrastructure/ci_cd/forklift/forklift/-/blob/main/README.MD?ref_type=heads)
for detailed description)

All settings are made in [`.forklift` job's
before_script](https://github.com/paritytech/polkadot-sdk/blob/es/forklift-test/.gitlab-ci.yml#L119)
and affect all jobs that extend `.docker-env` job

To disable feature set `FORKLIFT_BYPASS` variable to true in [project
settings in
gitlab](https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/settings/ci_cd)

Docker images from paritypr are also used in testnets. PR reverts docker
tags naming to a more convenient.

cc @PierreBesson

Includes:
 - bump `pjs` modules versions
- re-enable test disable in
https://github.com/paritytech/polkadot-sdk/pull/2294

---------

Co-authored-by: Bastian Köcher <[email protected]>

New release includes logic to move all jobs to `spot instances`.
Thx!

PR prepares CI to the GitHub Merge Queues. All github actions that were
running in PR adjusted so they can run in the merge queues. Zombienet
jobs will do nothing during PRs but they will run during merge queues.

Jobs that will be skipped during PR:
 - all zombienet jobs
 - all publish docker jobs

Jobs that will be skipped during merge queue:
 - check-labels
 - check-prdoc
 - pr-custom-review
 - review trigger

cc https://github.com/paritytech/ci_cd/issues/862

This PR prepares chains specs for _native-runtime-free_  world.

This PR has following changes:
- `substrate`:
  - adds support for:
- JSON based `GenesisConfig` to `ChainSpec` allowing interaction with
runtime `GenesisBuilder` API.
- interacting with arbitrary runtime wasm blob to[
`chain-spec-builder`](https://github.com/paritytech/substrate/blob/3ef576eaeb3f42610e85daecc464961cf1295570/bin/utils/chain-spec-builder/src/lib.rs#L46)
command line util,
- removes
[`code`](https://github.com/paritytech/substrate/blob/3ef576eaeb3f42610e85daecc464961cf1295570/frame/system/src/lib.rs#L660)
from `system_pallet`
  - adds `code` to the `ChainSpec`
- deprecates
[`ChainSpec::from_genesis`](https://github.com/paritytech/substrate/blob/3ef576eaeb3f42610e85daecc464961cf1295570/client/chain-spec/src/chain_spec.rs#L263),
but also changes the signature of this method extending it with `code`
argument.
[`ChainSpec::builder()`](https://github.com/paritytech/substrate/blob/20bee680ed098be7239cf7a6b804cd4de267983e/client/chain-spec/src/chain_spec.rs#L507)
should be used instead.
- `polkadot`:
- all references to `RuntimeGenesisConfig` in `node/service` are
removed,
- all
`(kusama|polkadot|versi|rococo|wococo)_(staging|dev)_genesis_config`
functions now return the JSON patch for default runtime `GenesisConfig`,
  - `ChainSpecBuilder` is used, `ChainSpec::from_genesis` is removed,

- `cumulus`:
  - `ChainSpecBuilder` is used, `ChainSpec::from_genesis` is removed,
- _JSON_ patch configuration used instead of `RuntimeGenesisConfig
struct` in all chain specs.
  
---------

Co-authored-by: command-bot <>
Co-authored-by: Javier Viola <[email protected]>
Co-authored-by: Davide Galassi <[email protected]>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Kevin Krone <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

CI image has been updated in the shared snippet, reverting the variable
back.

Run CI using new image with nightly 2023-11-01

cc https://github.com/paritytech/polkadot-sdk/issues/2113
cc https://github.com/paritytech/ci_cd/issues/896

This version includes:
- Move `spot` usage in CI to 50%
- Fix `PodMonitor`, metrics will be relayed to grafana

Includes:
-  PoC for new `spot instance` infra for jobs.
- Fixes in test-runner/cli

Co-authored-by: command-bot <>

Bump zombiente version. This version includes the fixes needed for
`mixnet`.
Thx!

Includes:
- fix for https://github.com/paritytech/zombienet/issues/1360 (root
cause of https://github.com/paritytech/polkadot-sdk/issues/1647)
- Improve default concurrency to spawn nodes.

Also, fix `var` name in CI file.

cc @skunert

Update CI tag for docker vm runners

---------

Co-authored-by: alvicsam <[email protected]>

* [ci] Move additional tests back to test-linux-stable

* run slow tests with all tests

* check if test upgrade_version_checks_should_work works

* rm comment

* [WIP][ci] Return publish-rustdoc

* rm files

* fix ref

* add build doc

* add comment

* move test-deterministic-wasm to test stage

* rm test-deterministic-wasm from root

* test publish

* enable pipeline

* add test_deterministic_wasm.sh for resolving conflicts

* rm unused bash script

* zombienet: use test-parachain image for the slashing test

* use the right image

* try polkadot-parachain image

* try naming collator alice 🙈



* add needed job for the pipeline

* fix user id in polkadot-parachain-debug image

* small tweaks to the test

* another small tweak

* yet another small tweak

* bump zombienet version

---------

Co-authored-by: Javier Viola <[email protected]>

* bump zombienet to latest version

* add env var

* fix upgrade node text, env var for downloading artifacts

* [ci] Add DAG

* add dag

* add more dag and disable deny

* test cancel pipeline

* fix clippy

* [ci] Add missing components to CI

* add timestamp

* add pipeline-stopper-artifacts to jobs

* break clippy

* mv definition

* fix clippy

* comment timestamp, rm verbose

* commen cancel

* rm --verbose from build jobs

* disable rusty-cachier before_script

* enable timestamp back

* rollback .gitignore

* rollback .gitignore

* rollback .gitignore