Tests run as part of the `test-linux-stable` jobs could hang as shown
[here](https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5341393).
This PR adds a failsafe for such cases.

Fixes https://github.com/paritytech/polkadot-sdk/issues/3144

Builds on top of https://github.com/paritytech/polkadot-sdk/pull/3229

### Summary
Some preparations for Runtime to support elastic scaling, guarded by
config node features bit `FeatureIndex::ElasticScalingMVP`. This PR
introduces a per-candidate `CoreIndex` but does it in a hacky way to
avoid changing `CandidateCommitments`, `CandidateReceipts` primitives
and networking protocols.

#### Including `CoreIndex` in `BackedCandidate`
If the `ElasticScalingMVP` feature bit is enabled then
`BackedCandidate::validator_indices` is extended by 8 bits.
The value stored in these bits represents the assumed core index for the
candidate.

It is temporary solution which works by creating a mapping from
`BackedCandidate` to `CoreIndex` by assuming the `CoreIndex` can be
discovered by checking in which validator group the validator that
signed the statement is.

TODO:
- [x] fix tests
- [x] add new tests
- [x] Bump runtime API for Kusama, so we have that node features thing!
-> https://github.com/polkadot-fellows/runtimes/pull/194

---------

Signed-off-by: Andrei Sandu <[email protected]>
Signed-off-by: alindima <[email protected]>
Co-authored-by: alindima <[email protected]>

# Runtime side for PoV Reclaim

## Implementation Overview
- Hostfunction to fetch the storage proof size has been added to the
PVF. It uses the size tracking recorder that was introduced in my
previous PR.
- Mechanisms to use the reclaim HostFunction have been introduced.
- 1. A SignedExtension that checks the node-reported proof size before
and after application of an extrinsic. Then it reclaims the difference.
- 2. A manual helper to make reclaiming easier when manual interaction
is required, for example in `on_idle` or other hooks.
- In order to utilize the manual reclaiming, I modified `WeightMeter` to
support the reduction of consumed weight, at least for storage proof
size.

## How to use
To enable the general functionality for a parachain:
1. Add the SignedExtension to your parachain runtime. 
2. Provide the HostFunction to the node
3. Enable proof recording during block import

## TODO
- [x] PRDoc

---------

Co-authored-by: Dmitry Markin <[email protected]>
Co-authored-by: Davide Galassi <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

The rationale behind this, is that it may be useful for some users
actually disable RPC batch requests or limit them by length instead of
the total size bytes of the batch.

This PR adds two new CLI options:

```
--rpc-disable-batch-requests - disable batch requests on the server
--rpc-max-batch-request-len <LEN> - limit batches to LEN on the server.
```

Fixes https://github.com/paritytech/polkadot-sdk/issues/3352

Fix image tag to use in merge queues tests, currently is failing because
is using the wrong tag (e.g
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5224342).

Thx!

PR allows `build-runtimes-polkavm` to fail. Also added DAG to run the
job immediately.

cc https://github.com/paritytech/ci_cd/issues/945
cc https://github.com/paritytech/polkadot-sdk/issues/3352

Related to https://github.com/paritytech/polkadot-sdk/issues/3242

Bridges: add test 0002 to CI

Related to https://github.com/paritytech/polkadot-sdk/issues/3176

This PR only adds the first bridge zombienet test back to the CI after
fixing it, reverting
https://github.com/paritytech/polkadot-sdk/pull/3071

Credits to @svyatonik for building all the CI infrastructure around
this.

Add [forklift
caching](https://gitlab.parity.io/parity/infrastructure/ci_cd/forklift/forklift)
to remainig jobs

by .sh and .py scripts:
- cargo-check-each-crate x6 (`.gitlab/check-each-crate.py`)
- build-linux-stable (`polkadot/scripts/build-only-wasm.sh`)

by before_script:
- build-linux-substrate
- build-subkey-linux (with `.build-subkey` job)
- cargo-check-benches x2

**To disable feature set FORKLIFT_BYPASS variable to true in [project
settings in
gitlab](https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/settings/ci_cd)**
(forklift now handles FORKLIFT_BYPASS by itself)

This PR improves compatibility with RISC-V and PolkaVM, allowing more
runtimes to successfully compile.

In particular, it makes the following changes:

- The `sp-mmr-primitives` and `sp-consensus-beefy` crates
unconditionally required an `std`-only dependency; now they only require
those dependencies when the `std` feature is actually enabled. (Our
RISC-V target is, unlike WASM, a true `no_std` target where you can't
accidentally use stuff from `std` anymore.)
- One of our dependencies (the `bitvec` trace) uses a crate called
`radium` which doesn't compile under RISC-V due to incomplete
autodetection logic in their `build.rs` file. The good news is that this
is already fixed in the newest upstream version of `radium`, and the
newest version of `bitvec` uses it. The bad news is that the newest
version of `bitvec` is not currently released on crates.io, so we can't
use it. I've [created an
issue](https://github.com/ferrilab/ferrilab/issues/5) asking for a new
release, but in the meantime I forked the currently used `radium` 0.7,
[fixed the faulty
logic](https://github.com/paritytech/radium-0.7-fork/commit/ed66c8a294b138c67f93499644051d97d4c7fbda)
and used cargo's patching capabilities to use it for the RISC-V runtime
builds. This might be a little hacky, but it is the least intrusive way
to fix the problem, doesn't affect WASM builds at all, and we can
trivially remove it once a new `bitvec` is released.
- The new runtimes are added to the CI to make sure their compilation
doesn't break.

This PR adds initial support for building RISC-V runtimes targeting
PolkaVM.

- Setting the `SUBSTRATE_RUNTIME_TARGET=riscv` environment variable will
now build a RISC-V runtime instead of a WASM runtime.
- This only adds support for *building* runtimes; running them will need
a PolkaVM-based executor, which I will add in a future PR.
- Only building the minimal runtime is supported (building the Polkadot
runtime doesn't work *yet* due to one of the dependencies).
- The builder now sets a `substrate_runtime` cfg flag when building the
runtimes, with the idea being that instead of doing `#[cfg(not(feature =
"std"))]` or `#[cfg(target_arch = "wasm32")]` to detect that we're
building a runtime you'll do `#[cfg(substrate_runtime)]`. (Switching the
whole codebase to use this will be done in a future PR; I deliberately
didn't do this here to keep this PR minimal and reviewable.)
- Further renaming of things (e.g. types, environment variables and proc
macro attributes having "wasm" in their name) to be target-agnostic will
also be done in a future refactoring PR (while keeping backwards
compatibility where it makes sense; I don't intend to break anyone's
workflow or create unnecessary churn).
- This PR also fixes two bugs in the `wasm-builder` crate:
* The `RUSTC` environment variable is now removed when invoking the
compiler. This prevents the toolchain version from being overridden when
called from a `build.rs` script.
* When parsing the `rustup toolchain list` output the `(default)` is now
properly stripped and not treated as part of the version.
- I've also added a minimal CI job that makes sure this doesn't break in
the future. (cc @paritytech/ci)

cc @athei



------

Also, just a fun little tidbit: quickly comparing the size of the built
runtimes it seems that the PolkaVM runtime is slightly smaller than the
WASM one. (`production` build, with the `names` section substracted from
the WASM's size to keep things fair, since for the PolkaVM runtime we're
currently stripping out everything)

- `.wasm`: 625505 bytes
- `.wasm` (after wasm-opt -O3): 563205 bytes
- `.wasm` (after wasm-opt -Os): 562987 bytes
- `.wasm` (after wasm-opt -Oz): 536852 bytes
- `.polkavm`: ~~580338 bytes~~ 550476 bytes (after enabling extra target
features; I'll add those in another PR once we have an executor working)

---------

Co-authored-by: Bastian Köcher <[email protected]>

Fixes: https://github.com/paritytech/polkadot-sdk/issues/3165

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>

Found it when trying to run: 
```
bot bench-all pallet --pallet=pallet_balances
```
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5081585

## TODO
- [x] check/fix command bot for `people-westend-dev` /
`people-rococo-dev`
https://github.com/paritytech/command-bot-scripts/pull/67

closes https://github.com/paritytech/parity-bridges-common/issues/2796

This partially reverts the #2439 - there are some changes (unrelated to
CI) that we still want to keep. The reason of that removal is that with
async backing enabled for Rococo AH (and for other chains in the near
future), we see a lot of issues there (because we run `14` nodes +
additional standalone process within a same container and it causes a
lot of timeouts). There's no way known to me to fix it right now, so
we're removing those tests hopefully temporarily to keep CI green

Closes https://github.com/paritytech/polkadot-sdk/issues/2787

Regenerating the DB snapshot after fixing:
https://github.com/paritytech/polkadot-sdk/pull/2919

Now we can add some BEEFY checks to `0002-validators-warp-sync`

This version includes
- Performance improvements.
- Minor fixes.

Brridges zombienet tests are non-standard - zombienet currently missing
multiple relay chains support (see e.g.
https://github.com/paritytech/zombienet/pull/796), so we need to go live
with two relay networks, their parachains + custom test runner (which
e.g. doesn't shutdown net when its tests are finished and instead waits
for both networks tests to complete). So we are stuck with native
zombienet provider => this PR is an attempt to gather everything in a
single docker container and run tests there ~Draft, because it is far
from finishing - what I want now is to see how it works on CI~

<del>PR custom review is deprecated. Now `review-bot` performs these
functions.</del>
PR removes unused ci jobs and adjusts zombienet jobs for merge queues

cc @Bullrich

Part of https://github.com/paritytech/polkadot-sdk/issues/2787

This is an initial PR that adds some basic BEEFY checks to the warp sync
zombienet tests. To be more specific, it does the following:
- Changes the snapshot used by the warp sync zombienet tests to one
built from an updated version of the kitchensink runtime, that supports
BEEFY
- Adds some basic BEEFY checks to the warp sync zombienet tests
- Deduplicates some params of the warp sync zombienet tests, making them
easier to extend

Closes #1591.

The purpose of this PR is filter out backing statements from the network
signed by disabled validators. This is just an optimization, since we
will do filtering in the runtime in #1863 to avoid nodes to filter
garbage out at block production time.

- [x] Ensure it's ok to fiddle with the mask of manifests
- [x] Write more unit tests
- [x] Test locally
- [x] simple zombienet test
- [x] PRDoc

---------

Co-authored-by: Tsvetomir Dimitrov <[email protected]>

This adds a Zombienet test for Coretime.

Requires: https://github.com/paritytech/polkadot-sdk/pull/2862

---------

Co-authored-by: Javier Viola <[email protected]>
Co-authored-by: Javier Viola <[email protected]>

- Keep feature name short s/riscv-experimental/riscv
- Add to feature to CI build

---------

Co-authored-by: Bastian Köcher <[email protected]>

Many clippy lints usually enforced by `-Dcomplexity` and `-Dcorrectness`
are not caught by CI as they are gated by `features`, like
`runtime-benchmarks`, while the clippy CI job runs with only the default
features for all targets.

This PR also adds a CI step to run clippy with `--all-features` to
ensure the code quality is maintained behind feature gates from now on.

To improve local development, clippy lints are downgraded to warnings,
but they still will result in an error at CI due to the `-Dwarnings`
rustflag.

---------

Co-authored-by: Liam Aharon <[email protected]>

New runtimes for the Coretime Chain (a.k.a. "Broker Chain") described in
RFC-1.

Replaces https://github.com/paritytech/cumulus/pull/2889


- [x] Add Agile Coretime pallet
https://github.com/paritytech/substrate/pull/14568
- [x] Generate chain specs for local and testnets
- [x] Deploy parachain on Rococo - Done:
[rococo-coretime-rpc.polkadot.io](https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Frococo-coretime-rpc.polkadot.io#/explorer)

DevOps issue for Aura keygen:
https://github.com/paritytech/devops/issues/2725

Edit (Dónal): This PR is mainly for Rococo, the Westend runtime is a
shell with no `Broker` pallet. The Rococo runtime has the broker calls
filtered for initial deployment.

---------

Co-authored-by: Dónal Murray <[email protected]>
Co-authored-by: 0xmovses <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Marcin S. <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: Branislav Kontur <[email protected]>

After GitLab update it started to remove artifacts from master if PR has
more than 20 commits. PR fixes this for `node-bench-regression-guard`
job

cc https://github.com/paritytech/ci_cd/issues/915

Remove old version for `cli_args`, since this was fixed in the latest
version of zombienet and the `latest` version of polkadot introduce the
new flag `--insecure-validator-i-know-what-i-do`.

Fix jobs like
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4726174
Thx!

We currently use a bit of a hack in `.cargo/config` to make sure that
clippy isn't too annoying by specifying the list of lints.

There is now a stable way to define lints for a workspace. The only down
side is that every crate seems to have to opt into this so there's a
*few* files modified in this PR.

Dependencies:

- [x] PR that upgrades CI to use rust 1.74 is merged.

---------

Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: Branislav Kontur <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>

Initial implementation for the plan discussed here: https://github.com/paritytech/polkadot-sdk/issues/701
Built on top of https://github.com/paritytech/polkadot-sdk/pull/1178
v0: https://github.com/paritytech/polkadot/pull/7554,

## Overall idea

When approval-voting checks a candidate and is ready to advertise the
approval, defer it in a per-relay chain block until we either have
MAX_APPROVAL_COALESCE_COUNT candidates to sign or a candidate has stayed
MAX_APPROVALS_COALESCE_TICKS in the queue, in both cases we sign what
candidates we have available.

This should allow us to reduce the number of approvals messages we have
to create/send/verify. The parameters are configurable, so we should
find some values that balance:

- Security of the network: Delaying broadcasting of an approval
shouldn't but the finality at risk and to make sure that never happens
we won't delay sending a vote if we are past 2/3 from the no-show time.
- Scalability of the network: MAX_APPROVAL_COALESCE_COUNT = 1 &
MAX_APPROVALS_COALESCE_TICKS =0, is what we have now and we know from
the measurements we did on versi, it bottlenecks
approval-distribution/approval-voting when increase significantly the
number of validators and parachains
- Block storage: In case of disputes we have to import this votes on
chain and that increase the necessary storage with
MAX_APPROVAL_COALESCE_COUNT * CandidateHash per vote. Given that
disputes are not the normal way of the network functioning and we will
limit MAX_APPROVAL_COALESCE_COUNT in the single digits numbers, this
should be good enough. Alternatively, we could try to create a better
way to store this on-chain through indirection, if that's needed.

## Other fixes:
- Fixed the fact that we were sending random assignments to
non-validators, that was wrong because those won't do anything with it
and they won't gossip it either because they do not have a grid topology
set, so we would waste the random assignments.
- Added metrics to be able to debug potential no-shows and
mis-processing of approvals/assignments.

## TODO:
- [x] Get feedback, that this is moving in the right direction. @ordian
@sandreim @eskimor

 @burdges, let me know what you think.
- [x] More and more testing.
- [x]  Test in versi.
- [x] Make MAX_APPROVAL_COALESCE_COUNT &
MAX_APPROVAL_COALESCE_WAIT_MILLIS a parachain host configuration.
- [x] Make sure the backwards compatibility works correctly
- [x] Make sure this direction is compatible with other streams of work:
https://github.com/paritytech/polkadot-sdk/issues/635 &
https://github.com/paritytech/polkadot-sdk/issues/742
- [x] Final versi burn-in before merging

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>

## Summary

This PR turns on `-D warnings` for `cargo-check-each-crate job` job to
fail on warnings e.g. like this:
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4673130

Before this PR, there was a warning and `cargo-check-each-crate` job did
not fail:
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4641444
```
warning: unused import: `ToTokens`
  --> substrate/primitives/api/proc-macro/src/utils.rs:22:34
   |
22 | use quote::{format_ident, quote, ToTokens};
   |                                  ^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` on by default
warning: `sp-api-proc-macro` (lib) generated 1 warning (run `cargo fix --lib -p sp-api-proc-macro` to apply 1 suggestion)
```

Fixes on the way:
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4641444
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4673265
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4673410
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4673681
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4673836
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4673941
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4674256
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4679328



## Questions
- [ ] why does this check triggers only `cargo check --locked`?
`--all-features` or `--all-targets` are not needed? Or aren't they
avoided intentionally?

---------

Co-authored-by: command-bot <>

Co-authored-by: Javier Viola <[email protected]>

This PR is a continuation of
https://github.com/paritytech/polkadot-sdk/pull/2102 and part of an
initiative started here https://hackmd.io/@romanp/rJ318ZCEp
What has been done:
- The content under `docs/*` (with the exception of `docs/mermaid`) has
been moved to `docs/contributor/`
- Developer Hub has been renamed to Polkadot SDK Docs, and the crate has
been renamed from `developer-hub` to `polkadot-sdk-docs`
- The content under `developer-hub/*` has been moved to `docs/sdk`

---
Original PR https://github.com/paritytech/polkadot-sdk/pull/2565, it has
been close due to too many rebase conflicts

---------

Co-authored-by: Serban Iorga <[email protected]>
Co-authored-by: Chevdor <[email protected]>
Co-authored-by: Egor_P <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Using taplo, fixes all our broken and inconsistent toml formatting and
adds CI to keep them tidy.

If people want we can customise the format rules as described here
https://taplo.tamasfe.dev/configuration/formatter-options.html

@ggwpez

, I suggest zepter is used only for checking features are
propagated, and leave formatting for taplo to avoid duplicate work and
conflicts.

TODO
- [x] Use `exclude = [...]` syntax in taplo file to ignore zombienet
tests instead of deleting the dir

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

This PR introduces the new crate `developer_hub` into the polkadot-sdk
repo. The vision for the developer-hub crate is detailed in [this
document](https://docs.google.com/document/d/1XLLkFNE8v8HLvZpI2rzsa8N2IN1FcKntc8q-Sc4xBAk/edit?usp=sharing).

<img width="1128" alt="Screenshot 2023-11-02 at 10 45 48"
src="https://github.com/paritytech/polkadot-sdk/assets/5588131/1e12b60f-fef5-42c4-8503-a3ba234077c3">


Other than adding the new crate, it also does the following: 

* Remove the `substrate` crate, as there is now a unique umbrella crate
for multiple things in `developer_hub::polkadot_sdk`.
* (backport candidate) A minor change to `frame-support` macros that
allows `T::RuntimeOrigin` to also be acceptable as the origin type.
* (backport candidate) A minor change to `frame-system` that allows us
to deposit events at genesis because now the real genesis config is
generated via wasm, and we can safely assume `cfg!(feature = "std")`
means only testing. related to #62.
* (backport candidate) Introduces a small `read_events_for_pallet` to
`frame_system` for easier event reading in tests.
* From https://github.com/paritytech/polkadot-sdk-docs/issues/31, it
takes action on improving the `pallet::call` docs.
* From https://github.com/paritytech/polkadot-sdk-docs/issues/31, it
takes action on improving the `UncheckedExtrinsic` docs.

## Way Forward

First, a version of this is deployed temporarily
[here](https://blog.kianenigma.nl/polkadot-sdk/developer_hub/index.html).
I will keep it up to date on a daily basis.

### This Pull Request

I see two ways forward: 

1. We acknowledge that everything in `developer-hub` is going to be WIP,
and merge this asap. We should not yet use links to this crate anywhere.
2. We make this be the feature branch, make PRs against this, and either
gradually backport it, or only merge to master once it is done.

I am personally in favor of option 1. If we stick to option 2, we need a
better way to deploy a staging version of this to gh-pages.

### Issue Tracking

The main issues related to the future of `developer_hub` are: 

- https://github.com/paritytech/polkadot-sdk-docs/issues/31
- https://github.com/paritytech/polkadot-sdk-docs/issues/4
- https://github.com/paritytech/polkadot-sdk-docs/issues/26 
- https://github.com/paritytech/polkadot-sdk-docs/issues/32
- https://github.com/paritytech/polkadot-sdk-docs/issues/36


### After This Pull Request

- [ ] create a redirect for
https://paritytech.github.io/polkadot-sdk/master/substrate/
- [x] analytics 
- [ ] link checker
- [ ] the matter of publishing, and how all of these relative links for
when we do, that is still an open question. There is section on this in
the landing page.
- [ ] updated https://paritytech.github.io/

---------

Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Juan Girini <[email protected]>
Co-authored-by: bader y <[email protected]>
Co-authored-by: Sebastian Kunert <[email protected]>
Co-authored-by: James Wilson <[email protected]>
Co-authored-by: Michal Kucharczyk <[email protected]>

Since the Polkadot and Kusama runtimes are no longer in the repo, the
relevant systems parachains runtimes also need to be removed. More
context [here](https://github.com/paritytech/polkadot-sdk/issues/603)
and [here](https://github.com/paritytech/polkadot-sdk/pull/1731).

Removes the following:
- `asset-hub-kusama` and `asset-hub-polkadot`
- `bridge-hub-kusama` and `bridge-hub-polkadot`
- `collectives-polkadot`
- `glutton-kusama`

Partially solves #603 and adds to #1731.