Adds a `NodeFeatures` bitfield value to the runtime `HostConfiguration`,
with the purpose of coordinating the enabling of node-side features,
such as: https://github.com/paritytech/polkadot-sdk/issues/628 and
https://github.com/paritytech/polkadot-sdk/issues/598.
These are features that require all validators enable them at the same
time, assuming all/most nodes have upgraded their node versions.

This PR doesn't add any feature yet. These are coming in future PRs.

Also adds a runtime API for querying the state of the client features
and an extrinsic for setting/unsetting a feature by its index in the bitfield.

Note: originally part of:
https://github.com/paritytech/polkadot-sdk/pull/1644, but posted as
standalone to be reused by other PRs until the initial PR is merged

Fixes #1437

Co-authored-by: Sophia Gold <[email protected]>

Co-authored-by: Marcin S <[email protected]>

Collators were previously reencoding the available data and checking the
erasure root.
Replace that with just checking the PoV hash, which consumes much less
CPU and takes less time.

We also don't need to check the `PersistedValidationData` hash, as
collators don't use it.

Reason:
https://github.com/paritytech/polkadot-sdk/issues/575#issuecomment-1806572230



After systematic chunks recovery is merged, collators will no longer do
any reed-solomon encoding/decoding, which has proven to be a great CPU
consumer.

Signed-off-by: alindima <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>

The staging- rename commit was missing from the last PR for some reason.

Missing descriptions (47):  

- [x] `cumulus/client/collator/Cargo.toml`
- [x] `cumulus/client/relay-chain-inprocess-interface/Cargo.toml`
- [x] `cumulus/client/cli/Cargo.toml`
- [x] `cumulus/client/service/Cargo.toml`
- [x] `cumulus/client/relay-chain-rpc-interface/Cargo.toml`
- [x] `cumulus/client/relay-chain-interface/Cargo.toml`
- [x] `cumulus/client/relay-chain-minimal-node/Cargo.toml`
- [x] `cumulus/parachains/pallets/parachain-info/Cargo.toml`
- [x] `cumulus/parachains/pallets/ping/Cargo.toml`
- [x] `cumulus/primitives/utility/Cargo.toml`
- [x] `cumulus/primitives/aura/Cargo.toml`
- [x] `cumulus/primitives/core/Cargo.toml`
- [x] `cumulus/primitives/parachain-inherent/Cargo.toml`
- [x] `cumulus/test/relay-sproof-builder/Cargo.toml`
- [x] `cumulus/pallets/xcmp-queue/Cargo.toml`
- [x] `cumulus/pallets/dmp-queue/Cargo.toml`
- [x] `cumulus/pallets/xcm/Cargo.toml`
- [x] `polkadot/erasure-coding/Cargo.toml`
- [x] `polkadot/statement-table/Cargo.toml`
- [x] `polkadot/primitives/Cargo.toml`
- [x] `polkadot/rpc/Cargo.toml`
- [x] `polkadot/node/service/Cargo.toml`
- [x] `polkadot/node/core/parachains-inherent/Cargo.toml`
- [x] `polkadot/node/core/approval-voting/Cargo.toml`
- [x] `polkadot/node/core/dispute-coordinator/Cargo.toml`
- [x] `polkadot/node/core/av-store/Cargo.toml`
- [x] `polkadot/node/core/chain-api/Cargo.toml`
- [x] `polkadot/node/core/prospective-parachains/Cargo.toml`
- [x] `polkadot/node/core/backing/Cargo.toml`
- [x] `polkadot/node/core/provisioner/Cargo.toml`
- [x] `polkadot/node/core/runtime-api/Cargo.toml`
- [x] `polkadot/node/core/bitfield-signing/Cargo.toml`
- [x] `polkadot/node/network/dispute-distribution/Cargo.toml`
- [x] `polkadot/node/network/bridge/Cargo.toml`
- [x] `polkadot/node/network/collator-protocol/Cargo.toml`
- [x] `polkadot/node/network/approval-distribution/Cargo.toml`
- [x] `polkadot/node/network/availability-distribution/Cargo.toml`
- [x] `polkadot/node/network/bitfield-distribution/Cargo.toml`
- [x] `polkadot/node/network/gossip-support/Cargo.toml`
- [x] `polkadot/node/network/availability-recovery/Cargo.toml`
- [x] `polkadot/node/collation-generation/Cargo.toml`
- [x] `polkadot/node/overseer/Cargo.toml`
- [x] `polkadot/runtime/parachains/Cargo.toml`
- [x] `polkadot/runtime/common/slot_range_helper/Cargo.toml`
- [x] `polkadot/runtime/metrics/Cargo.toml`
- [x] `polkadot/xcm/pallet-xcm-benchmarks/Cargo.toml`
- [x] `polkadot/utils/generate-bags/Cargo.toml`
- [x]  `substrate/bin/minimal/runtime/Cargo.toml`

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Signed-off-by: alindima <[email protected]>
Co-authored-by: ordian <[email protected]>
Co-authored-by: Tsvetomir Dimitrov <[email protected]>
Co-authored-by: Marcin S <[email protected]>
Co-authored-by: alindima <[email protected]>
Co-authored-by: Sebastian Kunert <[email protected]>
Co-authored-by: Dmitry Markin <[email protected]>
Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>

One for local networks with `fast-runtime` feature activated (1 minute
sessions) and one without the feature activated that will be the default
that runs with 1 hour long sessions.

availability-distribution subsystem is not sending availability-recovery
messages. Update the overseer declaration to reflect this

**_PR migrated from https://github.com/paritytech/polkadot/pull/6782_** 

This PR will upgrade the network protocol to version 3 -> VStaging which
will later be renamed to V3. This version introduces a new kind of
assignment certificate that will be used for tranche0 assignments.
Instead of issuing/importing one tranche0 assignment per candidate,
there will be just one certificate per relay chain block per validator.
However, we will not be sending out the new assignment certificates,
yet. So everything should work exactly as before. Once the majority of
the validators have been upgraded to the new protocol version we will
enable the new certificates (starting at a specific relay chain block)
with a new client update.

There are still a few things that need to be done:

- [x] Use bitfield instead of Vec<CandidateIndex>:
https://github.com/paritytech/polkadot/pull/6802


  - [x] Fix existing approval-distribution and approval-voting tests
  - [x] Fix bitfield-distribution and statement-distribution tests
  - [x] Fix network bridge tests
  - [x] Implement todos in the code
  - [x] Add tests to cover new code
  - [x] Update metrics
  - [x] Remove the approval distribution aggression levels: TBD PR
  - [x] Parachains DB migration 
  - [x] Test network protocol upgrade on Versi
  - [x] Versi Load test
  - [x] Add Zombienet test
  - [x] Documentation updates
- [x] Fix for sending DistributeAssignment for each candidate claimed by
a v2 assignment (warning: Importing locally an already known assignment)
 - [x]  Fix AcceptedDuplicate
 - [x] Fix DB migration so that we can still keep old data.
 - [x] Final Versi burn in

---------

Signed-off-by: Andrei Sandu <[email protected]>
Signed-off-by: Alexandru Gheorghe <[email protected]>
Co-authored-by: Alexandru Gheorghe <[email protected]>

... see https://github.com/paritytech/polkadot-sdk/issues/2138

 for why
is not good, until we fix it let's add a warning to understand if this
is happening in the wild.

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

This PR updates the version of `serde_json` to `1.0.108` throughout the
codebase.

This PR prepares chains specs for _native-runtime-free_  world.

This PR has following changes:
- `substrate`:
  - adds support for:
- JSON based `GenesisConfig` to `ChainSpec` allowing interaction with
runtime `GenesisBuilder` API.
- interacting with arbitrary runtime wasm blob to[
`chain-spec-builder`](https://github.com/paritytech/substrate/blob/3ef576eaeb3f42610e85daecc464961cf1295570/bin/utils/chain-spec-builder/src/lib.rs#L46)
command line util,
- removes
[`code`](https://github.com/paritytech/substrate/blob/3ef576eaeb3f42610e85daecc464961cf1295570/frame/system/src/lib.rs#L660)
from `system_pallet`
  - adds `code` to the `ChainSpec`
- deprecates
[`ChainSpec::from_genesis`](https://github.com/paritytech/substrate/blob/3ef576eaeb3f42610e85daecc464961cf1295570/client/chain-spec/src/chain_spec.rs#L263),
but also changes the signature of this method extending it with `code`
argument.
[`ChainSpec::builder()`](https://github.com/paritytech/substrate/blob/20bee680ed098be7239cf7a6b804cd4de267983e/client/chain-spec/src/chain_spec.rs#L507

)
should be used instead.
- `polkadot`:
- all references to `RuntimeGenesisConfig` in `node/service` are
removed,
- all
`(kusama|polkadot|versi|rococo|wococo)_(staging|dev)_genesis_config`
functions now return the JSON patch for default runtime `GenesisConfig`,
  - `ChainSpecBuilder` is used, `ChainSpec::from_genesis` is removed,

- `cumulus`:
  - `ChainSpecBuilder` is used, `ChainSpec::from_genesis` is removed,
- _JSON_ patch configuration used instead of `RuntimeGenesisConfig
struct` in all chain specs.
  
---------

Co-authored-by: command-bot <>
Co-authored-by: Javier Viola <[email protected]>
Co-authored-by: Davide Galassi <[email protected]>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Kevin Krone <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

# Description

Update the bootnode of kusama parachains before decommissioning the
nodes. This will avoid connecting to non-existing bootnodes.

The `BlockBuilderProvider` was a trait that was defined in
`sc-block-builder`. The trait was implemented for `Client`. This
basically meant that you needed to import `sc-block-builder` any way to
have access to the block builder. So, this trait was not providing any
real value. This pull request is removing the said trait. Instead of the
trait it introduces a builder for creating a `BlockBuilder`. The builder
currently has the quite fabulous name `BlockBuilderBuilder` (I'm open to
any better name 😅

). The rest of the pull request is about
replacing the old trait with the new builder.

# Downstream code changes

If you used `new_block` or `new_block_at` before you now need to switch
it over to the new `BlockBuilderBuilder` pattern:

```rust
// `new` requires a type that implements `CallApiAt`. 
let mut block_builder = BlockBuilderBuilder::new(client)
                // Then you need to specify the hash of the parent block the block will be build on top of
		.on_parent_block(at)
                // The block builder also needs the block number of the parent block. 
                // Here it is fetched from the given `client` using the `HeaderBackend`
                // However, there also exists `with_parent_block_number` for directly passing the number
		.fetch_parent_block_number(client)
		.unwrap()
                // Enable proof recording if required. This call is optional.
		.enable_proof_recording()
                // Pass the digests. This call is optional.
                .with_inherent_digests(digests)
		.build()
		.expect("Creates new block builder");
```

---------

Co-authored-by: Sebastian Kunert <[email protected]>
Co-authored-by: command-bot <>

The check_hardware functions does not give us too much information as to
what is failing, so let's return the list of failed metrics, so that callers can print 
it.

This would make debugging easier, rather than try to guess which
dimension is actually failing.

Signed-off-by: Alexandru Gheorghe <[email protected]>

Co-authored-by: Marcin S <[email protected]>

While investigating some db migrations that make the node startup fail,
I noticed that the node wasn't exiting and that the log file were
growing exponentially, until my whole system was freezing and that makes
it really hard to actually find why it was failing in the first place.

E.g:
```
 ls -lh /tmp/zombie-01a04c2a2c0265d85f6440cf01c0f44a_-51319-uyggzuD4wEpV/bob.log
 32,6G oct 27 11:16 /tmp/zombie-01a04c2a2c0265d85f6440cf01c0f44a_-51319-uyggzuD4wEpV/bob.log
```

This was happening because the following errors were being printed
continously without the subsystem main loop exiting:

From dispute-coordinator:
```
WARN tokio-runtime-worker parachain::dispute-coordinator: error=Subsystem(Generated(Context("Signal channel is terminated and empty.")))
```

From availability recovery:
```
Erasure task channel closed. Node shutting down ?
```

Signed-off-by: Alexandru Gheorghe <[email protected]>

Fixes #768.

in-progress PR adding new tests and solving bugs

---------

Co-authored-by: Bradley Olson <[email protected]>
Co-authored-by: eskimor <[email protected]>
Co-authored-by: eskimor <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>

The validators are checking if async backing is enabled by checking the
version of the runtime api. If the runtime api is upgraded by a runtime
upgrade, the validators start to also enable the async backing logic.
However, just because async backing is enabled, it doesn't mean that all
collators and parachain runtimes have upgraded. This pull request fixes
an issue about advertising collations to the relay chain when it has
async backing enabled, but the collator is still using the old
networking protocol. The implementation is actually backwards compatible
as we can not expect that everyone directly upgrades. However, the
collation advertisement logic was requiring V2 networking messages after
async backing was enabled, which was wrong. This is now fixed by this
pull request.

Closes: https://github.com/paritytech/polkadot-sdk/issues/1923



---------

Co-authored-by: eskimor <[email protected]>

closes: https://github.com/paritytech/devops/issues/2090

## Changes
1. Updated the list of bootnodes.
2. Merged the Connect node and bootnode into a single node.
3. Decreased the number of nodes.
4. Updated the DNS name.

## Description
The initial 8 bootnodes were planned to be replaced by community
bootnodes, the community node was added but we did not bother to reduce
the Parity managed bootnodes. Fixing it now.

Fellowship companion:
https://github.com/polkadot-fellows/runtimes/pull/65

This starts the BEEFY client by default for Polkadot nodes.

Governance/sudo call is later required to enable/start consensus.

Part of https://github.com/paritytech/parity-bridges-common/issues/2420

closes #695

Could potentially be helpful to preserving caches when applicable, as
discussed in #685

kusama address: FvpsvV1GQAAbwqX6oyRjemgdKV11QU5bXsMg9xsonD1FLGK

closes #622 

Pros:
* simpler interface, just functions:
`create_runtime_from_artifact_bytes()` and `execute_artifact()`

Cons:
* extra overhead of constructing executor semantics each time

I could make it a combination of
* `create_runtime_config(params)` (such that we could clone the
constructed semantics)
* `create_runtime(blob, config)`
* `execute_artifact(blob, config, params)`

Not sure if it's worth it though.

---------

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Marcin S <[email protected]>

# Description

In a couple of cases, there were links pointing to the w3f github pages
domain. In other instances, there were links pointing to the old
polkadot repo's github pages. Both of these are now pointing to the
relevant links in
https://paritytech.github.io/polkadot-sdk/book/index.html.

These changes were made specifically because the w3f github pages
returns a 404, and while fixing the links, the old polkadot repo links
were touched up as well even if they do redirect properly.

This shouldn't affect anything as these are documentation link changes
only.

Exposes disabled validators list via a runtime API.

---------

Co-authored-by: ordian <[email protected]>
Co-authored-by: ordian <[email protected]>

Co-authored-by: command-bot <>

# Description
Update the DNS name of bootnodes to unify the deployment. 

Each bootnode have 3 port exposed: `30333, 30334, 443`. Before, we had
different DNS names for `30333, 30334` and `443` ports. It may confuse
people and give the impression that it is two different nodes. Fixing it
by using a single domain for all

Bumps the known_good_semver group with 1 update:
[syn](https://github.com/dtolnay/syn).

<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/dtolnay/syn/releases">syn's
releases</a>.</em></p>
<blockquote>
<h2>2.0.38</h2>
<ul>
<li>Fix <em>&quot;method 'peek' has an incompatible type for
trait&quot;</em> error when defining <code>bool</code> as a custom
keyword (<a
href="https://redirect.github.com/dtolnay/syn/issues/1518">#1518</a>,
thanks <a
href="https://github.com/Vanille-N"><code>@​Vanille-N</code></a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/dtolnay/syn/commit/43632bfb6c78ee1f952645a268ab1ac4af162977"><code>43632bf</code></a>
Release 2.0.38</li>
<li><a
href="https://github.com/dtolnay/syn/commit/abd2c214b44da64a5e420d72919308300eebc23d"><code>abd2c21</code></a>
Merge pull request <a
href="https://redirect.github.com/dtolnay/syn/issues/1518">#1518</a>
from Vanille-N/master</li>
<li><a
href="https://github.com/dtolnay/syn/commit/6701e6077e15013ef34b15e3ffdae2657e499d83"><code>6701e60</code></a>
Absolute path to <code>bool</code> in
<code>custom_punctuation.rs</code></li>
<li><a
href="https://github.com/dtolnay/syn/commit/7313d242398111423f046386aa0a75548f63d236"><code>7313d24</code></a>
Resolve single_match_else pedantic clippy lint in code generator</li>
<li><a
href="https://github.com/dtolnay/syn/commit/67ab64f3c09e17b23493c7cda498e7edb8830f21"><code>67ab64f</code></a>
Include unexpected token in the test failure message</li>
<li><a
href="https://github.com/dtolnay/syn/commit/137ae33486de3f2652487f8f64436ad1429df496"><code>137ae33</code></a>
Check no remaining token after the first literal</li>
<li><a
href="https://github.com/dtolnay/syn/commit/258e9e8a11d188c1ee1ffb2b069819239999f9ac"><code>258e9e8</code></a>
Ignore single_match_else pedantic clippy lint in test</li>
<li><a
href="https://github.com/dtolnay/syn/commit/92fd50ee8cb52968d9c66fbe6d67638c1f838e26"><code>92fd50e</code></a>
Test docs.rs documentation build in CI</li>
<li>See full diff in <a
href="https://github.com/dtolnay/syn/compare/2.0.37...2.0.38">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=syn&package-manager=cargo&previous-version=2.0.37&new-version=2.0.38)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores

)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>