The staging- rename commit was missing from the last PR for some reason.

When warp syncing we import the target block with all its state.
However, we didn't store the `LAST_PRUNED` block which would then lead
to `pruning` to forget about the imported block after a restart of the
node. We just set `LAST_PRUNED` to the parent block of the warp sync
target block to fix this issue.

Missing descriptions (47):  

- [x] `cumulus/client/collator/Cargo.toml`
- [x] `cumulus/client/relay-chain-inprocess-interface/Cargo.toml`
- [x] `cumulus/client/cli/Cargo.toml`
- [x] `cumulus/client/service/Cargo.toml`
- [x] `cumulus/client/relay-chain-rpc-interface/Cargo.toml`
- [x] `cumulus/client/relay-chain-interface/Cargo.toml`
- [x] `cumulus/client/relay-chain-minimal-node/Cargo.toml`
- [x] `cumulus/parachains/pallets/parachain-info/Cargo.toml`
- [x] `cumulus/parachains/pallets/ping/Cargo.toml`
- [x] `cumulus/primitives/utility/Cargo.toml`
- [x] `cumulus/primitives/aura/Cargo.toml`
- [x] `cumulus/primitives/core/Cargo.toml`
- [x] `cumulus/primitives/parachain-inherent/Cargo.toml`
- [x] `cumulus/test/relay-sproof-builder/Cargo.toml`
- [x] `cumulus/pallets/xcmp-queue/Cargo.toml`
- [x] `cumulus/pallets/dmp-queue/Cargo.toml`
- [x] `cumulus/pallets/xcm/Cargo.toml`
- [x] `polkadot/erasure-coding/Cargo.toml`
- [x] `polkadot/statement-table/Cargo.toml`
- [x] `polkadot/primitives/Cargo.toml`
- [x] `polkadot/rpc/Cargo.toml`
- [x] `polkadot/node/service/Cargo.toml`
- [x] `polkadot/node/core/parachains-inherent/Cargo.toml`
- [x] `polkadot/node/core/approval-voting/Cargo.toml`
- [x] `polkadot/node/core/dispute-coordinator/Cargo.toml`
- [x] `polkadot/node/core/av-store/Cargo.toml`
- [x] `polkadot/node/core/chain-api/Cargo.toml`
- [x] `polkadot/node/core/prospective-parachains/Cargo.toml`
- [x] `polkadot/node/core/backing/Cargo.toml`
- [x] `polkadot/node/core/provisioner/Cargo.toml`
- [x] `polkadot/node/core/runtime-api/Cargo.toml`
- [x] `polkadot/node/core/bitfield-signing/Cargo.toml`
- [x] `polkadot/node/network/dispute-distribution/Cargo.toml`
- [x] `polkadot/node/network/bridge/Cargo.toml`
- [x] `polkadot/node/network/collator-protocol/Cargo.toml`
- [x] `polkadot/node/network/approval-distribution/Cargo.toml`
- [x] `polkadot/node/network/availability-distribution/Cargo.toml`
- [x] `polkadot/node/network/bitfield-distribution/Cargo.toml`
- [x] `polkadot/node/network/gossip-support/Cargo.toml`
- [x] `polkadot/node/network/availability-recovery/Cargo.toml`
- [x] `polkadot/node/collation-generation/Cargo.toml`
- [x] `polkadot/node/overseer/Cargo.toml`
- [x] `polkadot/runtime/parachains/Cargo.toml`
- [x] `polkadot/runtime/common/slot_range_helper/Cargo.toml`
- [x] `polkadot/runtime/metrics/Cargo.toml`
- [x] `polkadot/xcm/pallet-xcm-benchmarks/Cargo.toml`
- [x] `polkadot/utils/generate-bags/Cargo.toml`
- [x]  `substrate/bin/minimal/runtime/Cargo.toml`

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Signed-off-by: alindima <[email protected]>
Co-authored-by: ordian <[email protected]>
Co-authored-by: Tsvetomir Dimitrov <[email protected]>
Co-authored-by: Marcin S <[email protected]>
Co-authored-by: alindima <[email protected]>
Co-authored-by: Sebastian Kunert <[email protected]>
Co-authored-by: Dmitry Markin <[email protected]>
Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>

This was discovered during a debugging session, and it only served to
mask the underlying error, which was not great.

# Description

We would like to add our bootnodes to the following parachains:

Westend: Westmint, Bridgehub

Kusama: Statemine, Bridgehub

Polkadot: Statemint, Bridgehub, Collectives

Thank you.

---------

Co-authored-by: Oliver Tale-Yazdi <[email protected]>

One for local networks with `fast-runtime` feature activated (1 minute
sessions) and one without the feature activated that will be the default
that runs with 1 hour long sessions.

This PR changes the registration order of the `MessageQueue` pallet so
that it is registered last.

This is necessary so that the
[on_initialize](https://github.com/Snowfork/snowbridge/blob/df8d5da8/parachain/pallets/outbound-queue/src/lib.rs#L267)
hooks for Snowbridge can run before `MessageQueue` delivers messages
using its own `on_initialize`.

Generally, I think this is preferable regardless of Snowbridge's
particular requirements. Other pallets may want to do housekeeping
before MessageQueue starts delivering messages.

I'm hoping this PR, if accepted, can be included in the same release as
https://github.com/paritytech/polkadot-sdk/pull/1246. As otherwise,
changing the order of pallet registration is an ABI-breaking change.

PalletInfo fields were private, preventing a user from actually using
the QueryPallet instruction in a meaningful way since they couldn't read
the received data.

This PR contains some fixes and cleanups for parachain nodes:

1. When using async backing, node no longer complains about being unable
to reach the prospective-parachain subsystem.
2. Parachain warp sync now informs users that the finalized para block
has been retrieved.
```
2023-11-08 13:24:42 [Parachain] 🎉 Received finalized parachain header #5747719 (0xa0aa…674b) from the relay chain.
```
3. When a user supplied an invalid `--relay-chain-rpc-url`, we were
crashing with a very verbose message. Removed the `expect` and improved
the error message.
```
2023-11-08 13:57:56 [Parachain] No valid RPC url found. Stopping RPC worker.
2023-11-08 13:57:56 [Parachain] Essential task `relay-chain-rpc-worker` failed. Shutting down service.
Error: Service(Application(WorkerCommunicationError("RPC worker channel closed. This can hint and connectivity issues with the supplied RPC endpoints. Message: oneshot canceled")))
```

Some legacy tests were mistakenly merged in #1256 for `emulated-integration-tests-common` crate.
This PR fixes the function name `build_genesis_storage` (no need to use `legacy` suffix, even though the genesis is built from `RuntimeGenesisConfig`).

Genesis building in runtime may involve calling some custom host
functions. This PR allows to pass `HostFunctions` into the `ChainSpec`
struct, which in turn are passed to `WasmExecutor`. The `ChainSpec` now
has extended host functions type parameter:
```
pub struct ChainSpec<G, E = NoExtension, EHF = ()>
```
which will be combined with the default set
(`sp_io::SubstrateHostFunctions`) in an instance of `WasmExecutor` used
to build the genesis config.

Fix for #2188

---------

Co-authored-by: Davide Galassi <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Closes:
- #1383 
- Declared chains can be now be imported and reused in a different
crate.
- Chain declaration are now generic over a generic type `N` (the
Network)
- #1389
- Solved #1383, chains and networks declarations can be restructure to
avoid having to compile all chains when running integrations tests where
are not needed.
- Chains are now declared on its own crate (removed from
`integration-tests-common`)
- Networks are now declared on its own crate (removed from
`integration-tests-common`)
    - Integration tests will import only the relevant Network crate
- `integration-tests-common` is renamed to
`emulated-integration-tests-common`

All this is necessary to be able to implement what is described here:
https://github.com/paritytech/roadmap/issues/56#issuecomment-1777010553

---------

Co-authored-by: command-bot <>

The trie cache implementation was ignoring the `storage_root` when
setting up the value cache. The problem with this is that the value
cache works using `storage_keys` and these keys are not unique across
different tries. A block can actually have different tries (main trie
and multiple child tries). This pull request fixes the issue by not
ignoring the `storage_root` and returning an unique `value_cache` per
`storage_root`. It also adds a test for the seen bug and improves
documentation that this doesn't happen again.

GitHub has a setting that requires manual click for executing GHA on the
branch, for the first-time contributors:
https://docs.github.com/en/actions/managing-workflow-runs/approving-workflow-runs-from-public-forks.

After this PR, gitspiegel will respect that setting. So, for PRs from
first-time contributors, gitspiegel won't do mirroring until the button
in PR is clicked. More info:
https://github.com/paritytech/gitspiegel/issues/169

Changes:
- Adds a new call `remove_key` to the sudo pallet to permanently remove
the sudo key.
- Remove some clones and general maintenance

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: command-bot <>

fixes #2196

`bridge-hub-westend-runtime` was added to cumulus/parachains, but wasn't
hooked up to xcm-emulator to run tests against it.

This commit addresses that ^.

Signed-off-by: Adrian Catangiu <[email protected]>

Added a proc macro to be able to write XCMs using the builder pattern.
This means we go from having to do this:

```rust
let message: Xcm<()> = Xcm(vec![
  WithdrawAsset(assets),
  BuyExecution { fees: asset, weight_limit: Unlimited },
  DepositAsset { assets, beneficiary },
]);
```

to this:

```rust
let message: Xcm<()> = Xcm::builder()
  .withdraw_asset(assets)
  .buy_execution(asset, Unlimited),
  .deposit_asset(assets, beneficiary)
  .build();
```

---------

Co-authored-by: Keith Yeung <[email protected]>
Co-authored-by: command-bot <>

When running with `--relay-chain-rpc-url` we received multiple reports
of high traffic that disappears when `--in-peers-light 0` is set. Indeed
it does not make much sense for light clients to connect to the minimal
node since it is not running the block announce protocol and the
request/response protocol for light clients.

This is intended to alleviate the traffic issues for now.

closes #1896
probably related https://github.com/paritytech/cumulus/issues/2563

# Description

This merely fixes a typo in the documentation, replacing the typo
"slashaed" with "slashed". Since external entities use the comments for
explanations of events, this will then be shown externally. I noticed
this when reviewing [this
event](https://polkadot.subscan.io/extrinsic/0xb6bc1e3abde0c2ed9c500c74cfc64cdb8179e5d9af97f4bf53242ce4cdd15a1d?event=18064194-6)
on Subscan.

This is not related to any other issues or PRs.

Part of #2186

The only usage of pallet-asset-rate is guarded by `runtime-benchmarks`
feature. I don't want ORML to be forced to include this pallet in deps
for no good reason.

There was a race in merging between
https://github.com/paritytech/polkadot-sdk/pull/1256 and
https://github.com/paritytech/polkadot-sdk/pull/1178, so this newly
added tests wasn't updated with the new path for the configuration, so
fix that.

Signed-off-by: Alexandru Gheorghe <[email protected]>

Original PR https://github.com/paritytech/substrate/pull/14641

---

Closes https://github.com/paritytech/polkadot-sdk/issues/109

### Problem
Quoting from the above issue:

> When adding a pallet to chain after genesis we currently don't set the
StorageVersion. So, when calling on_chain_storage_version it returns 0
while the pallet is maybe already at storage version 9 when it was added
to the chain. This could lead to issues when running migrations.

### Solution

- Create a new trait `BeforeAllRuntimeMigrations` with a single method
`fn before_all_runtime_migrations() -> Weight` trait with a noop default
implementation
- Modify `Executive` to call
`BeforeAllRuntimeMigrations::before_all_runtime_migrations` for all
pallets before running any other hooks
- Implement `BeforeAllRuntimeMigrations` in the pallet proc macro to
initialize the on-chain version to the current pallet version if the
pallet has no storage set (indicating it has been recently added to the
runtime and needs to have its version initialised).

### Other changes in this PR

- Abstracted repeated boilerplate to access the `pallet_name` in the
pallet expand proc macro.

### FAQ

#### Why create a new hook instead of adding this logic to the pallet
`pre_upgrade`?

`Executive` currently runs `COnRuntimeUpgrade` (custom migrations)
before `AllPalletsWithSystem` migrations. We need versions to be
initialized before the `COnRuntimeUpgrade` migrations are run, because
`COnRuntimeUpgrade` migrations may use the on-chain version for critical
logic. e.g. `VersionedRuntimeUpgrade` uses it to decide whether or not
to execute.

We cannot reorder `COnRuntimeUpgrade` and `AllPalletsWithSystem` so
`AllPalletsWithSystem` runs first, because `AllPalletsWithSystem` have
some logic in their `post_upgrade` hooks to verify that the on-chain
version and current pallet version match. A common use case of
`COnRuntimeUpgrade` migrations is to perform a migration which will
result in the versions matching, so if they were reordered these
`post_upgrade` checks would fail.

#### Why init the on-chain version for pallets without a current storage
version?

We must init the on-chain version for pallets even if they don't have a
defined storage version so if there is a future version bump, the
on-chain version is not automatically set to that new version without a
proper migration.

e.g. bad scenario:

1. A pallet with no 'current version' is added to the runtime
2. Later, the pallet is upgraded with the 'current version' getting set
to 1 and a migration is added to Executive Migrations to migrate the
storage from 0 to 1
    a. Runtime upgrade occurs
    b. `before_all` hook initializes the on-chain version to 1
c. `on_runtime_upgrade` of the migration executes, and sees the on-chain
version is already 1 therefore think storage is already migrated and
does not execute the storage migration
Now, on-chain version is 1 but storage is still at version 0.

By always initializing the on-chain version when the pallet is added to
the runtime we avoid that scenario.

---------

Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

### This PR is a port of this [PR for
substrate](https://github.com/paritytech/substrate/pull/13013) by
@Kianenigma



Add infrastructure needed to have a Pallet::decode_entire_state(), which
makes sure all "typed" storage items defined in the pallet are
decode-able.

This is not enforced in any way at the moment. Teams who wish to
integrate/use this in the try-runtime feature flag should add
frame_support::storage::migration::EnsureStateDecodes as the LAST ITEM
of the runtime's custom migrations, and pass it to frame-executive. This
will make it usable in try-runtime on-runtime-upgrade.

This now catches cases like
https://github.com/paritytech/polkadot-sdk/pull/1969:
```pre
ERROR runtime::executive] failed to decode the value at key: Failed to decode value at key: 0x94eadf0156a8ad5156507773d0471e4ab8ebad86f546c7e0b135a4212aace339. Storage info StorageInfo { pallet_name: Ok("ParaScheduler"), storage_name: Ok("AvailabilityCores"), prefix: Err(Utf8Error { valid_up_to: 0, error_len: Some(1) }), max_values: Some(1), max_size: None }. Raw value: Some("0x0c010101010101")
```

... or:

![image](https://github.com/paritytech/polkadot-sdk/assets/10380170/73052d4f-4da5-4b21-a8dd-b17004e5965e)

Closes #241

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>

availability-distribution subsystem is not sending availability-recovery
messages. Update the overseer declaration to reflect this

**_PR migrated from https://github.com/paritytech/polkadot/pull/6782_** 

This PR will upgrade the network protocol to version 3 -> VStaging which
will later be renamed to V3. This version introduces a new kind of
assignment certificate that will be used for tranche0 assignments.
Instead of issuing/importing one tranche0 assignment per candidate,
there will be just one certificate per relay chain block per validator.
However, we will not be sending out the new assignment certificates,
yet. So everything should work exactly as before. Once the majority of
the validators have been upgraded to the new protocol version we will
enable the new certificates (starting at a specific relay chain block)
with a new client update.

There are still a few things that need to be done:

- [x] Use bitfield instead of Vec<CandidateIndex>:
https://github.com/paritytech/polkadot/pull/6802
  - [x] Fix existing approval-distribution and approval-voting tests
  - [x] Fix bitfield-distribution and statement-distribution tests
  - [x] Fix network bridge tests
  - [x] Implement todos in the code
  - [x] Add tests to cover new code
  - [x] Update metrics
  - [x] Remove the approval distribution aggression levels: TBD PR
  - [x] Parachains DB migration 
  - [x] Test network protocol upgrade on Versi
  - [x] Versi Load test
  - [x] Add Zombienet test
  - [x] Documentation updates
- [x] Fix for sending DistributeAssignment for each candidate claimed by
a v2 assignment (warning: Importing locally an already known assignment)
 - [x]  Fix AcceptedDuplicate
 - [x] Fix DB migration so that we can still keep old data.
 - [x] Final Versi burn in

---------

Signed-off-by: Andrei Sandu <[email protected]>
Signed-off-by: Alexandru Gheorghe <[email protected]>
Co-authored-by: Alexandru Gheorghe <[email protected]>

A quick fix where a benchmark test was wrongly renamed in this PR
https://github.com/paritytech/polkadot-sdk/pull/1868

This PR exposes a `force_remove_vesting` through a ROOT call. 
See linked
[issue](https://github.com/paritytech/polkadot-sdk/issues/269)

---------

Co-authored-by: georgepisaltu <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: Dónal Murray <[email protected]>

... see https://github.com/paritytech/polkadot-sdk/issues/2138 for why
is not good, until we fix it let's add a warning to understand if this
is happening in the wild.

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

This PR updates the version of `serde_json` to `1.0.108` throughout the
codebase.

Move peer banning from `ChainSync` to `SyncingEngine`.

Otherwise the return code is not correctly propagated (ref
https://github.com/ggwpez/zepter/pull/48).

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Related https://github.com/paritytech/polkadot-sdk/issues/2013

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

This PR prepares chains specs for _native-runtime-free_  world.

This PR has following changes:
- `substrate`:
  - adds support for:
- JSON based `GenesisConfig` to `ChainSpec` allowing interaction with
runtime `GenesisBuilder` API.
- interacting with arbitrary runtime wasm blob to[
`chain-spec-builder`](https://github.com/paritytech/substrate/blob/3ef576eaeb3f42610e85daecc464961cf1295570/bin/utils/chain-spec-builder/src/lib.rs#L46)
command line util,
- removes
[`code`](https://github.com/paritytech/substrate/blob/3ef576eaeb3f42610e85daecc464961cf1295570/frame/system/src/lib.rs#L660)
from `system_pallet`
  - adds `code` to the `ChainSpec`
- deprecates
[`ChainSpec::from_genesis`](https://github.com/paritytech/substrate/blob/3ef576eaeb3f42610e85daecc464961cf1295570/client/chain-spec/src/chain_spec.rs#L263),
but also changes the signature of this method extending it with `code`
argument.
[`ChainSpec::builder()`](https://github.com/paritytech/substrate/blob/20bee680ed098be7239cf7a6b804cd4de267983e/client/chain-spec/src/chain_spec.rs#L507)
should be used instead.
- `polkadot`:
- all references to `RuntimeGenesisConfig` in `node/service` are
removed,
- all
`(kusama|polkadot|versi|rococo|wococo)_(staging|dev)_genesis_config`
functions now return the JSON patch for default runtime `GenesisConfig`,
  - `ChainSpecBuilder` is used, `ChainSpec::from_genesis` is removed,

- `cumulus`:
  - `ChainSpecBuilder` is used, `ChainSpec::from_genesis` is removed,
- _JSON_ patch configuration used instead of `RuntimeGenesisConfig
struct` in all chain specs.
  
---------

Co-authored-by: command-bot <>
Co-authored-by: Javier Viola <[email protected]>
Co-authored-by: Davide Galassi <[email protected]>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Kevin Krone <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

# Description

Update the bootnode of kusama parachains before decommissioning the
nodes. This will avoid connecting to non-existing bootnodes.