Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

Related to https://github.com/paritytech/polkadot-sdk/issues/7360

Derive `DecodeWithMemTracking` for the structures in the cumulus pallets
and for the structures in the `polkadot-sdk` runtimes.

The PR contains no functional changes and no manual implementation. Just
deriving `DecodeWithMemTracking`.

Co-authored-by: Bastian Köcher <git@kchr.de>

Co-authored-by: Bastian Köcher <git@kchr.de>

Original PR https://github.com/paritytech/polkadot-sdk/pull/2280
reverted in https://github.com/paritytech/polkadot-sdk/pull/3665

This PR reintroduces the reverted functionality with additional changes,
related effort
[here](https://github.com/paritytech/polkadot-sdk/pull/3623).
Description is copied over from the original PR

First part of [Extrinsic
Horizon](https://github.com/paritytech/polkadot-sdk/issues/2415)

Introduces a new trait `TransactionExtension` to replace
`SignedExtension`. Introduce the idea of transactions which obey the
runtime's extensions and have according Extension data (né Extra data)
yet do not have hard-coded signatures.

Deprecate the terminology of "Unsigned" when used for
transactions/extrinsics owing to there now being "proper" unsigned
transactions which obey the extension framework and "old-style" unsigned
which do not. Instead we have __*General*__ for the former and
__*Bare*__ for the latter. (Ultimately, the latter will be phased out as
a type of transaction, and Bare will only be used for Inherents.)

Types of extrinsic are now therefore:
- Bare (no hardcoded signature, no Extra data; used to be known as
"Unsigned")
- Bare transactions (deprecated): Gossiped, validated with
`ValidateUnsigned` (deprecated) and the `_bare_compat` bits of
`TransactionExtension` (deprecated).
  - Inherents: Not gossiped, validated with `ProvideInherent`.
- Extended (Extra data): Gossiped, validated via `TransactionExtension`.
  - Signed transactions (with a hardcoded signature) in extrinsic v4.
- General transactions (without a hardcoded signature) in extrinsic v5.

`TransactionExtension` differs from `SignedExtension` because:
- A signature on the underlying transaction may validly not be present.
- It may alter the origin during validation.
- `pre_dispatch` is renamed to `prepare` and need not contain the checks
present in `validate`.
- `validate` and `prepare` is passed an `Origin` rather than a
`AccountId`.
- `validate` may pass arbitrary information into `prepare` via a new
user-specifiable type `Val`.
- `AdditionalSigned`/`additional_signed` is renamed to
`Implicit`/`implicit`. It is encoded *for the entire transaction* and
passed in to each extension as a new argument to `validate`. This
facilitates the ability of extensions to acts as underlying crypto.

There is a new `DispatchTransaction` trait which contains only default
function impls and is impl'ed for any `TransactionExtension` impler. It
provides several utility functions which reduce some of the tedium from
using `TransactionExtension` (indeed, none of its regular functions
should now need to be called directly).

Three transaction version discriminator ("versions") are now permissible
(RFC [here](https://github.com/polkadot-fellows/RFCs/pull/84)) in
extrinsic version 5:
- 0b00000100 or 0b00000101: Bare (used to be called "Unsigned"):
contains Signature or Extra (extension data). After bare transactions
are no longer supported, this will strictly identify an Inherents only.
Available in both extrinsic versions 4 and 5.
- 0b10000100: Old-school "Signed" Transaction: contains Signature, Extra
(extension data) and an extension version byte, introduced as part of
[RFC99](https://github.com/polkadot-fellows/RFCs/blob/main/text/0099-transaction-extension-version.md).
Still available as part of extrinsic v4.
- 0b01000101: New-school "General" Transaction: contains Extra
(extension data) and an extension version byte, as per RFC99, but no
Signature. Only available in extrinsic v5.

For the New-school General Transaction, it becomes trivial for authors
to publish extensions to the mechanism for authorizing an Origin, e.g.
through new kinds of key-signing schemes, ZK proofs, pallet state,
mutations over pre-authenticated origins or any combination of the
above.

`UncheckedExtrinsic` still maintains encode/decode backwards
compatibility with extrinsic version 4, where the first byte was encoded
as:
- 0b00000100 - Unsigned transactions
- 0b10000100 - Old-school Signed transactions, without the extension
version byte

Now, `UncheckedExtrinsic` contains a `Preamble` and the actual call. The
`Preamble` describes the type of extrinsic as follows:
```rust
/// A "header" for extrinsics leading up to the call itself. Determines the type of extrinsic and
/// holds any necessary specialized data.
#[derive(Eq, PartialEq, Clone)]
pub enum Preamble<Address, Signature, Extension> {
	/// An extrinsic without a signature or any extension. This means it's either an inherent or
	/// an old-school "Unsigned" (we don't use that terminology any more since it's confusable with
	/// the general transaction which is without a signature but does have an extension).
	///
	/// NOTE: In the future, once we remove `ValidateUnsigned`, this will only serve Inherent
	/// extrinsics and thus can be renamed to `Inherent`.
	Bare(ExtrinsicVersion),
	/// An old-school transaction extrinsic which includes a signature of some hard-coded crypto.
	/// Available only on extrinsic version 4.
	Signed(Address, Signature, ExtensionVersion, Extension),
	/// A new-school transaction extrinsic which does not include a signature by default. The
	/// origin authorization, through signatures or other means, is performed by the transaction
	/// extension in this extrinsic. Available starting with extrinsic version 5.
	General(ExtensionVersion, Extension),
}
```

## Code Migration

### NOW: Getting it to build

Wrap your `SignedExtension`s in `AsTransactionExtension`. This should be
accompanied by renaming your aggregate type in line with the new
terminology. E.g. Before:

```rust
/// The SignedExtension to the basic transaction logic.
pub type SignedExtra = (
	/* snip */
	MySpecialSignedExtension,
);
/// Unchecked extrinsic type as expected by this runtime.
pub type UncheckedExtrinsic =
	generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, SignedExtra>;
```

After:

```rust
/// The extension to the basic transaction logic.
pub type TxExtension = (
	/* snip */
	AsTransactionExtension<MySpecialSignedExtension>,
);
/// Unchecked extrinsic type as expected by this runtime.
pub type UncheckedExtrinsic =
	generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, TxExtension>;
```

You'll also need to alter any transaction building logic to add a
`.into()` to make the conversion happen. E.g. Before:

```rust
fn construct_extrinsic(
		/* snip */
) -> UncheckedExtrinsic {
	let extra: SignedExtra = (
		/* snip */
		MySpecialSignedExtension::new(/* snip */),
	);
	let payload = SignedPayload::new(call.clone(), extra.clone()).unwrap();
	let signature = payload.using_encoded(|e| sender.sign(e));
	UncheckedExtrinsic::new_signed(
		/* snip */
		Signature::Sr25519(signature),
		extra,
	)
}
```

After:

```rust
fn construct_extrinsic(
		/* snip */
) -> UncheckedExtrinsic {
	let tx_ext: TxExtension = (
		/* snip */
		MySpecialSignedExtension::new(/* snip */).into(),
	);
	let payload = SignedPayload::new(call.clone(), tx_ext.clone()).unwrap();
	let signature = payload.using_encoded(|e| sender.sign(e));
	UncheckedExtrinsic::new_signed(
		/* snip */
		Signature::Sr25519(signature),
		tx_ext,
	)
}
```

### SOON: Migrating to `TransactionExtension`

Most `SignedExtension`s can be trivially converted to become a
`TransactionExtension`. There are a few things to know.

- Instead of a single trait like `SignedExtension`, you should now
implement two traits individually: `TransactionExtensionBase` and
`TransactionExtension`.
- Weights are now a thing and must be provided via the new function `fn
weight`.

#### `TransactionExtensionBase`

This trait takes care of anything which is not dependent on types
specific to your runtime, most notably `Call`.

- `AdditionalSigned`/`additional_signed` is renamed to
`Implicit`/`implicit`.
- Weight must be returned by implementing the `weight` function. If your
extension is associated with a pallet, you'll probably want to do this
via the pallet's existing benchmarking infrastructure.

#### `TransactionExtension`

Generally:
- `pre_dispatch` is now `prepare` and you *should not reexecute the
`validate` functionality in there*!
- You don't get an account ID any more; you get an origin instead. If
you need to presume an account ID, then you can use the trait function
`AsSystemOriginSigner::as_system_origin_signer`.
- You get an additional ticket, similar to `Pre`, called `Val`. This
defines data which is passed from `validate` into `prepare`. This is
important since you should not be duplicating logic from `validate` to
`prepare`, you need a way of passing your working from the former into
the latter. This is it.
- This trait takes a `Call` type parameter. `Call` is the runtime call
type which used to be an associated type; you can just move it to become
a type parameter for your trait impl.
- There's no `AccountId` associated type any more. Just remove it.

Regarding `validate`:
- You get three new parameters in `validate`; all can be ignored when
migrating from `SignedExtension`.
- `validate` returns a tuple on success; the second item in the tuple is
the new ticket type `Self::Val` which gets passed in to `prepare`. If
you use any information extracted during `validate` (off-chain and
on-chain, non-mutating) in `prepare` (on-chain, mutating) then you can
pass it through with this. For the tuple's last item, just return the
`origin` argument.

Regarding `prepare`:
- This is renamed from `pre_dispatch`, but there is one change:
- FUNCTIONALITY TO VALIDATE THE TRANSACTION NEED NOT BE DUPLICATED FROM
`validate`!!
- (This is different to `SignedExtension` which was required to run the
same checks in `pre_dispatch` as in `validate`.)

Regarding `post_dispatch`:
- Since there are no unsigned transactions handled by
`TransactionExtension`, `Pre` is always defined, so the first parameter
is `Self::Pre` rather than `Option<Self::Pre>`.

If you make use of `SignedExtension::validate_unsigned` or
`SignedExtension::pre_dispatch_unsigned`, then:
- Just use the regular versions of these functions instead.
- Have your logic execute in the case that the `origin` is `None`.
- Ensure your transaction creation logic creates a General Transaction
rather than a Bare Transaction; this means having to include all
`TransactionExtension`s' data.
- `ValidateUnsigned` can still be used (for now) if you need to be able
to construct transactions which contain none of the extension data,
however these will be phased out in stage 2 of the Transactions Horizon,
so you should consider moving to an extension-centric design.

---------

Signed-off-by: georgepisaltu <george.pisaltu@parity.io>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>

On top of #5082.

## Background

Previously, before #3479, we would
[include](https://github.com/paritytech/polkadot-sdk/blame/75074952/polkadot/runtime/parachains/src/builder.rs#L508C12-L508C44)
the cost enacting the candidate into the cost of processing a single
bitfield.
[Now](https://github.com/paritytech/polkadot-sdk/blame/dd48544a/polkadot/runtime/parachains/src/builder.rs#L529)
it is different, although the benchmarks seems to be not-up-to date.
Including the cost of enacting a candidate into a processing a single
bitfield cost was incorrect, since we multiple that by the number of
bitfields we have. Instead, we should separate calculate the cost of
processing a single bitfield without enactment, and multiple the cost of
enactment by the actual number of processed candidates (which is limited
by the number cores, not validators).

## Bench

Previously, the weight of `enact_candidate` was calculated manually
(without a benchmark) and then neglected:
https://github.com/paritytech/polkadot-sdk/blob/dd48544a

/polkadot/runtime/parachains/src/inclusion/mod.rs#L584

In this PR, we have a benchmark for it and it's based on the number of
ump and sent hrmp messages as well as whether the candidate has a
runtime upgrade (new_validation_code).
The differences from the previous attempt
https://github.com/paritytech/polkadot/pull/6929 are that
* we don't include the cost of enactment into the cost of processing a
backed candidate.
The reason for it is that enactment happens not in the same block as
backing (typically the next one), since we process bitfields before
backing votes.
* we don't take into account the size of the runtime upgrade, the
benchmark weight doesn't seem to depend much on it, but rather whether
there was one or not.

Similarly to the previous attempt, we don't account for dmp messages
(fixed cost). Also we don't account properly for received hrmp messages
(hrmp_watermark) because the cost of it depends on the runtime state and
can't be statically deduced in the benchmark (unless we pass the
information about channels as benchmark u32 arguments).

The total weight cost of processing a parainherent now includes the cost
of enactment of each candidate, but we don't do filtering based on that
(because we enact after processing bitfields and making other changes to
the storage).

## Numbers

```
Reads = 7 + (0 * u) + (3 * h) + (8 * c)
Writes = 10 + (1 * u) + (3 * h) + (7 * c)
```
In addition, there is a fixed cost of a few of ms (!) per candidate. 

This might result a full block slightly overflowing its weight with 200
enacted candidates, which in turn could prevent non-mandatory
transactions from being included in a block.

Given our modest limits on max ump and hrmp messages:
```
  maxUpwardMessageNumPerCandidate: 16
  hrmpMaxMessageNumPerCandidate: 10
```
and the fact that runtime upgrades are can't happen very frequently
(`validation_upgrade_cooldown`), we might only go over the limits in
case of many disputes.

TODOs:
- [x] Fix the overweight test
- [x] Generate the weights for Westend and Rococo
- [x] PRDoc

---------

Co-authored-by: command-bot <>
Co-authored-by: Alin Dima <alin@parity.io>

Short-term addresses
https://github.com/paritytech/polkadot-sdk/issues/4737.

- [x] Resolve benchmarking
I've digged into benchmarking mentioned
https://github.com/paritytech/polkadot-sdk/issues/4737#issuecomment-2155084660,
but it seemed to me that this code is different proof/path. @acatangiu
could you confirm? (btw, in this
[bench](https://github.com/paritytech/polkadot-sdk/blob/b65313e8

/bridges/modules/parachains/src/benchmarking.rs#L57),
where do you actually set the `fn parachains()` to a reasonable number?
i've only seen 1)
- [ ] Communicate to Snowfork team:
This seems to be the relevant code:
https://github.com/Snowfork/snowbridge/blob/1e18e010331777042aa7e8fff3c118094af856ba/relayer/cmd/parachain_head_proof.go#L95-L120
- [x] Is it preferred to iter() in some random order as suggested in
https://github.com/paritytech/polkadot-sdk/issues/4737#issuecomment-2155084660
or take lowest para ids instead as implemented here currently?
- [x] PRDoc

## Updating Polkadot and Kusama runtimes:

New weights need to be generated (`pallet_mmr`) and configs updated
similar to Rococo/Westend:
```patch
diff --git a/polkadot/runtime/rococo/src/lib.rs b/polkadot/runtime/rococo/src/lib.rs
index 5adffbd7422..c7da339b981 100644
--- a/polkadot/runtime/rococo/src/lib.rs
+++ b/polkadot/runtime/rococo/src/lib.rs
@@ -1307,9 +1307,11 @@ impl pallet_mmr::Config for Runtime {
        const INDEXING_PREFIX: &'static [u8] = mmr::INDEXING_PREFIX;
        type Hashing = Keccak256;
        type OnNewRoot = pallet_beefy_mmr::DepositBeefyDigest<Runtime>;
-       type WeightInfo = ();
        type LeafData = pallet_beefy_mmr::Pallet<Runtime>;
        type BlockHashProvider = pallet_mmr::DefaultBlockHashProvider<Runtime>;
+       type WeightInfo = weights::pallet_mmr::WeightInfo<Runtime>;
+       #[cfg(feature = "runtime-benchmarks")]
+       type BenchmarkHelper = parachains_paras::benchmarking::mmr_setup::MmrSetup<Runtime>;
 }

 parameter_types! {
@@ -1319,13 +1321,8 @@ parameter_types! {
 pub struct ParaHeadsRootProvider;
 impl BeefyDataProvider<H256> for ParaHeadsRootProvider {
        fn extra_data() -> H256 {
-               let mut para_heads: Vec<(u32, Vec<u8>)> = parachains_paras::Parachains::<Runtime>::get()
-                       .into_iter()
-                       .filter_map(|id| {
-                               parachains_paras::Heads::<Runtime>::get(&id).map(|head| (id.into(), head.0))
-                       })
-                       .collect();
-               para_heads.sort();
+               let para_heads: Vec<(u32, Vec<u8>)> =
+                       parachains_paras::Pallet::<Runtime>::sorted_para_heads();
                binary_merkle_tree::merkle_root::<mmr::Hashing, _>(
                        para_heads.into_iter().map(|pair| pair.encode()),
                )
@@ -1746,6 +1743,7 @@ mod benches {
                [pallet_identity, Identity]
                [pallet_indices, Indices]
                [pallet_message_queue, MessageQueue]
+               [pallet_mmr, Mmr]
                [pallet_multisig, Multisig]
                [pallet_parameters, Parameters]
                [pallet_preimage, Preimage]
```

---------

Co-authored-by: Adrian Catangiu <adrian@parity.io>

This should remove nearly all usage of `sp-std` except:
- bridge and bridge-hubs
- a few of frames re-export `sp-std`, keep them for now
- there is a usage of `sp_std::Writer`, I don't have an idea how to move
it

Please review proc-macro carefully. I'm not sure I'm doing it the right
way.

Note: need `/bot fmt`

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: command-bot <>

Inherited workspace dependencies cannot be renamed by the crate using
them (see [1](https://github.com/rust-lang/cargo/issues/12546),
[2](https://stackoverflow.com/questions/76792343/can-inherited-dependencies-in-rust-be-aliased-in-the-cargo-toml-file)).
Since we want to use inherited workspace dependencies everywhere, we
first need to unify all aliases that we use for a dependency throughout
the workspace.
The umbrella crate is currently excluded from this procedure, since it
should be able to export the crates by their original name without much
hassle.

For example: one crate may alias `parity-scale-codec` to `codec`, while
another crate does not alias it at all. After this change, all crates
have to use `codec` as name. The problematic combinations were:
- conflicting aliases: most crates aliases as `A` but some use `B`.
- missing alias: most of the crates alias a dep but some dont.
- superfluous alias: most crates dont alias a dep but some do.

The script that i used first determines whether most crates opted to
alias a dependency or not. From that info it decides whether to use an
alias or not. If it decided to use an alias, the most common one is used
everywhere.

To reproduce, i used
[this](https://github.com/ggwpez/substrate-scripts/blob/master/uniform-crate-alias.py)
python script in combination with
[this](https://github.com/ggwpez/zepter/blob/38ad10585fe98a5a86c1d2369738bc763a77057b/renames.json)
error output from Zepter.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>

Part of #3326 

@Kianenigma @ggwpez

 

polkadot address: 12poSUQPtcF1HUPQGY3zZu2P8emuW9YnsPduA4XG3oCEfJVp

---------

Signed-off-by: Matteo Muraca <mmuraca247@gmail.com>
Co-authored-by: ordian <write@reusable.software>

When using `schedule_code_upgrade` to change the code of a parachain in
the relay chain runtime, we had already fixed to not set the `GoAhead`
signal. This was done to not brick any parachain after the upgrade,
because they were seeing the signal without having any upgrade prepared.
The remaining problem is that the parachain code is only upgraded after
a parachain header was enacted, aka the parachain made some progress.
However, this is quite complicated if the parachain is bricked (which is
the most common scenario why to manually schedule a code upgrade). Thus,
this pull request replaces `SetGoAhead` with `UpgradeStrategy` to signal
to the logic kind of strategy want to use. The strategies are either
`SetGoAheadSignal` or `ApplyAtExpectedBlock`. `SetGoAheadSignal` sets
the go ahead signal as before and awaits a parachain block.
`ApplyAtExpectedBlock` schedules the upgrade and applies it directly at
the `expected_block` without waiting for the parachain to make any kind
of progress.

**Update:** Pushed additional changes based on the review comments.

**This pull request fixes various spelling mistakes in this
repository.**

Most of the changes are contained in the first **3** commits:

- `Fix spelling mistakes in comments and docs`

- `Fix spelling mistakes in test names`

- `Fix spelling mistakes in error messages, panic messages, logs and
tracing`

Other source code spelling mistakes are separated into individual
commits for easier reviewing:

- `Fix the spelling of 'authority'`

- `Fix the spelling of 'REASONABLE_HEADERS_IN_JUSTIFICATION_ANCESTRY'`

- `Fix the spelling of 'prev_enqueud_messages'`

- `Fix the spelling of 'endpoint'`

- `Fix the spelling of 'children'`

- `Fix the spelling of 'PenpalSiblingSovereignAccount'`

- `Fix the spelling of 'PenpalSudoAccount'`

- `Fix the spelling of 'insufficient'`

- `Fix the spelling of 'PalletXcmExtrinsicsBenchmark'`

- `Fix the spelling of 'subtracted'`

- `Fix the spelling of 'CandidatePendingAvailability'`

- `Fix the spelling of 'exclusive'`

- `Fix the spelling of 'until'`

- `Fix the spelling of 'discriminator'`

- `Fix the spelling of 'nonexistent'`

- `Fix the spelling of 'subsystem'`

- `Fix the spelling of 'indices'`

- `Fix the spelling of 'committed'`

- `Fix the spelling of 'topology'`

- `Fix the spelling of 'response'`

- `Fix the spelling of 'beneficiary'`

- `Fix the spelling of 'formatted'`

- `Fix the spelling of 'UNKNOWN_PROOF_REQUEST'`

- `Fix the spelling of 'succeeded'`

- `Fix the spelling of 'reopened'`

- `Fix the spelling of 'proposer'`

- `Fix the spelling of 'InstantiationNonce'`

- `Fix the spelling of 'depositor'`

- `Fix the spelling of 'expiration'`

- `Fix the spelling of 'phantom'`

- `Fix the spelling of 'AggregatedKeyValue'`

- `Fix the spelling of 'randomness'`

- `Fix the spelling of 'defendant'`

- `Fix the spelling of 'AquaticMammal'`

- `Fix the spelling of 'transactions'`

- `Fix the spelling of 'PassingTracingSubscriber'`

- `Fix the spelling of 'TxSignaturePayload'`

- `Fix the spelling of 'versioning'`

- `Fix the spelling of 'descendant'`

- `Fix the spelling of 'overridden'`

- `Fix the spelling of 'network'`

Let me know if this structure is adequate.

**Note:** The usage of the words `Merkle`, `Merkelize`, `Merklization`,
`Merkelization`, `Merkleization`, is somewhat inconsistent but I left it
as it is.

~~**Note:** In some places the term `Receival` is used to refer to
message reception, IMO `Reception` is the correct word here, but I left
it as it is.~~

~~**Note:** In some places the term `Overlayed` is used instead of the
more acceptable version `Overlaid` but I also left it as it is.~~

~~**Note:** In some places the term `Applyable` is used instead of the
correct version `Applicable` but I also left it as it is.~~

**Note:** Some usage of British vs American english e.g. `judgement` vs
`judgment`, `initialise` vs `initialize`, `optimise` vs `optimize` etc.
are both present in different places, but I suppose that's
understandable given the number of contributors.

~~**Note:** There is a spelling mistake in `.github/CODEOWNERS` but it
triggers errors in CI when I make changes to it, so I left it as it
is.~~

Signed-off-by: cuinix <915115094@qq.com>
Co-authored-by: Bastian Köcher <git@kchr.de>

Currently, anyone can registrar a code that exceeds the code size limit
when performing the upgrade from the registrar. This PR fixes that and
adds a new test to cover this.

cc @bkchr @eskimor

Also fixes: https://github.com/paritytech/polkadot-sdk/issues/1417

- [x] CoreIndex -> AssignmentProvider mapping will be able to change any
time.
- [x] Implement
- [x] Provide Migrations
- [x] Add and fix tests
- [x] Implement bulk assigner logic
- [x] bulk assigner tests
- [x] Port over current assigner to use bulk designer (+ share on-demand
with bulk): top-level assigner has core ranges: legacy, bulk
- [x] Adjust migrations to reflect new assigner structure
- [x] Move migration code to Assignment code directly and make it
recursive (make it possible to skip releases) -> follow up ticket.
- [x] Test migrations
- [x] Add migration PR to runtimes repo -> follow up ticket.
- [x] Wire up with actual UMP messages
- [x] Write PR docs

---------

Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com>
Co-authored-by: BradleyOlson64 <lotrftw9@gmail.com>
Co-authored-by: Anton Vilhelm Ásgeirsson <antonva@users.noreply.github.com>
Co-authored-by: antonva <anton.asgeirsson@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Marcin S. <marcin@realemail.net>
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: command-bot <>

## Motivation

`pallet-xcm` is the main user-facing interface for XCM functionality,
including assets manipulation functions like `teleportAssets()` and
`reserve_transfer_assets()` calls.

While `teleportAsset()` works both ways, `reserve_transfer_assets()`
works only for sending reserve-based assets to a remote destination and
beneficiary when the reserve is the _local chain_.

## Solution

This PR enhances `pallet_xcm::(limited_)reserve_withdraw_assets` to
support transfers when reserves are other chains.
This will allow complete, **bi-directional** reserve-based asset
transfers user stories using `pallet-xcm`.

Enables following scenarios:
- transferring assets with local reserve (was previously supported iff
asset used as fee also had local reserve - now it works in all cases),
- transferring assets with reserve on destination,
- transferring assets with reserve on remote/third-party chain (iff
assets and fees have same remote reserve),
- transferring assets with reserve different than the reserve of the
asset to be used as fees - meaning can be used to transfer random asset
with local/dest reserve while using DOT for fees on all involved chains,
even if DOT local/dest reserve doesn't match asset reserve,
- transferring assets with any type of local/dest reserve while using
fees which can be teleported between involved chains.

All of the above is done by pallet inner logic without the user having
to specify which scenario/reserves/teleports/etc. The correct scenario
and corresponding XCM programs are identified, and respectively, built
automatically based on runtime configuration of trusted teleporters and
trusted reserves.

#### Current limitations:
- while `fees` and "non-fee" `assets` CAN have different reserves (or
fees CAN be teleported), the remaining "non-fee" `assets` CANNOT, among
themselves, have different reserve locations (this is also implicitly
enforced by `MAX_ASSETS_FOR_TRANSFER=2`, but this can be safely
increased in the future).
- `fees` and "non-fee" `assets` CANNOT have **different remote**
reserves (this could also be supported in the future, but adds even more
complexity while possibly not being worth it - we'll see what the future
holds).

Fixes https://github.com/paritytech/polkadot-sdk/issues/1584
Fixes https://github.com/paritytech/polkadot-sdk/issues/2055

---------

Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>

The runtime code of a parachain can be replaced on the relay-chain via:

[cumulus]:
[enact_authorized_upgrade](https://github.com/paritytech/polkadot-sdk/blob/1a38d6d6/cumulus/pallets/parachain-system/src/lib.rs#L661);
this is used for a runtime upgrade when a parachain is not bricked.

[polkadot] (these are used when a parachain is bricked):
-
[force_set_current_code](https://github.com/paritytech/polkadot-sdk/blob/1a38d6d6/polkadot/runtime/parachains/src/paras/mod.rs#L823):
immediately changes the runtime code of a given para without a pvf check
(root).
-
[force_schedule_code_upgrade](https://github.com/paritytech/polkadot-sdk/blob/1a38d6d6/polkadot/runtime/parachains/src/paras/mod.rs#L864):
schedules a change to the runtime code of a given para including a pvf
check of the new code (root).
-
[schedule_code_upgrade](https://github.com/paritytech/polkadot-sdk/blob/1a38d6d6/polkadot/runtime/common/src/paras_registrar.rs#L395):
schedules a change to the runtime code of a given para including a pvf
check of the new code. Besides root, the parachain or parachain manager
can call this extrinsic given that the parachain is unlocked.

Polkadot signals a parachain to be ready for a runtime upgrade through
the
[GoAhead](https://github.com/paritytech/polkadot-sdk/blob/e4949344

/polkadot/primitives/src/v5/mod.rs#L1229)
signal.

When in cumulus `enact_authorized_upgrade` is executed, the same
underlying helper function of `force_schedule_code_upgrade` &
`schedule_code_upgrade`:
[schedule_code_upgrade](https://github.com/paritytech/polkadot/blob/09b61286da11921a3dda0a8e4015ceb9ef9cffca/runtime/parachains/src/paras/mod.rs#L1778),
is called on the relay-chain, which sets the `GoAhead` signal (if the
pvf is accepted).

If Cumulus receives the `GoAhead` signal from polkadot without having
the `PendingValidationCode` ready, it will panic
([ref](https://github.com/paritytech/polkadot/pull/7412)). For
`enact_authorized_upgrade` we know for sure the `PendingValidationCode`
is set. On the contrary, for `force_schedule_code_upgrade` &
`schedule_code_upgrade` this is not the case.

This PR includes a flag such that the `GoAhead` signal will only be set
when a runtime upgrade is enacted by the parachain
(`enact_authorized_upgrade`).

additional info: https://github.com/paritytech/polkadot/pull/7412

Closes #641

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Bastian Köcher <info@kchr.de>

* rfc14

* Update polkadot/runtime/common/src/paras_registrar/mod.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update polkadot/runtime/common/src/paras_registrar/mod.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* Update polkadot/runtime/common/src/paras_registrar/mod.rs

Co-authored-by: Bastian Köcher <git@kchr.de>

* fmt

* fix

* Update polkadot/runtime/common/src/paras_registrar/migration.rs

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

* 2224 is unlocked

* update migration list

* update comment

* use VersionedMigration

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* inclusion emulator logic for asynchronous backing (#4790)

* initial stab at candidate_context

* fmt

* docs & more TODOs

* some cleanups

* reframe as inclusion_emulator

* documentations yes

* update types

* add constraint modifications

* watermark

* produce modifications

* v2 primitives: re-export all v1 for consistency

* vstaging primitives

* emulator constraints: handle code upgrades

* produce outbound HRMP modifications

* stack.

* method for applying modifications

* method just for sanity-checking modifications

* fragments produce modifications, not prospectives

* make linear

* add some TODOs

* remove stacking; handle code upgrades

* take `fragment` private

* reintroduce stacking.

* fragment constructor

* add TODO

* allow validating fragments against future constraints

* docs

* relay-parent number and min code size checks

* check code upgrade restriction

* check max hrmp per candidate

* fmt

* remove GoAhead logic because it wasn't helpful

* docs on code upgrade failure

* test stacking

* test modifications against constraints

* fmt

* test fragments

* descending or duplicate test

* fmt

* remove unused imports in vstaging

* wrong primitives

* spellcheck

* Runtime changes for Asynchronous Backing (#4786)

* inclusion: utility for allowed relay-parents

* inclusion: use prev number instead of prev hash

* track most recent context of paras

* inclusion: accept previous relay-parents

* update dmp  advancement rule for async backing

* fmt

* add a comment about validation outputs

* clean up a couple of TODOs

* weights

* fix weights

* fmt

* Resolve dmp todo

* Restore inclusion tests

* Restore paras_inherent tests

* MostRecentContext test

* Benchmark for new paras dispatchable

* Prepare check_validation_outputs for upgrade

* cargo run --quiet --profile=production  --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_paras.rs

* cargo run --quiet --profile=production  --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_paras.rs

* cargo run --quiet --profile=production  --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_paras.rs

* cargo run --quiet --profile=production  --features=runtime-benchmarks -- benchmark --chain=rococo-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/rococo/src/weights/runtime_parachains_paras.rs

* Implementers guide changes

* More tests for allowed relay parents

* Add a github issue link

* Compute group index based on relay parent

* Storage migration

* Move allowed parents tracker to shared

* Compile error

* Get group assigned to core at the next block

* Test group assignment

* fmt

* Error instead of panic

* Update guide

* Extend doc-comment

* Update runtime/parachains/src/shared.rs

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

Co-authored-by: Chris Sosnin <chris125_@live.com>
Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>

* Prospective Parachains Subsystem (#4913)

* docs and skeleton

* subsystem skeleton

* main loop

* fragment tree basics & fmt

* begin fragment trees & view

* flesh out more of view update logic

* further flesh out update logic

* some refcount functions for fragment trees

* add fatal/non-fatal errors

* use non-fatal results

* clear up some TODOs

* ideal format for scheduling info

* add a bunch of TODOs

* some more fluff

* extract fragment graph to submodule

* begin fragment graph API

* trees, not graphs

* improve docs

* scope and constructor for trees

* add some test TODOs

* limit max ancestors and store constraints

* constructor

* constraints: fix bug in HRMP watermarks

* fragment tree population logic

* set::retain

* extract population logic

* implement add_and_populate

* fmt

* add some TODOs in tests

* implement child-selection

* strip out old stuff based on wrong assumptions

* use fatality

* implement pruning

* remove unused ancestor constraints

* fragment tree instantiation

* remove outdated comment

* add message/request types and skeleton for handling

* fmt

* implement handle_candidate_seconded

* candidate storage: handle backed

* implement handle_candidate_backed

* implement answer_get_backable_candidate

* remove async where not needed

* implement fetch_ancestry

* add logic for run_iteration

* add some docs

* remove global allow(unused), fix warnings

* make spellcheck happy (despite English)

* fmt

* bump Cargo.lock

* replace tracing with gum

* introduce PopulateFrom trait

* implement GetHypotheticalDepths

* revise docs slightly

* first fragment tree scope test

* more scope tests

* test add_candidate

* fmt

* test retain

* refactor test code

* test populate is recursive

* test contiguity of depth 0 is maintained

* add_and_populate tests

* cycle tests

* remove PopulateFrom trait

* fmt

* test hypothetical depths (non-recursive)

* have CandidateSeconded return membership

* tree membership requests

* Add a ProspectiveParachainsSubsystem struct

* add a staging API for base constraints

* add a `From` impl

* add runtime API for staging_validity_constraints

* implement fetch_base_constraints

* implement `fetch_upcoming_paras`

* remove reconstruction of candidate receipt; no obvious usecase

* fmt

* export message to broader module

* remove last TODO

* correctly export

* fix compilation and add GetMinimumRelayParent request

* make provisioner into a real subsystem with proper mesage bounds

* fmt

* fix ChannelsOut in overseer test

* fix overseer tests

* fix again

* fmt

* Integrate prospective parachains subsystem into backing: Part 1 (#5557)

* BEGIN ASYNC candidate-backing CHANGES

* rename & document modes

* answer prospective validation data requests

* GetMinimumRelayParents request is now plural

* implement an implicit view utility for backing subsystems

* implicit-view: get allowed relay parents

* refactorings and improvements to implicit view

* add some TODOs for tests

* split implicit view updates into 2 functions

* backing: define State to prepare for functional refactor

* add some docs

* backing: implement bones of new leaf activation logic

* backing: create per-relay-parent-states

* use new handle_active_leaves_update

* begin extracting logic from CandidateBackingJob

* mostly extract statement import from job logic

* handle statement imports outside of job logic

* do some TODO planning for prospective parachains integration

* finish rewriting backing subsystem in functional style

* add prospective parachains mode to relay parent entries

* fmt

* add a RejectedByProspectiveParachains error

* notify prospective parachains of seconded and backed candidates

* always validate candidates exhaustively in backing.

* return persisted_validation_data from validation

* handle rejections by prospective parachains

* implement seconding sanity check

* invoke validate_and_second

* Alter statement table to allow multiple seconded messages per validator

* refactor backing to have statements carry PVD

* clean up all warnings

* Add tests for implicit view

* Improve doc comments

* Prospective parachains mode based on Runtime API version

* Add a TODO

* Rework seconding_sanity_check

* Iterate over responses

* Update backing tests

* collator-protocol: load PVD from runtime

* Fix validator side tests

* Update statement-distribution to fetch PVD

* Fix statement-distribution tests

* Backing tests with prospective paras #1

* fix per_relay_parent pruning in backing

* Test multiple leaves

* Test seconding sanity check

* Import statement order

Before creating an entry in `PerCandidateState` map
wait for the approval from the prospective parachains

* Add a test for correct state updates

* Second multiple candidates per relay parent test

* Add backing tests with prospective paras

* Second more than one test without prospective paras

* Add a test for prospective para blocks

* Update malus

* typos

Co-authored-by: Chris Sosnin <chris125_@live.com>

* Track occupied depth in backing per parachain (#5778)

* provisioner: async backing changes (#5711)

* Provisioner changes for async backing

* Select candidates based on prospective paras mode

* Revert naming

* Update tests

* Update TODO comment

* review

* provisioner: async backing changes (#5711)

* Provisioner changes for async backing

* Select candidates based on prospective paras mode

* Revert naming

* Update tests

* Update TODO comment

* review

* fmt

* Network bridge changes for asynchronous backing + update subsystems to handle versioned packets (#5991)

* BEGIN STATEMENT DISTRIBUTION WORK

create a vstaging network protocol which is the same as v1

* mostly make network bridge amenable to vstaging

* network-bridge: fully adapt to vstaging

* add some TODOs for tests

* fix fallout in bitfield-distribution

* bitfield distribution tests + TODOs

* fix fallout in gossip-support

* collator-protocol: fix message fallout

* collator-protocol: load PVD from runtime

* add TODO for vstaging tests

* make things compile

* set used network protocol version using a feature

* fmt

* get approval-distribution building

* fix approval-distribution tests

* spellcheck

* nits

* approval distribution net protocol test

* bitfield distribution net protocol test

* Revert "collator-protocol: fix message fallout"

This reverts commit 07cc887303e16c6b3843ecb25cdc7cc2080e2ed1.

* Network bridge tests

Co-authored-by: Chris Sosnin <chris125_@live.com>

* remove max_pov_size requirement from prospective pvd request (#6014)

* remove max_pov_size requirement from prospective pvd request

* fmt

* Extract legacy statement distribution to its own module (#6026)

* add compatibility type to v2 statement distribution message

* warning cleanup

* handle compatibility layer for v2

* clean up an unimplemented!() block

* circulate statements based on version

* extract legacy v1 code into separate module

* remove unimplemented

* clean up naming of from_requester/responder

* remove TODOs

* have backing share seconded statements with PVD

* fmt

* fix warning

* Quick fix unused warning for not yet implemented/used staging messages.

* Fix network bridge test

* Fix wrong merge.

We now have 23 subsystems (network bridge split + prospective
parachains)

Co-authored-by: Robert Klotzner <robert.klotzner@gmx.at>

* Version 3 is already live.

* Fix tests (#6055)

* Fix backing tests

* Fix warnings.

* fmt

* collator-protocol: asynchronous backing changes (#5740)

* Draft collator side changes

* Start working on collations management

* Handle peer's view change

* Versioning on advertising

* Versioned collation fetching request

* Handle versioned messages

* Improve docs for collation requests

* Add spans

* Add request receiver to overseer

* Fix collator side tests

* Extract relay parent mode to lib

* Validator side draft

* Add more checks for advertisement

* Request pvd based on async backing mode

* review

* Validator side improvements

* Make old tests green

* More fixes

* Collator side tests draft

* Send collation test

* fmt

* Collator side network protocol versioning

* cleanup

* merge artifacts

* Validator side net protocol versioning

* Remove fragment tree membership request

* Resolve todo

* Collator side core state test

* Improve net protocol compatibility

* Validator side tests

* more improvements

* style fixes

* downgrade log

* Track implicit assignments

* Limit the number of seconded candidates per para

* Add a sanity check

* Handle fetched candidate

* fix tests

* Retry fetch

* Guard against dequeueing while already fetching

* Reintegrate connection management

* Timeout on advertisements

* fmt

* spellcheck

* update tests after merge

* validator assignment fixes for backing and collator protocol (#6158)

* Rename depth->ancestry len in tests

* Refactor group assignments

* Remove implicit assignments

* backing: consider occupied core assignments

* Track a single para on validator side

* Refactor prospective parachains mode request (#6179)

* Extract prospective parachains mode into util

* Skip activations depending on the mode

* backing: don't send backed candidate to provisioner (#6185)

* backing: introduce `CanSecond` request for advertisements filtering (#6225)

* Drop BoundToRelayParent

* draft changes

* fix backing tests

* Fix genesis ancestry

* Fix validator side tests

* more tests

* cargo generate-lockfile

* Implement `StagingValidityConstraints` Runtime API method (#6258)

* Implement StagingValidityConstraints

* spellcheck

* fix ump params

* Update hrmp comment

* Introduce ump per candidate limit

* hypothetical earliest block

* refactor primitives usage

* hypothetical earliest block number test

* fix build

* Prepare the Runtime for asynchronous backing upgrade (#6287)

* Introduce async backing params to runtime config

* fix cumulus config

* use config

* finish runtimes

* Introduce new staging API

* Update collator protocol

* Update provisioner

* Update prospective parachains

* Update backing

* Move async backing params lower in the config

* make naming consistent

* misc

* Use real prospective parachains subsystem (#6407)

* Backport `HypotheticalFrontier` into the feature branch (#6605)

* implement more general HypotheticalFrontier

* fmt

* drop unneeded request

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

* Resolve todo about legacy leaf activation (#6447)

* fix bug/warning in handling membership answers

* Remove `HypotheticalDepthRequest` in favor of `HypotheticalFrontierRequest` (#6521)

* Remove `HypotheticalDepthRequest` for `HypotheticalFrontierRequest`

* Update tests

* Fix (removed wrong docstring)

* Fix can_second request

* Patch some dead_code errors

---------

Co-authored-by: Chris Sosnin <chris125_@live.com>

* Async Backing: Send Statement Distribution "Backed" messages (#6634)

* Backing: Send Statement Distribution "Backed" messages

Closes #6590.

**TODO:**

- [ ] Adjust tests

* Fix compile errors

* (Mostly) fix tests

* Fix comment

* Fix test and compile error

* Test that `StatementDistributionMessage::Backed` is sent

* Fix compile error

* Fix some clippy errors

* Add prospective parachains subsystem tests (#6454)

* Add prospective parachains subsystem test

* Add `should_do_no_work_if_async_backing_disabled_for_leaf` test

* Implement `activate_leaf` helper, up to getting ancestry

* Finish implementing `activate_leaf`

* Small refactor in `activate_leaf`

* Get `CandidateSeconded` working

* Finish `send_candidate_and_check_if_found` test

* Refactor; send more leaves & candidates

* Refactor test

* Implement `check_candidate_parent_leaving_view` test

* Start work on `check_candidate_on_multiple_forks` test

* Don’t associate specific parachains with leaf

* Finish `correctly_updates_leaves` test

* Fix cycle due to reused head data

* Fix `check_backable_query` test

* Fix `check_candidate_on_multiple_forks` test

* Add `check_depth_and_pvd_queries` test

* Address review comments

* Remove TODO

* add a new index for output head data to candidate storage

* Resolve test TODOs

* Fix compile errors

* test candidate storage pruning, make sure new index is cleaned up

---------

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

* Node-side metrics for asynchronous backing (#6549)

* Add metrics for `prune_view_candidate_storage`

* Add metrics for `request_unblocked_collations`

* Fix docstring

* Couple fixes from review comments

* Fix `check_depth_query` test

* inclusion-emulator: mirror advancement rule check (#6361)

* inclusion-emulator: mirror advancement rule check

* fix build

* prospective-parachains: introduce `backed_in_path_only` flag for advertisements (#6649)

* Introduce `backed_in_path_only` flag for depth request

* fmt

* update doc comment

* fmt

* Add async-backing zombienet tests (#6314)

* Async backing: impl guide for statement distribution (#6738)

Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com>
Co-authored-by: alexgparity <115470171+alexgparity@users.noreply.github.com>

* Asynchronous backing statement distribution: Take III (#5999)

* add notification types for v2 statement-distribution

* improve protocol docs

* add empty vstaging module

* fmt

* add backed candidate packet request types

* start putting down structure of new logic

* handle activated leaf

* some sanity-checking on outbound statements

* fmt

* update vstaging share to use statements with PVD

* tiny refactor, candidate_hash location

* import local statements

* refactor statement import

* first stab at broadcast logic

* fmt

* fill out some TODOs

* start on handling incoming

* split off session info into separate map

* start in on a knowledge tracker

* address some grumbles

* format

* missed comment

* some docs for direct

* add note on slashing

* amend

* simplify 'direct' code

* finish up the 'direct' logic

* add a bunch of tests for the direct-in-group logic

* rename 'direct' to 'cluster', begin a candidate_entry module

* distill candidate_entry

* start in on a statement-store module

* some utilities for the statement store

* rewrite 'send_statement_direct' using new tools

* filter sending logic on peers which have the relay-parent in their view.

* some more logic for handling incoming statements

* req/res: BackedCandidatePacket -> AttestedCandidate + tweaks

* add a `validated_in_group` bitfield to BackedCandidateInventory

* BackedCandidateInventory -> Manifest

* start in on requester module

* add outgoing request for attested candidate

* add a priority mechanism for requester

* some request dispatch logic

* add seconded mask to tagged-request

* amend manifest to hold group index

* handle errors and set up scaffold for response validation

* validate attested candidate responses

* requester -> requests

* add some utilities for manipulating requests

* begin integrating requester

* start grid module

* tiny

* refactor grid topology to expose more info to subsystems

* fix grid_topology test

* fix overseer test

* implement topology group-based view construction logic

* fmt

* flesh out grid slightly more

* add indexed groups utility

* integrate Groups into per-session info

* refactor statement store to borrow Groups

* implement manifest knowledge utility

* add a test for topology setup

* don't send to group members

* test for conflicting manifests

* manifest knowledge tests

* fmt

* rename field

* garbage collection for grid tracker

* routines for finding correct/incorrect advertisers

* add manifest import logic

* tweak naming

* more tests for manifest import

* add comment

* rework candidates into a view-wide tracker

* fmt

* start writing boilerplate for grid sending

* fmt

* some more group boilerplate

* refactor handling of topology and authority IDs

* fmt

* send statements directly to grid peers where possible

* send to cluster only if statement belongs to cluster

* improve handling of cluster statements

* handle incoming statements along the grid

* API for introduction of candidates into the tree

* backing: use new prospective parachains API

* fmt prospective parachains changes

* fmt statement-dist

* fix condition

* get ready for tracking importable candidates

* prospective parachains: add Cow logic

* incomplete and complete hypothetical candidates

* remove keep_if_unneeded

* fmt

* implement more general HypotheticalFrontier

* fmt, cleanup

* add a by_parent_hash index to candidate tracker

* more framework for future code

* utilities for getting all hypothetical candidates for frontier

* track origin in statement store

* fmt

* requests should return peer

* apply post-confirmation reckoning

* flesh out import/announce/circulate logic on new statements

* adjust

* adjust TODO comment

* fix  backing tests

* update statement-distribution to use new indexedvec

* fmt

* query hypothetical candidates

* implement `note_importable_under`

* extract common utility of fragment tree updates

* add a helper function for getting statements unknown by backing

* import fresh statements to backing

* send announcements and acknowledgements over grid

* provide freshly importable statements

also avoid tracking backed candidates in statement distribution

* do not issue requests on newly importable candidates

* add TODO for later when confirming candidate

* write a routine for handling backed candidate notifications

* simplify grid substantially

* add some test TODOs

* handle confirmed candidates & grid announcements

* finish implementing manifest handling, including follow up statements

* send follow-up statements when acknowledging freshly backed

* fmt

* handle incoming acknowledgements

* a little DRYing

* wire up network messages to handlers

* fmt

* some skeleton code for peer view update handling

* more peer view skeleton stuff

* Fix async backing statement distribution tests (#6621)

* Fix compile errors in tests

* Cargo fmt

* Resolve some todos in async backing statement-distribution branch (#6482)

* Implement `remove_by_relay_parent`

* Extract `minimum_votes` to shared primitives.

* Add `can_send_statements_received_with_prejudice` test

* Fix test

* Update docstrings

* Cargo fmt

* Fix compile error

* Fix compile errors in tests

* Cargo fmt

* Add module docs; write `test_priority_ordering` (first draft)

* Fix `test_priority_ordering`

* Move `insert_or_update_priority`: `Drop` -> `set_cluster_priority`

* Address review comments

* Remove `Entry::get_mut`

* fix test compilation

* add a TODO for a test

* clean up a couple of TODOs

* implement sending pending cluster statements

* refactor utility function for sending acknowledgement and statements

* mostly implement catching peers up via grid

* Fix clippy error

* alter grid to track all pending statements

* fix more TODOs and format

* tweak a TODO in requests

* some logic for dispatching requests

* fmt

* skeleton for response receiving

* Async backing statement distribution: cluster tests (#6678)

* Add `pending_statements_set_when_receiving_fresh_statements`

* Add `pending_statements_updated_when_sending_statements` test

* fix up

* fmt

* update TODO

* rework seconded mask in requests

* change doc

* change unhandledresponse not to borrow request manager

* only accept responses sufficient to back

* finish implementing response handling

* extract statement filter to protocol crate

* rework requests: use statement filter in network protocol

* dispatch cluster requests correctly

* rework cluster statement sending

* implement request answering

* fmt

* only send confirmed candidate statement messages on unified relay-parent

* Fix Tests In Statement Distribution Branch

* Async Backing: Integrate `vstaging` of statement distribution into `lib.rs` (#6715)

* Integrate `handle_active_leaves_update`

* Integrate `share_local_statement`/`handle_backed_candidate_message`

* Start hooking up request/response flow

* Finish hooking up request/response flow

* Limit number of parallel requests in responder

* Fix test compilation errors

* Fix missing check for prospective parachains mode

* Fix some more compile errors

* clean up some review comments

* clean up warnings

* Async backing statement distribution: grid tests (#6673)

* Add `manifest_import_returns_ok_true` test

* cargo fmt

* Add pending_communication_receiving_manifest_on_confirmed_candidate

* Add `senders_can_provide_manifests_in_acknowledgement` test

* Add a couple of tests for pending statements

* Add `pending_statements_cleared_when_sending` test

* Add `pending_statements_respect_remote_knowledge` test

* Refactor group creation in tests

* Clarify docs

* Address some review comments

* Make some clarifications

* Fix post-merge errors

* Clarify test `senders_can_provide_manifests_in_acknowledgement`

* Try writing `pending_statements_are_updated_after_manifest_exchange`

* Document "seconding limit" and `reject_overflowing_manifests` test

* Test that seconding counts are not updated for validators on error

* Fix tests

* Fix manifest exchange test

* Add more tests in `requests.rs` (#6707)

This resolves remaining TODOs in this file.

* remove outdated inventory terminology

* Async backing statement distribution: `Candidates` tests (#6658)

* Async Backing: Fix clippy errors in statement distribution branch (#6720)

* Integrate `handle_active_leaves_update`

* Integrate `share_local_statement`/`handle_backed_candidate_message`

* Start hooking up request/response flow

* Finish hooking up request/response flow

* Limit number of parallel requests in responder

* Fix test compilation errors

* Fix missing check for prospective parachains mode

* Fix some more compile errors

* Async Backing: Fix clippy errors in statement distribution branch

* Fix some more clippy lints

* add tests module

* fix warnings in existing tests

* create basic test harness

* create a test state struct

* fmt

* create empty cluster & grid modules for tests

* some TODOs for cluster test suite

* describe test-suite for grid logic

* describe request test suite

* fix seconding-limit bug

* Remove extraneous `pub`

This somehow made it into my clippy PR.

* Fix some test compile warnings

* Remove some unneeded `allow`s

* adapt some new test helpers from Marcin

* add helper for activating a gossip topology

* add utility for signing statements

* helpers for connecting/disconnecting peers

* round out network utilities

* fmt

* fix bug in initializing validator-meta

* fix compilation

* implement first cluster test

* TODOs for incoming request tests

* Remove unneeded `make_committed_candidate` helper

* fmt

* some more tests for cluster

* add a TODO about grid senders

* integrate inbound req/res into test harness

* polish off initial cluster test suite

* keep introduce candidate request

* fix tests after introduce candidate request

* fmt

* Add grid protocol to module docs

* Fix comments

* Test `backed_in_path_only: true`

* Update node/network/protocol/src/lib.rs

Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>

* Update node/network/protocol/src/request_response/mod.rs

Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>

* Mark receiver with `vstaging`

* validate grid senders based on manifest kind

* fix mask_seconded/valid

* fix unwanted-mask check

* fix build

* resolve todo on leaf mode

* Unify protocol naming to vstaging

* fmt, fix grid test after topology change

* typo

Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>

* address review

* adjust comment, make easier to understand

* Fix typo

---------

Co-authored-by: Marcin S <marcin@bytedude.com>
Co-authored-by: Marcin S <marcin@realemail.net>
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
Co-authored-by: Chris Sosnin <chris125_@live.com>

* miscellaneous fixes to make asynchronous backing work (#6791)

* propagate network-protocol-staging feature

* add feature to adder-collator as well

* allow collation-generation of occupied cores

* prospective parachains: special treatment for pending availability candidates

* runtime: fetch candidates pending availability

* lazily construct PVD for pending candidates

* fix fallout in prospective parachains hypothetical/select_child

* runtime: enact candidates when creating paras-inherent

* make tests compile

* test pending availability in the scope

* add prospective parachains test

* fix validity constraints leftovers

* drop prints

* Fix typos

---------

Co-authored-by: Chris Sosnin <chris125_@live.com>
Co-authored-by: Marcin S <marcin@realemail.net>

* Remove restart from test (#6840)

* Async Backing: Statement Distribution Tests (#6755)

* start on handling incoming

* split off session info into separate map

* start in on a knowledge tracker

* address some grumbles

* format

* missed comment

* some docs for direct

* add note on slashing

* amend

* simplify 'direct' code

* finish up the 'direct' logic

* add a bunch of tests for the direct-in-group logic

* rename 'direct' to 'cluster', begin a candidate_entry module

* distill candidate_entry

* start in on a statement-store module

* some utilities for the statement store

* rewrite 'send_statement_direct' using new tools

* filter sending logic on peers which have the relay-parent in their view.

* some more logic for handling incoming statements

* req/res: BackedCandidatePacket -> AttestedCandidate + tweaks

* add a `validated_in_group` bitfield to BackedCandidateInventory

* BackedCandidateInventory -> Manifest

* start in on requester module

* add outgoing request for attested candidate

* add a priority mechanism for requester

* some request dispatch logic

* add seconded mask to tagged-request

* amend manifest to hold group index

* handle errors and set up scaffold for response validation

* validate attested candidate responses

* requester -> requests

* add some utilities for manipulating requests

* begin integrating requester

* start grid module

* tiny

* refactor grid topology to expose more info to subsystems

* fix grid_topology test

* fix overseer test

* implement topology group-based view construction logic

* fmt

* flesh out grid slightly more

* add indexed groups utility

* integrate Groups into per-session info

* refactor statement store to borrow Groups

* implement manifest knowledge utility

* add a test for topology setup

* don't send to group members

* test for conflicting manifests

* manifest knowledge tests

* fmt

* rename field

* garbage collection for grid tracker

* routines for finding correct/incorrect advertisers

* add manifest import logic

* tweak naming

* more tests for manifest import

* add comment

* rework candidates into a view-wide tracker

* fmt

* start writing boilerplate for grid sending

* fmt

* some more group boilerplate

* refactor handling of topology and authority IDs

* fmt

* send statements directly to grid peers where possible

* send to cluster only if statement belongs to cluster

* improve handling of cluster statements

* handle incoming statements along the grid

* API for introduction of candidates into the tree

* backing: use new prospective parachains API

* fmt prospective parachains changes

* fmt statement-dist

* fix condition

* get ready for tracking importable candidates

* prospective parachains: add Cow logic

* incomplete and complete hypothetical candidates

* remove keep_if_unneeded

* fmt

* implement more general HypotheticalFrontier

* fmt, cleanup

* add a by_parent_hash index to candidate tracker

* more framework for future code

* utilities for getting all hypothetical candidates for frontier

* track origin in statement store

* fmt

* requests should return peer

* apply post-confirmation reckoning

* flesh out import/announce/circulate logic on new statements

* adjust

* adjust TODO comment

* fix  backing tests

* update statement-distribution to use new indexedvec

* fmt

* query hypothetical candidates

* implement `note_importable_under`

* extract common utility of fragment tree updates

* add a helper function for getting statements unknown by backing

* import fresh statements to backing

* send announcements and acknowledgements over grid

* provide freshly importable statements

also avoid tracking backed candidates in statement distribution

* do not issue requests on newly importable candidates

* add TODO for later when confirming candidate

* write a routine for handling backed candidate notifications

* simplify grid substantially

* add some test TODOs

* handle confirmed candidates & grid announcements

* finish implementing manifest handling, including follow up statements

* send follow-up statements when acknowledging freshly backed

* fmt

* handle incoming acknowledgements

* a little DRYing

* wire up network messages to handlers

* fmt

* some skeleton code for peer view update handling

* more peer view skeleton stuff

* Fix async backing statement distribution tests (#6621)

* Fix compile errors in tests

* Cargo fmt

* Resolve some todos in async backing statement-distribution branch (#6482)

* Implement `remove_by_relay_parent`

* Extract `minimum_votes` to shared primitives.

* Add `can_send_statements_received_with_prejudice` test

* Fix test

* Update docstrings

* Cargo fmt

* Fix compile error

* Fix compile errors in tests

* Cargo fmt

* Add module docs; write `test_priority_ordering` (first draft)

* Fix `test_priority_ordering`

* Move `insert_or_update_priority`: `Drop` -> `set_cluster_priority`

* Address review comments

* Remove `Entry::get_mut`

* fix test compilation

* add a TODO for a test

* clean up a couple of TODOs

* implement sending pending cluster statements

* refactor utility function for sending acknowledgement and statements

* mostly implement catching peers up via grid

* Fix clippy error

* alter grid to track all pending statements

* fix more TODOs and format

* tweak a TODO in requests

* some logic for dispatching requests

* fmt

* skeleton for response receiving

* Async backing statement distribution: cluster tests (#6678)

* Add `pending_statements_set_when_receiving_fresh_statements`

* Add `pending_statements_updated_when_sending_statements` test

* fix up

* fmt

* update TODO

* rework seconded mask in requests

* change doc

* change unhandledresponse not to borrow request manager

* only accept responses sufficient to back

* finish implementing response handling

* extract statement filter to protocol crate

* rework requests: use statement filter in network protocol

* dispatch cluster requests correctly

* rework cluster statement sending

* implement request answering

* fmt

* only send confirmed candidate statement messages on unified relay-parent

* Fix Tests In Statement Distribution Branch

* Async Backing: Integrate `vstaging` of statement distribution into `lib.rs` (#6715)

* Integrate `handle_active_leaves_update`

* Integrate `share_local_statement`/`handle_backed_candidate_message`

* Start hooking up request/response flow

* Finish hooking up request/response flow

* Limit number of parallel requests in responder

* Fix test compilation errors

* Fix missing check for prospective parachains mode

* Fix some more compile errors

* clean up some review comments

* clean up warnings

* Async backing statement distribution: grid tests (#6673)

* Add `manifest_import_returns_ok_true` test

* cargo fmt

* Add pending_communication_receiving_manifest_on_confirmed_candidate

* Add `senders_can_provide_manifests_in_acknowledgement` test

* Add a couple of tests for pending statements

* Add `pending_statements_cleared_when_sending` test

* Add `pending_statements_respect_remote_knowledge` test

* Refactor group creation in tests

* Clarify docs

* Address some review comments

* Make some clarifications

* Fix post-merge errors

* Clarify test `senders_can_provide_manifests_in_acknowledgement`

* Try writing `pending_statements_are_updated_after_manifest_exchange`

* Document "seconding limit" and `reject_overflowing_manifests` test

* Test that seconding counts are not updated for validators on error

* Fix tests

* Fix manifest exchange test

* Add more tests in `requests.rs` (#6707)

This resolves remaining TODOs in this file.

* remove outdated inventory terminology

* Async backing statement distribution: `Candidates` tests (#6658)

* Async Backing: Fix clippy errors in statement distribution branch (#6720)

* Integrate `handle_active_leaves_update`

* Integrate `share_local_statement`/`handle_backed_candidate_message`

* Start hooking up request/response flow

* Finish hooking up request/response flow

* Limit number of parallel requests in responder

* Fix test compilation errors

* Fix missing check for prospective parachains mode

* Fix some more compile errors

* Async Backing: Fix clippy errors in statement distribution branch

* Fix some more clippy lints

* add tests module

* fix warnings in existing tests

* create basic test harness

* create a test state struct

* fmt

* create empty cluster & grid modules for tests

* some TODOs for cluster test suite

* describe test-suite for grid logic

* describe request test suite

* fix seconding-limit bug

* Remove extraneous `pub`

This somehow made it into my clippy PR.

* Fix some test compile warnings

* Remove some unneeded `allow`s

* adapt some new test helpers from Marcin

* add helper for activating a gossip topology

* add utility for signing statements

* helpers for connecting/disconnecting peers

* round out network utilities

* fmt

* fix bug in initializing validator-meta

* fix compilation

* implement first cluster test

* TODOs for incoming request tests

* Remove unneeded `make_committed_candidate` helper

* fmt

* Hook up request sender

* Add `valid_statement_without_prior_seconded_is_ignored` test

* Fix `valid_statement_without_prior_seconded_is_ignored` test

* some more tests for cluster

* add a TODO about grid senders

* integrate inbound req/res into test harness

* polish off initial cluster test suite

* keep introduce candidate request

* fix tests after introduce candidate request

* fmt

* Add grid protocol to module docs

* Remove obsolete test

* Fix comments

* Test `backed_in_path_only: true`

* Update node/network/protocol/src/lib.rs

Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>

* Update node/network/protocol/src/request_response/mod.rs

Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>

* Mark receiver with `vstaging`

* First draft of `ensure_seconding_limit_is_respected` test

* validate grid senders based on manifest kind

* fix mask_seconded/valid

* fix unwanted-mask check

* fix build

* resolve todo on leaf mode

* Unify protocol naming to vstaging

* Fix `ensure_seconding_limit_is_respected` test

* Start `backed_candidate_leads_to_advertisement` test

* fmt, fix grid test after topology change

* Send Backed notification

* Finish `backed_candidate_leads_to_advertisement` test

* Finish `peer_reported_for_duplicate_statements` test

* Finish `received_advertisement_before_confirmation_leads_to_request`

* Add `advertisements_rejected_from_incorrect_peers` test

* Add `manifest_rejected_*` tests

* Add `manifest_rejected_when_group_does_not_match_para` test

* Add `local_node_sanity_checks_incoming_requests` test

* Add `local_node_respects_statement_mask` test

* Add tests where peer is reported for providing invalid signatures

* Add `cluster_peer_allowed_to_send_incomplete_statements` test

* Add `received_advertisement_after_backing_leads_to_acknowledgement`

* Add `received_advertisement_after_confirmation_before_backing` test

* peer_reported_for_advertisement_conflicting_with_confirmed_candidate

* Add `peer_reported_for_not_enough_statements` test

* Add `peer_reported_for_providing_statements_meant_to_be_masked_out`

* Add `additional_statements_are_shared_after_manifest_exchange`

* Add `grid_statements_imported_to_backing` test

* Add `relay_parent_entering_peer_view_leads_to_advertisement` test

* Add `advertisement_not_re_sent_when_peer_re_enters_view` test

* Update node/network/statement-distribution/src/vstaging/tests/grid.rs

Co-authored-by: asynchronous rob <rphmeier@gmail.com>

* Resolve TODOs, update test

* Address unused code

* Add check after every test for unhandled requests

* Refactor (`make_dummy_leaf` and `handle_sent_request`)

* Refactor (`make_dummy_topology`)

* Minor refactor

---------

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
Co-authored-by: Chris Sosnin <chris125_@live.com>

* Fix some clippy lints in tests

* Async backing: minor fixes (#6920)

* bitfield-distribution test

* implicit view tests

* Refactor parameters -> params

* scheduler: update storage migration (#6963)

* update scheduler migration

* Adjust weight to account for storage read

* Statement Distribution Guide Edits (#7025)

* Statement distribution guide edits

* Addressed Marcin's comments

* Add attested candidate request retry timeouts (#6833)

Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
Co-authored-by: asynchronous rob <rphmeier@gmail.com>
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Chris Sosnin <chris125_@live.com>
Fix async backing statement distribution tests (#6621)
Resolve some todos in async backing statement-distribution branch (#6482)
Fix clippy errors in statement distribution branch (#6720)

* Async backing: add Prospective Parachains impl guide (#6933)

Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com>

* Updates to Provisioner Guide for Async Backing (#7106)

* Initial corrections and clarifications

* Partial first draft

* Finished first draft

* Adding back wrongly removed test bit

* fmt

* Update roadmap/implementers-guide/src/node/utility/provisioner.md

Co-authored-by: Marcin S. <marcin@realemail.net>

* Addressing comments

* Reorganization

* fmt

---------

Co-authored-by: Marcin S. <marcin@realemail.net>

* fmt

* Renaming Parathread Mentions (#7287)

* Renaming parathreads

* Renaming module to pallet

* More updates

* PVF: Refactor workers into separate crates, remove host dependency (#7253)

* PVF: Refactor workers into separate crates, remove host dependency

* Fix compile error

* Remove some leftover code

* Fix compile errors

* Update Cargo.lock

* Remove worker main.rs files

I accidentally copied these from the other PR. This PR isn't intended to
introduce standalone workers yet.

* Address review comments

* cargo fmt

* Update a couple of comments

* Update log targets

* Update quote to 1.0.27 (#7280)

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: parity-processbot <>

* pallets: implement `Default` for `GenesisConfig` in `no_std` (#7271)

* pallets: implement Default for GenesisConfig in no_std

This change is follow-up of: https://github.com/paritytech/substrate/pull/14108

It is a step towards: https://github.com/paritytech/substrate/issues/13334

* Cargo.lock updated

* update lockfile for {"substrate"}

---------

Co-authored-by: parity-processbot <>

* cli: enable BEEFY by default on test networks (#7293)

We consider BEEFY mature enough to run by default on all nodes
for test networks (Rococo/Wococo/Versi).

Right now, most nodes are not running it since it's opt-in using
--beefy flag. Switch to an opt-out model for test networks.

Replace --beefy flag from CLI with --no-beefy and have BEEFY
client start by default on test networks.

Signed-off-by: acatangiu <adrian@parity.io>

* runtime: past session slashing runtime API (#6667)

* runtime/vstaging: unapplied_slashes runtime API

* runtime/vstaging: key_ownership_proof runtime API

* runtime/ParachainHost: submit_report_dispute_lost

* fix key_ownership_proof API

* runtime: submit_report_dispute_lost runtime API

* nits

* Update node/subsystem-types/src/messages.rs

Co-authored-by: Marcin S. <marcin@bytedude.com>

* revert unrelated fmt changes

* post merge fixes

* fix compilation

---------

Co-authored-by: Marcin S. <marcin@bytedude.com>

* Correcting git mishap

* Document usage of `gum` crate (#7294)

* Document usage of gum crate

* Small fix

* Add some more basic info

* Update node/gum/src/lib.rs

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

* Update target docs

---------

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

* XCM: Fix issue with RequestUnlock (#7278)

* XCM: Fix issue with RequestUnlock

* Leave API changes for v4

* Fix clippy errors

* Fix tests

---------

Co-authored-by: parity-processbot <>

* Companion for Substrate#14228 (#7295)

* Companion for Substrate#14228

https://github.com/paritytech/substrate/pull/14228

* update lockfile for {"substrate"}

---------

Co-authored-by: parity-processbot <>

* Companion for #14237: Use latest sp-crates (#7300)

* To revert: Update substrate branch to "lexnv/bump_sp_crates"

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Revert "To revert: Update substrate branch to "lexnv/bump_sp_crates""

This reverts commit 5f1db84eac4a226c37b7f6ce6ee19b49dc7e2008.

* Update cargo lock

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update cargo.lock

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update cargo.lock

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* bounded-collections bump to 0.1.7 (#7305)

* bounded-collections bump to 0.1.7

Companion for: paritytech/substrate#14225

* update lockfile for {"substrate"}

---------

Co-authored-by: parity-processbot <>

* bump to quote 1.0.28 (#7306)

* `RollingSessionWindow` cleanup (#7204)

* Replace `RollingSessionWindow` with `RuntimeInfo` - initial commit

* Fix tests in import

* Fix the rest of the tests

* Remove dead code

* Fix todos

* Simplify session caching

* Comments for `SessionInfoProvider`

* Separate `SessionInfoProvider` from `State`

* `cache_session_info_for_head` becomes freestanding function

* Remove unneeded `mut` usage

* fn session_info -> fn get_session_info() to avoid name clashes. The function also tries to initialize `SessionInfoProvider`

* Fix SessionInfo retrieval

* Code cleanup

* Don't wrap `SessionInfoProvider` in an `Option`

* Remove `earliest_session()`

* Remove pre-caching -> wip

* Fix some tests and code cleanup

* Fix all tests

* Fixes in tests

* Fix comments, variable names and small style changes

* Fix a warning

* impl From<SessionWindowSize> for NonZeroUsize

* Fix logging for `get_session_info` - remove redundant logs and decrease log level to DEBUG

* Code review feedback

* Storage migration removing `COL_SESSION_WINDOW_DATA` from parachains db

* Remove `col_session_data` usages

* Storage migration clearing columns w/o removing them

* Remove session data column usages from `approval-voting` and `dispute-coordinator` tests

* Add some test cases from `RollingSessionWindow` to `dispute-coordinator` tests

* Fix formatting in initialized.rs

* Fix a corner case in `SessionInfo` caching for `dispute-coordinator`

* Remove `RollingSessionWindow` ;(

* Revert "Fix formatting in initialized.rs"

This reverts commit 0f94664ec9f3a7e3737a30291195990e1e7065fc.

* v2 to v3 migration drops `COL_DISPUTE_COORDINATOR_DATA` instead of clearing it

* Fix `NUM_COLUMNS` in `approval-voting`

* Use `columns::v3::NUM_COLUMNS` when opening db

* Update node/service/src/parachains_db/upgrade.rs

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

* Don't write in `COL_DISPUTE_COORDINATOR_DATA` for `test_rocksdb_migrate_2_to_3`

* Fix `NUM+COLUMNS` in approval_voting

* Fix formatting

* Fix columns usage

* Clarification comments about the different db versions

---------

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

* pallet-para-config: Remove remnant WeightInfo functions (#7308)

* pallet-para-config: Remove remnant WeightInfo functions

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* set_config_with_weight begone

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* ".git/.scripts/commands/bench/bench.sh" runtime kusama-dev runtime_parachains::configuration

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>

* XCM: PayOverXcm config (#6900)

* Move XCM query functionality to trait

* Fix tests

* Add PayOverXcm implementation

* fix the PayOverXcm trait to compile

* moved doc comment out of trait implmeentation and to the trait

* PayOverXCM documentation

* Change documentation a bit

* Added empty benchmark methods implementation and changed docs

* update PayOverXCM to convert AccountIds to MultiLocations

* Implement benchmarking method

* Change v3 to latest

* Descend origin to an asset sender (#6970)

* descend origin to an asset sender

* sender as tuple of dest and sender

* Add more variants to the QueryResponseStatus enum

* Change Beneficiary to Into<[u8; 32]>

* update PayOverXcm to return concrete errors and use AccountId as sender

* use polkadot-primitives for AccountId

* fix dependency to use polkadot-core-primitives

* force Unpaid instruction to the top of the instructions list

* modify report_outcome to accept interior argument

* use new_query directly for building final xcm query, instead of report_outcome

* fix usage of new_query to use the XcmQueryHandler

* fix usage of new_query to use the XcmQueryHandler

* tiny method calling fix

* xcm query handler (#7198)

* drop redundant query status

* rename ReportQueryStatus to OuterQueryStatus

* revert rename of QueryResponseStatus

* update mapping

* Update xcm/xcm-builder/src/pay.rs

Co-authored-by: Gavin Wood <gavin@parity.io>

* Updates

* Docs

* Fix benchmarking stuff

* Destination can be determined based on asset_kind

* Tweaking API to minimise clones

* Some repotting and docs

---------

Co-authored-by: Anthony Alaribe <anthonyalaribe@gmail.com>
Co-authored-by: Muharem Ismailov <ismailov.m.h@gmail.com>
Co-authored-by: Anthony Alaribe <anthony.alaribe@parity.io>
Co-authored-by: Gavin Wood <gavin@parity.io>

* Companion for #14265 (#7307)

* Update Cargo.lock

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

* Update Cargo.lock

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: parity-processbot <>

* bump serde to 1.0.163 (#7315)

* bump serde to 1.0.163

* bump ci

* update lockfile for {"substrate"}

---------

Co-authored-by: parity-processbot <>

* fmt

* Updated fmt

* Removing changes accidentally pulled from master

* fix another master pull issue

* Another master pull fix

* fmt

* Fixing implementers guide build

* Revert "Merge branch 'rh-async-backing-feature-while-frozen' of https://github.com/paritytech/polkadot into brad-rename-parathread"

This reverts commit bebc24af52ab61155e3fe02cb3ce66a592bce49c, reversing
changes made to 1b2de662dfb11173679d6da5bd0da9d149c85547.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: acatangiu <adrian@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Marcin S <marcin@realemail.net>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: ordian <write@reusable.software>
Co-authored-by: Marcin S. <marcin@bytedude.com>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Sam Johnson <sam@durosoft.com>
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
Co-authored-by: Anthony Alaribe <anthonyalaribe@gmail.com>
Co-authored-by: Muharem Ismailov <ismailov.m.h@gmail.com>
Co-authored-by: Anthony Alaribe <anthony.alaribe@parity.io>
Co-authored-by: Gavin Wood <gavin@parity.io>

* fix bitfield distribution test

* approval distribution tests

* fix bridge tests

* update Cargo.lock

* [async-backing-branch] Optimize collator-protocol validator-side request fetching (#7457)

* Optimize collator-protocol validator-side request fetching

* address feedback: replace tuples with structs

* feedback: add doc comments

* move collation types to subfolder

---------

Signed-off-by: alindima <alin@parity.io>

* Update collation generation for asynchronous backing (#7405)

* break candidate receipt construction and distribution into own function

* update implementers' guide to include SubmitCollation

* implement SubmitCollation for collation-generation

* fmt

* fix test compilation & remove unnecessary submodule

* add some TODOs for a test suite.

* Update roadmap/implementers-guide/src/types/overseer-protocol.md

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

* add new test harness and first test

* refactor to avoid requiring background sender

* ensure collation gets packaged and distributed

* tests for the fallback case with no hint

* add parent rp-number hint tests

* fmt

* update uses of CollationGenerationConfig

* fix remaining test

* address review comments

* use subsystemsender for background tasks

* fmt

* remove ValidationCodeHashHint and related tests

---------

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

* fix some more fallout from merge

* fmt

* remove staging APIs from Rococo & Westend (#7513)

* send network messages on main protocol name (#7515)

* misc async backing improvements for allowed ancestry blocks (#7532)

* shared: fix acquire_info

* backwards-compat test for prospective parachains

* same relay parent is allowed

* provisioner: request candidate receipt by relay parent (#7527)

* return candidates hash from prospective parachains

* update provisioner

* update tests

* guide changes

* send a single message to backing

* fix test

* revert to old `handle_new_activations` logic in some cases (#7514)

* revert to old `handle_new_activations` logic

* gate sending messages on scheduled cores to max_depth >= 2

* fmt

* 2->1

* Omnibus asynchronous backing bugfix PR (#7529)

* fix a bug in backing

* add some more logs

* prospective parachains: take ancestry only up to session bounds

* add test

* fix zombienet tests (#7614)

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

* fix runtime compilation

* make bitfield distribution tests compile

* attempt to fix zombienet disputes (#7618)

* update metric name

* update some metric names

* avoid cycles when creating fake candidates

* make undying collator more friendly to malformed parents

* fix a bug in malus

* fmt

* clippy

* add RUN_IN_CONTAINER to new ZombieNet tests (#7631)

* remove duplicated migration

happened because of master-merge

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: acatangiu <adrian@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: alindima <alin@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: Chris Sosnin <chris125_@live.com>
Co-authored-by: Parity Bot <admin@parity.io>
Co-authored-by: Chris Sosnin <48099298+slumber@users.noreply.github.com>
Co-authored-by: Robert Klotzner <robert.klotzner@gmx.at>
Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Co-authored-by: Marcin S <marcin@bytedude.com>
Co-authored-by: Marcin S <marcin@realemail.net>
Co-authored-by: Mattia L.V. Bradascio <28816406+bredamatt@users.noreply.github.com>
Co-authored-by: Bradley Olson <34992650+BradleyOlson64@users.noreply.github.com>
Co-authored-by: alexgparity <115470171+alexgparity@users.noreply.github.com>
Co-authored-by: BradleyOlson64 <lotrftw9@gmail.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: ordian <write@reusable.software>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Sam Johnson <sam@durosoft.com>
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
Co-authored-by: Anthony Alaribe <anthonyalaribe@gmail.com>
Co-authored-by: Muharem Ismailov <ismailov.m.h@gmail.com>
Co-authored-by: Anthony Alaribe <anthony.alaribe@parity.io>
Co-authored-by: Gavin Wood <gavin@parity.io>
Co-authored-by: Alin Dima <alin@parity.io>

* Fix clippy check

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Autofix clippy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix trivial

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* suppress warnings

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Quiet clippy 



Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use same rustfmt.toml as Substrate

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* format format file

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Format with new config

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add Substrate Clippy config

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Print Clippy version in CI

Otherwise its difficult to reproduce locally.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Make fmt happy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update node/core/pvf/src/error.rs

Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>

* Update node/core/pvf/src/error.rs

Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>

* paras: count upgrade delay from inclusion

* fix warning

* rename check cause block number field

* rename inclusion_parent -> included_at

Moves `Block` to `frame_system` instead of `construct_runtime` and removes `Header` and `BlockNumber` (#7431)

* Companion for substrate

* Minor update

* Formatting

* Fixes for cumulus

* Fixes tests in polkadot-runtime-parachains

* Minor update

* Removes unused import

* Fixes tests in polkadot-runtime-common

* Minor fix

* Update roadmap/implementers-guide/src/runtime/configuration.md

Co-authored-by: ordian <write@reusable.software>

* ".git/.scripts/commands/fmt/fmt.sh"

* update lockfile for {"substrate"}

---------

Co-authored-by: ordian <write@reusable.software>
Co-authored-by: command-bot <>

* GenesisBuild<T,I> deprecated. BuildGenesisConfig added

* fmt

* fixes

* more fixes

* more fixes

* fixes

* update lockfile for {"substrate"}

* fix

---------

Co-authored-by: parity-processbot <>

After an upgrade cooldown triggered to remove a signal we also need to
prune it from the list.

* ParaId: Serialize/Deserialize trait implemented in no-std

* Serialize/Deserialize trait implemented in no-std for more types

* serde in deps enabled

* fix

* missing types added

* update lockfile for {"substrate"}

* ".git/.scripts/commands/fmt/fmt.sh"

---------

Co-authored-by: parity-processbot <>