Shouldn't matter much, but this is run on every produced block so free
performance

**Summary:** This PR enables authoring of multiple blocks in one AURA
slot in the slot-based collator and stabilizes the slot-based collator.

## CLI Changes
The flag `--experimental-use-slot-based` is now marked as deprecated. I
opted to introduce `--authoring slot-based` instead of just removing the
`experimental` prefix. By introducing the `authoring` variant, we get
some future-proofing in case we want to introduce further options.

## Change Description
With elastic-scaling, we are able to author multiple blocks with a
single relay-chain parent. In the initial iteration, the interval
between two blocks was determined by the `slot_duration` of the
parachain. This PR introduces a more flexible model, where we try to
author multiple blocks in a single slot if the runtime allows it.

The block authoring loop is largely the same. The
[`SlotTimer`](https://github.com/paritytech/polkadot-sdk/blob/f1935bd9/cumulus/cli...

Closes #7573

@skunert  @bkchr



---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

- Closes https://github.com/paritytech/license-scanner/issues/44
- Silent because only comments are changed in the crates.

## What's inside

First, we change the file traversal mechanism from shell globbing to
walking through files which happens inside the `license-scanner` - so it
has knowledge about directory structure and can correlate files with
corresponding Cargo manifest.

Next, added `MIT-0` and `Unlicense` to the allowed list of licenses.
`Unlicense` appears only once across {Substrate,Cumulus,Polkadot} - in
`penpal-runtime`.

Finally, updated headers in files that do not match the corresponding
manifest license.

---------

Co-authored-by: cornholio <0@mcornholio.ru>

Part of https://github.com/paritytech/polkadot-sdk/issues/5079.

Removes all usage of the static async backing params, replacing them
with dynamically computed equivalent values (based on the claim queue
and scheduling lookahead).

Adds a new runtime API for querying the scheduling lookahead value. If
not present, falls back to 3 (the default value that is backwards
compatible with values we have on production networks for
allowed_ancestry_len)

Also resolves most of
https://github.com/paritytech/polkadot-sdk/issues/4447, removing code
that handles async backing not yet being enabled.
While doing this, I removed the support for collation protocol version 1
on collators, as it only worked for leaves not supporting async backing
(which are none).
I also unhooked the legacy v1 statement-distribution (for the same
reason as above). That subsystem is basically dead code now, so I had to
remove some of its tests as they would no longer pass (since the
subsystem no longer sends messages to the legacy variant). I did not
remove the entire legacy subsystem yet, as that would pollute this PR
too much. We can remove the entire v1 and v2 validation protocols in a
follow up PR.

In another PR: remove test files with names `prospective_parachains`
(it'd pollute this PR if we do now)

TODO:
- [x] add deprecation warnings
- [x] prdoc

---------

Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>

On top of https://github.com/paritytech/polkadot-sdk/pull/6757

Fixes https://github.com/paritytech/polkadot-sdk/issues/6858 by bumping
the `PARENT_SEARCH_DEPTH` constant to a larger value (30) and adds a
zombienet-sdk test that exercises the 12-core scenario.

This is a node-side limit that restricts the number of allowed pending
availability candidates when choosing the parent parablock during
authoring.
This limit is rather redundant, as the parachain runtime already
restricts the unincluded segment length to the configured value in the
[FixedVelocityConsensusHook](https://github.com/paritytech/polkadot-sdk/blob/88d900af

/cumulus/pallets/aura-ext/src/consensus_hook.rs#L35)
(which ideally should be equal to this `PARENT_SEARCH_DEPTH`).

For 12 cores, a value of 24 should be enough, but I bumped it to 30 to
have some extra buffer.

There are two other potential ways of fixing this:
- remove this constant altogether, as the parachain runtime already
makes those guarantees. Chose not to do this, as it can't hurt to have
an extra safeguard
- set this value to be equal to the uninlcuded segment size. This value
however is not exposed to the node-side and would require a new runtime
API, which seems overkill for a redundant check.

---------

Co-authored-by: Javier Viola <javier@parity.io>

Closes #7055

@skunert @bkchr



---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: command-bot <>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

Follow-up to #6825, which introduced this bug.

We use the `can_build_upon` method to ask the runtime if it is fine to
build another block. The runtime checks this based on the
[`ConsensusHook`](https://github.com/paritytech/polkadot-sdk/blob/c1b7c302/cumulus/pallets/aura-ext/src/consensus_hook.rs#L110-L110)
implementation, the most popular one being the `FixedConsensusHook`.

In #6825 I removed a check that would always allow us to build when we
are building on an included block. Turns out this check is still
required when:
1. The [`UnincludedSegment`
](https://github.com/paritytech/polkadot-sdk/blob/c1b7c302

/cumulus/pallets/parachain-system/src/lib.rs#L758-L758)
storage item in pallet-parachain-system is equal or larger than the
unincluded segment.
2. We are calling the `can_build_upon` runtime API where the included
block has progressed offchain to the current parent block (so last entry
in the `UnincludedSegment` storage item).

In this scenario the last entry in `UnincludedSegment` does not have a
hash assigned yet (because it was not available in `on_finalize` of the
previous block). So the unincluded segment will be reported at its
maximum length which will forbid building another block.

Ideally we would have a more elegant solution than to rely on the
node-side here. But for now the check is reintroduced and a test is
added to not break it again by accident.

---------

Co-authored-by: command-bot <>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

closes #3967 

## Changes
We now use relay chain slots to measure velocity on chain. Previously we
were storing the current parachain slot. Then in `on_state_proof` of the
`ConsensusHook` we were checking how many blocks were athored in the
current parachain slot. This works well when the parachain slot time and
relay chain slot time is the same. With elastic scaling, we can have
parachain slot times lower than that of the relay chain. In these cases
we want to measure velocity in relation to the relay chain. This PR
adjusts that.


##  Migration
This PR includes a migration. Storage item `SlotInfo` of pallet
`aura-ext` is renamed to `RelaySlotInfo` to better reflect its new
content. A migration has been added that just kills the old storage
item. `RelaySlotInfo` will be `None` initially but its value will be
adjusted after one new relay chain slot arrives.

---------

Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <git@kchr.de>

## Problem
In the parachain template we use the [fully verifying import queue

](https://github.com/paritytech/polkadot-sdk/blob/3d9eddbe/cumulus/client/consensus/aura/src/equivocation_import_queue.rs#L224-L224)
which does extra equivocation checks.

However, when we import a warp synced block with state, we don't set a
fork choice, leading to an incomplete block import pipeline and error
here:
https://github.com/paritytech/polkadot-sdk/blob/3d9eddbe/substrate/client/service/src/client/client.rs#L488-L488

This renders warp sync useless for chains using this import queue.

## Fix
The fix is to always import a block with state as best block, as we
already do in the normal Aura Verifier.
In a follow up we should also take another look into unifying the usage
of the different import queues.

fixes https://github.com/paritytech/project-mythical/issues/256

---------

Co-authored-by: command-bot <>

It doesn't make sense to only reorder the features array.

For example:

This makes it hard for me to compare the dependencies and features,
especially some crates have a really really long dependencies list.
```toml​
[dependencies]
c = "*"
a = "*"
b = "*"

[features]
std = [
  "a",
  "b",
  "c",
]
```

This makes my life easier.
```toml​
[dependencies]
a = "*"
b = "*"
c = "*"

[features]
std = [
  "a",
  "b",
  "c",
]
```

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: command-bot <>

Move `RelayChainDataFetcher` to its own module and rename it to
`RelayChainDataCache`. Also move the `core_selector` function to the
`slot_based` module.

Related issue: https://github.com/paritytech/polkadot-sdk/issues/6495

The `SlotBasedBlockImport` job is to collect the storage proofs of all
blocks getting imported. These storage proofs alongside the block are
being forwarded to the collation task. Right now they are just being
thrown away. More logic will follow later. Basically this will be
required to include multiple blocks into one `PoV` which will then be
done by the collation task.

---------

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: GitHub Action <action@github.com>

Move spawning of the slot-based collator into the `run` function. Also
the tasks are being spawned as blocking task and not just as normal
tasks.

---------

Co-authored-by: GitHub Action <action@github.com>

Resolves https://github.com/paritytech/polkadot-sdk/issues/6179

on top of https://github.com/paritytech/polkadot-sdk/pull/5423

This PR implements the plumbing work required for
https://github.com/paritytech/polkadot-sdk/issues/5047 . I also added
additional helper methods gated by feature "test" in primitives.

TODO:
- [x] PRDoc

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: GitHub Action <action@github.com>

Runtime side of https://github.com/paritytech/polkadot-sdk/issues/5048

Send the core selector ump signal in cumulus. Guarded by a compile time
feature until nodes are upgraded to a version that includes
https://github.com/paritytech/polkadot-sdk/pull/5423 for gracefully
handling ump signals.

---------

Co-authored-by: GitHub Action <action@github.com>

Partially implements
https://github.com/paritytech/polkadot-sdk/issues/5048

- adds a core selection runtime API to cumulus and a generic way of
configuring it for a parachain
- modifies the slot based collator to utilise the claim queue and the
generic core selection

What's left to be implemented (in a follow-up PR):
- add the UMP signal for core selection into the parachain-system pallet

View the RFC for more context:
https://github.com/polkadot-fellows/RFCs/pull/103

---------

Co-authored-by: command-bot <>

When using the relay chain though a `Arc<dyn RelayChainInterface>` there
is no way to call arbitrary runtime apis. Both implementations of that
trait allow this, so it feels natural to expose this functionality in
the trait.

This PR adds a `call_runtime_api` method to RelayChainInterface trait,
and a separate function also named `call_runtime_api` which allows the
caller to specify the input and output types, as opposed to having to
encode them. This generic function cannot be part of the trait because a
`dyn Trait` object cannot have generic methods.

---------

Co-authored-by: Bastian Köcher <git@kchr.de>

Co-authored-by: kianenigma <kian@parity.io>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

This PR introduces a feature that allows to optionally enable using the
full PoV size.

Technically, we're ready to enable it by default, but as corresponding
runtime changes have not been propagated to the system parachain
runtimes yet, doing so could put them at risk. On the other hand, there
are teams that could benefit from it right now, and it makes no sense
for them to wait for the fellowship release and everything.

---------

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

There was no need for it to be `&mut self` since block import can happen
concurrently for different blocks and in many cases it was `&mut Arc<dyn
BlockImport>` anyway :man_shrugging:

Similar in nature to
https://github.com/paritytech/polkadot-sdk/pull/4844

We only want to build one block per slot for Aura on parachains.
However, we still need to build on each relay chain fork, which is using
the same slot.

Closes: https://github.com/paritytech/polkadot-sdk/issues/5349

---------

Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

This pull requests adds a new CLI flag to `polkadot-parachains`
`--export-pov-to-path`. This CLI flag will instruct the node to export
any `PoV` that it build locally to export to the given folder. Then
these `PoV` files can be validated using the introduced
`cumulus-pov-validator`. The combination of export and validation can be
used for debugging parachain validation issues that may happen on the
relay chain.

Part of #3168 
On top of #3568

### Changes Overview
- Introduces a new collator variant in
`cumulus/client/consensus/aura/src/collators/slot_based/mod.rs`
- Two tasks are part of that module, one for block building and one for
collation building and submission.
- Introduces a new variant of `cumulus-test-runtime` which has 2s slot
duration, used for zombienet testing
- Zombienet tests for the new collator

**Note:** This collator is considered experimental and should only be
used for testing and exploration for now.

### Comparison with `lookahead` collator
- The new variant is slot based, meaning it waits for the next slot of
the parachain, then starts authoring
- The search for potential parents remains mostly unchanged from
lookahead
- As anchor, we use the current best relay parent
- In general, the new collator tends to be anchored to one relay parent
earlier. `lookahead` generally waits for a new relay block to arrive
before it attempts to build a block. ...

A few refactorings to block import and block verification that should
not be controversial.

Block verification before block import is stateless by design as
described in https://substrate.stackexchange.com/a/1322/25 and the fact
that it wasn't yet I consider to be a bug. Some code that requires it
had to use `Mutex`, but I do not expect it to have a measurable
performance impact.

Similarly with block import checking whether block preconditions should
not be an exclusive operation, there is nothing fundamentally wrong with
checking a few competing blocks whose parent blocks exist at the same
time (and even import them concurrently later, though IIRC this is not
yet implemented either).

They were originally a part of
https://github.com/paritytech/polkadot-sdk/pull/4842 and upstreaming
will help us to reduce the size of the patch we need to apply on top of
upstream code at Subspace every time we upgrade. There are no new
features introduced here, just refactoring to get rid of unnecessary
requirements.

After preparing in https://github.com/paritytech/polkadot-sdk/pull/4633,
we can lift also all internal dependencies up to the workspace.

This does not actually change anything, but uses `workspace = true` for
all dependencies. You can check it with:
```bash
git checkout -q $(git merge-base oty-lift-all-deps origin/master)
cargo tree -e features > master.out

git checkout -q oty-lift-all-deps
cargo tree -e features > new.out
diff master.out new.out
```

It did not yet lift 100% of dependencies, some inside of `target.*` or
some that had conflicting aliases introduced recently. But i will do
these together in a follow-up with CI checks.

Can be reproduced with [zepter](https://github.com/ggwpez/zepter/):
`zepter transpose d lift-to-workspace "regex:.*" --version-resolver
highest --skip-package "polkadot-sdk" --ignore-errors --fix`.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

This unifies the logic between `CallExecutor` and `Client` when it comes
to fetching the `code` for a given block. The actual `code` depends on
potential overrides/substitutes.

Besides that it changes the logic in the lookahead collator on which
`ValidationCodeHash` it sends to the validator alongside the `POV`. We
are now sending the code hash as found on the relay chain. This is done
as the local node could run with an override which is compatible to the
validation code on the relay chain, but has a different hash.

- unit tests for pov-recovery
- elastic scaling support (recovering multiple candidates in a single
relay chain block)
- also some small cleanups
- also switches to candidates_pending_availability in
`handle_empty_block_announce_data`

Fixes https://github.com/paritytech/polkadot-sdk/issues/3577

After https://github.com/paritytech/polkadot-sdk/pull/4097 is merged, we
should also add a zombienet test, similar to the existing
`0002-pov_recovery.toml` but which has a single collator using elastic
scaling on multiple cores.

Inherited workspace dependencies cannot be renamed by the crate using
them (see [1](https://github.com/rust-lang/cargo/issues/12546),
[2](https://stackoverflow.com/questions/76792343/can-inherited-dependencies-in-rust-be-aliased-in-the-cargo-toml-file)).
Since we want to use inherited workspace dependencies everywhere, we
first need to unify all aliases that we use for a dependency throughout
the workspace.
The umbrella crate is currently excluded from this procedure, since it
should be able to export the crates by their original name without much
hassle.

For example: one crate may alias `parity-scale-codec` to `codec`, while
another crate does not alias it at all. After this change, all crates
have to use `codec` as name. The problematic combinations were:
- conflicting aliases: most crates aliases as `A` but some use `B`.
- missing alias: most of the crates alias a dep but some dont.
- superfluous alias: most crates dont alias a dep but some do.

The script that i used first determines whether most crates opted to
alias a dependency or not. From that info it decides whether to use an
alias or not. If it decided to use an alias, the most common one is used
everywhere.

To reproduce, i used
[this](https://github.com/ggwpez/substrate-scripts/blob/master/uniform-crate-alias.py)
python script in combination with
[this](https://github.com/ggwpez/zepter/blob/38ad10585fe98a5a86c1d2369738bc763a77057b/renames.json)
error output from Zepter.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>

This bump of versions:

https://github.com/paritytech/polkadot-sdk/pull/4409/files#diff-13ee4b2252c9e516a0547f2891aa2105c3ca71c6d7a1e682c69be97998dfc87eR11936

reintroduced a dependency to proc-macro-crate 2.0.0 which is suffering
from: https://github.com/bkchr/proc-macro-crate/pull/42 this, so bump
parity-scale-codec to a newer version to eliminate the bad
proc-macro-crate 2.0.0 dependency.

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: command-bot <>

Some traits are already included in the 2021 prelude and so shouldn't be
needed to use explicitly:

use `convert::TryFrom`, `convert::TryInto`, and `iter::FromIterator` are
removed.

( https://doc.rust-lang.org/core/prelude/rust_2021/ )

No breaking changes or change of functionality, so I think no PR doc is
needed in this case.

(Motivation: Removes some references to `sp-std`)

This is phase 2 of async backing enablement for the system parachains on
the production networks. ~~It should be merged after
polkadot-fellows/runtimes#228 is enacted. After it is released,~~ all
the system parachain collators should be upgraded, and then we can
proceed with phase 3, which will enable async backing in the runtimes.

UPDATE: Indeed, we don't need to wait for the runtime upgrade enactions.
The lookahead collator handles the transition by itself, so we can
upgrade ASAP.

## Scope of changes

Here, we eliminate the dichotomy of having "generic Aura collators" for
the production system parachains and "lookahead Aura collators" for the
testnet system parachains. Now, all the collators are started as
lookahead ones, preserving the logic of transferring from the shell node
to Aura-enabled collators for the asset hubs. So, indeed, it simplifies
the parachain service logic, which cannot but rejoice.

Cumulus test-parachain node and test runtime were still using relay
chain consensus and 12s blocktimes. With async backing around the corner
on the major chains we should switch our tests too.

Also needed to nicely test the changes coming to collators in #3168.

### Changes Overview
- Followed the [migration
guide](https://wiki.polkadot.network/docs/maintain-guides-async-backing)
for async backing for the cumulus-test-runtime
- Adjusted the cumulus-test-service to use the correct import-queue,
lookahead collator etc.
- The block validation function now uses the Aura Ext Executor so that
the seal of the block is validated
- Previous point requires that we seal block before calling into
`validate_block`, I introduced a helper function for that
- Test client adjusted to provide a slot to the relay chain proof and
the aura pre-digest

Also while doing this, move slot duration fetching into the AURA code.

With Coretime enabled we can no longer assume there is a static 1:1
mapping between core index and para id. This mapping should be obtained
from the scheduler/claimqueue on block by block basis.

This PR modifies `para_id()` (from `CoreState`) to return the scheduled
`ParaId` for occupied cores and removes its usages in the code.

Closes https://github.com/paritytech/polkadot-sdk/issues/3948

---------

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

Working towards migrating the `parity-bridges-common` repo inside
`polkadot-sdk`. This PR upgrades some dependencies in order to align
them with the versions used in `parity-bridges-common`

Related to
https://github.com/paritytech/parity-bridges-common/issues/2538

This works only for collators that implement the `collator_fn` allowing
`collation-generation` subsystem to pull collations triggered on new
heads.

Also enables
`request_v2::CollationFetchingResponse::CollationWithParentHeadData` for
test adder/undying collators.

TODO:
- [x] fix tests
- [x] new tests
- [x] PR doc

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

**Update:** Pushed additional changes based on the review comments.

**This pull request fixes various spelling mistakes in this
repository.**

Most of the changes are contained in the first **3** commits:

- `Fix spelling mistakes in comments and docs`

- `Fix spelling mistakes in test names`

- `Fix spelling mistakes in error messages, panic messages, logs and
tracing`

Other source code spelling mistakes are separated into individual
commits for easier reviewing:

- `Fix the spelling of 'authority'`

- `Fix the spelling of 'REASONABLE_HEADERS_IN_JUSTIFICATION_ANCESTRY'`

- `Fix the spelling of 'prev_enqueud_messages'`

- `Fix the spelling of 'endpoint'`

- `Fix the spelling of 'children'`

- `Fix the spelling of 'PenpalSiblingSovereignAccount'`

- `Fix the spelling of 'PenpalSudoAccount'`

- `Fix the spelling of 'insufficient'`

- `Fix the spelling of 'PalletXcmExtrinsicsBenchmark'`

- `Fix the spelling of 'subtracted'`

- `Fix the spelling of 'CandidatePendingAvailability'`

- `Fix the spelling of 'exclusive'`

- `Fix the spelling of 'until'`

- `Fix the spelling of 'discriminator'`

- `Fix the spelling of 'nonexistent'`

- `Fix the spelling of 'subsystem'`

- `Fix the spelling of 'indices'`

- `Fix the spelling of 'committed'`

- `Fix the spelling of 'topology'`

- `Fix the spelling of 'response'`

- `Fix the spelling of 'beneficiary'`

- `Fix the spelling of 'formatted'`

- `Fix the spelling of 'UNKNOWN_PROOF_REQUEST'`

- `Fix the spelling of 'succeeded'`

- `Fix the spelling of 'reopened'`

- `Fix the spelling of 'proposer'`

- `Fix the spelling of 'InstantiationNonce'`

- `Fix the spelling of 'depositor'`

- `Fix the spelling of 'expiration'`

- `Fix the spelling of 'phantom'`

- `Fix the spelling of 'AggregatedKeyValue'`

- `Fix the spelling of 'randomness'`

- `Fix the spelling of 'defendant'`

- `Fix the spelling of 'AquaticMammal'`

- `Fix the spelling of 'transactions'`

- `Fix the spelling of 'PassingTracingSubscriber'`

- `Fix the spelling of 'TxSignaturePayload'`

- `Fix the spelling of 'versioning'`

- `Fix the spelling of 'descendant'`

- `Fix the spelling of 'overridden'`

- `Fix the spelling of 'network'`

Let me know if this structure is adequate.

**Note:** The usage of the words `Merkle`, `Merkelize`, `Merklization`,
`Merkelization`, `Merkleization`, is somewhat inconsistent but I left it
as it is.

~~**Note:** In some places the term `Receival` is used to refer to
message reception, IMO `Reception` is the correct word here, but I left
it as it is.~~

~~**Note:** In some places the term `Overlayed` is used instead of the
more acceptable version `Overlaid` but I also left it as it is.~~

~~**Note:** In some places the term `Applyable` is used instead of the
correct version `Applicable` but I also left it as it is.~~

**Note:** Some usage of British vs American english e.g. `judgement` vs
`judgment`, `initialise` vs `initialize`, `optimise` vs `optimize` etc.
are both present in different places, but I suppose that's
understandable given the number of contributors.

~~**Note:** There is a spelling mistake in `.github/CODEOWNERS` but it
triggers errors in CI when I make changes to it, so I left it as it
is.~~