Allow adding extra request-response protocols during polkadot service
initialization. This is required to add a request-response protocol
described in
[RFC-0008](https://polkadot-fellows.github.io/RFCs/approved/0008-parachain-bootnodes-dht.html)
to the relay chain side of the parachain node.

### Review notes
The PR might look scary due to a lot of code being moved. It is easier
to review it on a per-commit basis. The commits do not containing
changes to the code logic are named accordingly.

---------

Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>

# Description

Fixes #7413

## Integration

This PR updates the `DryRunApi`. The signature of the `dry_run_call` is
changed, and the XCM version of the return values of `dry_run_xcm` now
follows the version of the input XCM program.

## Review Notes

* **The `DryRunApi` is modified**
* **Added the `Router::clear_messages` to `dry_run_xcm` common
implementation**
* **Fixed the xcmp-queue's Router's clear_messages: channels details'
first_index and last_index are reset when clearing**
* **The MIN_XCM_VERSION is added**
* The common implementation in the `pallet-xcm` is modified accordingly
* The `DryRunApi` tests are modified to account for testing old XCM
versions
* The implementation from the `pallet-xcm` is used where it was not used
(including the `DryRunApi` tests)
* All the runtime implementations are modified according to the Runtime
API change

---------

Co-authored-by: Adrian Catangiu <adrian@parity.io>

Closes #3270

---------

Co-authored-by: command-bot <>

approval-voting-parallel has been running on all network types except
polkadot since release `1.17.0`, so it has around 2 months of baking and
running without any reported issues, so let's enable it on polkadot as
well.

After running in polkadot for awhile the flag will be entirely removed.

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>

Part of https://github.com/paritytech/polkadot-sdk/issues/5079.

Removes all usage of the static async backing params, replacing them
with dynamically computed equivalent values (based on the claim queue
and scheduling lookahead).

Adds a new runtime API for querying the scheduling lookahead value. If
not present, falls back to 3 (the default value that is backwards
compatible with values we have on production networks for
allowed_ancestry_len)

Also resolves most of
https://github.com/paritytech/polkadot-sdk/issues/4447, removing code
that handles async backing not yet being enabled.
While doing this, I removed the support for collation protocol version 1
on collators, as it only worked for leaves not supporting async backing
(which are none).
I also unhooked the legacy v1 statement-distribution (for the same
reason as above). That subsystem is basically dead code now, so I had to
remove some of its tests as they would no longer pass (since the
subsystem no longer sends messages to the legacy variant). I did not
remove the entire legacy subsystem yet, as that would pollute this PR
too much. We can remove the entire v1 and v2 validation protocols in a
follow up PR.

In another PR: remove test files with names `prospective_parachains`
(it'd pollute this PR if we do now)

TODO:
- [x] add deprecation warnings
- [x] prdoc

---------

Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Reference hardware requirements have been bumped to at least 8 cores so
we can no allocate 50% of that capacity to PVF execution.

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>

Implement cumulus StorageWeightReclaim as wrapping transaction extension + frame system ReclaimWeight (#6140)

(rebasing of https://github.com/paritytech/polkadot-sdk/pull/5234)

## Issues:

* Transaction extensions have weights and refund weight. So the
reclaiming of unused weight must happen last in the transaction
extension pipeline. Currently it is inside `CheckWeight`.
* cumulus storage weight reclaim transaction extension misses the proof
size of logic happening prior to itself.

## Done:

* a new storage `ExtrinsicWeightReclaimed` in frame-system. Any logic
which attempts to do some reclaim must use this storage to avoid double
reclaim.
* a new function `reclaim_weight` in frame-system pallet: info and post
info in arguments, read the already reclaimed weight, calculate the new
unused weight from info and post info. do the more accurate reclaim if
higher.
* `CheckWeight` is unchanged and still reclaim the weight in post
dispatch
* `ReclaimWeight` is a new transaction extension in frame system. For
s...

# Description
Issue #6476 
Collation-generation is not needed for validators node, and should be
removed.

## Implementation
Use a `DummySubsystem` for `collation_generation`

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: command-bot <>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>

# Description

This seems to be an old artifact of the long closed
https://github.com/paritytech/substrate/issues/6827 that I noticed when
working on related code earlier.

## Integration

`NetworkStarter` was removed, simply remove its usage:
```diff
-let (network, system_rpc_tx, tx_handler_controller, start_network, sync_service) =
+let (network, system_rpc_tx, tx_handler_controller, sync_service) =
    build_network(BuildNetworkParams {
...
-start_network.start_network();
```

## Review Notes

Changes are trivial, the only reason for this to not be accepted is if
it is desired to not start network automatically for whatever reason, in
which case the description of network starter needs to change.

# Checklist

* [x] My PR includes a detailed description as outlined in the
"Description" and its two subsections above.
* [ ] My PR follows the [labeling requirements](

https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process
) of this project (at minimum one label for `T` required)
* External contributors: ask maintainers to put the right label on your
PR.

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>

# Description

This is a continuation of
https://github.com/paritytech/polkadot-sdk/pull/5666 that finally fixes
https://github.com/paritytech/polkadot-sdk/issues/5333.

This should allow developers to create custom syncing strategies or even
the whole syncing engine if they so desire. It also moved syncing engine
creation and addition of corresponding protocol outside
`build_network_advanced` method, which is something Bastian expressed as
desired in
https://github.com/paritytech/polkadot-sdk/issues/5#issuecomment-1700816458

Here I replaced strategy-specific types and methods in `SyncingStrategy`
trait with generic ones. Specifically `SyncingAction` is now used by all
strategies instead of strategy-specific types with conversions.
`StrategyKey` was an enum with a fixed set of options and now replaced
with an opaque type that strategies create privately and send to upper
layers as an opaque type. Requests and responses are now handled in a
generic way regardless of the strategy, which reduced and simplified
strategy API.

`PolkadotSyncingStrategy` now lives in its dedicated module (had to edit
.gitignore for this) like other strategies.

`build_network_advanced` takes generic `SyncingService` as an argument
alongside with a few other low-level types (that can probably be
extracted in the future as well) without any notion of specifics of the
way syncing is actually done. All the protocol and tasks are created
outside and not a part of the network anymore. It still adds a bunch of
protocols like for light client and some others that should eventually
be restructured making `build_network_advanced` just building generic
network and not application-specific protocols handling.

## Integration

Just like https://github.com/paritytech/polkadot-sdk/pull/5666
introduced `build_polkadot_syncing_strategy`, this PR introduces
`build_default_block_downloader`, but for convenience and to avoid
typical boilerplate a simpler high-level function
`build_default_syncing_engine` is added that will take care of creating
typical block downloader, syncing strategy and syncing engine, which is
what most users will be using going forward. `build_network` towards the
end of the PR was renamed to `build_network_advanced` and
`build_network`'s API was reverted to
pre-https://github.com/paritytech/polkadot-sdk/pull/5666, so most users
will not see much of a difference during upgrade unless they opt-in to
use new API.

## Review Notes

For `StrategyKey` I was thinking about using something like private type
and then storing `TypeId` inside instead of a static string in it, let
me know if that would preferred.

The biggest change happened to requests that different strategies make
and how their responses are handled. The most annoying thing here is
that block response decoding, in contrast to all other responses, is
dependent on request. This meant request had to be sent throughout the
system. While originally `Response` was `Vec<u8>`, I didn't want to
re-encode/decode request and response just to fit into that API, so I
ended up with `Box<dyn Any + Send>`. This allows responses to be truly
generic and each strategy will know how to downcast it back to the
concrete type when handling the response.

Import queue refactoring was needed to move `SyncingEngine` construction
out of `build_network` that awkwardly implemented for `SyncingService`,
but due to `&mut self` wasn't usable on `Arc<SyncingService>` for no
good reason. `Arc<SyncingService>` itself is of course useless, but
refactoring to replace it with just `SyncingService` was unfortunately
rejected in https://github.com/paritytech/polkadot-sdk/pull/5454

As usual I recommend to review this PR as a series of commits instead of
as the final diff, it'll make more sense that way.

# Checklist

* [x] My PR includes a detailed description as outlined in the
"Description" and its two subsections above.
* [x] My PR follows the [labeling requirements](

https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process
) of this project (at minimum one label for `T` required)
* External contributors: ask maintainers to put the right label on your
PR.
* [x] I have made corresponding changes to the documentation (if
applicable)

The tests used the same paths. When run on CI, each test is run in its
own process and thus, this "serial_test" crate wasn't used. The tests
are now using their own thread local tempdir, which ensures that the
tests are working when running in parallel in the same program or when
being run individually.

# Benchmark Overhead Command for Parachains

This implements the `benchmark overhead` command for parachains. Full
context is available at:
https://github.com/paritytech/polkadot-sdk/issues/5303. Previous attempt
was this https://github.com/paritytech/polkadot-sdk/pull/5283, but here
we have integration into frame-omni-bencher and improved tooling.

## Changes Overview

Users are now able to use `frame-omni-bencher` to generate
`extrinsic_weight.rs` and `block_weight.rs` files for their runtime. The
core logic for generating these remains untouched; this PR provides
mostly machinery to make it work for parachains at all.

Similar to the pallet benchmarks, we gain the option to benchmark based
on just a runtime:

```
frame-omni-bencher v1 benchmark overhead --runtime {{runtime}}
```

or with a spec:

```
frame-omni-bencher v1 benchmark overhead --chain {{spec}} --genesis-builder spec
```

In this case, the genesis state is generated from the runtime presets.
However, it is also possible to use `--chain` and genesis builder `spec`
to generate the genesis state from the chain spec.

Additionally, we use metadata to perform some checks based on the
pallets the runtime exposes:

- If we see the `ParaInherent` pallet, we assume that we are dealing
with a relay chain. This means that we don't need proof recording during
import (since there is no storage weight).
- If we detect the `ParachainSystem` pallet, we assume that we are
dealing with a parachain and take corresponding actions like patching a
para id into the genesis state.

On the inherent side, I am currently supplying the standard inherents
every parachain needs.

In the current state, `frame-omni-bencher` supports all system chains.
In follow-up PRs, we could add additional inherents to increase
compatibility.

Since we are building a block during the benchmark, we also need to
build an extrinsic. By default, I am leveraging subxt to build the xt
dynamically. If a chain is not compatible with the `SubstrateConfig`
that comes with `subxt`, it can provide a custom extrinsic builder to
benchmarking-cli. This requires either a custom bencher implementation
or an integration into the parachains node.

Also cumulus-test-runtime has been migrated to provide genesis configs.

## Chain Compatibility
The current version here is compatible with the system chains and common
substrate chains. The way to go for others would be to customize the
frame-omni-bencher by providing a custom extrinsicbuilder. I did an
example implementation that works for mythical:
https://github.com/skunert/mythical-bencher

## Follow-Ups
- After #6040 is finished, we should integrate this here to make the
tooling truly useful. In the current form, the state is fairly small and
not representative.

## How to Review
I recommend starting from
[here](https://github.com/paritytech/polkadot-sdk/pull/5891/files#diff-50830ff756b3ac3403b7739d66c9e3a5185dbea550669ca71b28d19c7a2a54ecR264),
this method is the main entry point for omni-bencher and `polkadot`
binary.

TBD:
- [x] PRDoc

---------

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

on top of https://github.com/paritytech/polkadot-sdk/pull/5423

This PR implements the plumbing work required for
https://github.com/paritytech/polkadot-sdk/issues/5047 . I also added
additional helper methods gated by feature "test" in primitives.

TODO:
- [x] PRDoc

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: GitHub Action <action@github.com>

Closes #4896

The approval-voting-parallel introduced with
https://github.com/paritytech/polkadot-sdk/pull/4849 has been tested on
`versi` and approximately 3 weeks on parity's existing kusama nodes
https://github.com/paritytech/devops/issues/3583, things worked as
expected, so enable it by default on all kusama nodes in the next
release.

The next step will be enabling by default on polkadot if no issue
arrises while running on kusama.

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>

Original PR https://github.com/paritytech/polkadot-sdk/pull/2280
reverted in https://github.com/paritytech/polkadot-sdk/pull/3665

This PR reintroduces the reverted functionality with additional changes,
related effort
[here](https://github.com/paritytech/polkadot-sdk/pull/3623).
Description is copied over from the original PR

First part of [Extrinsic
Horizon](https://github.com/paritytech/polkadot-sdk/issues/2415)

Introduces a new trait `TransactionExtension` to replace
`SignedExtension`. Introduce the idea of transactions which obey the
runtime's extensions and have according Extension data (né Extra data)
yet do not have hard-coded signatures.

Deprecate the terminology of "Unsigned" when used for
transactions/extrinsics owing to there now being "proper" unsigned
transactions which obey the extension framework and "old-style" unsigned
which do not. Instead we have __*General*__ for the former and
__*Bare*__ for the latter. (Ultimately, the latter will be phased out as
a type of transaction, and Bare will only be used for Inherents.)

Types of extrinsic are now therefore:
- Bare (no hardcoded signature, no Extra data; used to be known as
"Unsigned")
- Bare transactions (deprecated): Gossiped, validated with
`ValidateUnsigned` (deprecated) and the `_bare_compat` bits of
`TransactionExtension` (deprecated).
  - Inherents: Not gossiped, validated with `ProvideInherent`.
- Extended (Extra data): Gossiped, validated via `TransactionExtension`.
  - Signed transactions (with a hardcoded signature) in extrinsic v4.
- General transactions (without a hardcoded signature) in extrinsic v5.

`TransactionExtension` differs from `SignedExtension` because:
- A signature on the underlying transaction may validly not be present.
- It may alter the origin during validation.
- `pre_dispatch` is renamed to `prepare` and need not contain the checks
present in `validate`.
- `validate` and `prepare` is passed an `Origin` rather than a
`AccountId`.
- `validate` may pass arbitrary information into `prepare` via a new
user-specifiable type `Val`.
- `AdditionalSigned`/`additional_signed` is renamed to
`Implicit`/`implicit`. It is encoded *for the entire transaction* and
passed in to each extension as a new argument to `validate`. This
facilitates the ability of extensions to acts as underlying crypto.

There is a new `DispatchTransaction` trait which contains only default
function impls and is impl'ed for any `TransactionExtension` impler. It
provides several utility functions which reduce some of the tedium from
using `TransactionExtension` (indeed, none of its regular functions
should now need to be called directly).

Three transaction version discriminator ("versions") are now permissible
(RFC [here](https://github.com/polkadot-fellows/RFCs/pull/84)) in
extrinsic version 5:
- 0b00000100 or 0b00000101: Bare (used to be called "Unsigned"):
contains Signature or Extra (extension data). After bare transactions
are no longer supported, this will strictly identify an Inherents only.
Available in both extrinsic versions 4 and 5.
- 0b10000100: Old-school "Signed" Transaction: contains Signature, Extra
(extension data) and an extension version byte, introduced as part of
[RFC99](https://github.com/polkadot-fellows/RFCs/blob/main/text/0099-transaction-extension-version.md).
Still available as part of extrinsic v4.
- 0b01000101: New-school "General" Transaction: contains Extra
(extension data) and an extension version byte, as per RFC99, but no
Signature. Only available in extrinsic v5.

For the New-school General Transaction, it becomes trivial for authors
to publish extensions to the mechanism for authorizing an Origin, e.g.
through new kinds of key-signing schemes, ZK proofs, pallet state,
mutations over pre-authenticated origins or any combination of the
above.

`UncheckedExtrinsic` still maintains encode/decode backwards
compatibility with extrinsic version 4, where the first byte was encoded
as:
- 0b00000100 - Unsigned transactions
- 0b10000100 - Old-school Signed transactions, without the extension
version byte

Now, `UncheckedExtrinsic` contains a `Preamble` and the actual call. The
`Preamble` describes the type of extrinsic as follows:
```rust
/// A "header" for extrinsics leading up to the call itself. Determines the type of extrinsic and
/// holds any necessary specialized data.
#[derive(Eq, PartialEq, Clone)]
pub enum Preamble<Address, Signature, Extension> {
	/// An extrinsic without a signature or any extension. This means it's either an inherent or
	/// an old-school "Unsigned" (we don't use that terminology any more since it's confusable with
	/// the general transaction which is without a signature but does have an extension).
	///
	/// NOTE: In the future, once we remove `ValidateUnsigned`, this will only serve Inherent
	/// extrinsics and thus can be renamed to `Inherent`.
	Bare(ExtrinsicVersion),
	/// An old-school transaction extrinsic which includes a signature of some hard-coded crypto.
	/// Available only on extrinsic version 4.
	Signed(Address, Signature, ExtensionVersion, Extension),
	/// A new-school transaction extrinsic which does not include a signature by default. The
	/// origin authorization, through signatures or other means, is performed by the transaction
	/// extension in this extrinsic. Available starting with extrinsic version 5.
	General(ExtensionVersion, Extension),
}
```

## Code Migration

### NOW: Getting it to build

Wrap your `SignedExtension`s in `AsTransactionExtension`. This should be
accompanied by renaming your aggregate type in line with the new
terminology. E.g. Before:

```rust
/// The SignedExtension to the basic transaction logic.
pub type SignedExtra = (
	/* snip */
	MySpecialSignedExtension,
);
/// Unchecked extrinsic type as expected by this runtime.
pub type UncheckedExtrinsic =
	generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, SignedExtra>;
```

After:

```rust
/// The extension to the basic transaction logic.
pub type TxExtension = (
	/* snip */
	AsTransactionExtension<MySpecialSignedExtension>,
);
/// Unchecked extrinsic type as expected by this runtime.
pub type UncheckedExtrinsic =
	generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, TxExtension>;
```

You'll also need to alter any transaction building logic to add a
`.into()` to make the conversion happen. E.g. Before:

```rust
fn construct_extrinsic(
		/* snip */
) -> UncheckedExtrinsic {
	let extra: SignedExtra = (
		/* snip */
		MySpecialSignedExtension::new(/* snip */),
	);
	let payload = SignedPayload::new(call.clone(), extra.clone()).unwrap();
	let signature = payload.using_encoded(|e| sender.sign(e));
	UncheckedExtrinsic::new_signed(
		/* snip */
		Signature::Sr25519(signature),
		extra,
	)
}
```

After:

```rust
fn construct_extrinsic(
		/* snip */
) -> UncheckedExtrinsic {
	let tx_ext: TxExtension = (
		/* snip */
		MySpecialSignedExtension::new(/* snip */).into(),
	);
	let payload = SignedPayload::new(call.clone(), tx_ext.clone()).unwrap();
	let signature = payload.using_encoded(|e| sender.sign(e));
	UncheckedExtrinsic::new_signed(
		/* snip */
		Signature::Sr25519(signature),
		tx_ext,
	)
}
```

### SOON: Migrating to `TransactionExtension`

Most `SignedExtension`s can be trivially converted to become a
`TransactionExtension`. There are a few things to know.

- Instead of a single trait like `SignedExtension`, you should now
implement two traits individually: `TransactionExtensionBase` and
`TransactionExtension`.
- Weights are now a thing and must be provided via the new function `fn
weight`.

#### `TransactionExtensionBase`

This trait takes care of anything which is not dependent on types
specific to your runtime, most notably `Call`.

- `AdditionalSigned`/`additional_signed` is renamed to
`Implicit`/`implicit`.
- Weight must be returned by implementing the `weight` function. If your
extension is associated with a pallet, you'll probably want to do this
via the pallet's existing benchmarking infrastructure.

#### `TransactionExtension`

Generally:
- `pre_dispatch` is now `prepare` and you *should not reexecute the
`validate` functionality in there*!
- You don't get an account ID any more; you get an origin instead. If
you need to presume an account ID, then you can use the trait function
`AsSystemOriginSigner::as_system_origin_signer`.
- You get an additional ticket, similar to `Pre`, called `Val`. This
defines data which is passed from `validate` into `prepare`. This is
important since you should not be duplicating logic from `validate` to
`prepare`, you need a way of passing your working from the former into
the latter. This is it.
- This trait takes a `Call` type parameter. `Call` is the runtime call
type which used to be an associated type; you can just move it to become
a type parameter for your trait impl.
- There's no `AccountId` associated type any more. Just remove it.

Regarding `validate`:
- You get three new parameters in `validate`; all can be ignored when
migrating from `SignedExtension`.
- `validate` returns a tuple on success; the second item in the tuple is
the new ticket type `Self::Val` which gets passed in to `prepare`. If
you use any information extracted during `validate` (off-chain and
on-chain, non-mutating) in `prepare` (on-chain, mutating) then you can
pass it through with this. For the tuple's last item, just return the
`origin` argument.

Regarding `prepare`:
- This is renamed from `pre_dispatch`, but there is one change:
- FUNCTIONALITY TO VALIDATE THE TRANSACTION NEED NOT BE DUPLICATED FROM
`validate`!!
- (This is different to `SignedExtension` which was required to run the
same checks in `pre_dispatch` as in `validate`.)

Regarding `post_dispatch`:
- Since there are no unsigned transactions handled by
`TransactionExtension`, `Pre` is always defined, so the first parameter
is `Self::Pre` rather than `Option<Self::Pre>`.

If you make use of `SignedExtension::validate_unsigned` or
`SignedExtension::pre_dispatch_unsigned`, then:
- Just use the regular versions of these functions instead.
- Have your logic execute in the case that the `origin` is `None`.
- Ensure your transaction creation logic creates a General Transaction
rather than a Bare Transaction; this means having to include all
`TransactionExtension`s' data.
- `ValidateUnsigned` can still be used (for now) if you need to be able
to construct transactions which contain none of the extension data,
however these will be phased out in stage 2 of the Transactions Horizon,
so you should consider moving to an extension-centric design.

---------

Signed-off-by: georgepisaltu <george.pisaltu@parity.io>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>

Closes #5705

---------

Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

### Fork-Aware Transaction Pool Implementation

This PR introduces a fork-aware transaction pool (fatxpool) enhancing
transaction management by maintaining the valid state of txpool for
different forks.

### High-level overview
The high level overview was added to
[`sc_transaction_pool::fork_aware_txpool`](https://github.com/paritytech/polkadot-sdk/blob/3ad0a1b7/substrate/client/transaction-pool/src/fork_aware_txpool/mod.rs#L21)
module. Use:
```
cargo  doc --document-private-items -p sc-transaction-pool --open
```
to build the doc. It should give a good overview and nice entry point
into the new pool's mechanics.

<details>
  <summary>Quick overview (documentation excerpt)</summary>

#### View
For every fork, a view is created. The view is a persisted state of the
transaction pool computed and updated at the tip of the fork. The view
is built around the existing `ValidatedPool` structure.

A view is created on every new best block notification. To create a
view, one of the existing views is chosen and cloned.

When the chain progresses, the view is kept in the cache
(`retracted_views`) to allow building blocks upon intermediary blocks in
the fork.

The views are deleted on finalization: views lower than the finalized
block are removed.

The views are updated with the transactions from the mempool—all
transactions are sent to the newly created views.
A maintain process is also executed for the newly created
views—basically resubmitting and pruning transactions from the
appropriate tree route.

##### View store
View store is the helper structure that acts as a container for all the
views. It provides some convenient methods.

##### Submitting transactions
Every transaction is submitted to every view at the tips of the forks.
Retracted views are not updated.
Every transaction also goes into the mempool.

##### Internal mempool
Shortly, the main purpose of an internal mempool is to prevent a
transaction from being lost. That could happen when a transaction is
invalid on one fork and could be valid on another. It also allows the
txpool to accept transactions when no blocks have been reported yet.

The mempool removes its transactions when they get finalized.
Transactions are also periodically verified on every finalized event and
removed from the mempool if no longer valid.

#### Events
Transaction events from multiple views are merged and filtered to avoid
duplicated events.
`Ready` / `Future` / `Inblock` events are originated in the Views and
are de-duplicated and forwarded to external listeners.
`Finalized` events are originated in fork-aware-txpool logic.
`Invalid` events requires special care and can be originated in both
view and fork-aware-txpool logic.

#### Light maintain
Sometime transaction pool does not have enough time to prepare fully
maintained view with all retracted transactions being revalidated. To
avoid providing empty ready transaction set to block builder (what would
result in empty block) the light maintain was implemented. It simply
removes the imported transactions from ready iterator.

#### Revalidation
Revalidation is performed for every view. The revalidation process is
started after a trigger is executed. The revalidation work is terminated
just after a new best block / finalized event is notified to the
transaction pool.
The revalidation result is applied to the newly created view which is
built upon the revalidated view.

Additionally, parts of the mempool are also revalidated to make sure
that no transactions are stuck in the mempool.


#### Logs
The most important log allowing to understand the state of the txpool
is:
```
              maintain: txs:(0, 92) views:[2;[(327, 76, 0), (326, 68, 0)]] event:Finalized { hash: 0x8...f, tree_route: [] }  took:3.463522ms
                             ^   ^         ^     ^   ^  ^      ^   ^  ^        ^                                                   ^
unwatched txs in mempool ────┘   │         │     │   │  │      │   │  │        │                                                   │
   watched txs in mempool ───────┘         │     │   │  │      │   │  │        │                                                   │
                     views  ───────────────┘     │   │  │      │   │  │        │                                                   │
                      1st view block # ──────────┘   │  │      │   │  │        │                                                   │
                           number of ready tx ───────┘  │      │   │  │        │                                                   │
                                numer of future tx ─────┘      │   │  │        │                                                   │
                                        2nd view block # ──────┘   │  │        │                                                   │
                                      number of ready tx ──────────┘  │        │                                                   │
                                           number of future tx ───────┘        │                                                   │
                                                                 event ────────┘                                                   │
                                                                       duration  ──────────────────────────────────────────────────┘
```
It is logged after the maintenance is done.

The `debug` level enables per-transaction logging, allowing to keep
track of all transaction-related actions that happened in txpool.
</details>


### Integration notes

For teams having a custom node, the new txpool needs to be instantiated,
typically in `service.rs` file, here is an example:

https://github.com/paritytech/polkadot-sdk/blob/9c547ff3

/cumulus/polkadot-omni-node/lib/src/common/spec.rs#L152-L161

To enable new transaction pool the following cli arg shall be specified:
`--pool-type=fork-aware`. If it works, there shall be information
printed in the log:
```
2024-09-20 21:28:17.528  INFO main txpool: [Parachain]  creating ForkAware txpool.
````

For debugging the following debugs shall be enabled:
```
      "-lbasic-authorship=debug",
      "-ltxpool=debug",
```
*note:* trace for txpool enables per-transaction logging.

### Future work
The current implementation seems to be stable, however further
improvements are required.
Here is the umbrella issue for future work:
- https://github.com/paritytech/polkadot-sdk/issues/5472


Partially fixes: #1202

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>

Jaeger tracing went mostly unused and it created bigger problems like
wasting CPU or memory leaks, so remove it entirely.

Fixes: https://github.com/paritytech/polkadot-sdk/issues/4995

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>

cc https://github.com/paritytech/ci_cd/issues/1035

cc https://github.com/paritytech/ci_cd/issues/1023

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>
Co-authored-by: Maksym H <1177472+mordamax@users.noreply.github.com>
Co-authored-by: gui <gui.thiolliere@gmail.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: ggwpez <ggwpez@users.noreply.github.com>

This is the implementation of the approach described here:
https://github.com/paritytech/polkadot-sdk/issues/1617#issuecomment-2150321612
&
https://github.com/paritytech/polkadot-sdk/issues/1617#issuecomment-2154357547
&
https://github.com/paritytech/polkadot-sdk/issues/1617#issuecomment-2154721395.

## Description of changes

The end goal is to have an architecture where we have single
subsystem(`approval-voting-parallel`) and multiple worker types that
would full-fill the work that currently is fulfilled by the
`approval-distribution` and `approval-voting` subsystems. The main loop
of the new subsystem would do just the distribution of work to the
workers.

The new subsystem will have:
- N approval-distribution workers: This would do the work that is
currently being done by the approval-distribution subsystem and in
addition to that will also perform the crypto-checks that an assignment
is valid and that a vote is correctly signed. Work is assigned via the
following formula: `worker_index = msg.validator % WORKER_COUNT`, this
guarantees that all assignments and approvals from the same validator
reach the same worker.
- 1 approval-voting worker: This would receive an already valid message
and do everything the approval-voting currently does, except the
crypto-checking that has been moved already to the approval-distribution
worker.

On the hot path of processing messages **no** synchronisation and
waiting is needed between approval-distribution and approval-voting
workers.

<img width="1431" alt="Screenshot 2024-06-07 at 11 28 08"
src="https://github.com/paritytech/polkadot-sdk/assets/49718502/a196199b-b705-4140-87d4-c6900ba8595e">



## Guidelines for reading

The full implementation is broken in 5 PRs and all of them are
self-contained and improve things incrementally even without the
parallelisation being implemented/enabled, the reason this approach was
taken instead of a big-bang PR, is to make things easier to review and
reduced the risk of breaking this critical subsystems.

After reading the full description of this PR, the changes should be
read in the following order:
1. https://github.com/paritytech/polkadot-sdk/pull/4848, some other
micro-optimizations for networks with a high number of validators. This
change gives us a speed up by itself without any other changes.
2. https://github.com/paritytech/polkadot-sdk/pull/4845 , this contains
only interface changes to decouple the subsystem from the `Context` and
be able to run multiple instances of the subsystem on different threads.
**No functional changes**
3. https://github.com/paritytech/polkadot-sdk/pull/4928, moving of the
crypto checks from approval-voting in approval-distribution, so that the
approval-distribution has no reason to wait after approval-voting
anymore. This change gives us a speed up by itself without any other
changes.
4. https://github.com/paritytech/polkadot-sdk/pull/4846, interface
changes to make approval-voting runnable on a separate thread. **No
functional changes**
5. This PR, where we instantiate an `approval-voting-parallel` subsystem
that runs on different workers the logic currently in
`approval-distribution` and `approval-voting`.
6. The next step after this changes get merged and deploy would be to
bring all the files from approval-distribution, approval-voting,
approval-voting-parallel into a single rust crate, to make it easier to
maintain and understand the structure.

## Results
Running subsystem-benchmarks with 1000 validators 100 fully ocuppied
cores and triggering all assignments and approvals for all tranches

#### Approval does not lags behind. 
 Master
```
Chain selection approved  after 72500 ms hash=0x0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
```
With this PoC
```
Chain selection approved  after 3500 ms hash=0x0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
```

#### Gathering enough assignments
 
Enough assignments are gathered in less than 500ms, so that gives un a
guarantee that un-necessary work does not get triggered, on master on
the same benchmark because the subsystems fall behind on work, that
number goes above 32 seconds on master.
 
<img width="2240" alt="Screenshot 2024-06-20 at 15 48 22"
src="https://github.com/paritytech/polkadot-sdk/assets/49718502/d2f2b29c-5ff6-44b4-a245-5b37ab8e58bc">


#### Cpu usage:
Master
```
CPU usage, seconds                     total   per block
approval-distribution                96.9436      9.6944
approval-voting                     117.4676     11.7468
test-environment                     44.0092      4.4009
```
With this PoC
```
CPU usage, seconds                     total   per block
approval-distribution                 0.0014      0.0001 --- unused
approval-voting                       0.0437      0.0044.  --- unused
approval-voting-parallel              5.9560      0.5956
approval-voting-parallel-0           22.9073      2.2907
approval-voting-parallel-1           23.0417      2.3042
approval-voting-parallel-2           22.0445      2.2045
approval-voting-parallel-3           22.7234      2.2723
approval-voting-parallel-4           21.9788      2.1979
approval-voting-parallel-5           23.0601      2.3060
approval-voting-parallel-6           22.4805      2.2481
approval-voting-parallel-7           21.8330      2.1833
approval-voting-parallel-db          37.1954      3.7195.  --- the approval-voting thread.
```

# Enablement strategy

Because just some trivial plumbing is needed in approval-distribution
and approval-voting to be able to run things in parallel and because
this subsystems plays a critical part in the system this PR proposes
that we keep both ways of running the approval work, as separated
subsystems and just a single subsystem(`approval-voting-parallel`) which
has multiple workers for the distribution work and one worker for the
approval-voting work and switch between them with a comandline flag.

The benefits for this is twofold.
1. With the same polkadot binary we can easily switch just a few
validators to use the parallel approach and gradually make this the
default way of running, if now issues arise.
2. In the worst case scenario were it becomes the default way of running
things, but we discover there are critical issues with it we have the
path to quickly disable it by asking validators to adjust their command
line flags.


# Next steps
- [x] Make sure through various testing we are not missing anything 
- [x] Polish the implementations to make them production ready
- [x] Add Unittest Tests for approval-voting-parallel.
- [x] Define and implement the strategy for rolling this change, so that
the blast radius is minimal(single validator) in case there are problems
with the implementation.
- [x]  Versi long running tests.
- [x] Add relevant metrics.

@ordian @eskimor @sandreim @AndreiEres

, let me know what you think.

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>

It is a first step for switching to the `frame-omni-bencher` for CI.

This PR includes several changes related to generating chain specs plus:

- [x] pallet `assigned_slots` fix missing `#[serde(skip)]` for phantom
- [x] pallet `paras_inherent` benchmark fix - cherry-picked from
https://github.com/paritytech/polkadot-sdk/pull/5688
- [x] migrates `get_preset` to the relevant runtimes
- [x] fixes Rococo genesis presets - does not work
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/7317249
- [x] fixes Rococo benchmarks for CI 
- [x] migrate westend genesis
- [x] remove wococo stuff

Closes: https://github.com/paritytech/polkadot-sdk/issues/5680

## Follow-ups
- Fix for frame-omni-bencher
https://github.com/paritytech/polkadot-sdk/pull/5655
- Enable new short-benchmarking CI -
https://github.com/paritytech/polkadot-sdk/pull/5706
- Remove gitlab pipelines for short benchmarking
- refactor all Cumulus runtimes to use `get_preset` -
https://github.com/paritytech/polkadot-sdk/issues/5704
- https://github.com/paritytech/polkadot-sdk/issues/5705
- https://github.com/paritytech/polkadot-sdk/issues/5700
- [ ] Backport to the stable

---------

Co-authored-by: command-bot <>
Co-authored-by: ordian <noreply@reusable.software>

# Description

Follow-up to https://github.com/paritytech/polkadot-sdk/pull/5469 and
mostly covering https://github.com/paritytech/polkadot-sdk/issues/5333.

The primary change here is that syncing strategy is no longer created
inside of syncing engine, instead syncing strategy is an argument of
syncing engine, more specifically it is an argument to `build_network`
that most downstream users will use. This also extracts addition of
request-response protocols outside of network construction, making sure
they are physically not present when they don't need to be (imagine
syncing strategy that uses none of Substrate's protocols in its
implementation for example).

This technically allows to completely replace syncing strategy with
whatever strategy chain might need.

There will be at least one follow-up PR that will simplify
`SyncingStrategy` trait and other public interfaces to remove mentions
of block/state/warp sync requests, replacing them with generic APIs,
such that strategies where warp sync is not applicable don't have to
provide dummy method implementations, etc.

## Integration

Downstream projects will have to write a bit of boilerplate calling
`build_polkadot_syncing_strategy` function to create previously default
syncing strategy.

## Review Notes

Please review PR through individual commits rather than the final diff,
it will be easier that way. The changes are mostly just moving code
around one step at a time.

# Checklist

* [x] My PR includes a detailed description as outlined in the
"Description" and its two subsections above.
* [x] My PR follows the [labeling requirements](

https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process
) of this project (at minimum one label for `T` required)
* External contributors: ask maintainers to put the right label on your
PR.
* [x] I have made corresponding changes to the documentation (if
applicable)

This is part of the work to further optimize the approval subsystems, if
you want to understand the full context start with reading
https://github.com/paritytech/polkadot-sdk/pull/4849#issue-2364261568,

# Description
This PR contain changes to make possible the run of single
approval-voting instance on a worker thread, so that it can be
instantiated by the approval-voting-parallel subsystem.

This does not contain any functional changes it just decouples the
subsystem from the subsystem Context and introduces more specific trait
dependencies for each function instead of all of them requiring a
context.

This change can be merged  independent of the followup PRs.

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

Fixes: https://github.com/paritytech/polkadot-sdk/issues/5122.

This PR extends the existing single core `benchmark_cpu` to also build a
score of the entire processor by spawning `EXPECTED_NUM_CORES(8)`
threads and averaging their throughput.

This is better than simply checking the number of cores, because also
covers multi-tenant environments where the OS sees a high number of
available CPUs, but because it has to share it with the rest of his
neighbours its total throughput does not satisfy the minimum
requirements.


## TODO
- [x] Obtain reference values on the reference hardware.

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>

closes https://github.com/paritytech/polkadot-sdk/issues/5044

This PR switches the runtime to the new receipts format (vstaging
primitives). I've implemented `From` to convert from new primitives to
`v7` primitives and used them in the node runtime api client
implementation. Until we implement the support in the node, it will
continue e to use the v7 primitives but the runtime apis already use the
new primitives.


An expected downside of RFC103 is decoding V2 receipts shows garbage
values if the input is V1:

_![ima_9ce77de](https://github.com/user-attachments/assets/71d80e78-e238-4518-8cd1-548ae0d74b70)_

TODO:
- [x] fix tests
- [x] A few more tests for the new primitives
- [x] PRDoc

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

This improves `sc-service` API by not requiring the whole
`&Configuration`, using specific configuration options instead.
`RpcConfiguration` was also extracted from `Configuration` to group all
RPC options together.

We don't use Substrate's CLI and would rather not use `Configuration`
either, but some key public functions require it even though they
ignored most of the fields anyway.

`RpcConfiguration` is very helpful not just for consolidation of the
fields, but also to finally make RPC optional for our use case, while
Substrate still runs RPC server on localhost even if listening address
is explicitly set to `None`, which is annoying (and I suspect there is a
reason for it, so didn't want to change the default just yet).

While this is a breaking change, most developers will not notice it if
they use higher-level APIs.

Fixes https://github.com/paritytech/polkadot-sdk/issues/2897

---------

Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>

# Prerequisite 
This is part of the work to further optimize the approval subsystems, if
you want to understand the full context start with reading
https://github.com/paritytech/polkadot-sdk/pull/4849#issue-2364261568,

# Description
This PR contain changes, so that the crypto checks are performed by the
approval-distribution subsystem instead of the approval-voting one. The
benefit for these, is twofold:
1. Approval-distribution won't have to wait every single time for the
approval-voting to finish its job, so the work gets to be pipelined
between approval-distribution and approval-voting.

2. By running in parallel multiple instances of approval-distribution as
described here
https://github.com/paritytech/polkadot-sdk/pull/4849#issue-2364261568,
this significant body of work gets to run in parallel.

## Changes:
1. When approval-voting send `ApprovalDistributionMessage::NewBlocks` it
needs to pass the core_index and candidate_hash of the candidates.
2. ApprovalDistribution needs to use `RuntimeInfo` to be able to fetch
the SessionInfo from the runtime.
3. Move `approval-voting` logic that checks VRF assignment into
`approval-distribution`
4. Move `approval-voting` logic that checks vote is correctly signed
into `approval-distribution`
5. Plumb `approval-distribution` and `approval-voting` tests to support
the new logic.

## Benefits
Even without parallelisation the gains are significant, for example on
my machine if we run approval subsystem bench for 500 validators and 100
cores and trigger all 89 tranches of assignments and approvals, the
system won't fall behind anymore because of late processing of messages.
```
Before change
Chain selection approved  after 11500 ms hash=0x0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a

After change

Chain selection approved  after 5500 ms hash=0x0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
```

## TODO:
- [x] Run on versi.
- [x] Update parachain host documentation.

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>

As Runtime release 1.3.0 includes all of the remaining staging
primitives and APIs we can now release primitives version 8.
No other changes other than renaming/moving done here.

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

rpc server: listen to `ipv6 socket` if available and `--experimental-rpc-endpoint` CLI option (#4792)

Close https://github.com/paritytech/polkadot-sdk/issues/3488,
https://github.com/paritytech/polkadot-sdk/issues/4331

This changes/adds the following:

1. The default setting is that substrate starts a rpc server that
listens to localhost both Ipv4 and Ipv6 on the same port. Ipv6 is
allowed to fail because some platforms may not support it
2. A new RPC CLI option `--experimental-rpc-endpoint` which allow to
configure arbitrary listen addresses including the port, if this is
enabled no other interfaces are enabled.
3. If the local addr is not found for any of the sockets the server is
not started throws an error.
4. Remove the deny_unsafe from the RPC implementations instead this is
an extension to allow different polices for different interfaces/sockets
such one may enable unsafe on local interface and safe on only the
external interface.

So for instance in this PR it's now possible to start up three RPC...

I'm not sure if this is exactly what
https://github.com/paritytech/polkadot-sdk/issues/3537 meant, but I
think it should be fine to wait for relay chain before initializing
parachain node fully, which removed the need for background task and
extra hacks throughout the stack just to know where warp sync should
start.

Previously there were both `WarpSyncParams` and `WarpSyncConfig`, but
there was no longer any point in having two data structures, so I
simplified it to just `WarpSyncConfig`.

Fixes https://github.com/paritytech/polkadot-sdk/issues/3537

### Context

Since Rococo is now deprecated, we need another testnet to detect
bleeding-edge changes to Substrate, Polkadot, & BEEFY consensus
protocols that could brick the bridge.

It's the mirror PR of https://github.com/Snowfork/polkadot-sdk/pull/157
which has reviewed by Snowbridge team internally.

Synced with @acatangiu

 about that in channel
https://matrix.to/#/!gxqZwOyvhLstCgPJHO:matrix.parity.io/$N0CvTfDSl3cOQLEJeZBh-wlKJUXx7EDHAuNN5HuYHY4?via=matrix.parity.io&via=parity.io&via=matrix.org

---------

Co-authored-by: Clara van Staden <claravanstaden64@gmail.com>

This PR replaces env_logger with sp_tracing because of an issue with
env_logger and gum #4660

---------

Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Andrei Eres <eresav@me.com>

This PR extends the metrics exposed by the peerstore with the total
number of banned peers.

The new metric is exposed under
`substrate_sub_libp2p_peerset_num_banned_peers`.

To easily extend metrics in the future, the `fn num_known_peers` is
removed in favor of `fn status`.

While at it, enable the metrics for litep2p:
- total number of peers from peerstore (needed to debug memory
consumption)
- total number of banned peers from peerstore (needed to debug
reputation bans and disconnects)

Have added a couple of tests to validate that the number of banned peers
is exposed properly.

Part of: https://github.com/paritytech/polkadot-sdk/issues/4681


### Testing Done
Using [subp2p-explorer](https://github.com/lexnv/subp2p-explorer) have
submitted random data on tx protocol.
The peer gets banned, the num of banned peers is incremented then the
peer is disconnected.

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>

Related to https://github.com/paritytech/polkadot-sdk/issues/4523

Add runtime API methods for:
- generating the ancestry proof
- submiting a fork voting report
- submitting a future voting report

Closes https://github.com/paritytech/polkadot-sdk/issues/4324
- On every active leaf candidate-validation subsystem checks if the node
is the next session authority.
- If it is, it fetches backed candidates and prepares unknown PVFs.
- We limit number of PVFs per block to not overload subsystem.

Related to https://github.com/paritytech/polkadot-sdk/issues/4523

Extracting part of https://github.com/paritytech/polkadot-sdk/pull/1903
(credits to @Lederstrumpf

 for the high-level strategy), but also
introducing significant adjustments both to the approach and to the
code. The main adjustment is the fact that the `ForkVotingProof` accepts
only one vote, compared to the original version which accepted a
`vec![]`. With this approach more calls are needed in order to report
multiple equivocated votes on the same commit, but it simplifies a lot
the checking logic. We can add support for reporting multiple signatures
at once in the future.

There are 2 things that are missing in order to consider this issue
done, but I would propose to do them in a separate PR since this one is
already pretty big:
- benchmarks/computing a weight for the new extrinsic (this wasn't
present in https://github.com/paritytech/polkadot-sdk/pull/1903 either)
- exposing an API for generating the ancestry proof. I'm not sure if we
should do this in the Mmr pallet or in the Beefy pallet

Co-authored-by: Robert Hambrock <roberthambrock@gmail.com>

---------

Co-authored-by: Adrian Catangiu <adrian@parity.io>

Closes: https://github.com/paritytech/polkadot-sdk/issues/4298

This PR also merges `xcm-fee-payment-runtime-api` module to the
`xcm-runtime-api`.


## TODO

- [x] rename `convert` to `convert_location` and add new one
`convert_account` (opposite direction)
- [x] add to the all testnet runtimes
- [x] check polkadot-js if supports that automatically or if needs to be
added manually https://github.com/polkadot-js/api/pull/5917
- [ ] backport/patch for fellows and release (asap)

## Open questions
- [x] should we merge `xcm-runtime-api` and
`xcm-fee-payment-runtime-api` to the one module `xcm-runtime-api` ?

## Usage
Input:
 - `location:  VersionedLocation`
 
Output:
 - account_id bytes


![image](https://github.com/paritytech/polkadot-sdk/assets/8159517/4607b15a-77d2-462b-806c-606107776c0d)

---------

Co-authored-by: Bastian Köcher <git@kchr.de>

Inherited workspace dependencies cannot be renamed by the crate using
them (see [1](https://github.com/rust-lang/cargo/issues/12546),
[2](https://stackoverflow.com/questions/76792343/can-inherited-dependencies-in-rust-be-aliased-in-the-cargo-toml-file)).
Since we want to use inherited workspace dependencies everywhere, we
first need to unify all aliases that we use for a dependency throughout
the workspace.
The umbrella crate is currently excluded from this procedure, since it
should be able to export the crates by their original name without much
hassle.

For example: one crate may alias `parity-scale-codec` to `codec`, while
another crate does not alias it at all. After this change, all crates
have to use `codec` as name. The problematic combinations were:
- conflicting aliases: most crates aliases as `A` but some use `B`.
- missing alias: most of the crates alias a dep but some dont.
- superfluous alias: most crates dont alias a dep but some do.

The script that i used first determines whether most crates opted to
alias a dependency or not. From that info it decides whether to use an
alias or not. If it decided to use an alias, the most common one is used
everywhere.

To reproduce, i used
[this](https://github.com/ggwpez/substrate-scripts/blob/master/uniform-crate-alias.py)
python script in combination with
[this](https://github.com/ggwpez/zepter/blob/38ad10585fe98a5a86c1d2369738bc763a77057b/renames.json)
error output from Zepter.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>

Revived version of https://github.com/paritytech/substrate/pull/13311 .
Except Signature is not generic and is dictated by AuthorityId.

---------

Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Robert Hambrock <roberthambrock@gmail.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>