Relates to: https://github.com/paritytech/polkadot-sdk/pull/4826
Relates to: https://github.com/paritytech/polkadot-sdk/issues/3214

## Description

`pallet-xcm` stores some operational data that uses `Versioned*` XCM
types. When we add a new XCM version (XV), we deprecate XV-2 and remove
XV-3. Without proper migration, this can lead to issues with
[undecodable
storage](https://github.com/paritytech/polkadot-sdk/actions/runs/11381324568/job/31662577532?pr=6092),
as was identified on the XCMv5 branch where XCMv2 was removed.

This PR extends the existing `MigrateToLatestXcmVersion` to include
migration for the `Queries`, `LockedFungibles`, and
`RemoteLockedFungibles` storage types. Additionally, more checks were
added to `try_state` for these types.

## TODO
- [x] create tracking issue for `polkadot-fellows`
https://github.com/polkadot-fellows/runtimes/issues/492
- [x] Add missing `MigrateToLatestXcmVersion` for westend
- [x] fix pallet-xcm `Queries`
- fails for Westen...

Closes #4896

This PR adds `build_struct_json_patch` which helps to generate a JSON
used for preset.

Here is doc and example:

https://github.com/paritytech/polkadot-sdk/blob/d868b858/substrate/frame/support/src/generate_genesis_config.rs#L168-L266

And real-world usage:

https://github.com/paritytech/polkadot-sdk/blob/d868b858

/cumulus/parachains/runtimes/assets/asset-hub-rococo/src/genesis_config_presets.rs#L37-L61

Closes #5700

---------

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

The approval-voting-parallel introduced with
https://github.com/paritytech/polkadot-sdk/pull/4849 has been tested on
`versi` and approximately 3 weeks on parity's existing kusama nodes
https://github.com/paritytech/devops/issues/3583, things worked as
expected, so enable it by default on all kusama nodes in the next
release.

The next step will be enabling by default on polkadot if no issue
arrises while running on kusama.

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>

Added missing API methods to Rococo and Westend parachains.
Preparatory work for making chopstick tests run smoothly.
Follow-up of
[PR#6039](https://github.com/paritytech/polkadot-sdk/pull/6039)

I noticed that hardware benchmarks are being run even though we pass the
--no-hardware-benchmarks cli flag. After some debugging, the cause is an
incorrect usage of the `then_some` method.

From [std
docs](https://doc.rust-lang.org/std/primitive.bool.html#method.then_some):

> Arguments passed to then_some are eagerly evaluated; if you are
passing the result of a function call, it is recommended to use
[then](https://doc.rust-lang.org/std/primitive.bool.html#method.then),
which is lazily evaluated.

```rust
let mut a = 0;
let mut function_with_side_effects = || { a += 1; };

true.then_some(function_with_side_effects());
false.then_some(function_with_side_effects());

// `a` is incremented twice because the value passed to `then_some` is
// evaluated eagerly.
assert_eq!(a, 2);
```

This PR fixes all the similar usages of the `then_some` method across
the codebase.

polkadot address: 138eUqXvUYT3o4GdbnWQfGRzM8yDWh5Q2eFrFULL7RAXzdWD

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Related to https://github.com/paritytech/polkadot-sdk/issues/6161

This seems to fix the `JavaScript heap out of memory` error encountered
in the bridge zombienet tests lately.

This is just a partial fix, since we also need to address
https://github.com/paritytech/polkadot-sdk/issues/6133 in order to fully
fix the bridge zombienet tests

This PR backports regular version bumps and prdocs reordering from the
current stable release back to master

Resolves https://github.com/paritytech/polkadot-sdk/issues/4776

This will enable proper core-sharing between paras, even if one of them
is not producing blocks.

TODO:
- [x] duplicate first entry in the claim queue if the queue used to be
empty
- [x] don't back anything if at the end of the block there'll be a
session change
- [x] write migration for removing the availability core storage
- [x] update and write unit tests
- [x] prdoc
- [x] add zombienet test for synchronous backing
- [x] add zombienet test for core-sharing paras where one of them is not
producing any blocks

_Important note:_
The `ttl` and `max_availability_timeouts` fields of the
HostConfiguration are not removed in this PR, due to #64.
Adding the workaround with the storage version check for every use of
the active HostConfiguration in all runtime APIs would be insane, as
it's used in almost all runtime APIs.

So even though the ttl and max_availability_timeouts fields will now be
unused, they will remain part of the host configuration.

These will be removed in a separate PR once #64 is fixed. Tracked by
https://github.com/paritytech/polkadot-sdk/issues/6067

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
Co-authored-by: command-bot <>

Changed returned type of API methods from `Result<bool,
xcm_runtime_apis::trusted_query::Error>` to a typed one `type
XcmTrustedQueryResult = Result<bool,
xcm_runtime_apis::trusted_query::Error>;`
Follow-up of
[PR-6039](https://github.com/paritytech/polkadot-sdk/pull/6039)

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Adrian Catangiu <adrian@parity.io>

Those files were introduced by mistake in
https://github.com/paritytech/polkadot-sdk/pull/6039

Original PR https://github.com/paritytech/polkadot-sdk/pull/2280
reverted in https://github.com/paritytech/polkadot-sdk/pull/3665

This PR reintroduces the reverted functionality with additional changes,
related effort
[here](https://github.com/paritytech/polkadot-sdk/pull/3623).
Description is copied over from the original PR

First part of [Extrinsic
Horizon](https://github.com/paritytech/polkadot-sdk/issues/2415)

Introduces a new trait `TransactionExtension` to replace
`SignedExtension`. Introduce the idea of transactions which obey the
runtime's extensions and have according Extension data (né Extra data)
yet do not have hard-coded signatures.

Deprecate the terminology of "Unsigned" when used for
transactions/extrinsics owing to there now being "proper" unsigned
transactions which obey the extension framework and "old-style" unsigned
which do not. Instead we have __*General*__ for the former and
__*Bare*__ for the latter. (Ultimately, the latter will be phased out as
a type of transaction, and Bare will only be used for Inherents.)

Types of extrinsic are now therefore:
- Bare (no hardcoded signature, no Extra data; used to be known as
"Unsigned")
- Bare transactions (deprecated): Gossiped, validated with
`ValidateUnsigned` (deprecated) and the `_bare_compat` bits of
`TransactionExtension` (deprecated).
  - Inherents: Not gossiped, validated with `ProvideInherent`.
- Extended (Extra data): Gossiped, validated via `TransactionExtension`.
  - Signed transactions (with a hardcoded signature) in extrinsic v4.
- General transactions (without a hardcoded signature) in extrinsic v5.

`TransactionExtension` differs from `SignedExtension` because:
- A signature on the underlying transaction may validly not be present.
- It may alter the origin during validation.
- `pre_dispatch` is renamed to `prepare` and need not contain the checks
present in `validate`.
- `validate` and `prepare` is passed an `Origin` rather than a
`AccountId`.
- `validate` may pass arbitrary information into `prepare` via a new
user-specifiable type `Val`.
- `AdditionalSigned`/`additional_signed` is renamed to
`Implicit`/`implicit`. It is encoded *for the entire transaction* and
passed in to each extension as a new argument to `validate`. This
facilitates the ability of extensions to acts as underlying crypto.

There is a new `DispatchTransaction` trait which contains only default
function impls and is impl'ed for any `TransactionExtension` impler. It
provides several utility functions which reduce some of the tedium from
using `TransactionExtension` (indeed, none of its regular functions
should now need to be called directly).

Three transaction version discriminator ("versions") are now permissible
(RFC [here](https://github.com/polkadot-fellows/RFCs/pull/84)) in
extrinsic version 5:
- 0b00000100 or 0b00000101: Bare (used to be called "Unsigned"):
contains Signature or Extra (extension data). After bare transactions
are no longer supported, this will strictly identify an Inherents only.
Available in both extrinsic versions 4 and 5.
- 0b10000100: Old-school "Signed" Transaction: contains Signature, Extra
(extension data) and an extension version byte, introduced as part of
[RFC99](https://github.com/polkadot-fellows/RFCs/blob/main/text/0099-transaction-extension-version.md).
Still available as part of extrinsic v4.
- 0b01000101: New-school "General" Transaction: contains Extra
(extension data) and an extension version byte, as per RFC99, but no
Signature. Only available in extrinsic v5.

For the New-school General Transaction, it becomes trivial for authors
to publish extensions to the mechanism for authorizing an Origin, e.g.
through new kinds of key-signing schemes, ZK proofs, pallet state,
mutations over pre-authenticated origins or any combination of the
above.

`UncheckedExtrinsic` still maintains encode/decode backwards
compatibility with extrinsic version 4, where the first byte was encoded
as:
- 0b00000100 - Unsigned transactions
- 0b10000100 - Old-school Signed transactions, without the extension
version byte

Now, `UncheckedExtrinsic` contains a `Preamble` and the actual call. The
`Preamble` describes the type of extrinsic as follows:
```rust
/// A "header" for extrinsics leading up to the call itself. Determines the type of extrinsic and
/// holds any necessary specialized data.
#[derive(Eq, PartialEq, Clone)]
pub enum Preamble<Address, Signature, Extension> {
	/// An extrinsic without a signature or any extension. This means it's either an inherent or
	/// an old-school "Unsigned" (we don't use that terminology any more since it's confusable with
	/// the general transaction which is without a signature but does have an extension).
	///
	/// NOTE: In the future, once we remove `ValidateUnsigned`, this will only serve Inherent
	/// extrinsics and thus can be renamed to `Inherent`.
	Bare(ExtrinsicVersion),
	/// An old-school transaction extrinsic which includes a signature of some hard-coded crypto.
	/// Available only on extrinsic version 4.
	Signed(Address, Signature, ExtensionVersion, Extension),
	/// A new-school transaction extrinsic which does not include a signature by default. The
	/// origin authorization, through signatures or other means, is performed by the transaction
	/// extension in this extrinsic. Available starting with extrinsic version 5.
	General(ExtensionVersion, Extension),
}
```

## Code Migration

### NOW: Getting it to build

Wrap your `SignedExtension`s in `AsTransactionExtension`. This should be
accompanied by renaming your aggregate type in line with the new
terminology. E.g. Before:

```rust
/// The SignedExtension to the basic transaction logic.
pub type SignedExtra = (
	/* snip */
	MySpecialSignedExtension,
);
/// Unchecked extrinsic type as expected by this runtime.
pub type UncheckedExtrinsic =
	generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, SignedExtra>;
```

After:

```rust
/// The extension to the basic transaction logic.
pub type TxExtension = (
	/* snip */
	AsTransactionExtension<MySpecialSignedExtension>,
);
/// Unchecked extrinsic type as expected by this runtime.
pub type UncheckedExtrinsic =
	generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, TxExtension>;
```

You'll also need to alter any transaction building logic to add a
`.into()` to make the conversion happen. E.g. Before:

```rust
fn construct_extrinsic(
		/* snip */
) -> UncheckedExtrinsic {
	let extra: SignedExtra = (
		/* snip */
		MySpecialSignedExtension::new(/* snip */),
	);
	let payload = SignedPayload::new(call.clone(), extra.clone()).unwrap();
	let signature = payload.using_encoded(|e| sender.sign(e));
	UncheckedExtrinsic::new_signed(
		/* snip */
		Signature::Sr25519(signature),
		extra,
	)
}
```

After:

```rust
fn construct_extrinsic(
		/* snip */
) -> UncheckedExtrinsic {
	let tx_ext: TxExtension = (
		/* snip */
		MySpecialSignedExtension::new(/* snip */).into(),
	);
	let payload = SignedPayload::new(call.clone(), tx_ext.clone()).unwrap();
	let signature = payload.using_encoded(|e| sender.sign(e));
	UncheckedExtrinsic::new_signed(
		/* snip */
		Signature::Sr25519(signature),
		tx_ext,
	)
}
```

### SOON: Migrating to `TransactionExtension`

Most `SignedExtension`s can be trivially converted to become a
`TransactionExtension`. There are a few things to know.

- Instead of a single trait like `SignedExtension`, you should now
implement two traits individually: `TransactionExtensionBase` and
`TransactionExtension`.
- Weights are now a thing and must be provided via the new function `fn
weight`.

#### `TransactionExtensionBase`

This trait takes care of anything which is not dependent on types
specific to your runtime, most notably `Call`.

- `AdditionalSigned`/`additional_signed` is renamed to
`Implicit`/`implicit`.
- Weight must be returned by implementing the `weight` function. If your
extension is associated with a pallet, you'll probably want to do this
via the pallet's existing benchmarking infrastructure.

#### `TransactionExtension`

Generally:
- `pre_dispatch` is now `prepare` and you *should not reexecute the
`validate` functionality in there*!
- You don't get an account ID any more; you get an origin instead. If
you need to presume an account ID, then you can use the trait function
`AsSystemOriginSigner::as_system_origin_signer`.
- You get an additional ticket, similar to `Pre`, called `Val`. This
defines data which is passed from `validate` into `prepare`. This is
important since you should not be duplicating logic from `validate` to
`prepare`, you need a way of passing your working from the former into
the latter. This is it.
- This trait takes a `Call` type parameter. `Call` is the runtime call
type which used to be an associated type; you can just move it to become
a type parameter for your trait impl.
- There's no `AccountId` associated type any more. Just remove it.

Regarding `validate`:
- You get three new parameters in `validate`; all can be ignored when
migrating from `SignedExtension`.
- `validate` returns a tuple on success; the second item in the tuple is
the new ticket type `Self::Val` which gets passed in to `prepare`. If
you use any information extracted during `validate` (off-chain and
on-chain, non-mutating) in `prepare` (on-chain, mutating) then you can
pass it through with this. For the tuple's last item, just return the
`origin` argument.

Regarding `prepare`:
- This is renamed from `pre_dispatch`, but there is one change:
- FUNCTIONALITY TO VALIDATE THE TRANSACTION NEED NOT BE DUPLICATED FROM
`validate`!!
- (This is different to `SignedExtension` which was required to run the
same checks in `pre_dispatch` as in `validate`.)

Regarding `post_dispatch`:
- Since there are no unsigned transactions handled by
`TransactionExtension`, `Pre` is always defined, so the first parameter
is `Self::Pre` rather than `Option<Self::Pre>`.

If you make use of `SignedExtension::validate_unsigned` or
`SignedExtension::pre_dispatch_unsigned`, then:
- Just use the regular versions of these functions instead.
- Have your logic execute in the case that the `origin` is `None`.
- Ensure your transaction creation logic creates a General Transaction
rather than a Bare Transaction; this means having to include all
`TransactionExtension`s' data.
- `ValidateUnsigned` can still be used (for now) if you need to be able
to construct transactions which contain none of the extension data,
however these will be phased out in stage 2 of the Transactions Horizon,
so you should consider moving to an extension-centric design.

---------

Signed-off-by: georgepisaltu <george.pisaltu@parity.io>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>

Implemented is_trusted_reserve and is_trusted_teleporter API methods. 
Tested them with regular and chopstick tests.
Fixes #97

---------

Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>

Closes #5705

---------

Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

### Fork-Aware Transaction Pool Implementation

This PR introduces a fork-aware transaction pool (fatxpool) enhancing
transaction management by maintaining the valid state of txpool for
different forks.

### High-level overview
The high level overview was added to
[`sc_transaction_pool::fork_aware_txpool`](https://github.com/paritytech/polkadot-sdk/blob/3ad0a1b7/substrate/client/transaction-pool/src/fork_aware_txpool/mod.rs#L21)
module. Use:
```
cargo  doc --document-private-items -p sc-transaction-pool --open
```
to build the doc. It should give a good overview and nice entry point
into the new pool's mechanics.

<details>
  <summary>Quick overview (documentation excerpt)</summary>

#### View
For every fork, a view is created. The view is a persisted state of the
transaction pool computed and updated at the tip of the fork. The view
is built around the existing `ValidatedPool` structure.

A view is created on every new best block notification. To create a
view, one of the existing views is chosen and cloned.

When the chain progresses, the view is kept in the cache
(`retracted_views`) to allow building blocks upon intermediary blocks in
the fork.

The views are deleted on finalization: views lower than the finalized
block are removed.

The views are updated with the transactions from the mempool—all
transactions are sent to the newly created views.
A maintain process is also executed for the newly created
views—basically resubmitting and pruning transactions from the
appropriate tree route.

##### View store
View store is the helper structure that acts as a container for all the
views. It provides some convenient methods.

##### Submitting transactions
Every transaction is submitted to every view at the tips of the forks.
Retracted views are not updated.
Every transaction also goes into the mempool.

##### Internal mempool
Shortly, the main purpose of an internal mempool is to prevent a
transaction from being lost. That could happen when a transaction is
invalid on one fork and could be valid on another. It also allows the
txpool to accept transactions when no blocks have been reported yet.

The mempool removes its transactions when they get finalized.
Transactions are also periodically verified on every finalized event and
removed from the mempool if no longer valid.

#### Events
Transaction events from multiple views are merged and filtered to avoid
duplicated events.
`Ready` / `Future` / `Inblock` events are originated in the Views and
are de-duplicated and forwarded to external listeners.
`Finalized` events are originated in fork-aware-txpool logic.
`Invalid` events requires special care and can be originated in both
view and fork-aware-txpool logic.

#### Light maintain
Sometime transaction pool does not have enough time to prepare fully
maintained view with all retracted transactions being revalidated. To
avoid providing empty ready transaction set to block builder (what would
result in empty block) the light maintain was implemented. It simply
removes the imported transactions from ready iterator.

#### Revalidation
Revalidation is performed for every view. The revalidation process is
started after a trigger is executed. The revalidation work is terminated
just after a new best block / finalized event is notified to the
transaction pool.
The revalidation result is applied to the newly created view which is
built upon the revalidated view.

Additionally, parts of the mempool are also revalidated to make sure
that no transactions are stuck in the mempool.


#### Logs
The most important log allowing to understand the state of the txpool
is:
```
              maintain: txs:(0, 92) views:[2;[(327, 76, 0), (326, 68, 0)]] event:Finalized { hash: 0x8...f, tree_route: [] }  took:3.463522ms
                             ^   ^         ^     ^   ^  ^      ^   ^  ^        ^                                                   ^
unwatched txs in mempool ────┘   │         │     │   │  │      │   │  │        │                                                   │
   watched txs in mempool ───────┘         │     │   │  │      │   │  │        │                                                   │
                     views  ───────────────┘     │   │  │      │   │  │        │                                                   │
                      1st view block # ──────────┘   │  │      │   │  │        │                                                   │
                           number of ready tx ───────┘  │      │   │  │        │                                                   │
                                numer of future tx ─────┘      │   │  │        │                                                   │
                                        2nd view block # ──────┘   │  │        │                                                   │
                                      number of ready tx ──────────┘  │        │                                                   │
                                           number of future tx ───────┘        │                                                   │
                                                                 event ────────┘                                                   │
                                                                       duration  ──────────────────────────────────────────────────┘
```
It is logged after the maintenance is done.

The `debug` level enables per-transaction logging, allowing to keep
track of all transaction-related actions that happened in txpool.
</details>


### Integration notes

For teams having a custom node, the new txpool needs to be instantiated,
typically in `service.rs` file, here is an example:

https://github.com/paritytech/polkadot-sdk/blob/9c547ff3

/cumulus/polkadot-omni-node/lib/src/common/spec.rs#L152-L161

To enable new transaction pool the following cli arg shall be specified:
`--pool-type=fork-aware`. If it works, there shall be information
printed in the log:
```
2024-09-20 21:28:17.528  INFO main txpool: [Parachain]  creating ForkAware txpool.
````

For debugging the following debugs shall be enabled:
```
      "-lbasic-authorship=debug",
      "-ltxpool=debug",
```
*note:* trace for txpool enables per-transaction logging.

### Future work
The current implementation seems to be stable, however further
improvements are required.
Here is the umbrella issue for future work:
- https://github.com/paritytech/polkadot-sdk/issues/5472


Partially fixes: #1202

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>

Added error logs in pallet-xcm to help in debugging, fixes #2408 

## TODO

- [x] change `log::error` to `tracing::error` format for `xcm-executor`
- [x] check existing logs, e.g. this one can be extended with more info
`tracing::error!(target: "xcm::reanchor", ?error, "Failed reanchoring
with error");`
- [x] use `tracing` instead of `log` for `pallet-xcm/src/lib.rs`

---------

Co-authored-by: Ayevbeosa Iyamu <aiyamu@vatebra.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>

Testing the approach of this before it goes live on Polkadot
https://github.com/polkadot-fellows/runtimes/pull/471

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Dónal Murray <donal.murray@parity.io>

# Description

When we send `QueueEvent::StartWork`, we have already completed the
execution. This may be a leftover of a previous logic change. Currently,
the name is misleading, so it would be better to rename it to
`FinishWork`.


https://github.com/paritytech/polkadot-sdk/blob/c52675ef/polkadot/node/core/pvf/src/execute/queue.rs#L632-L646


https://github.com/paritytech/polkadot-sdk/blob/c52675ef

/polkadot/node/core/pvf/src/execute/queue.rs#L361-L363

Fixes https://github.com/paritytech/polkadot-sdk/issues/5910

## Integration

Shouldn't affect downstream projects.

---------

Co-authored-by: GitHub Action <action@github.com>

Resolves https://github.com/paritytech/polkadot-sdk/issues/5972

Only needed to increase some timeouts

# Description

This PR addresses
https://github.com/paritytech/polkadot-sdk/issues/5878.

After dry running an xcm on asset hub, we had redundant xcms showing up
in the `forwarded_xcms` field of the dry run effects returned.
These were caused by two things:
- The `UpwardMessageSender` router always added an element even if there
were no messages.
- The two routers on asset hub westend related to bridging (to rococo
and sepolia) getting the message from their queues when their queues is
actually the same xcmp queue that was already contemplated.

In order to fix this, we check for no messages in UMP and clear the
implementation of `InspectMessageQueues` for these bridging routers.
Keep in mind that the bridged message is still sent, as normal via the
xcmp-queue to Bridge Hub.
To keep on dry-running the journey of the message, the next hop to
dry-run is Bridge Hub.
That'll be tackled in a different PR.

Added a test in `bridge-hub-westend-integration-tests` and
`bridge-hub-rococo-integration-tests` that show that dry-running a
transfer across the bridge from asset hub results in one and only one
message sent to bridge hub.

## TODO
- [x] Functionality
- [x] Test

---------

Co-authored-by: command-bot <>

Resolves https://github.com/paritytech/polkadot-sdk/issues/4632

The new logic optimizes the distribution of execution jobs for disputes,
approvals, and backings. Testing shows improved finality lag and
candidate checking times, especially under heavy network load.

### Approach

This update adds prioritization to the PVF execution queue. The logic
partially implements the suggestions from
https://github.com/paritytech/polkadot-sdk/issues/4632#issuecomment-2209188695.

We use thresholds to determine how much a current priority can "steal"
from lower ones:
-  Disputes: 70%
-  Approvals: 80%
-  Backing System Parachains: 100%
-  Backing: 100%

A threshold indicates the portion of the current priority that can be
allocated from lower priorities.

For example:
-  Disputes take 70%, leaving 30% for approvals and all backings.
- 80% of the remaining goes to approvals, which is 30% * 80% = 24% of
the original 100%.
- If we used parts of the original 100%, approvals couldn't take more
than 24%, even if there are no disputes.

Assuming a maximum of 12 executions per block, with a 6-second window, 2
CPU cores, and a 2-second run time, we get these distributions:

-  With disputes: 8 disputes, 3 approvals, 1 backing
-  Without disputes: 9 approvals, 3 backings

It's worth noting that when there are no disputes, if there's only one
backing job, we continue processing approvals regardless of their
fulfillment status.

### Versi Testing 40/20

Testing showed a slight difference in finality lag and candidate
checking time between this pull request and its base on the master
branch. The more loaded the network, the greater the observed
difference.

Testing Parameters:
-  40 validators (4 malicious)
-  20 gluttons with 2 seconds of PVF execution time
-  6 VRF modulo samples
-  12 required approvals

![Pasted Graphic
3](https://github.com/user-attachments/assets/8b6163a4-a1c9-44c2-bdba-ce1ef4b1eba7)
![Pasted Graphic
4](https://github.com/user-attachments/assets/9f016647-7727-42e8-afe9-04f303e6c862)

### Versi Testing 80/40

For this test, we compared the master branch with the branch from
https://github.com/paritytech/polkadot-sdk/pull/5616. The second branch
is based on the current one but removes backing jobs that have exceeded
their time limits. We excluded malicious nodes to reduce noise from
disputing and banning validators. The results show that, under the same
load, nodes experience less finality lag and reduced recovery and check
time. Even parachains are functioning with a shorter block time,
although it remains over 6 seconds.

Testing Parameters:
-  80 validators (0 malicious)
-  40 gluttons with 2 seconds of PVF execution time
-  6 VRF modulo samples
-  30 required approvals


![image](https://github.com/user-attachments/assets/42bcc845-9115-4ae3-9910-286b77a60bbf)

---------

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

# Description

We previously set the PoV request timeout to 1.2s based on synchronous
backing, which allowed for 5 PoVs per relay block. With asynchronous
backing, we no longer have a time budget and can increase the value to
2s.

Fixes https://github.com/paritytech/polkadot-sdk/issues/5885

## Integration

This PR shouldn't affect downstream projects.

## Review Notes

This PR can be followed by experiments with Gluttons on Kusama to
confirm that the timeout is sufficient.

Resolves https://github.com/polkadot-fellows/runtimes/issues/459

Tested with rococo/westend/kusama/polkadot runtimes

---------

Co-authored-by: DavidK <davidk@parity.io>
Co-authored-by: Muharem <ismailov.m.h@gmail.com>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Dónal Murray <donal.murray@parity.io>

# Description
## What?
Make it possible for other pallets to implement their own logic when a
slash on a balance occurs.

## Why?
In the [introduction of
holds](https://github.com/paritytech/substrate/pull/12951) @gavofyork
said:
> Since Holds are designed to be infallibly slashed, this means that any
logic using a Freeze must handle the possibility of the frozen amount
being reduced, potentially to zero. A permissionless function should be
provided in order to allow bookkeeping to be updated in this instance.

At Polimec we needed to find a way to reduce the vesting schedules of
our users after a slash was made, and after talking to @Kianenigma

 at
the Web3Summit, we realized there was no easy way to implement this with
the current traits, so we came up with this solution.



## How?
- First we abstract the `done_slash` function of holds::Balanced to it's
own trait that any pallet can implement.
- Then we add a config type in pallet-balances that accepts a callback
tuple of all the pallets that implement this trait.
- Finally implement done_slash for pallet-balances such that it calls
the config type.

## Integration
The default implementation of done_slash is still an empty function, and
the new config type of pallet-balances can be set to an empty tuple, so
nothing changes by default.

## Review Notes
- I suggest to focus on the first commit which contains the main logic
changes.
- I also have a working implementation of done_slash for pallet_vesting,
should I add it to this PR?
- If I run `cargo +nightly fmt --all` then I get changes to a lot of
unrelated crates, so not sure if I should run it to avoid the fmt
failure of the CI
- Should I hunt down references to fungible/fungibles documentation and
update it accordingly?

**Polkadot address:** `15fj1UhQp8Xes7y7LSmDYTy349mXvUwrbNmLaP5tQKBxsQY1`

# Checklist

* [x] My PR includes a detailed description as outlined in the
"Description" and its two subsections above.
* [x] My PR follows the [labeling requirements](

https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process
) of this project (at minimum one label for `T` required)
* External contributors: ask maintainers to put the right label on your
PR.
* [ ] I have made corresponding changes to the documentation (if
applicable)

---------

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: command-bot <>
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>

Closes https://github.com/paritytech/polkadot-sdk/issues/5045 and
https://github.com/paritytech/polkadot-sdk/issues/5046

<del>On top of
https://github.com/paritytech/polkadot-sdk/pull/5362</del>

TODO:
- [x] storage migration for allowed relay parents tracker 
- [x] check session index
- [x] PRdoc
- [x] tests
- [x] ensure UMP queue cannot be abused with this change
- [x] Zombienet runtime upgrade test

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Fixes #5900.

The AllowTopLevelPaidExecutionFrom allows ClearOrigin instructions
before the expected BuyExecution instruction, it also allows messages
without any origin altering instructions.

This commit enhances the barrier to also support messages that use
AliasOrigin, or DescendOrigin. This is sometimes desired in asset
transfer XCM programs that need to run the inbound assets instructions
using the origin chain root origin, but then want to drop privileges for
the rest of the program. Currently these programs drop privileges by
clearing the origin completely, but that also unnecessarily limits the
range of actions available to the rest of the program. Using
DescendOrigin or AliasOrigin allows the sending chain to instruct the
receiving chain what the deprivileged real origin is.

See https://github.com/polkadot-fellows/RFCs/pull/109 and
https://github.com/polkadot-fellows/RFCs/pull/122 for more details on
how DescendOrigin and AliasOrigin could be used instead of ClearOrigin.

---------

Signed-off-by: Adrian Catangiu <adrian@parity.io>

Jaeger tracing went mostly unused and it created bigger problems like
wasting CPU or memory leaks, so remove it entirely.

Fixes: https://github.com/paritytech/polkadot-sdk/issues/4995

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>

Added `HashedDescription<AccountId,
DescribeFamily<DescribeAllTerminal>>` foreign locations to local
accounts converter to all the parachains.

---------

Co-authored-by: command-bot <>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>

Quoting @bkchr (from
[here](https://github.com/paritytech/polkadot-sdk/issues/5334#issuecomment-2383815078)):

> That the hardcoded limit is used there again, is IMO not correct. The
`max_pov_size` should be controlled by the `HostConfiguration` and not
limited by some "random" constant.

This PR aims to change the hard limit to a not-so-random constant,
allowing more room for maneuvering in the future.

Fixes https://github.com/paritytech/polkadot-sdk/issues/5617

# Description

This PR removes the
`CandidateValidationMessage::ValidateFromChainState`, which was
previously used by backing, but is no longer relevant since initial
async backing implementation
https://github.com/paritytech/polkadot/pull/5557.

Fixes https://github.com/paritytech/polkadot-sdk/issues/5643

## Integration

This change should not affect downstream projects since
`ValidateFromChainState` was already unused.

## Review Notes

- Removed all occurrences of `ValidateFromChainState`.
- Moved utility functions, previously used in candidate validation tests
and malus, exclusively to candidate validation tests as they are no
longer used in malus.
- Deleted the
`polkadot_parachain_candidate_validation_validate_from_chain_state`
metric from Prometheus.
- Removed `Spawner` from `ReplaceValidationResult` in malus’
interceptors.
- `fake_validation_error` was only used for `ValidateFromChainState`
handling, while other cases directly used
`InvalidCandidate::Invalid...

# Description

close #5641

---------

Co-authored-by: Bastian Köcher <git@kchr.de>

This PR backports regular version bumps and prdocs reordering from the
`stable2409` release branch to `master`

---------

Co-authored-by: Morgan Adamiec <morgan@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Nazar Mokrynskyi <nazar@mokrynskyi.com>

Jaeger spans were not usable for debugging, see
https://github.com/paritytech/polkadot-sdk/issues/4995, but we pay a
price in CPU cost, subsystem-benchmarks show this brings a reduction of
about 10-15% in CPU usage per subsystem, so remove it.

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>

cc https://github.com/paritytech/ci_cd/issues/1035

cc https://github.com/paritytech/ci_cd/issues/1023

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>
Co-authored-by: Maksym H <1177472+mordamax@users.noreply.github.com>
Co-authored-by: gui <gui.thiolliere@gmail.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: ggwpez <ggwpez@users.noreply.github.com>

This is the implementation of the approach described here:
https://github.com/paritytech/polkadot-sdk/issues/1617#issuecomment-2150321612
&
https://github.com/paritytech/polkadot-sdk/issues/1617#issuecomment-2154357547
&
https://github.com/paritytech/polkadot-sdk/issues/1617#issuecomment-2154721395.

## Description of changes

The end goal is to have an architecture where we have single
subsystem(`approval-voting-parallel`) and multiple worker types that
would full-fill the work that currently is fulfilled by the
`approval-distribution` and `approval-voting` subsystems. The main loop
of the new subsystem would do just the distribution of work to the
workers.

The new subsystem will have:
- N approval-distribution workers: This would do the work that is
currently being done by the approval-distribution subsystem and in
addition to that will also perform the crypto-checks that an assignment
is valid and that a vote is correctly signed. Work is assigned via the
following formula: `worker_index = msg.validator % WORKER_COUNT`, this
guarantees that all assignments and approvals from the same validator
reach the same worker.
- 1 approval-voting worker: This would receive an already valid message
and do everything the approval-voting currently does, except the
crypto-checking that has been moved already to the approval-distribution
worker.

On the hot path of processing messages **no** synchronisation and
waiting is needed between approval-distribution and approval-voting
workers.

<img width="1431" alt="Screenshot 2024-06-07 at 11 28 08"
src="https://github.com/paritytech/polkadot-sdk/assets/49718502/a196199b-b705-4140-87d4-c6900ba8595e">



## Guidelines for reading

The full implementation is broken in 5 PRs and all of them are
self-contained and improve things incrementally even without the
parallelisation being implemented/enabled, the reason this approach was
taken instead of a big-bang PR, is to make things easier to review and
reduced the risk of breaking this critical subsystems.

After reading the full description of this PR, the changes should be
read in the following order:
1. https://github.com/paritytech/polkadot-sdk/pull/4848, some other
micro-optimizations for networks with a high number of validators. This
change gives us a speed up by itself without any other changes.
2. https://github.com/paritytech/polkadot-sdk/pull/4845 , this contains
only interface changes to decouple the subsystem from the `Context` and
be able to run multiple instances of the subsystem on different threads.
**No functional changes**
3. https://github.com/paritytech/polkadot-sdk/pull/4928, moving of the
crypto checks from approval-voting in approval-distribution, so that the
approval-distribution has no reason to wait after approval-voting
anymore. This change gives us a speed up by itself without any other
changes.
4. https://github.com/paritytech/polkadot-sdk/pull/4846, interface
changes to make approval-voting runnable on a separate thread. **No
functional changes**
5. This PR, where we instantiate an `approval-voting-parallel` subsystem
that runs on different workers the logic currently in
`approval-distribution` and `approval-voting`.
6. The next step after this changes get merged and deploy would be to
bring all the files from approval-distribution, approval-voting,
approval-voting-parallel into a single rust crate, to make it easier to
maintain and understand the structure.

## Results
Running subsystem-benchmarks with 1000 validators 100 fully ocuppied
cores and triggering all assignments and approvals for all tranches

#### Approval does not lags behind. 
 Master
```
Chain selection approved  after 72500 ms hash=0x0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
```
With this PoC
```
Chain selection approved  after 3500 ms hash=0x0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
```

#### Gathering enough assignments
 
Enough assignments are gathered in less than 500ms, so that gives un a
guarantee that un-necessary work does not get triggered, on master on
the same benchmark because the subsystems fall behind on work, that
number goes above 32 seconds on master.
 
<img width="2240" alt="Screenshot 2024-06-20 at 15 48 22"
src="https://github.com/paritytech/polkadot-sdk/assets/49718502/d2f2b29c-5ff6-44b4-a245-5b37ab8e58bc">


#### Cpu usage:
Master
```
CPU usage, seconds                     total   per block
approval-distribution                96.9436      9.6944
approval-voting                     117.4676     11.7468
test-environment                     44.0092      4.4009
```
With this PoC
```
CPU usage, seconds                     total   per block
approval-distribution                 0.0014      0.0001 --- unused
approval-voting                       0.0437      0.0044.  --- unused
approval-voting-parallel              5.9560      0.5956
approval-voting-parallel-0           22.9073      2.2907
approval-voting-parallel-1           23.0417      2.3042
approval-voting-parallel-2           22.0445      2.2045
approval-voting-parallel-3           22.7234      2.2723
approval-voting-parallel-4           21.9788      2.1979
approval-voting-parallel-5           23.0601      2.3060
approval-voting-parallel-6           22.4805      2.2481
approval-voting-parallel-7           21.8330      2.1833
approval-voting-parallel-db          37.1954      3.7195.  --- the approval-voting thread.
```

# Enablement strategy

Because just some trivial plumbing is needed in approval-distribution
and approval-voting to be able to run things in parallel and because
this subsystems plays a critical part in the system this PR proposes
that we keep both ways of running the approval work, as separated
subsystems and just a single subsystem(`approval-voting-parallel`) which
has multiple workers for the distribution work and one worker for the
approval-voting work and switch between them with a comandline flag.

The benefits for this is twofold.
1. With the same polkadot binary we can easily switch just a few
validators to use the parallel approach and gradually make this the
default way of running, if now issues arise.
2. In the worst case scenario were it becomes the default way of running
things, but we discover there are critical issues with it we have the
path to quickly disable it by asking validators to adjust their command
line flags.


# Next steps
- [x] Make sure through various testing we are not missing anything 
- [x] Polish the implementations to make them production ready
- [x] Add Unittest Tests for approval-voting-parallel.
- [x] Define and implement the strategy for rolling this change, so that
the blast radius is minimal(single validator) in case there are problems
with the implementation.
- [x]  Versi long running tests.
- [x] Add relevant metrics.

@ordian @eskimor @sandreim @AndreiEres

, let me know what you think.

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>

A change that I missed to add in
https://github.com/paritytech/polkadot-sdk/pull/5525 .

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

This change adds the required validation for stronger UX guarantees when
using `InitiateReserveWithdraw` or `InitiateTeleport` XCM instructions.
Execution of the instructions will fail if the local chain is not
configured to trust the "destination" or "reserve" chain as a
reserve/trusted-teleporter for the provided "assets".

With this change, misuse of `InitiateReserveWithdraw`/`InitiateTeleport`
fails on origin with no overall side-effects, rather than failing on
destination (with side-effects to origin's assets issuance).

The commit also makes the same validations for pallet-xcm transfers, and
adds regression tests.

---------

Signed-off-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>

Minor cleanup.

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>