This should be configured on the runtime level and not somewhere inside
the pallet.

---------

Co-authored-by: Adrian Catangiu <[email protected]>
Co-authored-by: Branislav Kontur <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>

Introduce types to define 1:1 balance conversion for different relative
asset ids/locations of native asset.

Examples:
native asset on Asset Hub presented as `VersionedLocatableAsset` type in
the context of Relay Chain is
```
{
  `location`: (0, Parachain(1000)),
  `asset_id`: (1, Here),
}
```
and it's balance should be converted 1:1 by implementations of
`ConversionToAssetBalance` trait.

---------

Co-authored-by: Branislav Kontur <[email protected]>

This PR introduces `BlockHashProvider` into `pallet_mmr::Config`
This type is used to get `block_hash` for a given `block_number` rather
than directly using `frame_system::Pallet::block_hash`

The `DefaultBlockHashProvider` uses `frame_system::Pallet::block_hash`
to get the `block_hash`

Closes: #4062

Fixes https://github.com/paritytech/polkadot-sdk/issues/3576

Required by elastic scaling collators.
Deprecates old API: `candidate_pending_availability`.

TODO:
- [x] PRDoc

---------

Signed-off-by: Andrei Sandu <[email protected]>

Implements https://github.com/polkadot-fellows/RFCs/issues/82
Allow any parachain to have bidirectional channel with any system
parachains

Looking for initial feedbacks before continue finish it

TODOs:
- [x] docs
- [x] benchmarks
- [x] tests

---------

Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: Adrian Catangiu <[email protected]>

Part of #3326 

@Kianenigma @ggwpez

 

polkadot address: 12poSUQPtcF1HUPQGY3zZu2P8emuW9YnsPduA4XG3oCEfJVp

---------

Signed-off-by: Matteo Muraca <[email protected]>
Co-authored-by: ordian <[email protected]>

This PR backports `spec_version`, `node_version` bumps and reordering of
the prdocs from the 1.10.0 release branch

closes https://github.com/paritytech/polkadot-sdk/issues/3894

Allows Nomination Pool configuration to be set by a custom origin
instead of root.

In runtimes, we would set this to be `StakingAdmin`, same as for
pallet-staking.

---------

Co-authored-by: Liam Aharon <[email protected]>

Signed-off-by: Andrei Sandu <[email protected]>

The runtime now can provide a number of predefined presets of
`RuntimeGenesisConfig` struct. This presets are intended to be used in
different deployments, e.g.: `local`, `staging`, etc, and should be
included into the corresponding chain-specs.

Having `GenesisConfig` presets in runtime allows to fully decouple node
from runtime types (the problem is described in #1984).

**Summary of changes:**
- The `GenesisBuilder` API was adjusted to enable this functionality
(and provide better naming - #150):
   ```rust
    fn preset_names() -> Vec<PresetId>;
fn get_preset(id: Option<PresetId>) -> Option<serde_json::Value>;
//`None` means default
    fn build_state(value: serde_json::Value);
    pub struct PresetId(Vec<u8>);
   ```

- **Breaking change**: Old `create_default_config` method was removed,
`build_config` was renamed to `build_state`. As a consequence a node
won't be able to interact with genesis config for older runtimes. The
cleanup was made for sake of API simplicity. Also IMO maintaining
compatibility with old API is not so crucial.
- Reference implementation was provided for `substrate-test-runtime` and
`rococo` runtimes. For rococo new
[`genesis_configs_presets`](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b/polkadot/runtime/rococo/src/genesis_config_presets.rs#L530)
module was added and is used in `GenesisBuilder`
[_presets-related_](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b/polkadot/runtime/rococo/src/lib.rs#L2462-L2485)
methods.

- The `chain-spec-builder` util was also improved and allows to
([_doc_](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b/substrate/bin/utils/chain-spec-builder/src/lib.rs#L19)):
   - list presets provided by given runtime (`list-presets`),
- display preset or default config provided by the runtime
(`display-preset`),
   - build chain-spec using named preset (`create ... named-preset`),


- The `ChainSpecBuilder` is extended with
[`with_genesis_config_preset_name`](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b/substrate/client/chain-spec/src/chain_spec.rs#L447)
method which allows to build chain-spec using named preset provided by
the runtime. Sample usage on the node side
[here](https://github.com/paritytech/polkadot-sdk/blob/2caffaae

/polkadot/node/service/src/chain_spec.rs#L404).

Implementation of #1984.
fixes: #150
part of: #25

---------

Co-authored-by: Sebastian Kunert <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>

Part of https://github.com/paritytech/polkadot-sdk/issues/226 
Related https://github.com/paritytech/polkadot-sdk/issues/1833

- Deprecate `CurrencyAdapter` and introduce `FungibleAdapter`
- Deprecate `ToStakingPot` and replace usage with `ResolveTo`
- Required creating a new `StakingPotAccountId` struct that implements
`TypedGet` for the staking pot account ID
- Update parachain common utils `DealWithFees`, `ToAuthor` and
`AssetsToBlockAuthor` implementations to use `fungible`
- Update runtime XCM Weight Traders to use `ResolveTo` instead of
`ToStakingPot`
- Update runtime Transaction Payment pallets to use `FungibleAdapter`
instead of `CurrencyAdapter`
- [x] Blocked by https://github.com/paritytech/polkadot-sdk/pull/1296,
needs the `Unbalanced::decrease_balance` fix

Rejoice! Rejoice! The story is nearly over.

This PR removes stale migrations, auxiliary structures, and package
dependencies, thus making Rococo and Westend totally free from any
`im-online`-related stuff.

`im-online` still stays a part of the Substrate node and its runtime:
https://github.com/paritytech/polkadot-sdk/blob/0d932484/substrate/bin/node/runtime/src/lib.rs#L2276-L2277
I'm not sure if it makes sense to remove it from there considering that
we're not removing `im-online` from FRAME. Please share your opinion.

Runtime release 1.2 includes bumping of the ParachainHost APIs up to
v10, so let's move all the released APIs out of vstaging folder, this PR
does not include any logic changes only renaming of the modules and some
moving around.

Signed-off-by: Alexandru Gheorghe <[email protected]>

Removes transient code introduced to clean up offchain database after
`im-online` pallet removal.

Should be merged after #2290 has been enacted.

This will make it possible to use remaining weight on idle for
processing enqueued messages.
More context here https://github.com/paritytech/polkadot-sdk/issues/3709

---------

Co-authored-by: Adrian Catangiu <[email protected]>

The PR provides API for obtaining:
- the weight required to execute an XCM message,
- a list of acceptable `AssetId`s for message execution payment,
- the cost of the weight in the specified acceptable `AssetId`.

It is meant to address an issue where one has to guess how much fee to
pay for execution. Also, at the moment, a client has to guess which
assets are acceptable for fee execution payment.
See the related issue
https://github.com/paritytech/polkadot-sdk/issues/690.
With this API, a client is supposed to query the list of the supported
asset IDs (in the XCM version format the client understands), weigh the
XCM program the client wants to execute and convert the weight into one
of the acceptable assets. Note that the client is supposed to know what
program will be executed on what chains. However, having a small
companion JS library for the pallet-xcm and xtokens should be enough to
determine what XCM programs will be executed and where (since these
pallets compose a known small set of programs).
```Rust
pub trait XcmPaymentApi<Call>
	where
		Call: Codec,
	{
		/// Returns a list of acceptable payment assets.
		///
		/// # Arguments
		///
		/// * `xcm_version`: Version.
		fn query_acceptable_payment_assets(xcm_version: Version) -> Result<Vec<VersionedAssetId>, Error>;
		/// Returns a weight needed to execute a XCM.
		///
		/// # Arguments
		///
		/// * `message`: `VersionedXcm`.
		fn query_xcm_weight(message: VersionedXcm<Call>) -> Result<Weight, Error>;
		/// Converts a weight into a fee for the specified `AssetId`.
		///
		/// # Arguments
		///
		/// * `weight`: convertible `Weight`.
		/// * `asset`: `VersionedAssetId`.
		fn query_weight_to_asset_fee(weight: Weight, asset: VersionedAssetId) -> Result<u128, Error>;
		/// Get delivery fees for sending a specific `message` to a `destination`.
		/// These always come in a specific asset, defined by the chain.
		///
		/// # Arguments
		/// * `message`: The message that'll be sent, necessary because most delivery fees are based on the
		///   size of the message.
		/// * `destination`: The destination to send the message to. Different destinations may use
		///   different senders that charge different fees.
		fn query_delivery_fees(destination: VersionedLocation, message: VersionedXcm<()>) -> Result<VersionedAssets, Error>;
	}
```
An
[example](https://gist.github.com/PraetorP/4bc323ff85401abe253897ba990ec29d)
of a client side code.

---------

Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Adrian Catangiu <[email protected]>
Co-authored-by: Daniel Shiposha <[email protected]>

This PR notifies broker pallet for any parachain slot swaps performed on
the relay chain. This is achieved by registering an `OnSwap` for the the
`coretime` pallet. The hook sends XCM message to the broker chain and
invokes a new extrinsic `swap_leases` which updates `Leases` storage
item (which keeps the legacy parachain leases).

I made two assumptions in this PR:
1.
[`Leases`](https://github.com/paritytech/polkadot-sdk/blob/4987d798/substrate/frame/broker/src/lib.rs#L120)
in `broker` pallet and
[`Leases`](https://github.com/paritytech/polkadot-sdk/blob/4987d798

/polkadot/runtime/common/src/slots/mod.rs#L118)
in `slots` pallet are in sync.
2. `swap_leases` extrinsic from `broker` pallet can be triggered only by
root or by the XCM message from the relay chain. If not - the extrinsic
will generate an error and do nothing.

As a side effect from the changes `OnSwap` trait is moved from
runtime/common/traits.rs to runtime/parachains. Otherwise it is not
accessible from `broker` pallet.

Closes https://github.com/paritytech/polkadot-sdk/issues/3552

TODOs:

- [x] Weights
- [x] Tests

---------

Co-authored-by: command-bot <>
Co-authored-by: eskimor <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Changes needed to implement the runtime part of elastic scaling:
https://github.com/paritytech/polkadot-sdk/issues/3131,
https://github.com/paritytech/polkadot-sdk/issues/3132,
https://github.com/paritytech/polkadot-sdk/issues/3202

Also fixes https://github.com/paritytech/polkadot-sdk/issues/3675

TODOs:

- [x] storage migration
- [x] optimise process_candidates from O(N^2)
- [x] drop backable candidates which form cycles
- [x] fix unit tests
- [x] add more unit tests
- [x] check the runtime APIs which use the pending availability storage.
We need to expose all of them, see
https://github.com/paritytech/polkadot-sdk/issues/3576
- [x] optimise the candidate selection. we're currently picking randomly
until we satisfy the weight limit. we need to be smart about not
breaking candidate chains while being fair to all paras -
https://github.com/paritytech/polkadot-sdk/pull/3573

Relies on the changes made in
https://github.com/paritytech/polkadot-sdk/pull/3233 in terms of the
inclusion policy and the candidate ordering

---------

Signed-off-by: alindima <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: eskimor <[email protected]>

This PR backports:
- node version bump
- `spec_vesion` bump
- reordering of the `prdocs` to the appropriate folder
from the `1.9.0` release branch

Step in https://github.com/paritytech/polkadot-sdk/issues/3688

Part of #3326 

cc @Kianenigma @ggwpez @liamaharon

 

polkadot address: 12poSUQPtcF1HUPQGY3zZu2P8emuW9YnsPduA4XG3oCEfJVp

---------

Signed-off-by: Matteo Muraca <[email protected]>

Step in https://github.com/paritytech/polkadot-sdk/issues/171

This PR removes `as [disambiguation_path]` syntax from `derive_impl`
usage across the polkadot-sdk as introduced in
https://github.com/paritytech/polkadot-sdk/pull/3505

Revert "FRAME: Create `TransactionExtension` as a replacement for `SignedExtension` (#2280)" (#3665)

This PR reverts #2280 which introduced `TransactionExtension` to replace
`SignedExtension`.

As a result of the discussion
[here](https://github.com/paritytech/polkadot-sdk/pull/3623#issuecomment-1986789700),
the changes will be reverted for now with plans to reintroduce the
concept in the future.

---------

Signed-off-by: georgepisaltu <[email protected]>

Closes #2160

First part of [Extrinsic
Horizon](https://github.com/paritytech/polkadot-sdk/issues/2415)

Introduces a new trait `TransactionExtension` to replace
`SignedExtension`. Introduce the idea of transactions which obey the
runtime's extensions and have according Extension data (né Extra data)
yet do not have hard-coded signatures.

Deprecate the terminology of "Unsigned" when used for
transactions/extrinsics owing to there now being "proper" unsigned
transactions which obey the extension framework and "old-style" unsigned
which do not. Instead we have __*General*__ for the former and
__*Bare*__ for the latter. (Ultimately, the latter will be phased out as
a type of transaction, and Bare will only be used for Inherents.)

Types of extrinsic are now therefore:
- Bare (no hardcoded signature, no Extra data; used to be known as
"Unsigned")
- Bare transactions (deprecated): Gossiped, validated with
`ValidateUnsigned` (deprecated) and the `_bare_compat` bits of
`TransactionExtension` (deprecated).
  - Inherents: Not gossiped, validated with `ProvideInherent`.
- Extended (Extra data): Gossiped, validated via `TransactionExtension`.
  - Signed transactions (with a hardcoded signature).
  - General transactions (without a hardcoded signature).

`TransactionExtension` differs from `SignedExtension` because:
- A signature on the underlying transaction may validly not be present.
- It may alter the origin during validation.
- `pre_dispatch` is renamed to `prepare` and need not contain the checks
present in `validate`.
- `validate` and `prepare` is passed an `Origin` rather than a
`AccountId`.
- `validate` may pass arbitrary information into `prepare` via a new
user-specifiable type `Val`.
- `AdditionalSigned`/`additional_signed` is renamed to
`Implicit`/`implicit`. It is encoded *for the entire transaction* and
passed in to each extension as a new argument to `validate`. This
facilitates the ability of extensions to acts as underlying crypto.

There is a new `DispatchTransaction` trait which contains only default
function impls and is impl'ed for any `TransactionExtension` impler. It
provides several utility functions which reduce some of the tedium from
using `TransactionExtension` (indeed, none of its regular functions
should now need to be called directly).

Three transaction version discriminator ("versions") are now
permissible:
- 0b000000100: Bare (used to be called "Unsigned"): contains Signature
or Extra (extension data). After bare transactions are no longer
supported, this will strictly identify an Inherents only.
- 0b100000100: Old-school "Signed" Transaction: contains Signature and
Extra (extension data).
- 0b010000100: New-school "General" Transaction: contains Extra
(extension data), but no Signature.

For the New-school General Transaction, it becomes trivial for authors
to publish extensions to the mechanism for authorizing an Origin, e.g.
through new kinds of key-signing schemes, ZK proofs, pallet state,
mutations over pre-authenticated origins or any combination of the
above.

## Code Migration

### NOW: Getting it to build

Wrap your `SignedExtension`s in `AsTransactionExtension`. This should be
accompanied by renaming your aggregate type in line with the new
terminology. E.g. Before:

```rust
/// The SignedExtension to the basic transaction logic.
pub type SignedExtra = (
	/* snip */
	MySpecialSignedExtension,
);
/// Unchecked extrinsic type as expected by this runtime.
pub type UncheckedExtrinsic =
	generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, SignedExtra>;
```

After:

```rust
/// The extension to the basic transaction logic.
pub type TxExtension = (
	/* snip */
	AsTransactionExtension<MySpecialSignedExtension>,
);
/// Unchecked extrinsic type as expected by this runtime.
pub type UncheckedExtrinsic =
	generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, TxExtension>;
```

You'll also need to alter any transaction building logic to add a
`.into()` to make the conversion happen. E.g. Before:

```rust
fn construct_extrinsic(
		/* snip */
) -> UncheckedExtrinsic {
	let extra: SignedExtra = (
		/* snip */
		MySpecialSignedExtension::new(/* snip */),
	);
	let payload = SignedPayload::new(call.clone(), extra.clone()).unwrap();
	let signature = payload.using_encoded(|e| sender.sign(e));
	UncheckedExtrinsic::new_signed(
		/* snip */
		Signature::Sr25519(signature),
		extra,
	)
}
```

After:

```rust
fn construct_extrinsic(
		/* snip */
) -> UncheckedExtrinsic {
	let tx_ext: TxExtension = (
		/* snip */
		MySpecialSignedExtension::new(/* snip */).into(),
	);
	let payload = SignedPayload::new(call.clone(), tx_ext.clone()).unwrap();
	let signature = payload.using_encoded(|e| sender.sign(e));
	UncheckedExtrinsic::new_signed(
		/* snip */
		Signature::Sr25519(signature),
		tx_ext,
	)
}
```

### SOON: Migrating to `TransactionExtension`

Most `SignedExtension`s can be trivially converted to become a
`TransactionExtension`. There are a few things to know.

- Instead of a single trait like `SignedExtension`, you should now
implement two traits individually: `TransactionExtensionBase` and
`TransactionExtension`.
- Weights are now a thing and must be provided via the new function `fn
weight`.

#### `TransactionExtensionBase`

This trait takes care of anything which is not dependent on types
specific to your runtime, most notably `Call`.

- `AdditionalSigned`/`additional_signed` is renamed to
`Implicit`/`implicit`.
- Weight must be returned by implementing the `weight` function. If your
extension is associated with a pallet, you'll probably want to do this
via the pallet's existing benchmarking infrastructure.

#### `TransactionExtension`

Generally:
- `pre_dispatch` is now `prepare` and you *should not reexecute the
`validate` functionality in there*!
- You don't get an account ID any more; you get an origin instead. If
you need to presume an account ID, then you can use the trait function
`AsSystemOriginSigner::as_system_origin_signer`.
- You get an additional ticket, similar to `Pre`, called `Val`. This
defines data which is passed from `validate` into `prepare`. This is
important since you should not be duplicating logic from `validate` to
`prepare`, you need a way of passing your working from the former into
the latter. This is it.
- This trait takes two type parameters: `Call` and `Context`. `Call` is
the runtime call type which used to be an associated type; you can just
move it to become a type parameter for your trait impl. `Context` is not
currently used and you can safely implement over it as an unbounded
type.
- There's no `AccountId` associated type any more. Just remove it.

Regarding `validate`:
- You get three new parameters in `validate`; all can be ignored when
migrating from `SignedExtension`.
- `validate` returns a tuple on success; the second item in the tuple is
the new ticket type `Self::Val` which gets passed in to `prepare`. If
you use any information extracted during `validate` (off-chain and
on-chain, non-mutating) in `prepare` (on-chain, mutating) then you can
pass it through with this. For the tuple's last item, just return the
`origin` argument.

Regarding `prepare`:
- This is renamed from `pre_dispatch`, but there is one change:
- FUNCTIONALITY TO VALIDATE THE TRANSACTION NEED NOT BE DUPLICATED FROM
`validate`!!
- (This is different to `SignedExtension` which was required to run the
same checks in `pre_dispatch` as in `validate`.)

Regarding `post_dispatch`:
- Since there are no unsigned transactions handled by
`TransactionExtension`, `Pre` is always defined, so the first parameter
is `Self::Pre` rather than `Option<Self::Pre>`.

If you make use of `SignedExtension::validate_unsigned` or
`SignedExtension::pre_dispatch_unsigned`, then:
- Just use the regular versions of these functions instead.
- Have your logic execute in the case that the `origin` is `None`.
- Ensure your transaction creation logic creates a General Transaction
rather than a Bare Transaction; this means having to include all
`TransactionExtension`s' data.
- `ValidateUnsigned` can still be used (for now) if you need to be able
to construct transactions which contain none of the extension data,
however these will be phased out in stage 2 of the Transactions Horizon,
so you should consider moving to an extension-centric design.

## TODO

- [x] Introduce `CheckSignature` impl of `TransactionExtension` to
ensure it's possible to have crypto be done wholly in a
`TransactionExtension`.
- [x] Deprecate `SignedExtension` and move all uses in codebase to
`TransactionExtension`.
  - [x] `ChargeTransactionPayment`
  - [x] `DummyExtension`
  - [x] `ChargeAssetTxPayment` (asset-tx-payment)
  - [x] `ChargeAssetTxPayment` (asset-conversion-tx-payment)
  - [x] `CheckWeight`
  - [x] `CheckTxVersion`
  - [x] `CheckSpecVersion`
  - [x] `CheckNonce`
  - [x] `CheckNonZeroSender`
  - [x] `CheckMortality`
  - [x] `CheckGenesis`
  - [x] `CheckOnlySudoAccount`
  - [x] `WatchDummy`
  - [x] `PrevalidateAttests`
  - [x] `GenericSignedExtension`
  - [x] `SignedExtension` (chain-polkadot-bulletin)
  - [x] `RefundSignedExtensionAdapter`
- [x] Implement `fn weight` across the board.
- [ ] Go through all pre-existing extensions which assume an account
signer and explicitly handle the possibility of another kind of origin.
- [x] `CheckNonce` should probably succeed in the case of a non-account
origin.
- [x] `CheckNonZeroSender` should succeed in the case of a non-account
origin.
- [x] `ChargeTransactionPayment` and family should fail in the case of a
non-account origin.
  - [ ] 
- [x] Fix any broken tests.

---------

Signed-off-by: georgepisaltu <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Signed-off-by: Alexandru Gheorghe <[email protected]>
Signed-off-by: Andrei Sandu <[email protected]>
Co-authored-by: Nikhil Gupta <[email protected]>
Co-authored-by: georgepisaltu <[email protected]>
Co-authored-by: Chevdor <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Maciej <[email protected]>
Co-authored-by: Javier Viola <[email protected]>
Co-authored-by: Marcin S. <[email protected]>
Co-authored-by: Tsvetomir Dimitrov <[email protected]>
Co-authored-by: Javier Bullrich <[email protected]>
Co-authored-by: Koute <[email protected]>
Co-authored-by: Adrian Catangiu <[email protected]>
Co-authored-by: Vladimir Istyufeev <[email protected]>
Co-authored-by: Ross Bulat <[email protected]>
Co-authored-by: Gonçalo Pestana <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Svyatoslav Nikolsky <[email protected]>
Co-authored-by: André Silva <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: s0me0ne-unkn0wn <[email protected]>
Co-authored-by: ordian <[email protected]>
Co-authored-by: Sebastian Kunert <[email protected]>
Co-authored-by: Aaro Altonen <[email protected]>
Co-authored-by: Dmitry Markin <[email protected]>
Co-authored-by: Alexandru Vasile <[email protected]>
Co-authored-by: Alexander Samusev <[email protected]>
Co-authored-by: Julian Eager <[email protected]>
Co-authored-by: Michal Kucharczyk <[email protected]>
Co-authored-by: Davide Galassi <[email protected]>
Co-authored-by: Dónal Murray <[email protected]>
Co-authored-by: yjh <[email protected]>
Co-authored-by: Tom Mi <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Will | Paradox | ParaNodes.io <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Joshy Orndorff <[email protected]>
Co-authored-by: Joshy Orndorff <[email protected]>
Co-authored-by: PG Herveou <[email protected]>
Co-authored-by: Alexander Theißen <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Juan Girini <[email protected]>
Co-authored-by: bader y <[email protected]>
Co-authored-by: James Wilson <[email protected]>
Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: asynchronous rob <[email protected]>
Co-authored-by: Parth <[email protected]>
Co-authored-by: Andrew Jones <[email protected]>
Co-authored-by: Jonathan Udd <[email protected]>
Co-authored-by: Serban Iorga <[email protected]>
Co-authored-by: Egor_P <[email protected]>
Co-authored-by: Branislav Kontur <[email protected]>
Co-authored-by: Evgeny Snitko <[email protected]>
Co-authored-by: Just van Stam <[email protected]>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: gupnik <[email protected]>
Co-authored-by: dzmitry-lahoda <[email protected]>
Co-authored-by: zhiqiangxu <[email protected]>
Co-authored-by: Nazar Mokrynskyi <[email protected]>
Co-authored-by: Anwesh <[email protected]>
Co-authored-by: cheme <[email protected]>
Co-authored-by: Sam Johnson <[email protected]>
Co-authored-by: kianenigma <[email protected]>
Co-authored-by: Jegor Sidorenko <[email protected]>
Co-authored-by: Muharem <[email protected]>
Co-authored-by: joepetrowski <[email protected]>
Co-authored-by: Alexandru Gheorghe <[email protected]>
Co-authored-by: Gabriel Facco de Arruda <[email protected]>
Co-authored-by: Squirrel <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>
Co-authored-by: georgepisaltu <[email protected]>
Co-authored-by: command-bot <>

This PR backports Node version and `spec_version` bumps to `1.8.0` from
the latest release and orders prdoc files related to it.

If an XCM execution fails or ends with leftover assets, these will be
trapped.
In order to claim them, a custom XCM has to be executed, with the
`ClaimAsset` instruction.
However, arbitrary XCM execution is not allowed everywhere yet and XCM
itself is still not easy enough to use for users out there with trapped
assets.
This new extrinsic in `pallet-xcm` will allow these users to easily
claim their assets, without concerning themselves with writing arbitrary
XCMs.

Part of fixing https://github.com/paritytech/polkadot-sdk/issues/3495

---------

Co-authored-by: command-bot <>
Co-authored-by: Adrian Catangiu <[email protected]>

This PR removes `AssignmentProviderConfig` and uses the corresponding
ondemand parameters from `HostConfiguration` instead. Additionally
`scheduling_lookahead` and all coretime/ondemand related parameters are
extracted in a separate struct - `SchedulerParams`.

The most relevant commit from the PR is [this
one](https://github.com/paritytech/polkadot-sdk/pull/3181/commits/830bc0f5).

Fixes https://github.com/paritytech/polkadot-sdk/issues/2268

---------

Co-authored-by: command-bot <>

This MR is the merge of
https://github.com/paritytech/substrate/pull/14414 and
https://github.com/paritytech/substrate/pull/14275. It implements
[RFC#13](https://github.com/polkadot-fellows/RFCs/pull/13), closes
https://github.com/paritytech/polkadot-sdk/issues/198.

----- 

This Merge request introduces three major topicals:

1. Multi-Block-Migrations
1. New pallet `poll` hook for periodic service work
1. Replacement hooks for `on_initialize` and `on_finalize` in cases
where `poll` cannot be used

and some more general changes to FRAME.  
The changes for each topical span over multiple crates. They are listed
in topical order below.

# 1.) Multi-Block-Migrations

Multi-Block-Migrations are facilitated by creating `pallet_migrations`
and configuring `System::Config::MultiBlockMigrator` to point to it.
Executive picks this up and triggers one step of the migrations pallet
per block.
The chain is in lockdown mode for as long as an MBM is ongoing.
Executive does this by polling `MultiBlockMigrator::ongoing` and not
allowing any transaction in a block, if true.

A MBM is defined through trait `SteppedMigration`. A condensed version
looks like this:
```rust
/// A migration that can proceed in multiple steps.
pub trait SteppedMigration {
	type Cursor: FullCodec + MaxEncodedLen;
	type Identifier: FullCodec + MaxEncodedLen;

	fn id() -> Self::Identifier;

	fn max_steps() -> Option<u32>;

	fn step(
		cursor: Option<Self::Cursor>,
		meter: &mut WeightMeter,
	) -> Result<Option<Self::Cursor>, SteppedMigrationError>;
}
```

`pallet_migrations` can be configured with an aggregated tuple of these
migrations. It then starts to migrate them one-by-one on the next
runtime upgrade.
Two things are important here:
- 1. Doing another runtime upgrade while MBMs are ongoing is not a good
idea and can lead to messed up state.
- 2. **Pallet Migrations MUST BE CONFIGURED IN `System::Config`,
otherwise it is not used.**

The pallet supports an `UpgradeStatusHandler` that can be used to notify
external logic of upgrade start/finish (for example to pause XCM
dispatch).

Error recovery is very limited in the case that a migration errors or
times out (exceeds its `max_steps`). Currently the runtime dev can
decide in `FailedMigrationHandler::failed` how to handle this. One
follow-up would be to pair this with the `SafeMode` pallet and enact
safe mode when an upgrade fails, to allow governance to rescue the
chain. This is currently not possible, since governance is not
`Mandatory`.

## Runtime API

- `Core`: `initialize_block` now returns `ExtrinsicInclusionMode` to
inform the Block Author whether they can push transactions.

### Integration

Add it to your runtime implementation of `Core` and `BlockBuilder`:
```patch
diff --git a/runtime/src/lib.rs b/runtime/src/lib.rs
@@ impl_runtime_apis! {
	impl sp_block_builder::Core<Block> for Runtime {
-		fn initialize_block(header: &<Block as BlockT>::Header) {
+		fn initialize_block(header: &<Block as BlockT>::Header) -> RuntimeExecutiveMode {
			Executive::initialize_block(header)
		}

		...
	}
```

# 2.) `poll` hook

A new pallet hook is introduced: `poll`. `Poll` is intended to replace
mostly all usage of `on_initialize`.
The reason for this is that any code that can be called from
`on_initialize` cannot be migrated through an MBM. Currently there is no
way to statically check this; the implication is to use `on_initialize`
as rarely as possible.
Failing to do so can result in broken storage invariants.

The implementation of the poll hook depends on the `Runtime API` changes
that are explained above.

# 3.) Hard-Deadline callbacks

Three new callbacks are introduced and configured on `System::Config`:
`PreInherents`, `PostInherents` and `PostTransactions`.
These hooks are meant as replacement for `on_initialize` and
`on_finalize` in cases where the code that runs cannot be moved to
`poll`.
The reason for this is to make the usage of HD-code (hard deadline) more
explicit - again to prevent broken invariants by MBMs.

# 4.) FRAME (general changes)

## `frame_system` pallet

A new memorize storage item `InherentsApplied` is added. It is used by
executive to track whether inherents have already been applied.
Executive and can then execute the MBMs directly between inherents and
transactions.

The `Config` gets five new items:
- `SingleBlockMigrations` this is the new way of configuring migrations
that run in a single block. Previously they were defined as last generic
argument of `Executive`. This shift is brings all central configuration
about migrations closer into view of the developer (migrations that are
configured in `Executive` will still work for now but is deprecated).
- `MultiBlockMigrator` this can be configured to an engine that drives
MBMs. One example would be the `pallet_migrations`. Note that this is
only the engine; the exact MBMs are injected into the engine.
- `PreInherents` a callback that executes after `on_initialize` but
before inherents.
- `PostInherents` a callback that executes after all inherents ran
(including MBMs and `poll`).
- `PostTransactions` in symmetry to `PreInherents`, this one is called
before `on_finalize` but after all transactions.

A sane default is to set all of these to `()`. Example diff suitable for
any chain:
```patch
@@ impl frame_system::Config for Test {
 	type MaxConsumers = ConstU32<16>;
+	type SingleBlockMigrations = ();
+	type MultiBlockMigrator = ();
+	type PreInherents = ();
+	type PostInherents = ();
+	type PostTransactions = ();
 }
```

An overview of how the block execution now looks like is here. The same
graph is also in the rust doc.

<details><summary>Block Execution Flow</summary>
<p>

![Screenshot 2023-12-04 at 19 11
29](https://github.com/paritytech/polkadot-sdk/assets/10380170/e88a80c4-ef11-4faa-8df5-8b33a724c054)

</p>
</details> 

## Inherent Order

Moved to https://github.com/paritytech/polkadot-sdk/pull/2154

---------------


## TODO

- [ ] Check that `try-runtime` still works
- [ ] Ensure backwards compatibility with old Runtime APIs
- [x] Consume weight correctly
- [x] Cleanup

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Juan Girini <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Gavin Wood <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

## Problem
During the bumping of the `polkadot-fellows` repository to
`[email protected]`, I encountered a situation where the benchmarks
`teleport_assets` and `reserve_transfer_assets` in AssetHubKusama
started to fail. This issue arose due to a decreased ED balance for
AssetHubs introduced
[here](https://github.com/polkadot-fellows/runtimes/pull/158/files#diff-80668ff8e793b64f36a9a3ec512df5cbca4ad448c157a5d81abda1b15f35f1daR213),
and also because of a [missing CI
pipeline](https://github.com/polkadot-fellows/runtimes/issues/197) to
check the benchmarks, which went unnoticed.

These benchmarks expect the `caller` to have enough:
1. balance to transfer (BTT)
2. balance for paying delivery (BFPD).
 
So the initial balance was calculated as `ED * 100`, which seems
reasonable:
```
const ED_MULTIPLIER: u32 = 100;
let balance = existential_deposit.saturating_mul(ED_MULTIPLIER.into());`
```
The problem arises when the price for delivery is 100 times higher than
the existential deposit. In other words, when `ED * 100` does not cover
`BTT` + `BFPD`.

I check AHR/AHW/AHK/AHP and this problem has only AssetHubKusama
```
ED: 3333333
calculated price to parent delivery:  1031666634  (from xcm logs from the benchmark)
---

3333333 * 100 - BTT(3333333) - BFPD(1031666634) = −701666667
```
which results in the error;
```
2024-02-23 09:19:42 Unable to charge fee with error Module(ModuleError { index: 31, error: [17, 0, 0, 0], message: Some("FeesNotMet") })
Error: Input("Benchmark pallet_xcm::reserve_transfer_assets failed: FeesNotMet")
     
```

## Solution

The benchmarks `teleport_assets` and `reserve_transfer_assets` were
fixed by removing `ED * 100` and replacing it with `DeliveryHelper`
logic, which calculates the (almost real) price for delivery and sets it
along with the existential deposit as the initial balance for the
account used in the benchmark.


## TODO

- [ ] patch for 1.6 -
https://github.com/paritytech/polkadot-sdk/pull/3466
- [ ] patch for 1.7 -
https://github.com/paritytech/polkadot-sdk/pull/3465
- [ ] patch for 1.8 - TODO: PR

---------

Co-authored-by: Francisco Aguirre <[email protected]>

from Westend and Rococo.

---------

Co-authored-by: eskimor <[email protected]>
Co-authored-by: command-bot <>

Adds the coretime and on demand pallets to enable Coretime on Westend. 
In order for the migration to run successfully, we need the
Broker/Coretime parachain to be live.

TODO:
- [ ] Broker parachain is live
https://github.com/paritytech/polkadot-sdk/pull/3272

---------

Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <[email protected]>

This is a follow-up for `im-online` pallet removal that is cleaning up
its off-chain storage. Must be merged no earlier than #2265 is enacted.
Related: #1964

---------

Co-authored-by: Bastian Köcher <[email protected]>

This PR backports version bumps from `1.7.0` release branch and moves
related prdoc files to the appropriate folder.

The `TotalLockedValue` storage value in nomination pools pallet may get
out of sync if the staking pallet does implicit withdrawal of unlocking
chunks belonging to a bonded pool stash. This fix is based on a new
method in the `OnStakingUpdate` traits, `on_withdraw`, which allows the
nomination pools pallet to adjust the `TotalLockedValue` every time
there is an implicit or explicit withdrawal from a bonded pool's stash.

This PR also adds a migration that checks and updates the on-chain TVL
if it got out of sync due to the bug this PR fixes.

**Changes to `trait OnStakingUpdate`**

In order for staking to notify the nomination pools pallet that chunks
where withdrew, we add a new method, `on_withdraw` to the
`OnStakingUpdate` trait. The nomination pools pallet filters the
withdraws that are related to bonded pool accounts and updates the
`TotalValueLocked` accordingly.

**Others**
- Adds try-state checks to the EPM/staking e2e tests
- Adds tests for auto withdrawing in the context of nomination pools

**To-do**
- [x] check if we need a migration to fix the current `TotalValueLocked`
(run try-runtime)
- [x] migrations to fix the current on-chain TVL value 

 ✅ **Kusama**:
```
TotalValueLocked: 99.4559 kKSM
TotalValueLocked (calculated) 99.4559 kKSM
```
⚠

️ **Westend**:
```
TotalValueLocked: 18.4060 kWND
TotalValueLocked (calculated) 18.4050 kWND
```
**Polkadot**: TVL not released yet.

Closes https://github.com/paritytech/polkadot-sdk/issues/3055

---------

Co-authored-by: command-bot <>
Co-authored-by: Ross Bulat <[email protected]>
Co-authored-by: Dónal Murray <[email protected]>

Relates to: https://github.com/paritytech/polkadot-sdk/issues/3214

## TODO

- [ ] backport to the `1.7.0` release

[frame] `#[pallet::composite_enum]` improved variant count handling + removed `pallet_balances`'s `MaxHolds` config (#2657)

I started this investigation/issue based on @liamaharon

 question
[here](https://github.com/paritytech/polkadot-sdk/pull/1801#discussion_r1410452499).

## Problem

The `pallet_balances` integrity test should correctly detect that the
runtime has correct distinct `HoldReasons` variant count. I assume the
same situation exists for RuntimeFreezeReason.

It is not a critical problem, if we set `MaxHolds` with a sufficiently
large value, everything should be ok. However, in this case, the
integrity_test check becomes less useful.

**Situation for "any" runtime:**
- `HoldReason` enums from different pallets:
```rust
        /// from pallet_nis
        #[pallet::composite_enum]
	pub enum HoldReason {
		NftReceipt,
	}

        /// from pallet_preimage
        #[pallet::composite_enum]
	pub enum HoldReason {
		Preimage,
	}

        // from pallet_state-trie-migration
        #[pallet::composite_enum]
	pub enum HoldReason {
		SlashForContinueMigrate,
		SlashForMigrateCustomTop,
		SlashForMigrateCustomChild,
	}
```

- generated `RuntimeHoldReason` enum looks like:
```rust
pub enum RuntimeHoldReason {

    #[codec(index = 32u8)]
    Preimage(pallet_preimage::HoldReason),

    #[codec(index = 38u8)]
    Nis(pallet_nis::HoldReason),

    #[codec(index = 42u8)]
    StateTrieMigration(pallet_state_trie_migration::HoldReason),
}
```

- composite enum `RuntimeHoldReason` variant count is detected as `3`
- we set `type MaxHolds = ConstU32<3>`
- `pallet_balances::integrity_test` is ok with `3`(at least 3)

However, the real problem can occur in a live runtime where some
functionality might stop working. This is due to a total of 5 distinct
hold reasons (for pallets with multi-instance support, it is even more),
and not all of them can be used because of an incorrect `MaxHolds`,
which is deemed acceptable according to the `integrity_test`:
  ```
  // pseudo-code - if we try to call all of these:

T::Currency::hold(&pallet_nis::HoldReason::NftReceipt.into(),
&nft_owner, deposit)?;
T::Currency::hold(&pallet_preimage::HoldReason::Preimage.into(),
&nft_owner, deposit)?;

T::Currency::hold(&pallet_state_trie_migration::HoldReason::SlashForContinueMigrate.into(),
&nft_owner, deposit)?;

  // With `type MaxHolds = ConstU32<3>` these two will fail

T::Currency::hold(&pallet_state_trie_migration::HoldReason::SlashForMigrateCustomTop.into(),
&nft_owner, deposit)?;

T::Currency::hold(&pallet_state_trie_migration::HoldReason::SlashForMigrateCustomChild.into(),
&nft_owner, deposit)?;
  ```  


## Solutions

A macro `#[pallet::*]` expansion is extended of `VariantCount`
implementation for the `#[pallet::composite_enum]` enum type. This
expansion generates the `VariantCount` implementation for pallets'
`HoldReason`, `FreezeReason`, `LockId`, and `SlashReason`. Enum variants
must be plain enum values without fields to ensure a deterministic
count.

The composite runtime enum, `RuntimeHoldReason` and
`RuntimeFreezeReason`, now sets `VariantCount::VARIANT_COUNT` as the sum
of pallets' enum `VariantCount::VARIANT_COUNT`:
```rust
#[frame_support::pallet(dev_mode)]
mod module_single_instance {

	#[pallet::composite_enum]
	pub enum HoldReason {
		ModuleSingleInstanceReason1,
		ModuleSingleInstanceReason2,
	}
...
}

#[frame_support::pallet(dev_mode)]
mod module_multi_instance {

	#[pallet::composite_enum]
	pub enum HoldReason<I: 'static = ()> {
		ModuleMultiInstanceReason1,
		ModuleMultiInstanceReason2,
		ModuleMultiInstanceReason3,
	}
...
}


impl self::sp_api_hidden_includes_construct_runtime::hidden_include::traits::VariantCount
    for RuntimeHoldReason
{
    const VARIANT_COUNT: u32 = 0
        + module_single_instance::HoldReason::VARIANT_COUNT
        + module_multi_instance::HoldReason::<module_multi_instance::Instance1>::VARIANT_COUNT
        + module_multi_instance::HoldReason::<module_multi_instance::Instance2>::VARIANT_COUNT
        + module_multi_instance::HoldReason::<module_multi_instance::Instance3>::VARIANT_COUNT;
}
```

In addition, `MaxHolds` is removed (as suggested
[here](https://github.com/paritytech/polkadot-sdk/pull/2657#discussion_r1443324573))
from `pallet_balances`, and its `Holds` are now bounded to
`RuntimeHoldReason::VARIANT_COUNT`. Therefore, there is no need to let
the runtime specify `MaxHolds`.


## For reviewers

Relevant changes can be found here:
- `substrate/frame/support/procedural/src/lib.rs` 
-  `substrate/frame/support/procedural/src/pallet/parse/composite.rs`
-  `substrate/frame/support/procedural/src/pallet/expand/composite.rs`
-
`substrate/frame/support/procedural/src/construct_runtime/expand/composite_helper.rs`
-
`substrate/frame/support/procedural/src/construct_runtime/expand/hold_reason.rs`
-
`substrate/frame/support/procedural/src/construct_runtime/expand/freeze_reason.rs`
- `substrate/frame/support/src/traits/misc.rs`

And the rest of the files is just about removed `MaxHolds` from
`pallet_balances`

## Next steps

Do the same for `MaxFreezes`
https://github.com/paritytech/polkadot-sdk/issues/2997.

---------

Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Dónal Murray <[email protected]>
Co-authored-by: gupnik <[email protected]>

Moves `pallet_mmr` back behind `pallet_session` to address
polkadot-fellows/runtimes#160.

Opening draft for CI - should be merged or closed depending on outcome
of w3f/polkadot-spec#718.

---------

Co-authored-by: Adrian Catangiu <[email protected]>

Moved from: https://github.com/paritytech/polkadot/pull/6951

closes https://github.com/paritytech/polkadot-sdk/issues/490

- [x] update cumulus

--- 
This PR introduces transactional processing of certain xcm instructions.
For the list of instructions checkout
https://github.com/paritytech/polkadot-sdk/issues/490. The transactional
processing is implemented as an xcm-executor config item. The two
implementations in this PR are `FrameTransactionalProcessor` and `()`.
The `()` implementation does no transactional processing. Each
implementation of the `ProcessTransaction` trait has an
`IS_TRANSACTIONAL` const that tells the XCVM if transactional processing
is actually implemented. If Transactional processing is implemented,
changes to touched registers should also be rolled back to prevent
inconsistencies.


Note for reviewers:
Check out the following safety assumption:
https://github.com/paritytech/polkadot-sdk/pull/1222/files#diff-4effad7d8c1c9de19fd27e18661cbf2128c8718f3b2420a27d2f816e0749ea53R30

---------

Co-authored-by: Keith Yeung <[email protected]>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: command-bot <>