`execute` and `send` try to decode the xcm in the parameters before
reaching the filter line.
The new extrinsics decode only after the filter line.
These should be used instead of the old ones.

## TODO
- [x] Tests
- [x] Generate weights
- [x] Deprecation issue ->
https://github.com/paritytech/polkadot-sdk/issues/3771
- [x] PRDoc
- [x] Handle error in pallet-contracts

This would make writing XCMs in PJS Apps more difficult, but here's the
fix for that: https://github.com/polkadot-js/apps/pull/10350.
Already deployed! https://polkadot.js.org/apps/#/utilities/xcm

Supersedes https://github.com/paritytech/polkadot-sdk/pull/1798/

---------

Co-authored-by: PG Herveou <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: Adrian Catangiu <[email protected]>

The PR provides API for obtaining:
- the weight required to execute an XCM message,
- a list of acceptable `AssetId`s for message execution payment,
- the cost of the weight in the specified acceptable `AssetId`.

It is meant to address an issue where one has to guess how much fee to
pay for execution. Also, at the moment, a client has to guess which
assets are acceptable for fee execution payment.
See the related issue
https://github.com/paritytech/polkadot-sdk/issues/690.
With this API, a client is supposed to query the list of the supported
asset IDs (in the XCM version format the client understands), weigh the
XCM program the client wants to execute and convert the weight into one
of the acceptable assets. Note that the client is supposed to know what
program will be executed on what chains. However, having a small
companion JS library for the pallet-xcm and xtokens should be enough to
determine what XCM programs will be executed and where (since these
pallets compose a known small set of programs).
```Rust
pub trait XcmPaymentApi<Call>
	where
		Call: Codec,
	{
		/// Returns a list of acceptable payment assets.
		///
		/// # Arguments
		///
		/// * `xcm_version`: Version.
		fn query_acceptable_payment_assets(xcm_version: Version) -> Result<Vec<VersionedAssetId>, Error>;
		/// Returns a weight needed to execute a XCM.
		///
		/// # Arguments
		///
		/// * `message`: `VersionedXcm`.
		fn query_xcm_weight(message: VersionedXcm<Call>) -> Result<Weight, Error>;
		/// Converts a weight into a fee for the specified `AssetId`.
		///
		/// # Arguments
		///
		/// * `weight`: convertible `Weight`.
		/// * `asset`: `VersionedAssetId`.
		fn query_weight_to_asset_fee(weight: Weight, asset: VersionedAssetId) -> Result<u128, Error>;
		/// Get delivery fees for sending a specific `message` to a `destination`.
		/// These always come in a specific asset, defined by the chain.
		///
		/// # Arguments
		/// * `message`: The message that'll be sent, necessary because most delivery fees are based on the
		///   size of the message.
		/// * `destination`: The destination to send the message to. Different destinations may use
		///   different senders that charge different fees.
		fn query_delivery_fees(destination: VersionedLocation, message: VersionedXcm<()>) -> Result<VersionedAssets, Error>;
	}
```
An
[example](https://gist.github.com/PraetorP/4bc323ff85401abe253897ba990ec29d)
of a client side code.

---------

Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Adrian Catangiu <[email protected]>
Co-authored-by: Daniel Shiposha <[email protected]>

**Update:** Pushed additional changes based on the review comments.

**This pull request fixes various spelling mistakes in this
repository.**

Most of the changes are contained in the first **3** commits:

- `Fix spelling mistakes in comments and docs`

- `Fix spelling mistakes in test names`

- `Fix spelling mistakes in error messages, panic messages, logs and
tracing`

Other source code spelling mistakes are separated into individual
commits for easier reviewing:

- `Fix the spelling of 'authority'`

- `Fix the spelling of 'REASONABLE_HEADERS_IN_JUSTIFICATION_ANCESTRY'`

- `Fix the spelling of 'prev_enqueud_messages'`

- `Fix the spelling of 'endpoint'`

- `Fix the spelling of 'children'`

- `Fix the spelling of 'PenpalSiblingSovereignAccount'`

- `Fix the spelling of 'PenpalSudoAccount'`

- `Fix the spelling of 'insufficient'`

- `Fix the spelling of 'PalletXcmExtrinsicsBenchmark'`

- `Fix the spelling of 'subtracted'`

- `Fix the spelling of 'CandidatePendingAvailability'`

- `Fix the spelling of 'exclusive'`

- `Fix the spelling of 'until'`

- `Fix the spelling of 'discriminator'`

- `Fix the spelling of 'nonexistent'`

- `Fix the spelling of 'subsystem'`

- `Fix the spelling of 'indices'`

- `Fix the spelling of 'committed'`

- `Fix the spelling of 'topology'`

- `Fix the spelling of 'response'`

- `Fix the spelling of 'beneficiary'`

- `Fix the spelling of 'formatted'`

- `Fix the spelling of 'UNKNOWN_PROOF_REQUEST'`

- `Fix the spelling of 'succeeded'`

- `Fix the spelling of 'reopened'`

- `Fix the spelling of 'proposer'`

- `Fix the spelling of 'InstantiationNonce'`

- `Fix the spelling of 'depositor'`

- `Fix the spelling of 'expiration'`

- `Fix the spelling of 'phantom'`

- `Fix the spelling of 'AggregatedKeyValue'`

- `Fix the spelling of 'randomness'`

- `Fix the spelling of 'defendant'`

- `Fix the spelling of 'AquaticMammal'`

- `Fix the spelling of 'transactions'`

- `Fix the spelling of 'PassingTracingSubscriber'`

- `Fix the spelling of 'TxSignaturePayload'`

- `Fix the spelling of 'versioning'`

- `Fix the spelling of 'descendant'`

- `Fix the spelling of 'overridden'`

- `Fix the spelling of 'network'`

Let me know if this structure is adequate.

**Note:** The usage of the words `Merkle`, `Merkelize`, `Merklization`,
`Merkelization`, `Merkleization`, is somewhat inconsistent but I left it
as it is.

~~**Note:** In some places the term `Receival` is used to refer to
message reception, IMO `Reception` is the correct word here, but I left
it as it is.~~

~~**Note:** In some places the term `Overlayed` is used instead of the
more acceptable version `Overlaid` but I also left it as it is.~~

~~**Note:** In some places the term `Applyable` is used instead of the
correct version `Applicable` but I also left it as it is.~~

**Note:** Some usage of British vs American english e.g. `judgement` vs
`judgment`, `initialise` vs `initialize`, `optimise` vs `optimize` etc.
are both present in different places, but I suppose that's
understandable given the number of contributors.

~~**Note:** There is a spelling mistake in `.github/CODEOWNERS` but it
triggers errors in CI when I make changes to it, so I left it as it
is.~~

If an XCM execution fails or ends with leftover assets, these will be
trapped.
In order to claim them, a custom XCM has to be executed, with the
`ClaimAsset` instruction.
However, arbitrary XCM execution is not allowed everywhere yet and XCM
itself is still not easy enough to use for users out there with trapped
assets.
This new extrinsic in `pallet-xcm` will allow these users to easily
claim their assets, without concerning themselves with writing arbitrary
XCMs.

Part of fixing https://github.com/paritytech/polkadot-sdk/issues/3495

---------

Co-authored-by: command-bot <>
Co-authored-by: Adrian Catangiu <[email protected]>

Adds the coretime and on demand pallets to enable Coretime on Westend. 
In order for the migration to run successfully, we need the
Broker/Coretime parachain to be live.

TODO:
- [ ] Broker parachain is live
https://github.com/paritytech/polkadot-sdk/pull/3272

---------

Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <[email protected]>

Some more work regarding XCMv4. Two limits from v3 were not transferred
over, those are:
- The instructions limit
- The number of assets limit
Both of these are now in v4.

For some reason `AssetInstance` increased in size, don't know why CI
didn't catch that before.

---------

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: command-bot <>

Continues the work of https://github.com/paritytech/polkadot/pull/7312.
`MultiLocation` was modified to implement serialize in no-std in that
PR, but not the assets types.
This PR does the same for assets, and this also allows to inject them in
pallets that require them in genesis.

Moved from: https://github.com/paritytech/polkadot/pull/6951

closes https://github.com/paritytech/polkadot-sdk/issues/490

- [x] update cumulus

--- 
This PR introduces transactional processing of certain xcm instructions.
For the list of instructions checkout
https://github.com/paritytech/polkadot-sdk/issues/490. The transactional
processing is implemented as an xcm-executor config item. The two
implementations in this PR are `FrameTransactionalProcessor` and `()`.
The `()` implementation does no transactional processing. Each
implementation of the `ProcessTransaction` trait has an
`IS_TRANSACTIONAL` const that tells the XCVM if transactional processing
is actually implemented. If Transactional processing is implemented,
changes to touched registers should also be rolled back to prevent
inconsistencies.


Note for reviewers:
Check out the following safety assumption:
https://github.com/paritytech/polkadot-sdk/pull/1222/files#diff-4effad7d8c1c9de19fd27e18661cbf2128c8718f3b2420a27d2f816e0749ea53R30

---------

Co-authored-by: Keith Yeung <[email protected]>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: command-bot <>

# Note for reviewer

Most changes are just syntax changes necessary for the new version.
Most important files should be the ones under the `xcm` folder.

# Description 

Added XCMv4.

## Removed `Multi` prefix
The following types have been renamed:
- MultiLocation -> Location
- MultiAsset -> Asset
- MultiAssets -> Assets
- InteriorMultiLocation -> InteriorLocation
- MultiAssetFilter -> AssetFilter
- VersionedMultiAsset -> VersionedAsset
- WildMultiAsset -> WildAsset
- VersionedMultiLocation -> VersionedLocation

In order to fix a name conflict, the `Assets` in `xcm-executor` were
renamed to `HoldingAssets`, as they represent assets in holding.

## Removed `Abstract` asset id

It was not being used anywhere and this simplifies the code.

Now assets are just constructed as follows:

```rust
let asset: Asset = (AssetId(Location::new(1, Here)), 100u128).into();
```

No need for specifying `Concrete` anymore.

## Outcome is now a named fields struct

Instead of

```rust
pub enum Outcome {
  Complete(Weight),
  Incomplete(Weight, Error),
  Error(Error),
}
```

we now have

```rust
pub enum Outcome {
  Complete { used: Weight },
  Incomplete { used: Weight, error: Error },
  Error { error: Error },
}
```

## Added Reanchorable trait

Now both locations and assets implement this trait, making it easier to
reanchor both.

## New syntax for building locations and junctions

Now junctions are built using the following methods:

```rust
let location = Location {
    parents: 1,
    interior: [Parachain(1000), PalletInstance(50), GeneralIndex(1984)].into()
};
```

or

```rust
let location = Location::new(1, [Parachain(1000), PalletInstance(50), GeneralIndex(1984)]);
```

And they are matched like so:

```rust
match location.unpack() {
  (1, [Parachain(id)]) => ...
  (0, Here) => ...,
  (1, [_]) => ...,
}
```

This syntax is mandatory in v4, and has been also implemented for v2 and
v3 for easier migration.

This was needed to make all sizes smaller.

# TODO
- [x] Scaffold v4
- [x] Port github.com/paritytech/polkadot/pull/7236
- [x] Remove `Multi` prefix
- [x] Remove `Abstract` asset id

---------

Co-authored-by: command-bot <>
Co-authored-by: Keith Yeung <[email protected]>

Adds more logging to the XCM execution for better debugging.

## Summary

This pull request proposes a solution for improved control of the
versioned XCM flow over the bridge (across different consensus chains)
and resolves the situation where the sending chain/consensus has already
migrated to a higher XCM version than the receiving chain/consensus.

## Problem/Motivation

The current flow over the bridge involves a transfer from AssetHubRococo
(AHR) to BridgeHubRococo (BHR) to BridgeHubWestend (BHW) and finally to
AssetHubWestend (AHW), beginning with a reserve-backed transfer on AHR.

In this process:
1. AHR sends XCM `ExportMessage` through `XcmpQueue`, incorporating XCM
version checks using the `WrapVersion` feature, influenced by
`pallet_xcm::SupportedVersion` (managed by
`pallet_xcm::force_xcm_version` or version discovery).

2. BHR handles the `ExportMessage` instruction, utilizing the latest XCM
version. The `HaulBlobExporter` converts the inner XCM to
[`VersionedXcm::from`](https://github.com/paritytech/polkadot-sdk/blob/63ac2471/polkadot/xcm/xcm-builder/src/universal_exports.rs#L465-L467),
also using the latest XCM version.

However, challenges arise:
- Incompatibility when BHW uses a different version than BHR. For
instance, if BHR migrates to **XCMv4** while BHW remains on **XCMv3**,
BHR's `VersionedXcm::from` uses `VersionedXcm::V4` variant, causing
encoding issues for BHW.
  ```
	/// Just a simulation of possible error, which could happen on BHW
	/// (this code is based on actual master without XCMv4)
	let encoded = hex_literal::hex!("0400");
	println!("{:?}", VersionedXcm::<()>::decode(&mut &encoded[..]));

Err(Error { cause: None, desc: "Could not decode `VersionedXcm`, variant
doesn't exist" })
  ``` 
- Similar compatibility issues exist between AHR and AHW.

## Solution

This pull request introduces the following solutions:

1. **New trait `CheckVersion`** - added to the `xcm` module and exposing
`pallet_xcm::SupportedVersion`. This enhancement allows checking the
actual XCM version for desired destinations outside of the `pallet_xcm`
module.

2. **Version Check in `HaulBlobExporter`** uses `CheckVersion` to check
known/configured destination versions, ensuring compatibility. For
example, in the scenario mentioned, BHR can store the version `3` for
BHW. If BHR is on XCMv4, it will attempt to downgrade the message to
version `3` instead of using the latest version `4`.

3. **Version Check in `pallet-xcm-bridge-hub-router`** - this check
ensures compatibility with the real destination's XCM version,
preventing the unnecessary sending of messages to the local bridge hub
if versions are incompatible.

These additions aim to improve the control and compatibility of XCM
flows over the bridge and addressing issues related to version
mismatches.

## Possible alternative solution

_(More investigation is needed, and at the very least, it should extend
to XCMv4/5. If this proves to be a viable option, I can open an RFC for
XCM.)._

Add the `XcmVersion` attribute to the `ExportMessage` so that the
sending chain can determine, based on what is stored in
`pallet_xcm::SupportedVersion`, the version the destination is using.
This way, we may not need to handle the version in `HaulBlobExporter`.

```
ExportMessage {
	network: NetworkId,
	destination: InteriorMultiLocation,
	xcm: Xcm<()>
	destination_xcm_version: Version, // <- new attritbute
},
```

```
pub trait ExportXcm {
        fn validate(
		network: NetworkId,
		channel: u32,
		universal_source: &mut Option<InteriorMultiLocation>,
		destination: &mut Option<InteriorMultiLocation>,
		message: &mut Option<Xcm<()>>,
                destination_xcm_version: Version, , // <- new attritbute
	) -> SendResult<Self::Ticket>;
```

## Future Directions

This PR does not fix version discovery over bridge, further
investigation will be conducted here:
https://github.com/paritytech/polkadot-sdk/issues/2417.

## TODO

- [x] `pallet_xcm` mock for tests uses hard-coded XCM version `2` -
change to 3 or lastest?
- [x] fix `pallet-xcm-bridge-hub-router`
- [x] fix HaulBlobExporter with version determination
[here](https://github.com/paritytech/polkadot-sdk/blob/2183669d/polkadot/xcm/xcm-builder/src/universal_exports.rs#L465)
- [x] add unit-tests to the runtimes
- [x] run benchmarks for `ExportMessage`
- [x] extend local run scripts about `force_xcm_version(dest, version)`
- [ ] when merged, prepare governance calls for Rococo/Westend
- [ ] add PRDoc

Part of: https://github.com/paritytech/parity-bridges-common/issues/2719

---------

Co-authored-by: command-bot <>

# Description

- What does this PR do? Allows to generate JSON schema for subset of XCM
in std builds
- Why are these changes needed? To support XCM messages in CosmWasm
contracts which require Schemars to generate contract clients
- How were these changes implemented and what do they affect? We will
use schema feature flag to build XCM pallet with JSON schema enabled

# Checklist

- [x] My PR includes a detailed description as outlined in the
"Description" section above
- [x] My PR follows the [labeling requirements](CONTRIBUTING.md#Process)
of this project (at minimum one label for `T`
  required)
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works (if applicable)
- [x] If this PR alters any external APIs or interfaces used by
Polkadot, the corresponding Polkadot PR is ready as well
  as the corresponding Cumulus PR (optional)

We're going to bridge Polkadot Bridge Hub with [Polkadot Bulletin
chain](https://github.com/zdave-parity/polkadot-bulletin-chain) soon
(and Rococo Bridge Hub with 1:1 copy of Polkadot Bulletin chain even
sooner), so we need a variant for that chain in `NetworkId`. As
suggested, I'm adding a new variant for it to the `NetworkId` (we may
have used `ByGenesis(_)`, but decision was made to have a dedicated
variant for that).

On extrinsics/call, ensure local XCM execution is complete/successful.
Otherwise, fail the extrinsic so that state changes don't get committed
to the db.

Added regression tests that fail without the fix.

fixes #2237

---------

Co-authored-by: Adrian Catangiu <[email protected]>

Adding on top of the new builder pattern for creating XCM programs, I'm
adding some more APIs:

```rust
let paying_fees: Xcm<()> = Xcm::builder() // Only allow paying for fees
  .withdraw_asset() // First instruction has to load the holding register
  .buy_execution() // Second instruction has to be `buy_execution`
  .build();

let paying_fees_invalid: Xcm<()> = Xcm::builder()
  .withdraw_asset()
  .build(); // Invalid, need to pay for fees

let not_paying_fees: Xcm<()> = Xcm::builder_unpaid()
  .unpaid_execution() // Needed
  .withdraw_asset()
  .deposit_asset()
  .build();

let all_goes: Xcm<()> = Xcm::builder_unsafe() // You can do anything
  .withdraw_asset()
  .deposit_asset()
  .build();
```

The invalid bits are because the methods don't even exist on the types
that you'd want to call them on.

---------

Co-authored-by: command-bot <>

## Motivation

`pallet-xcm` is the main user-facing interface for XCM functionality,
including assets manipulation functions like `teleportAssets()` and
`reserve_transfer_assets()` calls.

While `teleportAsset()` works both ways, `reserve_transfer_assets()`
works only for sending reserve-based assets to a remote destination and
beneficiary when the reserve is the _local chain_.

## Solution

This PR enhances `pallet_xcm::(limited_)reserve_withdraw_assets` to
support transfers when reserves are other chains.
This will allow complete, **bi-directional** reserve-based asset
transfers user stories using `pallet-xcm`.

Enables following scenarios:
- transferring assets with local reserve (was previously supported iff
asset used as fee also had local reserve - now it works in all cases),
- transferring assets with reserve on destination,
- transferring assets with reserve on remote/third-party chain (iff
assets and fees have same remote reserve),
- transferring assets with reserve different than the reserve of the
asset to be used as fees - meaning can be used to transfer random asset
with local/dest reserve while using DOT for fees on all involved chains,
even if DOT local/dest reserve doesn't match asset reserve,
- transferring assets with any type of local/dest reserve while using
fees which can be teleported between involved chains.

All of the above is done by pallet inner logic without the user having
to specify which scenario/reserves/teleports/etc. The correct scenario
and corresponding XCM programs are identified, and respectively, built
automatically based on runtime configuration of trusted teleporters and
trusted reserves.

#### Current limitations:
- while `fees` and "non-fee" `assets` CAN have different reserves (or
fees CAN be teleported), the remaining "non-fee" `assets` CANNOT, among
themselves, have different reserve locations (this is also implicitly
enforced by `MAX_ASSETS_FOR_TRANSFER=2`, but this can be safely
increased in the future).
- `fees` and "non-fee" `assets` CANNOT have **different remote**
reserves (this could also be supported in the future, but adds even more
complexity while possibly not being worth it - we'll see what the future
holds).

Fixes https://github.com/paritytech/polkadot-sdk/issues/1584
Fixes https://github.com/paritytech/polkadot-sdk/issues/2055

---------

Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Branislav Kontur <[email protected]>

PalletInfo fields were private, preventing a user from actually using
the QueryPallet instruction in a meaningful way since they couldn't read
the received data.

Added a proc macro to be able to write XCMs using the builder pattern.
This means we go from having to do this:

```rust
let message: Xcm<()> = Xcm(vec![
  WithdrawAsset(assets),
  BuyExecution { fees: asset, weight_limit: Unlimited },
  DepositAsset { assets, beneficiary },
]);
```

to this:

```rust
let message: Xcm<()> = Xcm::builder()
  .withdraw_asset(assets)
  .buy_execution(asset, Unlimited),
  .deposit_asset(assets, beneficiary)
  .build();
```

---------

Co-authored-by: Keith Yeung <[email protected]>
Co-authored-by: command-bot <>

(imported from https://github.com/paritytech/cumulus/pull/2157)

## Changes

This MR refactores the XCMP, Parachains System and DMP pallets to use
the [MessageQueue](https://github.com/paritytech/substrate/pull/12485)
for delayed execution of incoming messages. The DMP pallet is entirely
replaced by the MQ and thereby removed. This allows for PoV-bounded
execution and resolves a number of issues that stem from the current
work-around.

All System Parachains adopt this change.  
The most important changes are in `primitives/core/src/lib.rs`,
`parachains/common/src/process_xcm_message.rs`,
`pallets/parachain-system/src/lib.rs`, `pallets/xcmp-queue/src/lib.rs`
and the runtime configs.

### DMP Queue Pallet

The pallet got removed and its logic refactored into parachain-system.
Overweight message management can be done directly through the MQ
pallet.

Final undeployment migrations are provided by
`cumulus_pallet_dmp_queue::UndeployDmpQueue` and `DeleteDmpQueue` that
can be configured with an aux config trait like:

```rust
parameter_types! {
	pub const DmpQueuePalletName: &'static str = \"DmpQueue\" < CHANGE ME;
	pub const RelayOrigin: AggregateMessageOrigin = AggregateMessageOrigin::Parent;
}

impl cumulus_pallet_dmp_queue::MigrationConfig for Runtime {
	type PalletName = DmpQueuePalletName;
	type DmpHandler = frame_support::traits::EnqueueWithOrigin<MessageQueue, RelayOrigin>;
	type DbWeight = <Runtime as frame_system::Config>::DbWeight;
}

// And adding them to your Migrations tuple:
pub type Migrations = (
	...
	cumulus_pallet_dmp_queue::UndeployDmpQueue<Runtime>,
	cumulus_pallet_dmp_queue::DeleteDmpQueue<Runtime>,
);
```

### XCMP Queue pallet

Removed all dispatch queue functionality. Incoming XCMP messages are now
either: Immediately handled if they are Signals, enqueued into the MQ
pallet otherwise.

New config items for the XCMP queue pallet:
```rust
/// The actual queue implementation that retains the messages for later processing.
type XcmpQueue: EnqueueMessage<ParaId>;

/// How a XCM over HRMP from a sibling parachain should be processed.
type XcmpProcessor: ProcessMessage<Origin = ParaId>;

/// The maximal number of suspended XCMP channels at the same time.
#[pallet::constant]
type MaxInboundSuspended: Get<u32>;
```

How to configure those:

```rust
// Use the MessageQueue pallet to store messages for later processing. The `TransformOrigin` is needed since
// the MQ pallet itself operators on `AggregateMessageOrigin` but we want to enqueue `ParaId`s.
type XcmpQueue = TransformOrigin<MessageQueue, AggregateMessageOrigin, ParaId, ParaIdToSibling>;

// Process XCMP messages from siblings. This is type-safe to only accept `ParaId`s. They will be dispatched
// with origin `Junction::Sibling(…)`.
type XcmpProcessor = ProcessFromSibling<
	ProcessXcmMessage<
		AggregateMessageOrigin,
		xcm_executor::XcmExecutor<xcm_config::XcmConfig>,
		RuntimeCall,
	>,
>;

// Not really important what to choose here. Just something larger than the maximal number of channels.
type MaxInboundSuspended = sp_core::ConstU32<1_000>;
```

The `InboundXcmpStatus` storage item was replaced by
`InboundXcmpSuspended` since it now only tracks inbound queue suspension
and no message indices anymore.

Now only sends the most recent channel `Signals`, as all prio ones are
out-dated anyway.

### Parachain System pallet

For `DMP` messages instead of forwarding them to the `DMP` pallet, it
now pushes them to the configured `DmpQueue`. The message processing
which was triggered in `set_validation_data` is now being done by the MQ
pallet `on_initialize`.

XCMP messages are still handed off to the `XcmpMessageHandler`
(XCMP-Queue pallet) - no change here.

New config items for the parachain system pallet:
```rust
/// Queues inbound downward messages for delayed processing. 
///
/// Analogous to the `XcmpQueue` of the XCMP queue pallet.
type DmpQueue: EnqueueMessage<AggregateMessageOrigin>;
``` 

How to configure:
```rust
/// Use the MQ pallet to store DMP messages for delayed processing.
type DmpQueue = MessageQueue;
``` 

## Message Flow

The flow of messages on the parachain side. Messages come in from the
left via the `Validation Data` and finally end up at the `Xcm Executor`
on the right.

![Untitled
(1)](https://github.com/paritytech/cumulus/assets/10380170/6cf8b377-88c9-4aed-96df-baace266e04d)

## Further changes

- Bumped the default suspension, drop and resume thresholds in
`QueueConfigData::default()`.
- `XcmpQueue::{suspend_xcm_execution, resume_xcm_execution}` errors when
they would be a noop.
- Properly validate the `QueueConfigData` before setting it.
- Marked weight files as auto-generated so they wont auto-expand in the
MR files view.
- Move the `hypothetical` asserts to `frame_support` under the name
`experimental_hypothetically`

Questions:
- [ ] What about the ugly `#[cfg(feature = \"runtime-benchmarks\")]` in
the runtimes? Not sure how to best fix. Just having them like this makes
tests fail that rely on the real message processor when the feature is
enabled.
- [ ] Need a good weight for `MessageQueueServiceWeight`. The scheduler
already takes 80% so I put it to 10% but that is quite low.

TODO:
- [x] Remove c&p code after
https://github.com/paritytech/polkadot/pull/6271
- [x] Use `HandleMessage` once it is public in Substrate
- [x] fix `runtime-benchmarks` feature
https://github.com/paritytech/polkadot/pull/6966
- [x] Benchmarks
- [x] Tests
- [ ] Migrate `InboundXcmpStatus` to `InboundXcmpSuspended`
- [x] Possibly cleanup Migrations (DMP+XCMP)
- [x] optional: create `TransformProcessMessageOrigin` in Substrate and
replace `ProcessFromSibling`
- [ ] Rerun weights on ref HW

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: command-bot <>

Fixes https://github.com/paritytech/polkadot-sdk/issues/2123

The `xcm` crate was renamed to `staging-xcm` to be able to publish it to
crates.io as someone as squatted `xcm`. The problem with this rename is
that the `TypeInfo` includes the crate name which ultimately lands in
the metadata. The metadata is consumed by downstream users like
`polkadot-js` or people building on top of `polkadot-js`. These people
are using the entire `path` to find the type in the type registry. Thus,
their code would break as the type path would now be [`staging_xcm`,
`VersionedXcm`] instead of [`xcm`, `VersionedXcm`]. This pull request
fixes this by renaming the path segment `staging_xcm` to `xcm`.

This requires: https://github.com/paritytech/scale-info/pull/197

---------

Co-authored-by: Francisco Aguirre <[email protected]>

Combination of paritytech/polkadot#7005, its addon PR
paritytech/polkadot#7585 and its companion paritytech/cumulus#2433.

This PR introduces a new XcmFeesToAccount struct which implements the
`FeeManager` trait, and assigns this struct as the `FeeManager` in the
XCM config for all runtimes.

The struct simply deposits all fees handled by the XCM executor to a
specified account. In all runtimes, the specified account is configured
as the treasury account.

XCM __delivery__ fees are now being introduced (unless the root origin
is sending a message to a system parachain on behalf of the originating
chain).

# Note for reviewers

Most file changes are tests that had to be modified to account for the
new fees.
Main changes are in:
- cumulus/pallets/xcmp-queue/src/lib.rs <- To make it track the delivery
fees exponential factor
- polkadot/xcm/xcm-builder/src/fee_handling.rs <- Added. Has the
FeeManager implementation
- All runtime xcm_config files <- To add the FeeManager to the XCM
configuration

# Important note

After this change, instructions that create and send a new XCM (Query*,
Report*, ExportMessage, InitiateReserveWithdraw, InitiateTeleport,
DepositReserveAsset, TransferReserveAsset, LockAsset and RequestUnlock)
will require the corresponding origin account in the origin register to
pay for transport delivery fees, and the onward message will fail to be
sent if the origin account does not have the required amount. This
delivery fee is on top of what we already collect as tx fees in
pallet-xcm and XCM BuyExecution fees!

Wallet UIs that want to expose the new delivery fee can do so using the
formula:

```
delivery_fee_factor * (base_fee + encoded_msg_len * per_byte_fee)
```

where the delivery fee factor can be obtained from the corresponding
pallet based on which transport you are using (UMP, HRMP or bridges),
the base fee is a constant, the encoded message length from the message
itself and the per byte fee is the same as the configured per byte fee
for txs (i.e. `TransactionByteFee`).

---------

Co-authored-by: Branislav Kontur <[email protected]>
Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: Giles Cope <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>

* Enforce a decoding limit in MultiAssets

* ".git/.scripts/commands/fmt/fmt.sh"

* Update polkadot/xcm/src/v3/multiasset.rs

Co-authored-by: Keith Yeung <[email protected]>

* Just use a BoundedVec

* Conflicts

---------

Co-authored-by: command-bot <>
Co-authored-by: Keith Yeung <[email protected]>

* Rename `polkadot-parachain` to `polkadot-parachain-primitives`

While doing this it also fixes some last `rustdoc` issues and fixes
another Cargo warning related to `pallet-paged-list`.

* Fix compilation

* ".git/.scripts/commands/fmt/fmt.sh"

* Fix XCM docs

---------

Co-authored-by: command-bot <>

* Add environmental variable to track decoded instructions

* Fix doc tests

* Fix manifest formatting

* ".git/.scripts/commands/fmt/fmt.sh"

* Add one more test

* Add SetAppendix in test

---------

Co-authored-by: command-bot <>

* Put `GetWeight` where it belongs

* add GetWeight to v2

* Re-export unchanged trait

---------

Co-authored-by: Just van Stam <[email protected]>
Co-authored-by: Keith Yeung <[email protected]>

* Add instruction limit when decoding XCMs

* Make the instruction limit a constant

* Use vec for buffer

* ".git/.scripts/commands/fmt/fmt.sh"

* Go back on std

* Use BoundedVec's Decode implementation

* ".git/.scripts/commands/fmt/fmt.sh"

* Use an actual BoundedVec to decode XCMs

* Change comment location

* ".git/.scripts/commands/fmt/fmt.sh"

* Remove unused imports

* ".git/.scripts/commands/fmt/fmt.sh"

---------

Co-authored-by: command-bot <>

* Polkadot gets topics

* Formatting

* Fixes

---------

Co-authored-by: Keith Yeung <[email protected]>

Co-authored-by: parity-processbot <>

* Use same rustfmt.toml as Substrate

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* format format file

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Format with new config

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add Substrate Clippy config

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Print Clippy version in CI

Otherwise its difficult to reproduce locally.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Make fmt happy

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update node/core/pvf/src/error.rs

Co-authored-by: Tsvetomir Dimitrov <[email protected]>

* Update node/core/pvf/src/error.rs

Co-authored-by: Tsvetomir Dimitrov <[email protected]>

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Tsvetomir Dimitrov <[email protected]>

* Document non-uniqueness of SetTopic IDs

* More comments on WithUniqueTopic

[xcm] `GlobalConsensusConvertsFor` for remote relay chain (based on pevious GlobalConsensusParachainConvertsFor) (#7517)

* [xcm] `GlobalConsensusConvertsFor` for remote relay chain (based on previous GlobalConsensusParachainConvertsFor)

* Typo

* PR fix (constants in test)

* Re-export of `GlobalConsensusConvertsFor`

* assert to panic

* Update xcm/src/v3/multiasset.rs

Co-authored-by: joe petrowski <[email protected]>

* Update xcm/xcm-builder/src/location_conversion.rs

Co-authored-by: joe petrowski <[email protected]>

* Update xcm/xcm-builder/src/location_conversion.rs

Co-authored-by: joe petrowski <[email protected]>

* Review fixes

---------

Co-authored-by: parity-processbot <>
Co-authored-by: joe petrowski <[email protected]>

* Add polkadot XCM benchmarks

* Add temp

* ".git/.scripts/commands/bench/bench.sh" xcm polkadot pallet_xcm_benchmarks::fungible

* ".git/.scripts/commands/bench/bench.sh" xcm polkadot pallet_xcm_benchmarks::generic

* Add weights to XCM on Polkadot

* Make CI fail on old files

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update template

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add reserve_asset_deposited benchmark

* ".git/.scripts/commands/bench/bench.sh" xcm kusama pallet_xcm_benchmarks::generic

* Update weights

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Change initiate_reserve_deposit in runtime weights

* Update weights

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Remove trusted reserves from runtimes

* Fix pallet-xcm-benchmarks mock

* Fix test

* Change pallet xcm weigher in kusama

* Fix

* Remove merge conflict artifact

* Remove initiate_reserve_withdraw from generic benchmarks

* Add missing implementation to XCM benchmark

* Fix failing karura test

* Remove dbg!

Co-authored-by: Keith Yeung <[email protected]>

* Fix fmt

* Revert "Fix fmt"

This reverts commit 676f2d8db07d7427750c79f95494d4988d06fda5.

* Fix fmt

* Remove duplicated template code

* Add back part of the template

* ".git/.scripts/commands/bench-vm/bench-vm.sh" xcm polkadot pallet_xcm_benchmarks::fungible

* Don't skip reserve asset deposited benchmark

* Remove call to non-generated benchmark yet

* Underscore unused parameter

* Skip not supported benchmarks and hardcode value

* Remove ReserveAssetDeposited benchmark

* ".git/.scripts/commands/bench-vm/bench-vm.sh" xcm polkadot pallet_xcm_benchmarks::fungible

* Add back ReserveAssetDeposited

* ".git/.scripts/commands/bench-vm/bench-vm.sh" xcm polkadot pallet_xcm_benchmarks::fungible

* Use default benchmark for ReserveAssetDeposited

* Add missing parameter

* Revert reserve asset deposited benchmark

* ".git/.scripts/commands/bench-vm/bench-vm.sh" xcm kusama pallet_xcm_benchmarks::fungible

* ".git/.scripts/commands/bench-vm/bench-vm.sh" xcm westend pallet_xcm_benchmarks::fungible

* ".git/.scripts/commands/bench/bench.sh" xcm rococo pallet_xcm_benchmarks::fungible

* Add 'real' benchmarks

* Add TrustedReserve to actual XcmConfig

* Add TrustedReserve to actual XcmConfig (fix)

* Whitelist from benchmarking XCM storage keys read each block (#6871)

* Whitelist from benchmarking XCM storage keys read each block

* ".git/.scripts/commands/bench/bench.sh" runtime polkadot pallet_xcm

* ".git/.scripts/commands/bench/bench.sh" runtime polkadot pallet_xcm

* ".git/.scripts/commands/bench/bench.sh" runtime westend pallet_xcm

* ".git/.scripts/commands/bench/bench.sh" runtime rococo pallet_xcm

* Remove XcmPallet SupportedVersion from the benchmark whitelist

* ".git/.scripts/commands/bench/bench.sh" runtime polkadot pallet_xcm

* ".git/.scripts/commands/bench/bench.sh" runtime kusama pallet_xcm

* ".git/.scripts/commands/bench/bench.sh" runtime westend pallet_xcm

* ".git/.scripts/commands/bench/bench.sh" runtime rococo pallet_xcm

* WIP

* Add necessary traits, remove unnecessary whitelisted keys

* Fix tests

* Remove unused file

* Remove unused import

---------

Co-authored-by: command-bot <>

* ".git/.scripts/commands/bench/bench.sh" xcm kusama pallet_xcm_benchmarks::fungible

* ".git/.scripts/commands/bench/bench.sh" xcm kusama pallet_xcm_benchmarks::fungible

* ".git/.scripts/commands/bench/bench.sh" xcm kusama pallet_xcm_benchmarks::fungible

* ".git/.scripts/commands/bench/bench.sh" xcm rococo pallet_xcm_benchmarks::fungible

* ".git/.scripts/commands/bench/bench.sh" xcm westend pallet_xcm_benchmarks::fungible

* Fix spellchecker issues

* Remove unused migration code

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Keith Yeung <[email protected]>

* Remove

* rm trait

* [XCM] Make NetworkId and Junction versioned

* Fixes

---------

Co-authored-by: parity-processbot <>

* ParaId: Serialize/Deserialize trait implemented in no-std

* Serialize/Deserialize trait implemented in no-std for more types

* serde in deps enabled

* fix

* missing types added

* update lockfile for {"substrate"}

* ".git/.scripts/commands/fmt/fmt.sh"

---------

Co-authored-by: parity-processbot <>

* Tools for unique topic references

* Formatting

* Naming

* Repot into routing.rs.

* More things done

* Universal Exporter supports topic-as-reference

* Some tests for the topic routing

* More tests

* Paid bridge tests

* Add message ID to sending events

* Formatting

* fix and integrate into test nets

* Move DenyThenTry and friend from Cumulus

* Append SetTopic rather than prepend

* Docs

* Docs

* Work with new ProcessMessage ID API

* Formatting

* Fix build

* Fixes

* Formatting

* Update xcm/xcm-builder/src/barriers.rs

Co-authored-by: Francisco Aguirre <[email protected]>

* Update xcm/xcm-builder/src/routing.rs

Co-authored-by: Francisco Aguirre <[email protected]>

* Docs

* Rename message_hash

* Formatting

* ".git/.scripts/commands/fmt/fmt.sh"

* Rename

* Another Rename

* ".git/.scripts/commands/fmt/fmt.sh"

* ".git/.scripts/commands/fmt/fmt.sh"

* Update xcm/xcm-builder/src/routing.rs

Co-authored-by: Keith Yeung <[email protected]>

---------

Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: Keith Yeung <[email protected]>

* Add to workspace

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* fmt

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Remove dumb clones

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update Cargo.lock

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: parity-processbot <>