Use Message Queue pallet for UMP dispatch (#6271)

* Add ProcessXcmMessage struct

* Migrate away from weights in host config

* New well-known key to report UMPQ capacity

* Add missing file

* Fixes

* Remove original UMP files

* Docs

* Update runtime/parachains/src/inclusion/mod.rs

Co-authored-by: asynchronous rob <[email protected]>

* Add benchmarking

* Benchmarks

* Mock example of using the QueueChangeHandler to update the WKK

* Use master Cargo.lock

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Merge remote-tracking branch 'origin/master' into gav-message-queue

* Update Cargo.lock

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update remove-weight migration

The migration got touched on master; just resolving conflicts here.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add message- to dispatch-origin conversion for XCM processing

Just using the `impl Into<MultiLocation>` was a bit inflexible.
Like this, the Relaychain can convert `UMP(para)` to a MultiLocation `para`.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* DNM: Temporarily comment code since XCMv3 is not merged yet

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Use u64 for queue-wide limits on UmpAcceptanceCheckErr

Using u32 here was one audit finding for the queue pallet.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Define one sub-queue per *MP queue

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Harden check_upward_messages

Using safe math and casts.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add type-safe well_known_keys

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add message-queue weights

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Deploy MessageQueue to Polkadot

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update Cargo.toml

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Migrate to parachain config V5

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update UMP tests

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Cleanup

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* fmt

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Revert messed up merge 🤦



Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update remove-weight migration

The migration got touched on master; just resolving conflicts here.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add message- to dispatch-origin conversion for XCM processing

Just using the `impl Into<MultiLocation>` was a bit inflexible.
Like this, the Relaychain can convert `UMP(para)` to a MultiLocation `para`.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* DNM: Temporarily comment code since XCMv3 is not merged yet

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Use u64 for queue-wide limits on UmpAcceptanceCheckErr

Using u32 here was one audit finding for the queue pallet.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Define one sub-queue per *MP queue

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Harden check_upward_messages

Using safe math and casts.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add type-safe well_known_keys

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add message-queue weights

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Deploy MessageQueue to Polkadot

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update Cargo.toml

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Migrate to parachain config V5

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update UMP tests

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Cleanup

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* fmt

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Make stuff compile

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Move DMP and HRMP messages to the MessageQueue

It currently does not compile in the CIbecause of some local
tweaks to Substrate.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Diener for CI

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* diener update cargo.lock

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Cleanup

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Revert wrong changes

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Revert "DNM: Temporarily comment code since XCMv3 is not merged yet"

This reverts commit 820aa235cb21dd1d2621843607f7682bf035434e.

* Make compile

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fixup runtimes

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* fmt

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Define benchmarks

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Cleanup

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix migration

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Use master Cargo.lock

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Lockfile

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix test

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add AggregateMessageOrigin

This enum currently only holds one value, but having it will make
it easier in the future to extend.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Forbid UMP for off-boarding paras

- Reject candidates with UMP messages for off-boarding paras
- Forbid scheduling off-boarding when a para has unprocess UMPs

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Delete stupid test

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Use BoundedVec for upward messages

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add weights and fix MessageProcessor

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Bound receive_upward_messages and check bound in configuration pallet

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Bound Debug impl

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* fmt

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* clippy

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix test runtime

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix xcm-simulator

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Properly fix xcm-simulator and fuzzer

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix tests

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* fmt

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* cargo update -p sp-io

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Adapt to upstream Substrate changes

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix ProcesseMessage impls

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Some tests

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Use master Cargo.lock

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* cargo update -p sp-io

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Use new MQ API

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix test

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix migration

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update Cargo.lock

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add UMP while Para offboarding tests

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* fmt

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Use Mocked message processor for benchmarking

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Use variables for constants

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add MQ pallet weights

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Use MQ pallet weights

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Configure QueueChangeHandler

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* fmt

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add config test

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix MQ serive weight

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Cleanup

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Cleanup outgoing UMP dispatch queues

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Use Master Cargo.lock

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update Cargo.lock

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Weight mul is not const

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Clippy

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Remove merge marker

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update runtime/parachains/src/inclusion/mod.rs

Co-authored-by: Gavin Wood <[email protected]>

* Update runtime/kusama/src/lib.rs

Co-authored-by: Gavin Wood <[email protected]>

* Use lowercase UMP

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Clarify comment

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Use Weight::from_parts

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix test

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix doc

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Emit event after the fact

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add defensive_proof to receive_upward_messages

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Reapply "Remove original UMP files"

Looks like they came back from the dead. Re-apply commit cf6d316f0

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Remove old files

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Rename MaxUmpMessageLen -> MaxUmpMessageLenOf

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Test defensive message dropping of receive_upward_messages

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fixup imports

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update implementors guide

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix tests

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Remove FAIL-CI mark

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Delete unused code

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add another test for MQ change hook

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Imports

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Keep Kusama runtime formatting

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Revert "Delete unused code"

This reverts commit dd76bca5025b7e1ef846a9539c3607eed185f16a.

* Feature gate mock functions

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Review: Use saturating_add

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Test RelayDispatchQueueSize storage key

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Move migration to own file to avoid merge conflicts

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Migration in own file

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fixup migration

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Rococo: configure MQ pallet

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fixup tests

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Revert "Rococo: configure MQ pallet"

Going to do this as follow up, since it needs Substrate changes
and i dont want to stall this MR any longer.

This reverts commit b9c15e8a8339c4e877d654ee3f09903af4210736.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Revert "Fixup tests"

This reverts commit 88f1cbe20774d20e5e1e554e798960ae39437af1.

* Fixup migration

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix migration

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* fmt

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix CI

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix migration

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix other migration

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Bump MAX_CODE_SIZE to 10MiB

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add ForceUpdateUmpLimits migration

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* fmt

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* clippy

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* clippy

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* imports

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Use defensive instead of defensive_proof

'defensive_proof' also prints the 'self', which spams the console
too much when running the tests. Just the length is enough.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Rename to ScheduleConfigUpdate

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fixup migration checks

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add MAX_CODE_SIZE to ScheduleConfigUpdate

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Set MAX_CODE_SIZE to 4MiB

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix benchmark

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix formatting

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Revert "Add MAX_CODE_SIZE to ScheduleConfigUpdate"

This reverts commit 7caffb09e83083b57affd548215e45b25c3d64dc.

* Revert "Set MAX_CODE_SIZE to 4MiB"

This reverts commit 103ffbaf686487d2fbe0082a16826af17cacc1a1.

* Revert "Bump MAX_CODE_SIZE to 10MiB"

This reverts commit 530734b7b0da5b7680054e0242348fcc79a666fe.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Remove consistency check from migration

Re-addig these checks is blocked on https://github.com/paritytech/polkadot/issues/7108



Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix constants

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Bump MAX_UPWARD_MESSAGE_SIZE_BOUND for Westend

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix migrations

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Use old nightly for fmt

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fixes

* cargo fmt

* Fix tests

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix tests

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fixes

* Add MQ pallet to fuzzer

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fmt

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix XMC simulator example

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Remove runtime-benchmarks from fuzzers

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Revert "Remove runtime-benchmarks from fuzzers"

This reverts commit e1f2bb01b6dea2dd465539d3658719895b58b557.

* Fix example simulator

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add V6 migration and remove old ones

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Actually make old migrations reusable

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Dont delete old migrations

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Future proof AggregateMessageOrigin and review fixes

There are indications that Loopback and Bridged will be needed soon.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* More cleanup

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* fmt

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* fix benchmarks

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix fuzzer build

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Review

Co-authored-by: muharem <[email protected]>

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Remove old migration

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Set MQ service weight to 20%

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix tabs in Markdown

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: asynchronous rob <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Keith Yeung <[email protected]>