Skip to content
  1. Jan 07, 2024
  3. Jan 04, 2024
  5. Dec 13, 2023
  7. Dec 07, 2023
  9. Dec 01, 2023
  11. Nov 30, 2023
  13. Nov 28, 2023
  15. Nov 02, 2023
    • Oliver Tale-Yazdi's avatar
      Use `Message Queue` as DMP and XCMP dispatch queue (#1246) · e1c033eb
      Oliver Tale-Yazdi authored 
      (imported from https://github.com/paritytech/cumulus/pull/2157)

## Changes

This MR refactores the XCMP, Parachains System and DMP pallets to use
the [MessageQueue](https://github.com/paritytech/substrate/pull/12485)
for delayed execution of incoming messages. The DMP pallet is entirely
replaced by the MQ and thereby removed. This allows for PoV-bounded
execution and resolves a number of issues that stem from the current
work-around.

All System Parachains adopt this change.  
The most important changes are in `primitives/core/src/lib.rs`,
`parachains/common/src/process_xcm_message.rs`,
`pallets/parachain-system/src/lib.rs`, `pallets/xcmp-queue/src/lib.rs`
and the runtime configs.

### DMP Queue Pallet

The pallet got removed and its logic refactored into parachain-system.
Overweight message management can be done directly through the MQ
pallet.

Final undeployment migrations are provided by
`cumulus_pallet_dmp_queue::UndeployDmpQueue` and `DeleteDmpQueue` that
can be configured with an aux config trait like:

```rust
parameter_types! {
	pub const DmpQueuePalletName: &'static str = \"DmpQueue\" < CHANGE ME;
	pub const RelayOrigin: AggregateMessageOrigin = AggregateMessageOrigin::Parent;
}

impl cumulus_pallet_dmp_queue::MigrationConfig for Runtime {
	type PalletName = DmpQueuePalletName;
	type DmpHandler = frame_support::traits::EnqueueWithOrigin<MessageQueue, RelayOrigin>;
	type DbWeight = <Runtime as frame_system::Config>::DbWeight;
}

// And adding them to your Migrations tuple:
pub type Migrations = (
	...
	cumulus_pallet_dmp_queue::UndeployDmpQueue<Runtime>,
	cumulus_pallet_dmp_queue::DeleteDmpQueue<Runtime>,
);
```

### XCMP Queue pallet

Removed all dispatch queue functionality. Incoming XCMP messages are now
either: Immediately handled if they are Signals, enqueued into the MQ
pallet otherwise.

New config items for the XCMP queue pallet:
```rust
/// The actual queue implementation that retains the messages for later processing.
type XcmpQueue: EnqueueMessage<ParaId>;

/// How a XCM over HRMP from a sibling parachain should be processed.
type XcmpProcessor: ProcessMessage<Origin = ParaId>;

/// The maximal number of suspended XCMP channels at the same time.
#[pallet::constant]
type MaxInboundSuspended: Get<u32>;
```

How to configure those:

```rust
// Use the MessageQueue pallet to store messages for later processing. The `TransformOrigin` is needed since
// the MQ pallet itself operators on `AggregateMessageOrigin` but we want to enqueue `ParaId`s.
type XcmpQueue = TransformOrigin<MessageQueue, AggregateMessageOrigin, ParaId, ParaIdToSibling>;

// Process XCMP messages from siblings. This is type-safe to only accept `ParaId`s. They will be dispatched
// with origin `Junction::Sibling(…)`.
type XcmpProcessor = ProcessFromSibling<
	ProcessXcmMessage<
		AggregateMessageOrigin,
		xcm_executor::XcmExecutor<xcm_config::XcmConfig>,
		RuntimeCall,
	>,
>;

// Not really important what to choose here. Just something larger than the maximal number of channels.
type MaxInboundSuspended = sp_core::ConstU32<1_000>;
```

The `InboundXcmpStatus` storage item was replaced by
`InboundXcmpSuspended` since it now only tracks inbound queue suspension
and no message indices anymore.

Now only sends the most recent channel `Signals`, as all prio ones are
out-dated anyway.

### Parachain System pallet

For `DMP` messages instead of forwarding them to the `DMP` pallet, it
now pushes them to the configured `DmpQueue`. The message processing
which was triggered in `set_validation_data` is now being done by the MQ
pallet `on_initialize`.

XCMP messages are still handed off to the `XcmpMessageHandler`
(XCMP-Queue pallet) - no change here.

New config items for the parachain system pallet:
```rust
/// Queues inbound downward messages for delayed processing. 
///
/// Analogous to the `XcmpQueue` of the XCMP queue pallet.
type DmpQueue: EnqueueMessage<AggregateMessageOrigin>;
``` 

How to configure:
```rust
/// Use the MQ pallet to store DMP messages for delayed processing.
type DmpQueue = MessageQueue;
``` 

## Message Flow

The flow of messages on the parachain side. Messages come in from the
left via the `Validation Data` and finally end up at the `Xcm Executor`
on the right.

![Untitled
(1)](https://github.com/paritytech/cumulus/assets/10380170/6cf8b377-88c9-4aed-96df-baace266e04d)

## Further changes

- Bumped the default suspension, drop and resume thresholds in
`QueueConfigData::default()`.
- `XcmpQueue::{suspend_xcm_execution, resume_xcm_execution}` errors when
they would be a noop.
- Properly validate the `QueueConfigData` before setting it.
- Marked weight files as auto-generated so they wont auto-expand in the
MR files view.
- Move the `hypothetical` asserts to `frame_support` under the name
`experimental_hypothetically`

Questions:
- [ ] What about the ugly `#[cfg(feature = \"runtime-benchmarks\")]` in
the runtimes? Not sure how to best fix. Just having them like this makes
tests fail that rely on the real message processor when the feature is
enabled.
- [ ] Need a good weight for `MessageQueueServiceWeight`. The scheduler
already takes 80% so I put it to 10% but that is quite low.

TODO:
- [x] Remove c&p code after
https://github.com/paritytech/polkadot/pull/6271
- [x] Use `HandleMessage` once it is public in Substrate
- [x] fix `runtime-benchmarks` feature
https://github.com/paritytech/polkadot/pull/6966


- [x] Benchmarks
- [x] Tests
- [ ] Migrate `InboundXcmpStatus` to `InboundXcmpSuspended`
- [x] Possibly cleanup Migrations (DMP+XCMP)
- [x] optional: create `TransformProcessMessageOrigin` in Substrate and
replace `ProcessFromSibling`
- [ ] Rerun weights on ref HW

---------

Signed-off-by: default avatarOliver Tale-Yazdi <[email protected]>
Co-authored-by: default avatarLiam Aharon <[email protected]>
Co-authored-by: default avatarjoe petrowski <[email protected]>
Co-authored-by: default avatarKian Paimani <[email protected]>
Co-authored-by: command-bot <>
      e1c033eb
  17. Oct 24, 2023
    • Oliver Tale-Yazdi's avatar
      Improve features dev-ex (#1831) · 4a443567
      Oliver Tale-Yazdi authored 
      Adds a config file that allows to run `zepter` without any arguments in
the workspace to address all issues.
A secondary workflow for the CI is provided as `zepter run check`. Both
the formatting and linting are now in one check for efficiancy.

The latest version also detects some more things that `featalign` was
already showing.

Error message [in the
CI](https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/3916205)
now looks like this:
```pre
...
crate 'test-parachains' (/Users/vados/Documents/work/polkadot-sdk/polkadot/parachain/test-parachains/Cargo.toml)
  feature 'std'
    must propagate to:
      parity-scale-codec
Found 55 issues (run with --fix to fix).
Error: Command 'lint propagate-feature' failed with exit code 1

Polkadot-SDK uses the Zepter CLI to detect abnormalities in the feature configuration.
It looks like one more more checks failed; please check the console output. You can try to automatically address them by running `zepter`.
Otherwise please ask directly in the Merge Request, GitHub Discussions or on Matrix Chat, thank you.

For more information, see:
  - https://github.com/paritytech/polkadot-sdk/issues/1831
  - https://github.com/ggwpez/zepter


```

TODO:
- [x] Check that CI fails correctly

---------

Signed-off-by: default avatarOliver Tale-Yazdi <[email protected]>
      4a443567
  19. Oct 20, 2023
    • Bastian Köcher's avatar
      `xcm`: Change `TypeInfo::path` to not include `staging` (#1948) · f3bf5c1a
      Bastian Köcher authored 
      The `xcm` crate was renamed to `staging-xcm` to be able to publish it to
crates.io as someone as squatted `xcm`. The problem with this rename is
that the `TypeInfo` includes the crate name which ultimately lands in
the metadata. The metadata is consumed by downstream users like
`polkadot-js` or people building on top of `polkadot-js`. These people
are using the entire `path` to find the type in the type registry. Thus,
their code would break as the type path would now be [`staging_xcm`,
`VersionedXcm`] instead of [`xcm`, `VersionedXcm`]. This pull request
fixes this by renaming the path segment `staging_xcm` to `xcm`.

This requires: https://github.com/paritytech/scale-info/pull/197



---------

Co-authored-by: default avatarFrancisco Aguirre <[email protected]>
      f3bf5c1a
  21. Oct 19, 2023
  23. Sep 05, 2023
  25. Sep 01, 2023
  27. Aug 30, 2023
  29. Aug 29, 2023
  31. Aug 25, 2023
  33. Aug 23, 2023
  35. Aug 14, 2023
  37. Aug 01, 2023
  39. Jul 18, 2023
  41. Jul 14, 2023
    • juangirini's avatar
      Replace system config `Index` for `Nonce` (#14290) · 6a29a70a
      juangirini authored 
      * replace Index by Nonce

* replace Index by Nonce

* replace Index by Nonce

* replace Index by Nonce

* replace Index by Nonce

* wip

* remove index in lieu of nonce

* wip

* remove accountnonce in lieu of nonce

* add minor improvement

* rebase and merge conflicts
      6a29a70a
  43. Jul 13, 2023
    • gupnik's avatar
      Moves `Block` to `frame_system` instead of `construct_runtime` and removes... · 5e7b27e9
      gupnik authored 
      
Moves `Block` to `frame_system` instead of `construct_runtime` and removes `Header` and `BlockNumber` (#14437)

* Initial setup

* Adds node block

* Uses UncheckedExtrinsic and removes Where section

* Updates frame_system to use Block

* Adds deprecation warning

* Fixes pallet-timestamp

* Removes Header and BlockNumber

* Addresses review comments

* Addresses review comments

* Adds comment about compiler bug

* Removes where clause

* Refactors code

* Fixes errors in cargo check

* Fixes errors in cargo check

* Fixes warnings in cargo check

* Formatting

* Fixes construct_runtime tests

* Uses import instead of full path for BlockNumber

* Uses import instead of full path for Header

* Formatting

* Fixes construct_runtime tests

* Fixes imports in benchmarks

* Formatting

* Fixes construct_runtime tests

* Formatting

* Minor updates

* Fixes construct_runtime ui tests

* Fixes construct_runtime ui tests with 1.70

* Fixes docs

* Fixes docs

* Adds u128 mock block type

* Fixes split example

* fixes for cumulus

* ".git/.scripts/commands/fmt/fmt.sh"

* Updates new tests

* Fixes fully-qualified path in few places

* Formatting

* Update frame/examples/default-config/src/lib.rs

Co-authored-by: default avatarJuan <[email protected]>

* Update frame/support/procedural/src/construct_runtime/mod.rs

Co-authored-by: default avatarJuan <[email protected]>

* ".git/.scripts/commands/fmt/fmt.sh"

* Addresses some review comments

* Fixes build

* ".git/.scripts/commands/fmt/fmt.sh"

* Update frame/democracy/src/lib.rs

Co-authored-by: default avatarOliver Tale-Yazdi <[email protected]>

* Update frame/democracy/src/lib.rs

Co-authored-by: default avatarOliver Tale-Yazdi <[email protected]>

* Update frame/support/procedural/src/construct_runtime/mod.rs

Co-authored-by: default avatarOliver Tale-Yazdi <[email protected]>

* Update frame/support/procedural/src/construct_runtime/mod.rs

Co-authored-by: default avatarOliver Tale-Yazdi <[email protected]>

* Addresses review comments

* Updates trait bounds

* Minor fix

* ".git/.scripts/commands/fmt/fmt.sh"

* Removes unnecessary bound

* ".git/.scripts/commands/fmt/fmt.sh"

* Updates test

* Fixes build

* Adds a bound for header

* ".git/.scripts/commands/fmt/fmt.sh"

* Removes where block

* Minor fix

* Minor fix

* Fixes tests

* ".git/.scripts/commands/update-ui/update-ui.sh" 1.70

* Updates test

* Update primitives/runtime/src/traits.rs

Co-authored-by: default avatarBastian Köcher <[email protected]>

* Update primitives/runtime/src/traits.rs

Co-authored-by: default avatarBastian Köcher <[email protected]>

* Updates doc

* Updates doc

---------

Co-authored-by: command-bot <>
Co-authored-by: default avatarJuan <[email protected]>
Co-authored-by: default avatarOliver Tale-Yazdi <[email protected]>
Co-authored-by: default avatarBastian Köcher <[email protected]>
      5e7b27e9
  45. Jul 12, 2023
    • Michal Kucharczyk's avatar
      `GenesisBuild<T,I>` deprecated. `BuildGenesisConfig` added. (#14306) · 87d41d0a
      Michal Kucharczyk authored 
      

* frame::support: GenesisConfig types for Runtime enabled

* frame::support: macro generating GenesisBuild::build for RuntimeGenesisConfig

* frame: ambiguity BuildStorage vs GenesisBuild fixed

* fix

* RuntimeGenesisBuild added

* Revert "frame: ambiguity BuildStorage vs GenesisBuild fixed"

This reverts commit 950f3d019d0e21c55a739c44cc19cdabd3ff0293.

* Revert "fix"

This reverts commit a2f76dd24e9a16cf9230d45825ed28787211118b.

* Revert "RuntimeGenesisBuild added"

This reverts commit 3c131b618138ced29c01ab8d15d8c6410c9e128b.

* Revert "Revert "frame: ambiguity BuildStorage vs GenesisBuild fixed""

This reverts commit 2b1ecd467231eddec69f8d328039ba48a380da3d.

* Revert "Revert "fix""

This reverts commit fd7fa629adf579d83e30e6ae9fd162637fc45e30.

* Code review suggestions

* frame: BuildGenesisConfig added, BuildGenesis deprecated

* frame: some pallets updated with BuildGenesisConfig

* constuct_runtime: support for BuildGenesisConfig

* frame::support: genesis_build macro supports BuildGenesisConfig

* frame: BuildGenesisConfig added, BuildGenesis deprecated

* Cargo.lock update

* test-runtime: fixes

* Revert "fix"

This reverts commit a2f76dd24e9a16cf9230d45825ed28787211118b.

* Revert "frame: ambiguity BuildStorage vs GenesisBuild fixed"

This reverts commit 950f3d019d0e21c55a739c44cc19cdabd3ff0293.

* self review

* doc fixed

* ui tests fixed

* fmt

* tests fixed

* genesis_build macrto fixed for non-generic GenesisConfig

* BuildGenesisConfig constraints added

* warning fixed

* some duplication removed

* fmt

* fix

* doc tests fix

* doc fix

* cleanup: remove BuildModuleGenesisStorage

* self review comments

* fix

* Update frame/treasury/src/tests.rs

Co-authored-by: default avatarSebastian Kunert <[email protected]>

* Update frame/support/src/traits/hooks.rs

Co-authored-by: default avatarSebastian Kunert <[email protected]>

* doc fix: GenesisBuild exposed

* ".git/.scripts/commands/fmt/fmt.sh"

* frame: more serde(skip) + cleanup

* Update frame/support/src/traits/hooks.rs

Co-authored-by: default avatarDavide Galassi <[email protected]>

* frame: phantom fields moved to the end of structs

* chain-spec: Default::default cleanup

* test-runtime: phantom at the end

* merge master fixes

* fix

* fix

* fix

* fix

* fix (facepalm)

* Update frame/support/procedural/src/pallet/expand/genesis_build.rs

Co-authored-by: default avatarBastian Köcher <[email protected]>

* fmt

* fix

* fix

---------

Co-authored-by: parity-processbot <>
Co-authored-by: default avatarSebastian Kunert <[email protected]>
Co-authored-by: default avatarDavide Galassi <[email protected]>
Co-authored-by: default avatarBastian Köcher <[email protected]>
      87d41d0a
  47. Jun 28, 2023
  49. Jun 21, 2023
  51. Jun 19, 2023
  53. May 31, 2023
  55. May 29, 2023
  57. May 21, 2023
  59. Apr 13, 2023
  61. Apr 12, 2023
  63. Mar 16, 2023
  65. Mar 13, 2023
    • Vivek Pandya's avatar
      Remove use of trait Store from all pallets and deprecate it. (#13535) · 2009821c
      Vivek Pandya authored 
      * Remove use of trait Store from staking pallet

* Remove use of trait Store from bounties pallet

* Remove use of trait Store from collective pallet

* Remove use of trait Store from babe pallet

* Remove use of trait Store from assets pallet

* Remove use of trait Store from grandpa pallet

* Remove use of trait Store from balances pallet

* Remove use of trait Store from authorship pallet

* Remove use of trait Store from authority-discovery pallet

* Remove use of trait Store from atomic-swap pallet

* Remove use of trait Store from sudo pallet

* Remove use of trait Store from scheduler pallet

* Remove use of trait Store from scored-pool pallet

* Remove use of trait Store from society pallet

* Remove use of trait Store from lottery pallet

* Remove use of trait Store from executive pallet

* Remove use of trait Store from democracy pallet

* Remove use of trait Store from elections-phragmen pallet

* Remove use of trait Store from indices pallet

* Remove use of trait Store from identity pallet

* Remove use of trait Store from multisig pallet

* Remove use of trait Store from merkle-mountain-range pallet

* Remove use of trait Store from im-online pallet

* Remove use of trait Store from membership pallet

* Remove use of trait Store from nicks pallet

* Remove use of trait Store from session pallet

* Remove use of trait Store from transaction-payment pallet

* Remove use of trait Store from utility pallet

* Remove use of trait Store from child-bounties pallet

* Remove use of trait Store from nis pallet

* Remove use of trait Store from nfts pallet

* Remove use of trait Store from conviction-voting pallet

* Remove use of trait Store from treasury pallet

* Remove use of trait Store from vesting pallet

* Remove use of trait Store from preimage pallet

* Remove use of trait Store from uniques pallet

* Remove use of trait Store from ranked-collective pallet

* Remove use of trait Store from beefy-mmr pallet

* Remove use of trait Store from referenda pallet

* Remove use of trait Store from whitelist pallet

* Remove use of trait Store from alliance pallet

* Remove use of trait Store from nomination-pools pallet

* Remove use of trait Store from state-trie-migration pallet

* Remove use of trait Store from message-queue pallet

* Remove use of trait Store from root-offences pallet

* Remove use of trait Store from root-testing pallet

* Remove use of trait Store from timestamps pallet

* Remove use of trait Store from system pallet

* Remove use of trait Store from offences pallet

* Remove use of trait Store from recovery pallet

* Remove use of trait Store from node-authorization pallet

* Remove use of trait Store from proxy pallet

* Remove use of trait Store from benchmarking pallet

* Remove use of trait Store from bags-list pallet

* Add deprecated warning in store_trait

* Change warning message

* Run cargo fmt

* Fix warning and update tests

* Remove unnecessary allow deprecated

* Remove use of trait Store

* Fix mismatch in expected output

* Minor update to warning message for deprecation of generate_store with Store trait attribute

* Fixes as per review comments

* Fixes as per review suggestions

* Remove use of Store trait from core-fellowship pallet

* Fix type in store_trait.rs

* Fixes as pre review comment
      2009821c
  67. Mar 07, 2023
  69. Mar 04, 2023
  71. Mar 02, 2023
  73. Feb 28, 2023
  75. Feb 25, 2023
  77. Feb 21, 2023
    • Vivek Pandya's avatar
      Remove years from copyright notes. (#13415) · bc53b9a0
      Vivek Pandya authored 
      * Change copyright year to 2023 from 2022

* Fix incorrect update of copyright year

* Remove years from copy right header

* Fix remaining files

* Fix typo in a header and remove update-copyright.sh
      bc53b9a0