1. Nov 03, 2023
  2. Nov 02, 2023
    • Oliver Tale-Yazdi's avatar
      Use `Message Queue` as DMP and XCMP dispatch queue (#1246) · e1c033eb
      Oliver Tale-Yazdi authored
      (imported from https://github.com/paritytech/cumulus/pull/2157)
      
      ## Changes
      
      This MR refactores the XCMP, Parachains System and DMP pallets to use
      the [MessageQueue](https://github.com/paritytech/substrate/pull/12485)
      for delayed execution of incoming messages. The DMP pallet is entirely
      replaced by the MQ and thereby removed. This allows for PoV-bounded
      execution and resolves a number of issues that stem from the current
      work-around.
      
      All System Parachains adopt this change.  
      The most important changes are in `primitives/core/src/lib.rs`,
      `parachains/common/src/process_xcm_message.rs`,
      `pallets/parachain-system/src/lib.rs`, `pallets/xcmp-queue/src/lib.rs`
      and the runtime configs.
      
      ### DMP Queue Pallet
      
      The pallet got removed and its logic refactored into parachain-system.
      Overweight message management can be done directly through the MQ
      pallet.
      
      Final undeployment migrations are provided by
      `cumulus_pallet_dmp_queue::UndeployDmpQueue` and `DeleteDmpQueue` that
      can be configured with an aux config trait like:
      
      ```rust
      parameter_types! {
      	pub const DmpQueuePalletName: &'static str = \"DmpQueue\" < CHANGE ME;
      	pub const RelayOrigin: AggregateMessageOrigin = AggregateMessageOrigin::Parent;
      }
      
      impl cumulus_pallet_dmp_queue::MigrationConfig for Runtime {
      	type PalletName = DmpQueuePalletName;
      	type DmpHandler = frame_support::traits::EnqueueWithOrigin<MessageQueue, RelayOrigin>;
      	type DbWeight = <Runtime as frame_system::Config>::DbWeight;
      }
      
      // And adding them to your Migrations tuple:
      pub type Migrations = (
      	...
      	cumulus_pallet_dmp_queue::UndeployDmpQueue<Runtime>,
      	cumulus_pallet_dmp_queue::DeleteDmpQueue<Runtime>,
      );
      ```
      
      ### XCMP Queue pallet
      
      Removed all dispatch queue functionality. Incoming XCMP messages are now
      either: Immediately handled if they are Signals, enqueued into the MQ
      pallet otherwise.
      
      New config items for the XCMP queue pallet:
      ```rust
      /// The actual queue implementation that retains the messages for later processing.
      type XcmpQueue: EnqueueMessage<ParaId>;
      
      /// How a XCM over HRMP from a sibling parachain should be processed.
      type XcmpProcessor: ProcessMessage<Origin = ParaId>;
      
      /// The maximal number of suspended XCMP channels at the same time.
      #[pallet::constant]
      type MaxInboundSuspended: Get<u32>;
      ```
      
      How to configure those:
      
      ```rust
      // Use the MessageQueue pallet to store messages for later processing. The `TransformOrigin` is needed since
      // the MQ pallet itself operators on `AggregateMessageOrigin` but we want to enqueue `ParaId`s.
      type XcmpQueue = TransformOrigin<MessageQueue, AggregateMessageOrigin, ParaId, ParaIdToSibling>;
      
      // Process XCMP messages from siblings. This is type-safe to only accept `ParaId`s. They will be dispatched
      // with origin `Junction::Sibling(…)`.
      type XcmpProcessor = ProcessFromSibling<
      	ProcessXcmMessage<
      		AggregateMessageOrigin,
      		xcm_executor::XcmExecutor<xcm_config::XcmConfig>,
      		RuntimeCall,
      	>,
      >;
      
      // Not really important what to choose here. Just something larger than the maximal number of channels.
      type MaxInboundSuspended = sp_core::ConstU32<1_000>;
      ```
      
      The `InboundXcmpStatus` storage item was replaced by
      `InboundXcmpSuspended` since it now only tracks inbound queue suspension
      and no message indices anymore.
      
      Now only sends the most recent channel `Signals`, as all prio ones are
      out-dated anyway.
      
      ### Parachain System pallet
      
      For `DMP` messages instead of forwarding them to the `DMP` pallet, it
      now pushes them to the configured `DmpQueue`. The message processing
      which was triggered in `set_validation_data` is now being done by the MQ
      pallet `on_initialize`.
      
      XCMP messages are still handed off to the `XcmpMessageHandler`
      (XCMP-Queue pallet) - no change here.
      
      New config items for the parachain system pallet:
      ```rust
      /// Queues inbound downward messages for delayed processing. 
      ///
      /// Analogous to the `XcmpQueue` of the XCMP queue pallet.
      type DmpQueue: EnqueueMessage<AggregateMessageOrigin>;
      ``` 
      
      How to configure:
      ```rust
      /// Use the MQ pallet to store DMP messages for delayed processing.
      type DmpQueue = MessageQueue;
      ``` 
      
      ## Message Flow
      
      The flow of messages on the parachain side. Messages come in from the
      left via the `Validation Data` and finally end up at the `Xcm Executor`
      on the right.
      
      ![Untitled
      (1)](https://github.com/paritytech/cumulus/assets/10380170/6cf8b377-88c9-4aed-96df-baace266e04d)
      
      ## Further changes
      
      - Bumped the default suspension, drop and resume thresholds in
      `QueueConfigData::default()`.
      - `XcmpQueue::{suspend_xcm_execution, resume_xcm_execution}` errors when
      they would be a noop.
      - Properly validate the `QueueConfigData` before setting it.
      - Marked weight files as auto-generated so they wont auto-expand in the
      MR files view.
      - Move the `hypothetical` asserts to `frame_support` under the name
      `experimental_hypothetically`
      
      Questions:
      - [ ] What about the ugly `#[cfg(feature = \"runtime-benchmarks\")]` in
      the runtimes? Not sure how to best fix. Just having them like this makes
      tests fail that rely on the real message processor when the feature is
      enabled.
      - [ ] Need a good weight for `MessageQueueServiceWeight`. The scheduler
      already takes 80% so I put it to 10% but that is quite low.
      
      TODO:
      - [x] Remove c&p code after
      https://github.com/paritytech/polkadot/pull/6271
      - [x] Use `HandleMessage` once it is public in Substrate
      - [x] fix `runtime-benchmarks` feature
      https://github.com/paritytech/polkadot/pull/6966
      
      
      - [x] Benchmarks
      - [x] Tests
      - [ ] Migrate `InboundXcmpStatus` to `InboundXcmpSuspended`
      - [x] Possibly cleanup Migrations (DMP+XCMP)
      - [x] optional: create `TransformProcessMessageOrigin` in Substrate and
      replace `ProcessFromSibling`
      - [ ] Rerun weights on ref HW
      
      ---------
      
      Signed-off-by: default avatarOliver Tale-Yazdi <[email protected]>
      Co-authored-by: default avatarLiam Aharon <[email protected]>
      Co-authored-by: default avatarjoe petrowski <[email protected]>
      Co-authored-by: default avatarKian Paimani <[email protected]>
      Co-authored-by: command-bot <>
      e1c033eb
    • Serban Iorga's avatar
  3. Nov 01, 2023
    • Oliver Tale-Yazdi's avatar
      [FRAME] Short-circuit fungible self transfer (#2118) · c66ae375
      Oliver Tale-Yazdi authored
      
      
      Changes:
      - Change the fungible(s) logic to treat a self-transfer as No-OP (as
      long as all pre-checks pass).
      
      Note that the self-transfer case will not emit an event since no state
      was changed.
      
      ---------
      
      Signed-off-by: default avatarOliver Tale-Yazdi <[email protected]>
      c66ae375
    • Julian Eager's avatar
      Build workers for testing on demand (#2018) · 4f05f9a6
      Julian Eager authored
      4f05f9a6
    • jserrat's avatar
      9987bbb1
    • Serban Iorga's avatar
    • jserrat's avatar
      remove gum dependency on jaeger (#2106) · 2726d5af
      jserrat authored
      
      
      Co-authored-by: default avatarMarcin S <[email protected]>
      2726d5af
    • Ankan's avatar
      [NPoS] Paging reward payouts in order to scale rewardable nominators (#1189) · 00b85c51
      Ankan authored
      helps https://github.com/paritytech/polkadot-sdk/issues/439.
      closes https://github.com/paritytech/polkadot-sdk/issues/473.
      
      PR link in the older substrate repository:
      https://github.com/paritytech/substrate/pull/13498.
      
      # Context
      Rewards payout is processed today in a single block and limited to
      `MaxNominatorRewardedPerValidator`. This number is currently 512 on both
      Kusama and Polkadot.
      
      This PR tries to scale the nominators payout to an unlimited count in a
      multi-block fashion. Exposures are stored in pages, with each page
      capped to a certain number (`MaxExposurePageSize`). Starting out, this
      number would be the same as `MaxNominatorRewardedPerValidator`, but
      eventually, this number can be lowered through new runtime upgrades to
      limit the rewardeable nominators per dispatched call instruction.
      
      The changes in the PR are backward compatible.
      
      ## How payouts would work like after this change
      Staking exposes two calls, 1) the existing `payout_stakers` and 2)
      `payout_stakers_by_page`.
      
      ### payout_stakers
      This remains backward compatible with no signature change. If for a
      given era a validator has multiple pages, they can call `payout_stakers`
      multiple times. The pages are executed in an ascending sequence and the
      runtime takes care of preventing double claims.
      
      ### payout_stakers_by_page
      Very similar to `payout_stakers` but also accepts an extra param
      `page_index`. An account can choose to payout rewards only for an
      explicitly passed `page_index`.
      
      **Lets look at an example scenario**
      Given an active validator on Kusama had 1100 nominators,
      `MaxExposurePageSize` set to 512 for Era e. In order to pay out rewards
      to all nominators, the caller would need to call `payout_stakers` 3
      times.
      
      - `payout_stakers(origin, stash, e)` => will pay the first 512
      nominators.
      - `payout_stakers(origin, stash, e)` => will pay the second set of 512
      nominators.
      - `payout_stakers(origin, stash, e)` => will pay the last set of 76
      nominators.
      ...
      - `payout_stakers(origin, stash, e)` => calling it the 4th time would
      return an error `InvalidPage`.
      
      The above calls can also be replaced by `payout_stakers_by_page` and
      passing a `page_index` explicitly.
      
      ## Commission note
      Validator commission is paid out in chunks across all the pages where
      each commission chunk is proportional to the total stake of the current
      page. This implies higher the total stake of a page, higher will be the
      commission. If all the pages of a validator's single era are paid out,
      the sum of commission paid to the validator across all pages should be
      equal to what the commission would have been if we had a non-paged
      exposure.
      
      ### Migration Note
      Strictly speaking, we did not need to bump our storage version since
      there is no migration of storage in this PR. But it is still useful to
      mark a storage upgrade for the following reasons:
      
      - New storage items are introduced in this PR while some older storage
      items are deprecated.
      - For the next `HistoryDepth` eras, the exposure would be incrementally
      migrated to its corresponding paged storage item.
      - Runtimes using staking pallet would strictly need to wait at least
      `HistoryDepth` eras with current upgraded version (14) for the migration
      to complete. At some era `E` such that `E >
      era_at_which_V14_gets_into_effect + HistoryDepth`, we will upgrade to
      version X which will remove the deprecated storage items.
      In other words, it is a strict requirement that E<sub>x</sub> -
      E<sub>14</sub> > `HistoryDepth`, where
      E<sub>x</sub> = Era at which deprecated storages are removed from
      runtime,
      E<sub>14</sub> = Era at which runtime is upgraded to version 14.
      - For Polkadot and Kusama, there is a [tracker
      ticket](https://github.com/paritytech/polkadot-sdk/issues/433) to clean
      up the deprecated storage items.
      
      ### Storage Changes
      
      #### Added
      - ErasStakersOverview
      - ClaimedRewards
      - ErasStakersPaged
      
      #### Deprecated
      The following can be cleaned up after 84 eras which is tracked
      [here](https://github.com/paritytech/polkadot-sdk/issues/433).
      
      - ErasStakers.
      - ErasStakersClipped.
      - StakingLedger.claimed_rewards, renamed to
      StakingLedger.legacy_claimed_rewards.
      
      ### Config Changes
      - Renamed MaxNominatorRewardedPerValidator to MaxExposurePageSize.
      
      ### TODO
      - [x] Tracker ticket for cleaning up the old code after 84 eras.
      - [x] Add companion.
      - [x] Redo benchmarks before merge.
      - [x] Add Changelog for pallet_staking.
      - [x] Pallet should be configurable to enable/disable paged rewards.
      - [x] Commission payouts are distributed across pages.
      - [x] Review documentation thoroughly.
      - [x] Rename `MaxNominatorRewardedPerValidator` ->
      `MaxExposurePageSize`.
      - [x] NMap for `ErasStakersPaged`.
      - [x] Deprecate ErasStakers.
      - [x] Integrity tests.
      
      ### Followup issues
      [Runtime api for deprecated ErasStakers storage
      item](https://github.com/paritytech/polkadot-sdk/issues/426
      
      )
      
      ---------
      
      Co-authored-by: default avatarJavier Viola <[email protected]>
      Co-authored-by: default avatarRoss Bulat <[email protected]>
      Co-authored-by: command-bot <>
      00b85c51
  4. Oct 31, 2023
    • Adel Arja's avatar
      1953 defensive testing extrinsic (#1998) · 6e2f94f8
      Adel Arja authored
      
      
      # Description
      
      The `trigger_defensive` call has been added to the `root-testing`
      pallet. The idea is to have this pallet running on `Rococo/Westend` and
      use it to verify if the runtime monitoring works end-to-end.
      
      To accomplish this, `trigger_defensive` dispatches an event when it is
      called.
      
      Closes #1953
      
      # Checklist
      
      - [x] My PR includes a detailed description as outlined in the
      "Description" section above
      - [ ] My PR follows the [labeling requirements](CONTRIBUTING.md#Process)
      of this project (at minimum one label for `T`
        required)
      - [ ] I have made corresponding changes to the documentation (if
      applicable)
      - [ ] I have added tests that prove my fix is effective or that my
      feature works (if applicable)
      
      You can remove the "Checklist" section once all have been checked. Thank
      you for your contribution!
      
      ✄
      -----------------------------------------------------------------------------
      
      ---------
      
      Signed-off-by: default avatarOliver Tale-Yazdi <[email protected]>
      Co-authored-by: default avatarOliver Tale-Yazdi <[email protected]>
      6e2f94f8
    • Alexandru Gheorghe's avatar
      polkadot: parachains: Fix v9 host configuration migration (#2103) · 64f4b156
      Alexandru Gheorghe authored
      
      
      We shouldn't override with their default fields that have been added in
      the previous version(v8), because we are going to lose whatever values have
      been set.
      
      Note, v8 & v9 seems to have landed at the same time on Rococo, probably
      they will land at the same time on westend and other chains, so functionally
      doesn't make much difference, but let's have this fixed for people that copy-paste
      :D, like me.
      
      ---------
      
      Signed-off-by: default avatarAlexandru Gheorghe <[email protected]>
      64f4b156
    • Javier Viola's avatar
      [DNM] bump zombienet version (#2083) · ada1ac3d
      Javier Viola authored
      This version includes:
      - Move `spot` usage in CI to 50%
      - Fix `PodMonitor`, metrics will be relayed to grafana
      ada1ac3d
    • Marcin S.'s avatar
  5. Oct 29, 2023
  6. Oct 27, 2023
  7. Oct 25, 2023
    • Joshy Orndorff's avatar
      `polkadot-parachain-primitives` should not depend on `frame-support`. (#1897) · bdf18687
      Joshy Orndorff authored
      
      
      This PR does not make any functional changes to the code. Rather, it
      restructures the dependency graph.
      
      Before this PR, the crate `polkadot-parachain-primitives` depended
      directly on the crate `frame-support`. This is wrong in principal
      because a parachain does not necessarily have anything to do with frame.
      
      This dependency was only for the `Weight` type which was just a
      re-export from `sp-weights` anyway. So this PR changes the dependency to
      be directly on the much lighter `sp-weights`.
      
      ---------
      
      Co-authored-by: default avatarJoshy Orndorff <[email protected]>
      Co-authored-by: command-bot <>
      bdf18687
  8. Oct 24, 2023
  9. Oct 23, 2023
  10. Oct 21, 2023
  11. Oct 20, 2023
    • Bastian Köcher's avatar
      `xcm`: Change `TypeInfo::path` to not include `staging` (#1948) · f3bf5c1a
      Bastian Köcher authored
      The `xcm` crate was renamed to `staging-xcm` to be able to publish it to
      crates.io as someone as squatted `xcm`. The problem with this rename is
      that the `TypeInfo` includes the crate name which ultimately lands in
      the metadata. The metadata is consumed by downstream users like
      `polkadot-js` or people building on top of `polkadot-js`. These people
      are using the entire `path` to find the type in the type registry. Thus,
      their code would break as the type path would now be [`staging_xcm`,
      `VersionedXcm`] instead of [`xcm`, `VersionedXcm`]. This pull request
      fixes this by renaming the path segment `staging_xcm` to `xcm`.
      
      This requires: https://github.com/paritytech/scale-info/pull/197
      
      
      
      ---------
      
      Co-authored-by: default avatarFrancisco Aguirre <[email protected]>
      f3bf5c1a
  12. Oct 19, 2023
    • Bastian Köcher's avatar
      Do not force collators to update after enabling async backing (#1920) · b967ba53
      Bastian Köcher authored
      The validators are checking if async backing is enabled by checking the
      version of the runtime api. If the runtime api is upgraded by a runtime
      upgrade, the validators start to also enable the async backing logic.
      However, just because async backing is enabled, it doesn't mean that all
      collators and parachain runtimes have upgraded. This pull request fixes
      an issue about advertising collations to the relay chain when it has
      async backing enabled, but the collator is still using the old
      networking protocol. The implementation is actually backwards compatible
      as we can not expect that everyone directly upgrades. However, the
      collation advertisement logic was requiring V2 networking messages after
      async backing was enabled, which was wrong. This is now fixed by this
      pull request.
      
      Closes: https://github.com/paritytech/polkadot-sdk/issues/1923
      
      
      
      ---------
      
      Co-authored-by: default avatareskimor <[email protected]>
      b967ba53
  13. Oct 18, 2023
    • Bulat Saifullin's avatar
      Update kusama/polkadot bootnodes (#1895) · 3e980219
      Bulat Saifullin authored
      closes: https://github.com/paritytech/devops/issues/2090
      
      ## Changes
      1. Updated the list of bootnodes.
      2. Merged the Connect node and bootnode into a single node.
      3. Decreased the number of nodes.
      4. Updated the DNS name.
      
      ## Description
      The initial 8 bootnodes were planned to be replaced by community
      bootnodes, the community node was added but we did not bother to reduce
      the Parity managed bootnodes. Fixing it now.
      3e980219
    • Keith Yeung's avatar
      Introduce XcmFeesToAccount fee manager (#1234) · 3dece311
      Keith Yeung authored
      
      
      Combination of paritytech/polkadot#7005, its addon PR
      paritytech/polkadot#7585 and its companion paritytech/cumulus#2433.
      
      This PR introduces a new XcmFeesToAccount struct which implements the
      `FeeManager` trait, and assigns this struct as the `FeeManager` in the
      XCM config for all runtimes.
      
      The struct simply deposits all fees handled by the XCM executor to a
      specified account. In all runtimes, the specified account is configured
      as the treasury account.
      
      XCM __delivery__ fees are now being introduced (unless the root origin
      is sending a message to a system parachain on behalf of the originating
      chain).
      
      # Note for reviewers
      
      Most file changes are tests that had to be modified to account for the
      new fees.
      Main changes are in:
      - cumulus/pallets/xcmp-queue/src/lib.rs <- To make it track the delivery
      fees exponential factor
      - polkadot/xcm/xcm-builder/src/fee_handling.rs <- Added. Has the
      FeeManager implementation
      - All runtime xcm_config files <- To add the FeeManager to the XCM
      configuration
      
      # Important note
      
      After this change, instructions that create and send a new XCM (Query*,
      Report*, ExportMessage, InitiateReserveWithdraw, InitiateTeleport,
      DepositReserveAsset, TransferReserveAsset, LockAsset and RequestUnlock)
      will require the corresponding origin account in the origin register to
      pay for transport delivery fees, and the onward message will fail to be
      sent if the origin account does not have the required amount. This
      delivery fee is on top of what we already collect as tx fees in
      pallet-xcm and XCM BuyExecution fees!
      
      Wallet UIs that want to expose the new delivery fee can do so using the
      formula:
      
      ```
      delivery_fee_factor * (base_fee + encoded_msg_len * per_byte_fee)
      ```
      
      where the delivery fee factor can be obtained from the corresponding
      pallet based on which transport you are using (UMP, HRMP or bridges),
      the base fee is a constant, the encoded message length from the message
      itself and the per byte fee is the same as the configured per byte fee
      for txs (i.e. `TransactionByteFee`).
      
      ---------
      
      Co-authored-by: default avatarBranislav Kontur <[email protected]>
      Co-authored-by: default avatarjoe petrowski <[email protected]>
      Co-authored-by: default avatarGiles Cope <[email protected]>
      Co-authored-by: command-bot <>
      Co-authored-by: default avatarFrancisco Aguirre <[email protected]>
      Co-authored-by: default avatarLiam Aharon <[email protected]>
      Co-authored-by: default avatarKian Paimani <[email protected]>
      3dece311
    • joe petrowski's avatar
      Add Runtime Missing Crate Descriptions (#1909) · d3ea69b7
      joe petrowski authored
      Adds descriptions needed for publishing to crates.io.
      d3ea69b7
    • Serban Iorga's avatar
      Start BEEFY client by default for Polkadot nodes (#1913) · 6c39bb4a
      Serban Iorga authored
      Fellowship companion:
      https://github.com/polkadot-fellows/runtimes/pull/65
      
      This starts the BEEFY client by default for Polkadot nodes.
      
      Governance/sudo call is later required to enable/start consensus.
      
      Part of https://github.com/paritytech/parity-bridges-common/issues/2420
      6c39bb4a
    • Adrian Catangiu's avatar
      cumulus: add asset-hub-rococo runtime based on asset-hub-kusama and add... · 8b3905d2
      Adrian Catangiu authored
      
      cumulus: add asset-hub-rococo runtime based on asset-hub-kusama and add asset-bridging support to it (#1215)
      
      This commit adds Rococo Asset Hub dedicated runtime so we can test new
      features here, before merging them in Kusama Asset Hub.
      Also adds one such feature: asset transfer over bridge (Rococo AssetHub
      <> Wococo AssetHub)
      
      - clone `asset-hub-kusama-runtime` -> `asset-hub-rococo-runtime`
      - make it use Rococo primitives, names, assets, constants, etc
      - add asset-transfer-over-bridge support to Rococo AssetHub <> Wococo
      AssetHub
      
      Fixes #1128
      
      ---------
      
      Co-authored-by: default avatarBranislav Kontur <[email protected]>
      Co-authored-by: default avatarjoe petrowski <[email protected]>
      Co-authored-by: default avatarFrancisco Aguirre <[email protected]>
      8b3905d2
  14. Oct 17, 2023