`testnets-common` was introduced recently to start to separate testnet
parachain configurations from those of Polkadot and Kusama.

The `locks-review` and `polkadot-review` requirements are removed from
`parachains-common` in
https://github.com/paritytech/polkadot-sdk/pull/2564 and there are
[plans](https://github.com/paritytech/polkadot-sdk/pull/2564#discussion_r1410882115)
to move the Polkadot and Kusama contents of that package to the
fellowship, `testnets-common` is no longer needed.

This PR removes the crate and replaces uses of it in
`collectives-westend`, the only place it is currently used.

# Motivation (+testing)

### Enable easy `ForeignAssets` transfers using `pallet-xcm` 

We had just previously added capabilities to teleport fees during
reserve-based transfers, but what about reserve-transferring fees when
needing to teleport some non-fee asset?

This PR aligns everything under either explicit reserve-transfer,
explicit teleport, or this new flexible `transfer_assets()` which can
mix and match as needed with fewer artificial constraints imposed to the
user.

This will enable, for example, a (non-system) parachain to teleport
their `ForeignAssets` assets to AssetHub while using DOT to pay fees.
(the assets are teleported - as foreign assets should from their owner
chain - while DOT used for fees can only be reserve-based transferred
between said parachain and AssetHub).

Added `xcm-emulator` tests for this scenario ^.

# Description

Reverts `(limited_)reserve_transfer_assets` to only allow reserve-based
transfers for all `assets` including fees.

Similarly `(limited_)teleport_assets` only allows teleports for all
`assets` including fees.
    
For complex combinations of asset transfers where assets and fees may
have different reserves or different reserve/teleport trust
configurations, users can use the newly added `transfer_assets()`
extrinsic which is more flexible in allowing more complex scenarios.

`assets` (excluding `fees`) must have same reserve location or otherwise
be teleportable to `dest`.
No limitations imposed on `fees`.

- for local reserve: transfer assets to sovereign account of destination
chain and forward a notification XCM to `dest` to mint and deposit
reserve-based assets to `beneficiary`.
- for destination reserve: burn local assets and forward a notification
to `dest` chain to withdraw the reserve assets from this chain's
sovereign account and deposit them to `beneficiary`.
- for remote reserve: burn local assets, forward XCM to reserve chain to
move reserves from this chain's SA to `dest` chain's SA, and forward
another XCM to `dest` to mint and deposit reserve-based assets to
`beneficiary`.
- for teleports: burn local assets and forward XCM to `dest` chain to
mint/teleport assets and deposit them to `beneficiary`.

## Review notes

Only around 500 lines are prod code (see `pallet_xcm/src/lib.rs`), the
rest of the PR is new tests and improving existing tests.

---------

Co-authored-by: command-bot <>

Removes obsolete fields from the `QueueConfigData` structure. For the
remaining fields, if they use the old defaults, we replace them with the
new defaults.

Resolves: https://github.com/paritytech/polkadot-sdk/issues/1795

Westend SP dmp queue pallet removal is complete.

<img width="1499" alt="Screenshot 2023-11-28 at 08 31 27"
src="https://github.com/paritytech/polkadot-sdk/assets/16665596/906246fb-3de9-4133-a827-431636a097ad">

<img width="1499" alt="Screenshot 2023-11-28 at 08 32 08"
src="https://github.com/paritytech/polkadot-sdk/assets/16665596/bde84891-b044-42c7-9a0b-59125cd24db1">

<img width="1499" alt="Screenshot 2023-11-28 at 08 31 45"
src="https://github.com/paritytech/polkadot-sdk/assets/16665596/38337484-0856-45c0-b9ff-8c785bc3c0e3">

## Overview

This PR aligns the `spec_version` formatting to the [recent
changes](https://github.com/polkadot-fellows/runtimes/pull/26/files#diff-efa4caeb17487ecb13d8f5eb7863c3241d84afa2e73fbf25909a2ca89df0f362R142)
made for the Polkadot/Kusama runtimes.

It also backports the latest version `v1.4.0` bumps as `1_004_000`.

## Details

During the switch from `v0.9` to `v1.x`, the format of the
`spec_version` was modified from: `(M)m_ppp` for a runtime considered on
version `M.m.pp`. For instance `0.9.42` had a `spec_version` of `9420`.

With the transition to `v1.x`, the format was changed to a bigger number
(still `u32`) formatted as `MM_mm_ppp` where `1.2.3` would be stored as
`01_02_003`.

This PR aligns the format with that has been introduced in the
fellowship repo: `MMM_mmm_ppp`.

---------

Co-authored-by: Bastian Köcher <[email protected]>

Add collectives and glutton parachain westend runtimes to prepare for
#1737.

The removal of system parachain native runtimes #1737 is blocked until
chainspecs and runtime APIs can be dealt with cleanly (merge of #1256
and follow up PRs).

In the meantime, these additions are ready to be merged to `master`, so
I have separated them out into this PR.

Also marked `bridge-hub-westend` as unimplemented in line with [this
issue](https://github.com/paritytech/parity-bridges-common/issues/2602).

TODO
- [x] add to `command-bot` benchmarks
- [x] add to `command-bot-scripts` benchmarks
- [x] generate weights

---------

Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: Muharem <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Branislav Kontur <[email protected]>

## Motivation

`pallet-xcm` is the main user-facing interface for XCM functionality,
including assets manipulation functions like `teleportAssets()` and
`reserve_transfer_assets()` calls.

While `teleportAsset()` works both ways, `reserve_transfer_assets()`
works only for sending reserve-based assets to a remote destination and
beneficiary when the reserve is the _local chain_.

## Solution

This PR enhances `pallet_xcm::(limited_)reserve_withdraw_assets` to
support transfers when reserves are other chains.
This will allow complete, **bi-directional** reserve-based asset
transfers user stories using `pallet-xcm`.

Enables following scenarios:
- transferring assets with local reserve (was previously supported iff
asset used as fee also had local reserve - now it works in all cases),
- transferring assets with reserve on destination,
- transferring assets with reserve on remote/third-party chain (iff
assets and fees have same remote reserve),
- transferring assets with reserve different than the reserve of the
asset to be used as fees - meaning can be used to transfer random asset
with local/dest reserve while using DOT for fees on all involved chains,
even if DOT local/dest reserve doesn't match asset reserve,
- transferring assets with any type of local/dest reserve while using
fees which can be teleported between involved chains.

All of the above is done by pallet inner logic without the user having
to specify which scenario/reserves/teleports/etc. The correct scenario
and corresponding XCM programs are identified, and respectively, built
automatically based on runtime configuration of trusted teleporters and
trusted reserves.

#### Current limitations:
- while `fees` and "non-fee" `assets` CAN have different reserves (or
fees CAN be teleported), the remaining "non-fee" `assets` CANNOT, among
themselves, have different reserve locations (this is also implicitly
enforced by `MAX_ASSETS_FOR_TRANSFER=2`, but this can be safely
increased in the future).
- `fees` and "non-fee" `assets` CANNOT have **different remote**
reserves (this could also be supported in the future, but adds even more
complexity while possibly not being worth it - we'll see what the future
holds).

Fixes https://github.com/paritytech/polkadot-sdk/issues/1584
Fixes https://github.com/paritytech/polkadot-sdk/issues/2055

---------

Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Branislav Kontur <[email protected]>

(imported from https://github.com/paritytech/cumulus/pull/2157)

## Changes

This MR refactores the XCMP, Parachains System and DMP pallets to use
the [MessageQueue](https://github.com/paritytech/substrate/pull/12485)
for delayed execution of incoming messages. The DMP pallet is entirely
replaced by the MQ and thereby removed. This allows for PoV-bounded
execution and resolves a number of issues that stem from the current
work-around.

All System Parachains adopt this change.  
The most important changes are in `primitives/core/src/lib.rs`,
`parachains/common/src/process_xcm_message.rs`,
`pallets/parachain-system/src/lib.rs`, `pallets/xcmp-queue/src/lib.rs`
and the runtime configs.

### DMP Queue Pallet

The pallet got removed and its logic refactored into parachain-system.
Overweight message management can be done directly through the MQ
pallet.

Final undeployment migrations are provided by
`cumulus_pallet_dmp_queue::UndeployDmpQueue` and `DeleteDmpQueue` that
can be configured with an aux config trait like:

```rust
parameter_types! {
	pub const DmpQueuePalletName: &'static str = \"DmpQueue\" < CHANGE ME;
	pub const RelayOrigin: AggregateMessageOrigin = AggregateMessageOrigin::Parent;
}

impl cumulus_pallet_dmp_queue::MigrationConfig for Runtime {
	type PalletName = DmpQueuePalletName;
	type DmpHandler = frame_support::traits::EnqueueWithOrigin<MessageQueue, RelayOrigin>;
	type DbWeight = <Runtime as frame_system::Config>::DbWeight;
}

// And adding them to your Migrations tuple:
pub type Migrations = (
	...
	cumulus_pallet_dmp_queue::UndeployDmpQueue<Runtime>,
	cumulus_pallet_dmp_queue::DeleteDmpQueue<Runtime>,
);
```

### XCMP Queue pallet

Removed all dispatch queue functionality. Incoming XCMP messages are now
either: Immediately handled if they are Signals, enqueued into the MQ
pallet otherwise.

New config items for the XCMP queue pallet:
```rust
/// The actual queue implementation that retains the messages for later processing.
type XcmpQueue: EnqueueMessage<ParaId>;

/// How a XCM over HRMP from a sibling parachain should be processed.
type XcmpProcessor: ProcessMessage<Origin = ParaId>;

/// The maximal number of suspended XCMP channels at the same time.
#[pallet::constant]
type MaxInboundSuspended: Get<u32>;
```

How to configure those:

```rust
// Use the MessageQueue pallet to store messages for later processing. The `TransformOrigin` is needed since
// the MQ pallet itself operators on `AggregateMessageOrigin` but we want to enqueue `ParaId`s.
type XcmpQueue = TransformOrigin<MessageQueue, AggregateMessageOrigin, ParaId, ParaIdToSibling>;

// Process XCMP messages from siblings. This is type-safe to only accept `ParaId`s. They will be dispatched
// with origin `Junction::Sibling(…)`.
type XcmpProcessor = ProcessFromSibling<
	ProcessXcmMessage<
		AggregateMessageOrigin,
		xcm_executor::XcmExecutor<xcm_config::XcmConfig>,
		RuntimeCall,
	>,
>;

// Not really important what to choose here. Just something larger than the maximal number of channels.
type MaxInboundSuspended = sp_core::ConstU32<1_000>;
```

The `InboundXcmpStatus` storage item was replaced by
`InboundXcmpSuspended` since it now only tracks inbound queue suspension
and no message indices anymore.

Now only sends the most recent channel `Signals`, as all prio ones are
out-dated anyway.

### Parachain System pallet

For `DMP` messages instead of forwarding them to the `DMP` pallet, it
now pushes them to the configured `DmpQueue`. The message processing
which was triggered in `set_validation_data` is now being done by the MQ
pallet `on_initialize`.

XCMP messages are still handed off to the `XcmpMessageHandler`
(XCMP-Queue pallet) - no change here.

New config items for the parachain system pallet:
```rust
/// Queues inbound downward messages for delayed processing. 
///
/// Analogous to the `XcmpQueue` of the XCMP queue pallet.
type DmpQueue: EnqueueMessage<AggregateMessageOrigin>;
``` 

How to configure:
```rust
/// Use the MQ pallet to store DMP messages for delayed processing.
type DmpQueue = MessageQueue;
``` 

## Message Flow

The flow of messages on the parachain side. Messages come in from the
left via the `Validation Data` and finally end up at the `Xcm Executor`
on the right.

![Untitled
(1)](https://github.com/paritytech/cumulus/assets/10380170/6cf8b377-88c9-4aed-96df-baace266e04d)

## Further changes

- Bumped the default suspension, drop and resume thresholds in
`QueueConfigData::default()`.
- `XcmpQueue::{suspend_xcm_execution, resume_xcm_execution}` errors when
they would be a noop.
- Properly validate the `QueueConfigData` before setting it.
- Marked weight files as auto-generated so they wont auto-expand in the
MR files view.
- Move the `hypothetical` asserts to `frame_support` under the name
`experimental_hypothetically`

Questions:
- [ ] What about the ugly `#[cfg(feature = \"runtime-benchmarks\")]` in
the runtimes? Not sure how to best fix. Just having them like this makes
tests fail that rely on the real message processor when the feature is
enabled.
- [ ] Need a good weight for `MessageQueueServiceWeight`. The scheduler
already takes 80% so I put it to 10% but that is quite low.

TODO:
- [x] Remove c&p code after
https://github.com/paritytech/polkadot/pull/6271
- [x] Use `HandleMessage` once it is public in Substrate
- [x] fix `runtime-benchmarks` feature
https://github.com/paritytech/polkadot/pull/6966
- [x] Benchmarks
- [x] Tests
- [ ] Migrate `InboundXcmpStatus` to `InboundXcmpSuspended`
- [x] Possibly cleanup Migrations (DMP+XCMP)
- [x] optional: create `TransformProcessMessageOrigin` in Substrate and
replace `ProcessFromSibling`
- [ ] Rerun weights on ref HW

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: command-bot <>

closes https://github.com/paritytech/polkadot-sdk/issues/1882

## Breaking Changes

This PR introduces a new item to `pallet_balances::Config`:

```diff
trait Config {
++    type RuntimeFreezeReasons;
}
```

This value is only used to check it against `type MaxFreeze`. A similar
check has been added for `MaxHolds` against `RuntimeHoldReasons`, which
is already given to `pallet_balances`.

In all contexts, you should pass the real `RuntimeFreezeReasons`
generated by `construct_runtime` to `type RuntimeFreezeReasons`. Passing
`()` would also work, but it would imply that the runtime uses no
freezes at all.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>

Combination of paritytech/polkadot#7005, its addon PR
paritytech/polkadot#7585 and its companion paritytech/cumulus#2433.

This PR introduces a new XcmFeesToAccount struct which implements the
`FeeManager` trait, and assigns this struct as the `FeeManager` in the
XCM config for all runtimes.

The struct simply deposits all fees handled by the XCM executor to a
specified account. In all runtimes, the specified account is configured
as the treasury account.

XCM __delivery__ fees are now being introduced (unless the root origin
is sending a message to a system parachain on behalf of the originating
chain).

# Note for reviewers

Most file changes are tests that had to be modified to account for the
new fees.
Main changes are in:
- cumulus/pallets/xcmp-queue/src/lib.rs <- To make it track the delivery
fees exponential factor
- polkadot/xcm/xcm-builder/src/fee_handling.rs <- Added. Has the
FeeManager implementation
- All runtime xcm_config files <- To add the FeeManager to the XCM
configuration

# Important note

After this change, instructions that create and send a new XCM (Query*,
Report*, ExportMessage, InitiateReserveWithdraw, InitiateTeleport,
DepositReserveAsset, TransferReserveAsset, LockAsset and RequestUnlock)
will require the corresponding origin account in the origin register to
pay for transport delivery fees, and the onward message will fail to be
sent if the origin account does not have the required amount. This
delivery fee is on top of what we already collect as tx fees in
pallet-xcm and XCM BuyExecution fees!

Wallet UIs that want to expose the new delivery fee can do so using the
formula:

```
delivery_fee_factor * (base_fee + encoded_msg_len * per_byte_fee)
```

where the delivery fee factor can be obtained from the corresponding
pallet based on which transport you are using (UMP, HRMP or bridges),
the base fee is a constant, the encoded message length from the message
itself and the per byte fee is the same as the configured per byte fee
for txs (i.e. `TransactionByteFee`).

---------

Co-authored-by: Branislav Kontur <[email protected]>
Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: Giles Cope <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>

This PR implements [`GenesisBuilder`
API](https://github.com/paritytech/polkadot-sdk/blob/a414ea75

/substrate/primitives/genesis-builder/src/lib.rs#L38)
for all the runtimes in polkadot repo.

Step towards: paritytech/polkadot-sdk#25

---------

Co-authored-by: ordian <[email protected]>

The Ambassador Program on Polkadot Collectives Parachain

The Polkadot Ambassador Program has existed for a while; more
information can be found
[here](https://wiki.polkadot.network/docs/ambassadors).
In this PR, the program is being brought on chain.

### On Chain Structure
The on-chain program consists of nine ranks, divided into four
categories ([full
list](https://github.com/paritytech/cumulus/blob/c238fb26b75569a11abb57437fd14acd26e05f18/parachains/runtimes/collectives/collectives-polkadot/src/ambassador/mod.rs#L52)):
- Ambassadors (1-2 tiers)
- Senior Ambassadors (3-4 tiers)
- Head Ambassadors (5-7 tiers)
- Master Ambassadors (8-9 tiers)

Each rank has a corresponding `Origin` (e.g., `HeadAmbassadorsTier5` -
[full
list](https://github.com/paritytech/cumulus/blob/c238fb26b75569a11abb57437fd14acd26e05f18/parachains/runtimes/collectives/collectives-polkadot/src/ambassador/origins.rs#L35)),
which represents the collective voice of members of that rank and above.

### Referendum

The `AmbassadorReferenda` instance of [referenda
pallet](https://docs.rs/pallet-referenda/latest/pallet_referenda/)
consists of [nine
tracks](https://github.com/paritytech/cumulus/blob/c238fb26b75569a11abb57437fd14acd26e05f18/parachains/runtimes/collectives/collectives-polkadot/src/ambassador/tracks.rs#L51),
each corresponding to an `Origin`. A referendum taken on `senior
ambassador tier 4` track invites all members from rank 4 or above to
vote and commands `SeniorAmbassadors` `Origin`. Every member gets one
vote plus an additional vote for every excess rank. The referendum
proposal can be submitted by any member of a senior rank or above.

### Membership Management

Initial members will be brought on chain via migration, with subsequent
member management handled through the `AmbassadorCollective` instance of
[ranked collective
pallet](https://docs.rs/pallet-ranked-collective/latest/pallet_ranked_collective/).
Both `Root` and `FellowshipAdmin` `Origins`, commanded via public
Polkadot referendum, can promote or demote members to and from any rank.
Members themselves also have the power to promote or demote via its
referendum, with a senior member vote by the rank two above the new /
current rank - [full
configuration](https://github.com/paritytech/cumulus/blob/9ab6aa47063d7e8b67ddc10d9c136037f99c03a3/parachains/runtimes/collectives/collectives-polkadot/src/ambassador/mod.rs#L67).

### Content Management

The program's on-chain content is managed via the collectives content
pallet, allowing for setting its charter and making announcements. The
voice of head ambassadors have the authority to set the charter, while
announcements can be made by any senior rank member or through a
referendum among all members.

### Additional Functionality

The `AmbassadorCore` instance of [core fellowship
pallet](https://docs.rs/pallet-core-fellowship/latest/pallet_core_fellowship/)
decorates the ranked collectives pallet with features like salary
determination, activity/passivity registration, and the handling of
promotion and demotion periods. While the usage of this pallet is
optional in the first version, future updates will make it the exclusive
method for induction/promotion.

Periodic salaries in USDt, payable on Asset Hub, are introduced through
the [salary
pallet](https://docs.rs/pallet-salary/latest/pallet_salary/). This
requires induction into the ambassador core pallet.

Please for more information on the pallets' functionality refer to their
documentations.

### Next Steps:
- Migrate to seed the program members
- Mint ambassador NFT badges on Asset Hub when promoting
- Treasury pallet instance for the Ambassador Program

---------

Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: joe petrowski <[email protected]>

## Summary 
The term "common good parachain" has been abandoned in favor of "system
parachain" - e.g. [Joe's speech at
Decoded2023](https://youtu.be/CSO-ERHK2gY?t=456). This pull request
tries to fix and align code with this vision.

## Impact

The important change is implementation of `trait IsSystem` for `Id`
[here](https://github.com/paritytech/polkadot-sdk/pull/1406/files#diff-0b7b4f5b962a18ce980354592b55ab2a27b5a2e9f6f8089ec803ca73853e8583R225-R229)
where we changed condition from `< 1000` to `<= 1999`, which means that
all parachain IDs bellow 1999 (included) are considered as "system
parachain" IDs. This change has a direct impact on the following
components:

####
[ChildSystemParachainAsSuperuser](https://github.com/paritytech/polkadot-sdk/blob/master/polkadot/xcm/xcm-builder/src/origin_conversion.rs#L72-L88)
This origin converter is used for allowing to process XCM `Transact`
from "system parachain" on the relay chain - e.g. see [configuration for
Kusama](https://github.com/paritytech/polkadot-sdk/blob/master/polkadot/runtime/kusama/src/xcm_config.rs#L92-L101).
Only configured for Kusama, Westend, Rococo runtimes.

**No need for this feature anymore.** See
[comment](https://github.com/paritytech/polkadot-sdk/pull/1406#issuecomment-1708218715).

####
[IsChildSystemParachain](https://github.com/paritytech/polkadot-sdk/blob/master/polkadot/xcm/xcm-builder/src/barriers.rs#L310-L317)
`IsChildSystemParachain` is used with `AllowExplicitUnpaidExecutionFrom`
barrier for checking XCM programs (they have to start with
`UnpaidExecution` instruction).
Only configured for Kusama, Westend, Rococo runtimes.

**Overall the impact is low or mostly ok because it only allows unpaid
execution for "system parachains" (e.g. AssetHub, BridgeHub...) on the
relay chain.**

####
[SiblingSystemParachainAsSuperuser](https://github.com/paritytech/polkadot-sdk/blob/master/polkadot/xcm/xcm-builder/src/origin_conversion.rs#L94-L114)

Not used anywhere in `polkadot-sdk` repo.


## Unresolved Questions
- [ ] constants `LOWEST_USER_ID` and `LOWEST_PUBLIC_ID` seem to express
the same thing now, do we want to keep them both or deprecated one of
them? If so, which one?
- [x] determine impact for `ChildSystemParachainAsSuperuser`

## TODO

- [ ] when merged here, open PR to the `polkadot-fellows`

## Related Material
https://youtu.be/CSO-ERHK2gY?t=456

https://forum.polkadot.network/t/polkadot-protocol-and-common-good-parachains/866
https://wiki.polkadot.network/docs/learn-system-chains

Make Preimage pallet use Consideration instead of handling deposits
directly.

Other half of paritytech/substrate#13666.
Depends/based on #1361.

Script for the lazy migration that should be run manually once:
[migrate-preimage-lazy.py](https://github.com/ggwpez/substrate-scripts/blob/master/migrate-preimage-lazy.py).

## TODO

- [x] Migration code.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>

* add common libs

* asset hubs

* add westend

* bridge hubs

* collectives

* contracts

* emulated tests

* parachain bin

* delete collectives constants and update docs

* integration tests should have apache license (some missing, some needed changing)

* propagate features

* fmt

* Add missing Cumulus licenses

* Typo

* Add missing Substrate licenses

* Single job checking the sub-repos in steps

* Remove dates

* Remove dates

* Add missing (C)

* Update FRAME UI tests

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update more UI tests

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>

* Fix aura config

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix Cargo.toml files

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Ignore feature-dependant tests

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Allow dead code

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* restructure macro related exports

* restructure macro related exports

* wip

* wip

* update cargo lock

* refactor RuntimeDebug on unincluded segment

* fmt

* Companion: restructure `benchmarking` macro related exports (#3039)

* wip

* wip

* restructure benchmarking macro related exports

* add cargo lock

---------

Co-authored-by: parity-processbot <>

* Bump crate versions

* Bump spec_version

* Update substrate & polkadot

* min changes to make async backing compile

* (async backing) parachain-system: track limitations for unincluded blocks (#2438)

* unincluded segment draft

* read para head from storage proof

* read_para_head -> read_included_para_head

* Provide pub interface

* add errors

* fix unincluded segment update

* BlockTracker -> Ancestor

* add a dmp limit

* Read para head depending on the storage switch

* doc comments

* storage items docs

* add a sanity check on block initialize

* Check watermark

* append to the segment on block finalize

* Move segment update into set_validation_data

* Resolve para head todo

* option watermark

* fix comment

* Drop dmq check

* fix weight

* doc-comments on inherent invariant

* Remove TODO

* add todo

* primitives tests

* pallet tests

* doc comments

* refactor unincluded segment length into a ConsensusHook (#2501)

* refactor unincluded segment length into a ConsensusHook

* add docs

* refactor bandwidth_out calculation

Co-authored-by: Chris Sosnin <[email protected]>

* test for limits from impl

* fmt

* make tests compile

* update comment

* uncomment test

* fix collator test by adding parent to state proof

* patch HRMP watermark rules for unincluded segment

* get consensus-common tests to pass, using unincluded segment

* fix unincluded segment tests

* get all tests passing

* fmt

* rustdoc CI

* aura-ext: limit the number of authored blocks per slot (#2551)

* aura_ext consensus hook

* reverse dependency

* include weight into hook

* fix tests

* remove stray println

Co-authored-by: Chris Sosnin <[email protected]>

* fix test warning

* fix doc link

---------

Co-authored-by: Chris Sosnin <[email protected]>
Co-authored-by: Chris Sosnin <[email protected]>

* parachain-system: ignore go ahead signal once upgrade is processed (#2594)

* handle goahead signal for unincluded segment

* doc comment

* add test

* parachain-system: drop processed messages from inherent data (#2590)

* implement `drop_processed_messages`

* drop messages based on relay parent number

* adjust tests

* drop changes to mqc

* fix comment

* drop test

* drop more dead code

* clippy

* aura-ext: check slot in consensus hook and remove all `CheckInherents` logic (#2658)

* aura-ext: check slot in consensus hook

* convert relay chain slot

* Make relay chain slot duration generic

* use fixed velocity hook for pallets with aura

* purge timestamp inherent

* fix warning

* adjust runtime tests

* fix slots in tests

* Make `xcm-emulator` test pass for new consensus hook (#2722)

* add pallets on_initialize

* tests pass

* add AuraExt on_init

* ".git/.scripts/commands/fmt/fmt.sh"

---------

Co-authored-by: command-bot <>

---------

Co-authored-by: Ignacio Palacios <[email protected]>

* update polkadot git refs

* CollationGenerationConfig closure is now optional (#2772)

* CollationGenerationConfig closure is now optional

* fix test

* propagate network-protocol-staging feature (#2899)

* Feature Flagging Consensus Hook Type Parameter (#2911)

* First pass

* fmt

* Added as default feature in tomls

* Changed to direct dependency feature

* Dealing with clippy error

* Update pallets/parachain-system/src/lib.rs

Co-authored-by: asynchronous rob <[email protected]>

---------

Co-authored-by: asynchronous rob <[email protected]>

* fmt

* bump deps and remove warning

* parachain-system: update RelevantMessagingState according to the unincluded segment (#2948)

* mostly address 2471 with a bug introduced

* adjust relevant messaging state after computing total

* fmt

* max -> min

* fix test implementation of xcmp source

* add test

* fix test message sending logic

* fix + test

* add more to unincluded segment test

* fmt

---------

Co-authored-by: Chris Sosnin <[email protected]>

* Integrate new Aura / Parachain Consensus Logic in Parachain-Template / Polkadot-Parachain (#2864)

* add a comment

* refactor client/service utilities

* deprecate start_collator

* update parachain-template

* update test-service in the same way

* update polkadot-parachain crate

* fmt

* wire up new SubmitCollation message

* some runtime utilities for implementing unincluded segment runtime APIs

* allow parachains to configure their level of sybil-resistance when starting the network

* make aura-ext compile

* update to specify sybil resistance levels

* fmt

* specify relay chain slot duration in milliseconds

* update Aura to explicitly produce Send futures

also, make relay_chain_slot_duration a Duration

* add authoring duration to basic collator and document params

* integrate new basic collator into parachain-template

* remove assert_send used for testing

* basic-aura: only author when parent included

* update polkadot-parachain-bin

* fmt

* some fixes

* fixes

* add a RelayNumberMonotonicallyIncreases

* add a utility function for initializing subsystems

* some logging for timestamp adjustment

* fmt

* some fixes for lookahead collator

* add a log

* update `find_potential_parents` to account for sessions

* bound the loop

* restore & deprecate old start_collator and start_full_node functions.

* remove unnecessary await calls

* fix warning

* clippy

* more clippy

* remove unneeded logic

* ci

* update comment

Co-authored-by: Marcin S. <[email protected]>

* (async backing) restore `CheckInherents` for backwards-compatibility (#2977)

* bring back timestamp

* Restore CheckInherents

* revert to empty CheckInherents

* make CheckInherents optional

* attempt

* properly end system blocks

* add some more comments

* ignore failing system parachain tests

* update refs after main feature branch merge

* comment out the offending tests because CI runs ignored tests

* fix warnings

* fmt

* revert to polkadot master

* cargo update -p polkadot-primitives -p sp-io

---------

Co-authored-by: asynchronous rob <[email protected]>
Co-authored-by: Ignacio Palacios <[email protected]>
Co-authored-by: Bradley Olson <[email protected]>
Co-authored-by: Marcin S. <[email protected]>
Co-authored-by: eskimor <[email protected]>
Co-authored-by: Andronik <[email protected]>

* diff minimisation between runtimes

* another nit

* Update parachains/runtimes/assets/asset-hub-westend/src/xcm_config.rs

Co-authored-by: joe petrowski <[email protected]>

* Update parachains/runtimes/assets/asset-hub-polkadot/src/xcm_config.rs

Co-authored-by: joe petrowski <[email protected]>

* Upgrade english

* Removed unused `CheckedExtrinsic` (#2972)

---------

Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: Branislav Kontur <[email protected]>

* Copy rustfmt.toml from Polkadot master

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Format with new config

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add Polkadot clippy config

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update Cargo.lock

Looks like https://github.com/paritytech/polkadot/pull/7611 did not
correctly update the lockfile. Maybe a different Rust Version?!

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* replace Index for Nonce

* update lockfile for {"substrate", "polkadot"}

---------

Co-authored-by: parity-processbot <>

Moves `Block` to `frame_system` instead of `construct_runtime` and removes `Header` and `BlockNumber` (#2790)

* Fixes

* Removes unused import

* Uses Block and removes BlockNumber/Header from Chain

* Fixes bridges

* Fixes

* Removes unused import

* Fixes build

* Uses correct RelayBlock

* Minor fix

* Fixes glutton-kusama

* Uses correct RelayBlock

* Minor fix

* Fixes benchmark for pallet-bridge-parachains

* Adds appropriate constraints

* Minor fixes

* Removes unused import

* Fixes integrity tests

* Minor fixes

* Updates trait bounds

* Uses custom bound for AsPrimitive

* Fixes trait bounds

* Revert "Fixes trait bounds"

This reverts commit 0b0f42f583f3a616a88afe45fcd06d31e7d9a06f.

* Revert "Uses custom bound for AsPrimitive"

This reverts commit 838e5281adf8b6e9632a2abb9cd550db4ae24126.

* No AsPrimitive trait bound for now

* Removes bounds on Number

* update lockfile for {"substrate", "polkadot"}

* Formatting

* ".git/.scripts/commands/fmt/fmt.sh"

* Minor fix

---------

Co-authored-by: parity-processbot <>

* GenesisBuild<T,I> deprecated. BuildGenesisConfig added

* ".git/.scripts/commands/fmt/fmt.sh"

* integration-tests/emulated: ..Default::default added to genesis configs

* Cargo.lock updated

* Cargo.lock updated

* update lockfile for {"polkadot", "substrate"}

* clippy fixes

* clippy fixes

* clippy fixes again

---------

Co-authored-by: command-bot <>

* remove candidate when to invulnerable

* fix

* candidates to collators

* make parameters consistent and more reasonable

* add call to kick invulnerable candidates

* factor removal into weight

* fix: use accrue instead of add

* make set_invulnerables non-atomic

* benchmark add_invulnerable to account for candidate removal

* don't remove from candidates with set_invulnerables

* fix bounds on benchmarking

* protect against zero min invulnerables underflow

* extra event and tests

* make candidates/invulnerables self-cleaning on session change

* add integrity test

* unused imports

* make rococo-contracts have 1 collator

* Temporary fix for frame_system::set_code benchmark

* Removed temprary fix

* BenchmarkHelper for frame-system (TODO: add stuff for ParachainSystem OnSetCode)

* BenchmarkHelper for frame-system (TODO: add stuff for ParachainSystem OnSetCode)

* Glutton

* Fix benchmarks for `set_code`

* Changed dummy values to non-zero

* update lockfile for {"polkadot", "substrate"}

---------

Co-authored-by: parity-processbot <>

* Bump crate versions

* Bump spec_version to 9430

* Update all uses of pallet-aura to disallow multiple blocks per slot

* use ConstBool

* update lockfile for {"substrate", "polkadot"}

---------

Co-authored-by: parity-processbot <>

* core fellowship

* core fellowship weights

* salary

* weights

* fellowship pot

* registration period 15 days

* use treasury account for salary pay, promotion origin

* decision period for tracks 30 days

* docs

* comment

* Couple of fixes and some refactoring

* Alter curves to be a bit more conservative

* Use `PayOverXcm` for fellowship salary payments

* Docs and remove unneeded code

* Fixes

* Move Fellowship stuff in line with whitepaper

* fix: induction by a single Fellow (not proficient)

* doc fix

* renames, pallet index, allow unpaid for salary pallet

* Fix budget units

* Fixes

* Test sovereign account for Fellowship salaries

* Nice address test

* Fixes

* test for PayOverXcm setup

* Update parachains/runtimes/collectives/collectives-polkadot/src/fellowship/mod.rs

Co-authored-by: Bastian Köcher <[email protected]>

---------

Co-authored-by: Gav <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* add add and remove invulnerable dispatchables

* add migration

* fix benchmarking code

* add weights

* add migration to runtimes

* clippy

* pass SafeCallFilter

* make try-runtime work

* typos

Co-authored-by: Ankan <[email protected]>

* better insert and added test

* fix try-runtime update

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* Update pallets/collator-selection/src/migration.rs

* check events in test

* Update pallets/collator-selection/src/migration.rs

Co-authored-by: Kian Paimani <[email protected]>

* just dispatchresult

* only sp_std for try-runtime

---------

Co-authored-by: Ankan <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>

* Companion for: Substrate#13869

https://github.com/paritytech/substrate/pull/13869

* Fix

* Warning

* update lockfile for {"polkadot", "substrate"}

---------

Co-authored-by: parity-processbot <>

* Bump crate versions

* Bump spec_version to 9420

* Bump transaction_version (#2520)

* bump trnsaction_version

* revert transaction_version bump for all except the collectives

* make cargo fmt happy again

* Import Clippy config from Polkadot

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Auto clippy fix

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* No tabs in comments

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Prefer matches

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Dont drop references

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Trivial

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Refactor

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* fmt

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* add clippy to ci

* Clippy reborrow

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update client/pov-recovery/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/pov-recovery/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Partially revert 'Prefer matches'

Using matches! instead of match does give less compiler
checks as per review from @chevdor

.

Partially reverts 8c0609677f3ea040f77fffd5be6facf7c3fec95c

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update .cargo/config.toml

Co-authored-by: Chevdor <[email protected]>

* Revert revert 💩



Should be fine to use matches! macro since it is an explicit whitelist,
not wildcard matching.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: alvicsam <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Chevdor <[email protected]>
Co-authored-by: parity-processbot <>

* max proposal weight config

* update deps

---------

Co-authored-by: parity-processbot <>

* Fellowship into Collectives

* cargo.lock

* tracks alias

* allow to send Fellows origin over XCM

* update todos, remove duplication of type

* use Collectives location for Fellows body

* alias for ranks constants

* benchmarks

* proxy for Fellowship

* docs

* correct copyright date

* Apply suggestions from code review

Co-authored-by: joe petrowski <[email protected]>

* rustfmt

* remove council, update origins

* renames

* remove tech committee from promote origin

* renames

* Fellowship import

* test

* rename mod

* fix import

* updated addresses (only ss58 version)

* update addresses

* doc nits

* weights with new api

* update addresses

* fix try runtime

* update addresses

* use pallet api to import the members

* merge fix

* hex-literal version

* add Bradley to the 1 rank

---------

Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: parity-processbot <>

* Bump crate versions

* Bump spec_version to 9400

* bump transaction versions (#2364)

* Fellowship into Collectives

* cargo.lock

* tracks alias

* allow to send Fellows origin over XCM

* update todos, remove duplication of type

* use Collectives location for Fellows body

* alias for ranks constants

* benchmarks

* proxy for Fellowship

* docs

* correct copyright date

* Apply suggestions from code review

Co-authored-by: joe petrowski <[email protected]>

* rustfmt

* remove council, update origins

* renames

* remove tech committee from promote origin

* renames

* doc nits

* weights with new api

* sane weights for pallet_scheduler

* XCM tweaks for OpenGov (#2305)

* updated xcm configs for collectives and statemint

* remove xcm send from safe filter

* remove prod_or_fast

* remove empty line

* drop redundant aggregation

* remove match arm duplication

---------

Co-authored-by: joe petrowski <[email protected]>

* Fix APIs

* Reflect API changes

* Everything builds

* Fixes

* Fixes

* Update Cargo.toml

* Fixes

* Fixes

* No networks use freezes/holds

* update lockfile for {"polkadot", "substrate"}

* Fix test

ED cannot be zero anymore.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix test

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: parity-processbot <>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>