* collation-generation: use persisted validation data

* node: remote FullValidationData API

* runtime: remove FullValidationData API

* backing tests: use persisted validation data

* FullCandidateReceipt: use persisted validation data

This is not a big change since this type is not used anywhere

* Remove ValidationData and TransientValidationData

Also update the guide

* Companion for Substrate#7692

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Drive by fixes

The visibility modifiers are remnants of the previous structure where
HRMP wasn't a standalone module, by rather a submodule of the router
module.

* Add Currency assoc type to Config

This would allow us to reserve balance for deposits. This commit also
integrates the HRMP module in rococo, test-runtime and mocks to use the
balances pallet.

* Fix a bug that doesn't increment the age

In case the request is not confirmed, the age would be incremented but
not persisted.

* Fix cleaning the indexes

Before that change, the cleaning of the channel indexes was wrong, because it
naively removed entire rows that was pertaining to the para we delete.
This approach is flawed because it doesn't account for the rows that are
pertaining to other paras that contain the outgoing one.

This clearly violates the invariant imposed on the indexes, that all
the index rows must contain alive paras, but apart from that it also
lead to the situation where ingress index would contain the a different
set of channels that an egress have.

* Reserve currency for opening the channels

Note the ugly `unique_saturated_into` calls. The reason for them is the
currency trait accepts and defines the `Balance` associated type and the
deposit values are coming from the `HostConfiguration` where they are
defined using the `Balance`.

I figured that parameterising `HostConfiguration` would be annoying. On
the other hand, I don't expect these `unique_saturated_into` calls to
give us problems since it seems to be a reasonable assumption that this
module will be instantiated within a runtime where the Currency provided
will have a Balance that matches the one used in the configuration.

* Tests: Adapt `run_to_block` so that it submits a proper config

* Tests: exercise the deposit logic

* runtime: add Babe::next_epoch runtime api method

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Companion for #7810

* Added missing trait items for tests

* Add another missing trait item

* fixup

* "Update Substrate"

Co-authored-by: parity-processbot <>

* migrate claims to weightinfo

* fix up

* fix benchmark

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=claims --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/

* fix test runtime

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=claims --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/

* Update runtime/kusama/src/weights/claims.rs

* use path for pallet

* Update Cargo.lock

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_common::claims --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/

* Delete runtime_common::claims.rs

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_common::claims --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_common_claims.rs

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_common::claims --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_common_claims.rs

* use full automation file

* patch import

* consolidate benchmark logic

* fix

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_common::claims --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_common_claims.rs

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_common::claims --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_common_claims.rs

* update weight comments

Co-authored-by: Parity Benchmarking Bot <[email protected]>

* point to branch

* update babe API implementations

* build

* Revert "point to branch"

This reverts commit 669fde1e58cac1eef1847d5e6b39520fa4953f33.

* update Cargo.lock

* bump

* guide: add candidate information to OccupiedCore

* add descriptor and hash to occupied core type

* guide: add candidate hash to inclusion

* runtime: return candidate info in core state

* bitfield signing: stop querying runtime as much

* minimize going to runtime in availability distribution

* fix availability distribution tests

* guide: remove para ID from Occupied core

* get all crates compiling

* plumbing for rewarding backers

* give validators reward points for participating

* fix tests

* add bitfield rewarding

* add mocks for backing rewards

* add testing for backing & availability rewards

* implement RewardValidators on top of staking

* add to test-runtime and rococo

* add to test-runtime & rococo

* point to source on rewards values

* fix common tests

* do not reward availability anymore

* guide: merge backing and approval keys

* bump substrate master & update primitives

* use new SessionInfo struct in session_info

* session keys upgrade for Polkadot

* kusama & westend runtimes

* bump westend, kusama, and polkadot versions

* add session key to rococo & test-runtime

* update prepare-test-net to latest subkey

* update chain specs to support new para_assignment session key

* get cargo.lock from master

* formatting

* update kill_storage based on substrate master

* fix test-service

* assgn -> asgn

* use session info module for assignment session key

* Change branch.

* Update runtime.

* Revert "Change branch."

This reverts commit 841c59f3398136c27cc235a29d7d459e8a4c8ce0.

* Update substrate.

* Fix tests.

* Fix compilation.

* Fix frame system imports.

* Fix usages of system

* Fix stuff.

* Fix compilation.

* Fixes.

* Fix block_weight usage.

* Bump substrate.

* rename Trait -> Config

* revert diener changes

* rename HostConfig to ActiveConfig as more meaningful

* fix merge

* "Update Substrate"

* cargo update -p sp-io

Co-authored-by: parity-processbot <>

* guide: move erasure-root to candidate descriptor

* primitives: move erasure root to descriptor

* guide: unify candidate commitments and validation outputs

* primitives: unify validation outputs and candidate commitments

* parachains-runtime: fix fallout

* runtimes: fix fallout

* collation generation: fix fallout

* fix stray reference in primitives

* fix fallout in node-primitives

* fix remaining fallout in collation generation

* fix fallout in candidate validation

* fix fallout in runtime API subsystem

* fix fallout in subsystem messages

* fix fallout in candidate backing

* fix fallout in availability distribution

* don't clone

* clone

Co-authored-by: Sergei Shulepov <[email protected]>

* guide: fix formatting for SessionInfo module

* primitives: SessionInfo type

* punt on approval keys

* ah, revert the type alias

* session info runtime module skeleton

* update the guide

* runtime/configuration: sync with the guide

* runtime/configuration: setters for newly added fields

* runtime/configuration: set codec indexes

* runtime/configuration: update test

* primitives: fix SessionInfo definition

* runtime/session_info: initial impl

* runtime/session_info: use initializer for session handling (wip)

* runtime/session_info: mock authority discovery trait

* guide: update the initializer's order

* runtime/session_info: tests skeleton

* runtime/session_info: store n_delay_tranches in Configuration

* runtime/session_info: punt on approval keys

* runtime/session_info: add some basic tests

* Update primitives/src/v1.rs

* small fixes

* remove codec index annotation on structs

* fix off-by-one error

* validator_discovery: accept a session index

* runtime: replace validator_discovery api with session_info

* Update runtime/parachains/src/session_info.rs

Co-authored-by: Sergei Shulepov <[email protected]>

* runtime/session_info: add a comment about missing entries

* runtime/session_info: define the keys

* util: expose connect_to_past_session_validators

* util: allow session_info requests for jobs

* runtime-api: add mock test for session_info

* collator-protocol: add session_index to test state

* util: fix error message for runtime error

* fix compilation

* fix tests after merge with master

Co-authored-by: Sergei Shulepov <[email protected]>

* update tiny-keccak to 0.2

* update deps except bitvec and shared_memory

* fix some warning after futures upgrade

* remove useless package rename caused by bug in cargo-upgrade

* revert parity-util-mem *

* remove unused import

* cargo update

* remove all renames on parity-scale-codec

* remove the leftovers

* remove unused dep

* Guide: Split router module in guide.

Now we have: DMP, UMP and Router module.

* Add a glossary entry for what used to be called Router

* Extract DMP

* Extract UMP

* Extract HRMP

* Switch over to new modules

* Router: goodbye sweet prince

* Link to messaging overview for details.

* Update missed rococo and test runtimes.

* Commit destroyed by rebase changes

* Don't deprecate Router but rather make it a meta-project

Co-authored-by: Bernhard Schuster <[email protected]>

* Fix typos suggestion

Co-authored-by: Bernhard Schuster <[email protected]>

* Fix repetition in the impl guide

* Clarify that processed_downward_messages has the u32 type

* Remove the router subdir.

* Deabbreviate DMP,UMP,HRMP

Co-authored-by: Bernhard Schuster <[email protected]>

* Introduce CollatorFn type alias

* Make test-runtime imports consistent with rococo-runtime

* Update node/primitives/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* fix warnings

Co-authored-by: Bastian Köcher <[email protected]>

* Adds integration test based on adder collator

This adds an integration test for parachains that uses the adder
collator. The test will start two relay chain nodes and one collator and
waits until 4 blocks are build and enacted by the parachain.

* Make sure the integration test is run in CI

* Fix wasm compilation

* Update parachain/test-parachains/adder/collator/src/lib.rs

Co-authored-by: Sergei Shulepov <[email protected]>

* Update cli/src/command.rs

Co-authored-by: Sergei Shulepov <[email protected]>

* HRMP: Update the impl guide

* HRMP: Incorporate the channel notifications into the guide

* HRMP: Renaming in the impl guide

* HRMP: Constrain the maximum number of HRMP messages per candidate

This commit addresses the HRMP part of https://github.com/paritytech/polkadot/issues/1869

* XCM: Introduce HRMP related message types

* HRMP: Data structures and plumbing

* HRMP: Configuration

* HRMP: Data layout

* HRMP: Acceptance & Enactment

* HRMP: Test base logic

* Update adder collator

* HRMP: Runtime API for accessing inbound messages

Also, removing some redundant fully-qualified names.

* HRMP: Add diagnostic logging in acceptance criteria

* HRMP: Additional tests

* Self-review fixes

* save test refactorings for the next time

* Missed a return statement.

* a formatting blip

* Add missing logic for appending HRMP digests

* Remove the channel contents vectors which became empty

* Tighten HRMP channel digests invariants.

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* Remove a note about sorting for channel id

* Add missing rustdocs to the configuration

* Clarify and update the invariant for HrmpChannelDigests

* Make the onboarding invariant less sloppy

Namely, introduce `Paras::is_valid_para` (in fact, it already is present
in the implementation) and hook up the invariant to that.

Note that this says "within a session" because I don't want to make it
super strict on the session boundary. The logic on the session boundary
should be extremely careful.

* Make `CandidateCheckContext` use T::BlockNumber for hrmp_watermark

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* fix: ensure candidate validation gets code based on occupied core assumption

* guide: runtime API for historical validation code

* add historical runtime API

* integrate into runtime API subsystem

* remove blocked TODO

* fix service build: enable notifications protocol only under real overseer

* Update node/subsystem/src/messages.rs

Co-authored-by: Sergei Shulepov <[email protected]>

* fix compilation

Co-authored-by: Robert Habermeier <[email protected]>
Co-authored-by: Sergei Shulepov <[email protected]>

* UMP: Update the impl guide

* UMP: Incorporate XCM related changes into the guide

* UMP: Data structures and configuration

* UMP: Initial plumbing

* UMP: Data layout

* UMP: Acceptance criteria & enactment

* UMP: Fix dispatcher bug and add the test for it

* UMP: Constrain the maximum size of an UMP message

This commit addresses the UMP part of https://github.com/paritytech/polkadot/issues/1869

* Fix failing test due to misconfiguration

* Make the type of RelayDispatchQueueSize be more apparent in the guide

* Revert renaming `max_upward_queue_capacity` to `max_upward_queue_count`

* convert spaces to tabs

Co-authored-by: Bernhard Schuster <[email protected]>

* Update runtime/parachains/src/router/ump.rs

Co-authored-by: Bernhard Schuster <[email protected]>

Co-authored-by: Bernhard Schuster <[email protected]>

* adjustments for substrate PR

https://github.com/paritytech/substrate/pull/6912

* Update runtime/kusama/src/lib.rs

Co-authored-by: Guillaume Thiolliere <[email protected]>

* Apply suggestions from code review

Co-authored-by: Guillaume Thiolliere <[email protected]>

* update substrate

Co-authored-by: Guillaume Thiolliere <[email protected]>

* DMP: data structures and plumbing

* DMP: Implement DMP logic in the router module

DMP: Integrate DMP parts into the inclusion module

* DMP: Introduce the max size limit for the size of a downward message

* DMP: Runtime API for accessing inbound messages

* OCD

Small clean ups

* DMP: fix the naming of the error

* DMP: add caution about a non-existent recipient

At some point we had aliases for all the substrate crates but at some point we've switched to the full names to avoid confusion. It seems that we missed this instance.

* Improve and unify testing facilities

This improves the testing facilities by making the test client easier to
use. It also removes code that is not required for the test client.
Besides that it also moves the test service and test client under
`node/test`.

* Update Cargo.lock

* Update node/test/client/src/block_builder.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Remove explicit lifetime annotation

* Fix warnings and add extra `BlockBuilderExt`

Co-authored-by: Andronik Ordian <[email protected]>

* annoying whitespaces

* update guide

Add `CheckValidationOutputs` runtime api and also change the
candidate-validation stuff

* promote ValidationOutputs to global primitives

i.e. move it from node specific primitives to global v1 primitives. This
will be needed when we share it later in the runtime inclusion module

* refactor acceptance checks in the inclusion module

factor out the common code to share it during the block inclusion and
for the forthcoming CheckValidationOutputs runtime api.

Also note that the acceptance criteria was updated to incorporate checks
that exist now in candidate-validation

* plumb the runtime api outside

* extract validation_data from ValidationOutputs

* use runtime-api to check validation outputs

apart from that refactor, update docs and tidy a bit

* Update the maxium code size

This is to fix a test that performs an upgrade.

* Make register/deregister parachain dispatchables

This makes `register_parachain` and `deregister_parachain` of
`Registrar` dispatchables.

Besides that it brings back the functionality of the test node to
register a parachain.

* Fix tests

* PR review comments

* make polkadot work again,

* Fix build

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Remove old service, 3rd try

i.e.
Revert "Revert "Remove Old Service, 2nd try (#1732)" (#1758)"

This reverts commit 9a0f08bf

.

Closes #1757.

We now have some evidence that the polkadot validator was producing
blocks after all; the reason the blocks_constructed metric was 0 was
that as a new metric it hadn't yet been incorporated into that
branch's codebase. See
https://github.com/paritytech/polkadot/issues/1757#issuecomment-700977602

As this PR is based on a newer `master` branch than the previous one,
that should hopefully no longer be an issue.

* paras trait now has an Origin type

* initial work running a two node local net

* use the right incantations so the nodes produce blocks together

* improve internal documentation

Co-authored-by: Bastian Köcher <[email protected]>

* Companion for substrate/pull/7215

* More fixes

* Fix build

* fix

* Fix again with normal

* Fix build

* Latest changes

* "Update Substrate"

Co-authored-by: parity-processbot <>

This reverts commit 96f26155.

* Restore "Remove service, migrate all to service-new (#1630)"

i.e.
Revert "Revert "Remove service, migrate all to service-new (#1630)" (#1731)"

This reverts commit b4457f55

.

This allows us to get the changeset from #1630 into a new branch
which can be merged sometime in the future after appropriate burnin
tests have completed.

* remove ',)' from codebase outside of macros

* restore bdfl-preferred formatting

* attempt to improve destructuring formatting

* rename polkadot-service-new -> polkadot-service

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* remove unused import

* Update runtime/rococo-v1/README.md

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Andronik Ordian <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* Companion for `ModuleToIndex` to `PalletInfo` rename

* "Update Substrate"

Co-authored-by: parity-processbot <>
Co-authored-by: Benjamin Kampmann <[email protected]>

* Add WeightInfo to traits

* Update Cargo.lock

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Update `pallet_offences` trait

* remove session weight tests

* "Update Substrate"

* Update Cargo.lock

* Update Cargo.lock

* remove unused code

Co-authored-by: parity-processbot <>

This reverts commit d531ba56.

* import rococo into chain-spec

* make a few stabs at moving forward

* wip: rococo readme

* remove /service crate

- Move the chain-spec files to node-service
- update sufficient cargo files that polkadot-service-new builds
- not everything else builds yet

* wip: chase down some build errors in polkadot-cli

There's a lot more to go, but some progress has happened.

* make more progress getting polkadot-cli to build

* don't ignore polkadot.json within the res directory

* don't recreate pathbufs

* Prepare Polkadot to be used by Cumulus

This begins to make Polkadot usable from Cumulus.

* Remove old test

* migrate new_chain_ops fix from /service

* partially remove node/test-service

* Reset some changes

* Revert "partially remove node/test-service"

This reverts commit 7b8f9ba5bfc286a309df89853ae11facf3277ffb.

* WIP: replace v0 ParachainHost impl with v1 for test runtime

This is necessary because one of the current errors when building
the test service boils down to:

the trait bound `polkadot_test_runtime::RuntimeApiImpl<...>`:
  `polkadot_primitives::v1::ParachainHost<...>` is not satisfied

This is WIP because it appears to be causing some std leakage into
the wasm environment, or something; the compiler is currently
complaining about duplicate definitions of `panic_handler` and `oom`.
Presumably I have to identify all std types (Vec etc) and replace
them with sp_std equivalents.

* fix test runtime build

it wasn't std leakage, after all

* bump westend spec version

* use service-new as service within cli

* to revert: demo that forwarding the test runtime to the real impl blows up

* Revert "to revert: demo that forwarding the test runtime to the real impl blows up"

This reverts commit 68d2f385f378721c7433e3e39133434610cd2a51.

* Revert "Revert "to revert: demo that forwarding the test runtime to the real impl blows up""

This reverts commit 04cb1cbf8873b4429cb9c9fdccb7f4bb137dc720.

Might have just forgotten to disable default features

* More reverts

* MOARE

* plug in the runtime as the generic instantiation

This feels closer to a solution, but it still has problems: in particular,
it's assumed that Runtime implements all appropriate Trait traits,
which this one apparently does not.

* implement necessary traits to get the test runtime compiling

This is almost certainly not correct in some way; it really
looks like I need to mess with the construct_runtime! macro
somehow, to inject the inclusion trait's event type as a Event
variant. Still, better lock down this changeset while it all
compiles.

* add inclusion::Event as variant into Event enum

* implement unimplemented bits in kusama

* implement unimplemented bits in polkadot runtime

* implement unimplemented bits in westend runtime

* migrate client upgrades from master

* update test service with new node changes

* package metadata--that wasn't intended to be removed

* add parachains v1 modules to each runtime

It's not clear what precisely this does, but it's probably the right
thing to do.

* enable cli to opt out of full node features

* adjust rococo chainspec per example

https://github.com/paritytech/polkadot/blob/26f1fa47f7836ab4bee5d4aad127ebce748320dd/service/src/chain_spec.rs#L362

* try to fix Cargo.lock

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Andronik Ordian <[email protected]>

* Add weights and new trait

* update more trait instances

* Use () for tests

* update for new type

* "Update Substrate"

* bump spec version

* bump more spec

Co-authored-by: parity-processbot <>

* remove v0 parachains modules and switch to v1 primitives

* get tests compiling for runtime-common

* remove registrar module

* Add a dummy module

* remove runtime-parachains

* mostly remove old parachains code from polkadot-runtime

* remove slots::Trait implementation

* remove sp_std prelude import

* add a ZeroSizedTypeDifferentiator to dummy

* finish porting over polkadot runtime

* ZeroSizedTypeDifferentiator was actually unnecessary

* westend

* kusama

* test-runtime (no dummy modules)

* fix warning

* fix chain-specs

* fix test-service

* test-client

* remove dead import

* remove unused needed_extrinsics parameter

* runtimes compile

* remove rococo-v0

* remove remaining references to Rococo

* bump versions