* Companion for Substrate#7318

* "Update Substrate"

Co-authored-by: parity-processbot <>

This reverts commit a7b6c91f.

The authority discovery module was initially enabled by default on
validator and sentry nodes with commit 59f5eb4c. This change was later on
reverted in a7b6c91f. With this commit the authority discovery module is
again enabled by default.

They are not node handles anymore ;)

The collator is now enabled in a different way and we don't require this
argument anymore.

* validator discovery: use incremental updates for priority_group

* validator discovery: fix compilation

* validator discovery: remove Sync bound on Net

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Fix keystore types

* Use SyncCryptoStorePtr

* Borrow keystore

* Fix unused imports

* Fix polkadot service

* Fix bitfield-distribution tests

* Fix indentation

* Fix backing tests

* Fix tests

* Fix provisioner tests

* Removed SyncCryptoStorePtr

* Fix services

* Address PR feedback

* Address PR feedback - 2

* Update CryptoStorePtr imports to be from sp_keystore

* Typo

* Fix CryptoStore import

* Document the reason behind using filesystem keystore

* Remove VALIDATORS

* Fix duplicate dependency

* Mark sp-keystore as optional

* Fix availability distribution

* Fix call to sign_with

* Fix keystore usage

* Remove tokio and fix parachains Cargo config

* Typos

* Fix keystore dereferencing

* Fix CryptoStore import

* Fix provisioner

* Fix node backing

* Update services

* Cleanup dependencies

* Use sync_keystore

* Fix node service

* Fix node service - 2

* Fix node service - 3

* Rename CryptoStorePtr to SyncCryptoStorePtr

* "Update Substrate"

* Apply suggestions from code review

* Update node/core/backing/Cargo.toml

* Update primitives/src/v0.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Fix wasm build

* Update Cargo.lock

Co-authored-by: parity-processbot <>
Co-authored-by: Bastian Köcher <[email protected]>

* make polkadot work again,

* Fix build

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Remove old service, 3rd try

i.e.
Revert "Revert "Remove Old Service, 2nd try (#1732)" (#1758)"

This reverts commit 9a0f08bf

.

Closes #1757.

We now have some evidence that the polkadot validator was producing
blocks after all; the reason the blocks_constructed metric was 0 was
that as a new metric it hadn't yet been incorporated into that
branch's codebase. See
https://github.com/paritytech/polkadot/issues/1757#issuecomment-700977602

As this PR is based on a newer `master` branch than the previous one,
that should hopefully no longer be an issue.

* paras trait now has an Origin type

* initial work running a two node local net

* use the right incantations so the nodes produce blocks together

* improve internal documentation

Co-authored-by: Bastian Köcher <[email protected]>

* Make `AllSubsystems` usage easier in tests

This makes the usage of `AllSubsystems` easier in tests by introducing
new methods.

- `dummy` initializes `AllSubsystems` with all systems set to dummy
- `replace_*` to replace any subsystem

Besides that this pr adds a `ForwardSubsystem` that is also useful for
tests. This subsystem will forward all incoming messages to the given channel.

* Update node/overseer/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Update node/subsystem/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Update node/subsystem/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Move ForwardSubsystem and add a test

* Break some lines

Co-authored-by: Andronik Ordian <[email protected]>

* Land XCM Builder

* Clean up Cargo dependencies

Co-authored-by: Gavin Wood <[email protected]>

* Advertise to already connected validators

* Merge the loops and check the view

* Extend a test to capture new logic

* Fix a comment

* Update node/network/collator-protocol/src/collator_side.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Update comment

Co-authored-by: Andronik Ordian <[email protected]>

This pr changes the collation function to also pass the current relay
parent the parachain block should be build on.

* chain-api subsystem: implement BlockHeader messsage

* update the guide

* collator: fix a typo

* collator: fix more typos

* collator: fix even more typos

* stupid, but it compiles

* redo

* cleanup

* add ValidatorDiscovery to msgs

* sketch network bridge code

* ConnectToAuthorities instead of validators

* more stuff

* cleanup

* more stuff

* complete ConnectToAuthoritiesState

* Update node/network/bridge/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* Collator protocol subsystem (#1659)

* WIP

* The initial implementation of the collator side.

* Improve comments

* Multiple collation requests

* Add more tests and comments to validator side

* Add comments, remove dead code

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* Fix build after suggested changes

* Also connect to the next validator group

* Remove a Future impl and move TimeoutExt to util

* Minor nits

* Fix build

* Change FetchCollations back to FetchCollation

* Try this

* Final fixes

* Fix build

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* handle multiple in-flight connection requests

* handle cancelled requests

* Update node/core/runtime-api/src/lib.rs

Co-authored-by: Bernhard Schuster <[email protected]>

* redo it again

* more stuff

* redo it again

* update comments

* workaround Future is not Send

* fix trailing spaces

* clarify comments

* bridge: fix compilation in tests

* update more comments

* small fixes

* port collator protocol to new validator discovery api

* collator tests compile

* collator tests pass

* do not revoke a request when the stream receiver is closed

* make revoking opt-in

* fix is_fulfilled

* handle request revokation in collator

* tests

* wait for validator connections asyncronously

* fix compilation

* relabel my todos

* apply Fedor's patch

* resolve reconnection TODO

* resolve revoking TODO

* resolve channel capacity TODO

* resolve peer cloning TODO

* resolve peer disconnected TODO

* resolve PeerSet TODO

* wip tests

* more tests

* resolve Arc TODO

* rename pending to non_revoked

* one more test

* extract utility function into util crate

* fix compilation in tests

* Apply suggestions from code review

Co-authored-by: Fedor Sakharov <[email protected]>

* revert pin_project removal

* fix while let loop

* Revert "revert pin_project removal"

This reverts commit ae7f529d8de982ef66c3007dd1ff74c6ddce80d2.

* fix compilation

* Update node/subsystem/src/messages.rs

* docs on pub items

* guide updates

* remove a TODO

* small guide update

* fix a typo

* link to the issue

* validator discovery: on_request docs

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>
Co-authored-by: Fedor Sakharov <[email protected]>
Co-authored-by: Bernhard Schuster <[email protected]>

* parachains: use the root origin for mock

* registrar: notify the router module about the cleanup

* add missing call to schedule_para_cleanup

This pr changes the collator interface function to return an optional
collation instead of a collation. This is required as the parachain
itself can fail to generate a valid collation for various reason. Now if
the collation fails it will return `None`.

Besides that the pr adds some `RuntimeDebug` derive for `ValidationData`
and removes some whitespaces.

Co-authored-by: Gavin Wood <[email protected]>

* Bump version

* Update RELEASE.md

Co-authored-by: Shawn Tabrizi <[email protected]>

* initial work running a two node local net

* use the right incantations so the nodes produce blocks together

* improve internal documentation

* Companion for substrate/pull/7215

* More fixes

* Fix build

* fix

* Fix again with normal

* Fix build

* Latest changes

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Remove stale upgrades

* add back origin migration

* remove unused

* Update benchmark list

* copy weights from substrate

* Run benchmarks on Polkadot

* Update kusama benchmark list

* update westend benchmark list

* Run benchmarks on Kusama

* update payout staking weights

* cargo update -p sp-io

* remove babe and grandpa from benchmark list

* remove babe and grandpa weights

* re-run staking weights

* update staking weights

* copy weights from Polkadot into Westend

* Update Cargo.lock

* Initial commit.

* Fix build

* Add comments, remove Event

* Dont expose calls

* Remove TODO and origins

* Fix merge

* Enable or disable parathread registration

* overseer metrics: messages relayed

* provisioner metrics: cosmetic changes

* candidate selection metrics: cosmetic changes

* availability bitfields metrics

* availability distribution metrics

* PoV distribution metrics

* statement-distribution: small simplification

* statement-distribution: extract log target into a const

* statement-distribution: metrics

* address review nits

* Companion for https://github.com/paritytech/substrate/pull/7236

* "Update Substrate"

Co-authored-by: parity-processbot <>

* change (CI): soon to be deprecated only changed to rules, build is now manual on PRs

* change (CI): add workflow to avoid doubled pipelines, remove second job

* debug (CI): unblock pipeline from waiting a manual action

This is required to ensure that on chain isn't seen as compatible to
current master.

Co-authored-by: Gavin Wood <[email protected]>

Co-authored-by: Gavin Wood <[email protected]>

* Register listeners in statement distribution

* Review fixes

* Metrics

* Dont punish late collations

* Fix metrics

* Update node/network/collator-protocol/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Change on_request arg to Result

Co-authored-by: Andronik Ordian <[email protected]>

This reverts commit 96f26155.