* Parachain improvements

- Set the parachains configuration in Rococo genesis
- Don't stop the overseer when a subsystem job is stopped
- Several small code changes

* Remove unused functionality

* Return error from the runtime instead of printing it

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Update primitives/src/v1.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Update primitives/src/v1.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Fix test

* Revert "Update primitives/src/v1.rs"

This reverts commit 11fce278.

* Revert "Update primitives/src/v1.rs"

This reverts commit d6439fed.

* Revert "Return error from the runtime instead of printing it"

This reverts commit cb4b5c08.

* Revert "Fix test"

This reverts commit 0c5fa1b5

.

* Update runtime/parachains/src/runtime_api_impl/v1.rs

Co-authored-by: Sergei Shulepov <sergei@parity.io>

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Sergei Shulepov <sergei@parity.io>

* Fix keystore types

* Use SyncCryptoStorePtr

* Borrow keystore

* Fix unused imports

* Fix polkadot service

* Fix bitfield-distribution tests

* Fix indentation

* Fix backing tests

* Fix tests

* Fix provisioner tests

* Removed SyncCryptoStorePtr

* Fix services

* Address PR feedback

* Address PR feedback - 2

* Update CryptoStorePtr imports to be from sp_keystore

* Typo

* Fix CryptoStore import

* Document the reason behind using filesystem keystore

* Remove VALIDATORS

* Fix duplicate dependency

* Mark sp-keystore as optional

* Fix availability distribution

* Fix call to sign_with

* Fix keystore usage

* Remove tokio and fix parachains Cargo config

* Typos

* Fix keystore dereferencing

* Fix CryptoStore import

* Fix provisioner

* Fix node backing

* Update services

* Cleanup dependencies

* Use sync_keystore

* Fix node service

* Fix node service - 2

* Fix node service - 3

* Rename CryptoStorePtr to SyncCryptoStorePtr

* "Update Substrate"

* Apply suggestions from code review

* Update node/core/backing/Cargo.toml

* Update primitives/src/v0.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Fix wasm build

* Update Cargo.lock

Co-authored-by: parity-processbot <>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* stupid, but it compiles

* redo

* cleanup

* add ValidatorDiscovery to msgs

* sketch network bridge code

* ConnectToAuthorities instead of validators

* more stuff

* cleanup

* more stuff

* complete ConnectToAuthoritiesState

* Update node/network/bridge/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Collator protocol subsystem (#1659)

* WIP

* The initial implementation of the collator side.

* Improve comments

* Multiple collation requests

* Add more tests and comments to validator side

* Add comments, remove dead code

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* Fix build after suggested changes

* Also connect to the next validator group

* Remove a Future impl and move TimeoutExt to util

* Minor nits

* Fix build

* Change FetchCollations back to FetchCollation

* Try this

* Final fixes

* Fix build

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>

* handle multiple in-flight connection requests

* handle cancelled requests

* Update node/core/runtime-api/src/lib.rs

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* redo it again

* more stuff

* redo it again

* update comments

* workaround Future is not Send

* fix trailing spaces

* clarify comments

* bridge: fix compilation in tests

* update more comments

* small fixes

* port collator protocol to new validator discovery api

* collator tests compile

* collator tests pass

* do not revoke a request when the stream receiver is closed

* make revoking opt-in

* fix is_fulfilled

* handle request revokation in collator

* tests

* wait for validator connections asyncronously

* fix compilation

* relabel my todos

* apply Fedor's patch

* resolve reconnection TODO

* resolve revoking TODO

* resolve channel capacity TODO

* resolve peer cloning TODO

* resolve peer disconnected TODO

* resolve PeerSet TODO

* wip tests

* more tests

* resolve Arc TODO

* rename pending to non_revoked

* one more test

* extract utility function into util crate

* fix compilation in tests

* Apply suggestions from code review

Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>

* revert pin_project removal

* fix while let loop

* Revert "revert pin_project removal"

This reverts commit ae7f529d

.

* fix compilation

* Update node/subsystem/src/messages.rs

* docs on pub items

* guide updates

* remove a TODO

* small guide update

* fix a typo

* link to the issue

* validator discovery: on_request docs

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Fedor Sakharov <fedor.sakharov@gmail.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* cargo.toml updates

* session and system

* more

* more

* more

* more

* more

* fix

* compiles

* fix tests

* fix more tests

* fix mock

* fix deleted space

* Update validation/Cargo.toml

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update Cargo.lock

* update rococo

* remove unused warning

* update add benchmarks

* rename weight file

* forgot a file

* Update chain_spec.rs

* Revert "remove unused warning"

This reverts commit 4227cd0d

.

* fix merge

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update scale codec to latest version to fix bug in future rustc version

Companion of: https://github.com/paritytech/substrate/pull/6746

* 'Update substrate'

Co-authored-by: parity-processbot <>

* create a README on Runtime APIs

* add ParaId type

* write up runtime APIs

* more preamble

* rename

* rejig runtime APIs

* add occupied_since to `BlockNumber`

* skeleton crate for runtime API subsystem

* improve group_for_core

* improve docs on availability cores runtime API

* guide: freed -> free

* add primitives for runtime APIs

* create a v1 ParachainHost API trait

* guide: make validation code return `Option`al.

* skeleton runtime API helpers

* make parachain-host runtime-generic

* skeleton for most runtime API implementation functions

* guide: add runtime API helper methods

* implement new helpers of the inclusion module

* guide: remove retries check, as it is unneeded

* implement helpers for scheduler module for Runtime APIs

* clean up `validator_groups` implementation

* implement next_rotation_at and last_rotation_at

* guide: more helpers on GroupRotationInfo

* almost finish implementing runtime APIs

* add explicit block parameter to runtime API fns

* guide: generalize number parameter

* guide: add group_responsible to occupied-core

* update primitives due to guide changes

* finishing touches on runtime API implementation; squash warnings

* break out runtime API impl to separate file

* add tests for next_up logic

* test group rotation info

* point to filed TODO

* remove unused TODO [now]

* indentation

* guide: para -> para_id

* rename para field to para_id for core meta

* remove reference to outdated AvailabilityCores type

* add an event in `inclusion` for candidates being included or timing out

* guide: candidate events

* guide: adjust language

* Candidate events type from guide and adjust inclusion event

* implement `candidate_events` runtime API

* fix runtime test compilation

* max -> min

* fix typos

* guide: add `RuntimeAPIRequest::CandidateEvents`

* Bump trie-db

* Bump substrate

* scheduler module skeleton

* update scheduler skeleton to match latest version of guide

* better session change notification

* add mock randomness and fix test compilation

* shuffle validators into groups

* finish implementing session change logic for scheduler

* tweak core assignment type to track retries of parathread

* reframe queued parathread core as offset

* implement initialzation and finalization routines

* implement parathread claim queuing

* implement core_para

* implement the group_validators routine and fix errors

* add a reason for freeing cores

* implement `schedule` function

* add some docs to the scheduled function

* implement `occupied` helper

* implement availability predicate

* fix some warnings

* integrate scheduler into initializer

* integrate scheduler into mock module

* avoid conflict with Substrate's scheduler storage

* add parathreads index to paras module

* implement parathreads map in paras module

* add is_parathread to paras

* test adding parathread claim

* test that you cannot add claims when no parathread cores exist

* check session change parathread queue pruning

* test validator shuffling

* add allow_unused to scheduler items

* add test for scheduling

* add some more tests for scheduling logic

* test core rotation

* check parathread claim pruning after retries

* add bound notes

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* more suggestions from review

* test availability predicate, add box to please compiler

* add changes to guide

Co-authored-by: Peter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* file structure and initializer skeleton

* ensure session changes happen before initialization

* add a couple tests for initializer flow

* integrate with session handling

* configuration update logic

* configuration methods

* move test mock to its own module

* integrate configuration into initializer

* add note about initialization order

* integrate configuration module into mock

* add some tests for config module

* paras module storage

* implement paras session change operation

* amend past code pruning to fully cover acceptance period

* update guide again

* do pruning of historical validation code

* add weight to initialization

* integrate into mock & leave notes for next session

* clean up un-ended sentence

* alter test to account for double index in past code meta

* port over code-at logic test

* clarify checking for conflicting code upgrades

* add genesis for paras, include in mock, ensure incoming paras are processed

* note on return value of `validation_code_at`

* implement paras routines from implementor's guide

* bring over some existing tests and begin porting

* port over code upgrade tests

* test parachain registration

* test code_at with intermediate block

* fix warnings

* clean up docs and extract to separate struct

* adjust implementor's guide to include replacementtimes

* kill stray println

* rename expected_at to applied_after

* rewrite ParaPastCodeMeta to avoid reversal

* clarify and test interface of validation_code_at

* make FutureCode optional

* rename do_old_code_pruning

* add comment on Option<()> to answer FAQ

* address some more grumbles

* Identity

* Bump spec

* Update runtime/polkadot/src/lib.rs

Co-authored-by: kaichao <kaichaosuna@gmail.com>

* Fix runtime

* Bump runtimes

Co-authored-by: kaichao <kaichaosuna@gmail.com>

* Bump

* Bump

* Bump

* companion for https://github.com/paritytech/substrate/pull/6132



* Fix more tests

* upsub

* Bump

* Update runtime/common/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* remove cleanup

* Update runtime/westend/src/lib.rs

* Apply suggestions from code review

* Fix build

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Test CC1 spec

* Adaptive default chainspec

* Strip -pre

* Update to next CC1 iteration

* Raw chainspec

* Update claim statements

* Update URLs

* Minor tweak to make enum agree with URL

* final html multihashes

* New hashes

* Version

* updated SAFT statement hash

* Update runtimes; this relies on substrate #6131

* Bump

* Share transaction filterer and fix Kusama

* Warning

* Gah!

* Tidy

* Westend fixes

* Westend fix

* Tweak constants

* Implement TCF stuff

* Fix

* Warning

* 15 minute lookahead

* 15 minute lookahead in Polkadot

* update max transactions

* Enable utility, disable vested_transfer

* Update runtime/common/src/lib.rs

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Filter calls for validate unsigned also

* Deduplicate

* Fix

* Introduce Polkadot (mainnet) chainspec

* Fix naming.

* Enable indices and fix comment.

* Fix compilation

* Enable indices and fix comment.

* polkadot: babe: enable secondary VRF slots

* Test JSON

* Allow set_heads

* Fix

* Raw chain spec added

Co-authored-by: keorn <github@i.keorn.org>
Co-authored-by: NikVolf <nikvolf@gmail.com>
Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Tomasz Drwięga <tomasz@parity.io>
Co-authored-by: André Silva <andre.beat@gmail.com>

* Introduce mandatory statement signing into claims

* Introduce SignedExtension

* Tests passing

* Bump runtime version

* Bump version, fix test

* Test for validate

* Another couple of tests

* Enable PrevalidateAttests on Polkadot

* Enable PrevalidateAttests on Polkadot

* Fix build

* Fixes

* More fixes

* Fix bench tests

* Fix & test Preclaim clobbering.

* Fix for errant claim logic

* Add test

* Update tests, always use Vec as input

* mint_claim can add signature, some_benchmarks

* Add claim with statement test

* finish benchmarks

* put the correct number of claims with benchmarks

* fix compiler warning

* Update weights

* Weight comments for validation

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* grandpa: report equivocations on all runtimes

* runtime: fix test-runtime compilation

* runtime: enable historical session manager on all runtimes

* runtime: fix indentation

* runtime: use strong key types in KeyOwnerProofSystem definitions

* update substrate

* bump spec_version of runtimes

* Bump Westend runtime version

* Bump Substrate

* Prep for release of 0.7.32

* Update lock

* add storage_append

* Bump versions

Co-authored-by: Gav Wood <gavin@parity.io>

* Release 0.7.30

* Update Substrate & libp2p

* Adds an offchain call to submit double vote reports

* Some tweaks

* Remove unnecessary IdentifyAccount impls

* Adds ValidateDoubleVoteReports to test runtime

* sp-application-crypto is only a dev dependency

* Release 0.7.29

* Update `Cargo.lock`

* use stronger types for HeadData and ValidationCode in runtime

* fix weird debug compile error

* fix runtime build

* update invocations invalidation.rs

* fix tests

* update substrate

* Hopefully fix compilation

* network: rename roles to role

* runtime: fix runtime-benchmarks compilation

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* Fixes for democracy using Scheduler

* Revert branch change

* Parachains double vote handler initial implementation.

* Make tests test the actual slashing.

* Implement SignedExtension validation of double vote reports.

* Fixes build after merge

* Review fixes

* Adds historical session proofs

* Review fixes.

* Bump runtime spec_version

* Get the session number from the proof

* Check that proof matches session

* Change signature type on DoubleVoteReport

* Adds docs and removes blank lines

* Removes leftover code

* Fix build

* Fix build after a merge

* Apply suggestions from code review

Co-Authored-By: Robert Habermeier <rphmeier@gmail.com>

* Prune ParentToSessionIndex

* Remove a clone and a warning

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Gavin Wood <gavin@parity.io>

* Updates for the new democracy

* Introduce progressive democracy

Also move to "master" branch of Substrate ready for continuous
sync with Substrate master.

* bump version to v0.7.27

* update to latest substrate polkadot-master

* bump runtime impl_version

* Bump version and Substrate

* Version

* Lock

* Insert storage for finality tracker

* Bump lock

* Bump version

* Bump version

* Cargo lock

* Bump runtime version

* Bump Substrate again to include recent fix