* Polkadot companion for 8114

https://github.com/paritytech/substrate/pull/8114

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Companion for substrate#8079

* More changes

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Notify collators about seconded collation

This pr adds functionality to inform a collator that its collation was
seconded by a parachain validator. Before this signed statement was only
gossiped over the validation substream. Now, we explicitly send the
seconded statement to the collator after it was validated successfully.

Besides that it changes the `CollatorFn` to return an optional result
sender that is informed when the build collation was seconded by a
parachain validator.

* Add test

* Make sure we only send `Seconded` statements

* Make sure we only receive valid statements

* Review feedback

* CancelProxy uses `reject_announcement` instead of `remove_announcement`

* add back remove_announcement

* Revert "add back remove_announcement"

This reverts commit 467b608a9d77c4303cb8daa33e8f14d391d6f417.

* skeleton

* skeleton aux-schema module

* start approval types

* start aux schema with aux store

* doc

* finish basic types

* start approval types

* doc

* finish basic types

* write out schema types

* add debug and codec impls to approval types

* add debug and codec impls to approval types

also add some key computation

* add debug and codec impls to approval types

* getters for block and candidate entries

* grumbles

* remove unused AssignmentId

* load_decode utility

* implement DB clearing

* function for adding new block entry to aux store

* start `canonicalize` implementation

* more skeleton

* finish implementing canonicalize

* tag TODO

* implement a test AuxStore

* add allow(unused)

* basic loading and deleting test

* block_entry test function

* add a test for `add_block_entry`

* ensure range is exclusive at end

* test clear()

* test that add_block sets children

* add a test for canonicalize

* extract Pre-digest from header

* utilities for extracting RelayVRFStory from the header-chain

* add approval voting message types

* approval distribution message type

* subsystem skeleton

* state struct

* add futures-timer

* prepare service for babe slot duration

* more skeleton

* better integrate AuxStore

* RelayVRF -> RelayVRFStory

* canonicalize

* implement some tick functionality

* guide: tweaks

* check_approval

* more tweaks and helpers

* guide: add core index to candidate event

* primitives: add core index to candidate event

* runtime: add core index to candidate events

* head handling (session window)

* implement `determine_new_blocks`

* add TODO

* change error type on functions

* compute RelayVRFModulo assignments

* compute RelayVRFDelay assignments

* fix delay tranche calc

* assignment checking

* pluralize

* some dummy code for fetching assignments

* guide: add babe epoch runtime API

* implement a current_epoch() runtime API

* compute assignments

* candidate events get backing group

* import blocks and assignments into DB

* push block approval meta

* add message types, no overseer integration yet

* notify approval distribution of new blocks

* refactor import into separate functions

* impl tranches_to_approve

* guide: improve function signatures

* guide: remove Tick from ApprovalEntry

* trigger and broadcast assignment

* most of approval launching

* remove byteorder crate

* load blocks back to finality, except on startup

* check unchecked assignments

* add claimed core to approval voting message

* fix checks

* assign only to backing group

* remove import_checked_assignment from guide

* newline

* import assignments

* abstract out a bit

* check and import approvals

* check full approvals from assignment import too

* comment

* create a Transaction utility

* must_use

* use transaction in `check_full_approvals`

* wire up wakeups

* add Ord to CandidateHash

* wakeup refactoring

* return candidate info from add_block_entry

* schedule wakeups

* background task: do candidate validation

* forward candidate validation requests

* issue approval votes when requested

* clean up a couple TODOs

* fix up session caching

* clean up last unimplemented!() items

* fix remaining warnings

* remove TODO

* implement handle_approved_ancestor

* update Cargo.lock

* fix runtime API tests

* guide: cleanup assignment checking

* use claimed candidate index instead of core

* extract time to a trait

* tests module

* write a mock clock for testing

* allow swapping out the clock

* make abstract over assignment criteria

* add some skeleton tests and simplify params

* fix backing group check

* do backing group check inside check_assignment_cert

* write some empty test functions to implement

* add a test for non-backing

* test that produced checks pass

* some empty test ideas

* runtime/inclusion: remove outdated TODO

* fix compilation

* av-store: fix tests

* dummy cert

* criteria tests

* move `TestStore` to main tests file

* fix unused warning

* test harness beginnings

* resolve slots renaming fallout

* more compilation fixes

* wip: extract pure data into a separate module

* wip: extract pure data into a separate module

* move types completely to v1

* add persisted_entries

* add conversion trait impls

* clean up some warnings

* extract import logic to own module

* schedule wakeups

* experiment with Actions

* uncomment approval-checking

* separate module for approval checking utilities

* port more code to use actions

* get approval pipeline using actions

* all logic is uncommented

* main loop processes actions

* all loop logic uncommented

* separate function for handling actions

* remove last unimplemented item

* clean up warnings

* State gives read-only access to underlying DB

* tests for approval checking

* tests for approval criteria

* skeleton test module for import

* list of import tests to do

* some test glue code

* test reject bad assignment

* test slot too far in future

* test reject assignment with unknown candidate

* remove loads_blocks tests

* determine_new_blocks back to finalized & harness

* more coverage for determining new blocks

* make `imported_block_info` have less reliance on State

* candidate_info tests

* tests for session caching

* remove println

* extricate DB and main TestStores

* rewrite approval checking logic to counteract early delays

* move state out of function

* update approval-checking tests

* tweak wakeups & scheduling logic

* rename check_full_approvals

* test that assignment import updates candidate

* some approval import tests

* some tests for check_and_apply_approval

* add 'full' qualifier to avoid confusion

* extract should-trigger logic to separate function

* some tests for all triggering

* tests for when we trigger assignments

* test wakeups

* add block utilities for testing

* some more tests for approval updates

* approved_ancestor tests

* new action type for launch approval

* process-wakeup tests

* clean up some warnings

* fix in_future test

* approval checking tests

* tighten up too-far-in-future

* special-case genesis when caching sessions

* fix bitfield len

Co-authored-by: Andronik Ordian <[email protected]>

* Introduce PerPeerSet utility that allows to segrate based on PeerSet

* Remove `repr(usize)` from PeerSet

* move lib.sh to common dir

* make check-labels a github action

* check out the repo

* fix secrets

* add labelled PR type

* remove check-labels job from gitlab

* trigger on unlabelling also

* runtime: set equivocation report longevity

* "Update Substrate"

* update substrate

Co-authored-by: parity-processbot <>
Co-authored-by: thiolliere <[email protected]>

* Fix up balances Events

* "Update Substrate"

* Update substrate

Co-authored-by: parity-processbot <>

* Wasm caching

* Fix compilation errors

* Rename `cache_path` to `cache_base_path`

* "Update Substrate"

* Fix check-web-wasm build

Co-authored-by: parity-processbot <>

* Claims

* Crowdloan

* Runtime common

* Parachains registrar

* Impls

* Purchase

* Slots

* parachains runtime mock

* Use MockGenesisConfig in tests

* Update runtime/common/src/claims.rs

Co-authored-by: Guillaume Thiolliere <[email protected]>

Co-authored-by: Guillaume Thiolliere <[email protected]>

* validator_discovery: pass PeerSet to the request

* validator_discovery: track PeerSet of connected peers

* validator_discovery: fix tests

* validator_discovery: fix long line

* some fixes

* some validator_discovery logs

* log validator discovery request

* Also connect to validators on `DistributePoV`.

* validator_discovery: store the whole state per peer_set

* bump spec versions in kusama, polkadot and westend

* Correcting doc.

* validator_discovery: bump channel capacity

* pov-distribution: some cleanup

* this should fix the test, but it does not

* I just got some brain damage while fixing this

Why are you even reading this???

* wrap long line

* address some review nits

Co-authored-by: Robert Klotzner <[email protected]>

* guide: more robust approval counting procedure

* elaborate

* redefine non-empty

* improve writing

* remove redundant

* CI: change manual triggering to auto based on rules [skip ci]

* CI: remove debug comments [skip ci]

* CI: chore

* CI: chore

* CI: cargo update -p sp_io

Bumps [trie-db](https://github.com/paritytech/trie) from 0.22.2 to 0.22.3.
- [Release notes](https://github.com/paritytech/trie/releases)
- [Commits](https://github.com/paritytech/trie/compare/trie-db-v0.22.2...trie-db-v0.22.3)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* send our view to new peers immediately

* guide: update

* Fix tests by expecting the view updates

* Add test that we send our view on connection

Co-authored-by: Sergei Shulepov <[email protected]>

* node: pass local authorship to the transaction pool

* update substrate

* initial implementation of lifecycles and upgrades

* clean up a bit

* fix doc comment

* more rigid lifecycle checks

* include paras which are transitioning, and lifecycle query

* format guide

* update api

* update guide

* explicit outgoing state, fix genesis

* handle outgoing with transitioning paras

* do not include transitioning paras in identifier

* Update roadmap/implementers-guide/src/runtime/paras.md

* Update roadmap/implementers-guide/src/runtime/paras.md

* Update roadmap/implementers-guide/src/runtime/paras.md

* Apply suggestions from code review

* Use matches macro

* Correct terms

* Apply suggestions from code review

* actions queue

* Revert "actions queue"

This reverts commit b2e9011ec8937d6c73e99292416c9692aeb30f73.

* collapse onboarding state

Co-authored-by: Gavin Wood <[email protected]>

* cancel proxy

* fix

* Remove reject_announcement and add CancelProxy to westend

Co-authored-by: David Dorgan <[email protected]>

`libp2p-swarm` `CHANGELOG.md`:

- Have `ToggleProtoHandler` ignore listen upgrade errors when disabled.
  [PR 1945](https://github.com/libp2p/rust-libp2p/pull/1945/files).

Fixes:

https://github.com/paritytech/polkadot/issues/2373

* Refactored the build steps for building with cargo

* Cargo build step needs pre requisites not mentioned

* Update README.md

* unused variable can be explicity ignored

Co-authored-by: Bastian Köcher <[email protected]>

* End multiplexer stream once one of its inputs end.

Also add test, that we don't panic once a stream is exhausted.

* Don't fuse already fused stream.

* Refactored the build steps for building with cargo

* Cargo build step needs pre requisites not mentioned

* Update README.md

Co-authored-by: Bastian Köcher <[email protected]>

* Implement `Debug` manually for CandidateHash

This will make candidate hashes printed consistently without the
`CandidateHash(` and `)` decorations.

* Do not print CompressedPov's guts

It can be overwhelming. Better just use the size.

* Log when candidate is generated

* Print para_id and candidate_hash upon receiving a collation

* Move NetworkBridgeEvent to subsystem::messages.

It is not protocol related at all, it is in fact only part of the
subsystem communication as it gets wrapped into messages of each
subsystem.

* Request/response infrastructure is taking shape.

WIP: Does not compile.

* Multiplexer variant not supported by Rusts type system.

* request_response::request type checks.

* Cleanup.

* Minor fixes for request_response.

* Implement request sending + move multiplexer.

Request multiplexer is moved to bridge, as there the implementation is
more straight forward as we can specialize on `AllMessages` for the
multiplexing target.

Sending of requests is mostly complete, apart from a few `From`
instances. Receiving is also almost done, initializtion needs to be
fixed and the multiplexer needs to be invoked.

* Remove obsolete multiplexer.

* Initialize bridge with multiplexer.

* Finish generic request sending/receiving.

Subsystems are now able to receive and send requests and responses via
the overseer.

* Doc update.

* Fixes.

* Link issue for not yet implemented code.

* Fixes suggested by @ordian

 - thanks!

- start encoding at 0
- don't crash on zero protocols
- don't panic on not yet implemented request handling

* Update node/network/protocol/src/request_response/v1.rs

Use index 0 instead of 1.

Co-authored-by: Andronik Ordian <[email protected]>

* Update node/network/protocol/src/request_response.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Fix existing tests.

* Better avoidance of division by zoro errors.

* Doc fixes.

* send_request -> start_request.

* Fix missing renamings.

* Update substrate.

* Pass TryConnect instead of true.

* Actually import `IfDisconnected`.

* Fix wrong import.

* Update node/network/bridge/src/lib.rs

typo

Co-authored-by: Pierre Krieger <[email protected]>

* Update node/network/bridge/src/multiplexer.rs

Remove redundant import.

Co-authored-by: Pierre Krieger <[email protected]>

* Stop doing tracing from within `From` instance.

Thanks for the catch @tomaka

!

* Get rid of redundant import.

* Formatting cleanup.

* Fix tests.

* Add link to issue.

* Clarify comments some more.

* Fix tests.

* Formatting fix.

* tabs

* Fix link

Co-authored-by: Bernhard Schuster <[email protected]>

* Use map_err.

Co-authored-by: Bernhard Schuster <[email protected]>

* Improvements inspired by suggestions by @drahnr.

- Channel size is now determined by function.
- Explicitely scope NetworkService::start_request.

Co-authored-by: Andronik Ordian <[email protected]>
Co-authored-by: Pierre Krieger <[email protected]>
Co-authored-by: Bernhard Schuster <[email protected]>

We only checked all validity votes, ignoring invalid votes. So, the
condidition could not hold. Besides fixing the panic, I removed some old
cruft that isn't required anymore.

* Add missing extrincis function

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Conpanion for Substrate#7127

* Use sp_session::OneSessionHandler

* .

* Fix pallet_session::OneSessionHandler

* OneSessionHandler is in frame_support now

* "Update Substrate"

Co-authored-by: parity-processbot <>

* PVD: `block_number`->`relay_parent_number`

* ValidationParams: `relay_chain_height`->`relay_parent_number`

* Expose DMQ MQC hash as a well-known-key

This way the relay storage merkle proofs will be able to obtain the DMQ
MQC hash and we will be able to remove the it from the
PersistedValidationData struct.

* PersistedValidationData: Remove HRMP MQC heads

* PersistedValidationData: Remove `dmq_mqc_head`

* Expose the HRMP ingress channel index as a well-known-key

This way a parachain (PVF and collator) can find all the parachains that
have an outbound channel to the given one. That allows in turn to find
all the inbound channels for the given para.

Having access to that allows the parachain to get the same information
as the hrmp_mqc_heads now provide.

* Rename `relay_storage_root` to `relay_parent_storage_root`

* DMP

* HRMP

* Update runtime/parachains/src/dmp.rs

Co-authored-by: Bastian Köcher <[email protected]>

* grab guide updates from approval-voting branch

* focus on control flow and behavior, not functions

* add approval voting preamble info

* tabs -> spaces

* upgrade codec and bitvec

* "Update Substrate"

Co-authored-by: parity-processbot <>