* WIP

* The initial implementation of the collator side.

* Improve comments

* Multiple collation requests

* Add more tests and comments to validator side

* Add comments, remove dead code

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* Fix build after suggested changes

* Also connect to the next validator group

* Remove a Future impl and move TimeoutExt to util

* Minor nits

* Fix build

* Change FetchCollations back to FetchCollation

* Try this

* Final fixes

* Fix build

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* update networking types

* port over overseer-protocol message types

* Add the collation protocol to network bridge

* message sending

* stub for ConnectToValidators

* add some helper traits and methods to protocol types

* add collator protocol message

* leaves-updating

* peer connection and disconnection

* add utilities for dispatching multiple events

* implement message handling

* add an observedrole enum with equality and no sentry nodes

* derive partial-eq on network bridge event

* add PartialEq impls for network message types

* add Into implementation for observedrole

* port over existing network bridge tests

* add some more tests

* port bitfield distribution

* port over bitfield distribution tests

* add codec indices

* port PoV distribution

* port over PoV distribution tests

* port over statement distribution

* port over statement distribution tests

* update overseer and service-new

* address review comments

* port availability distribution

* port over availability distribution tests

Closes #1237

* break out subsystem-util and subsystem-test-helpers into individual crates

* cause all packages to check successfully

* Update scale codec to latest version to fix bug in future rustc version

Companion of: https://github.com/paritytech/substrate/pull/6746

* 'Update substrate'

Co-authored-by: parity-processbot <>

* Initial commit

Forked at: 9637baea
Parent branch: origin/master

* Refactor

* Refactor

* Remove macro

* WIP

Forked at: 9637baea
Parent branch: origin/master

* CLEANUP

Forked at: 9637baea


Parent branch: origin/master

* small fix

* fix for browser

* Switch branch

* Rewrite service builds

* Update branch

* Fix sp-core branch

* Switch branch back and update

Co-authored-by: Cecile Tonglet <[email protected]>

* feat bitfield distribution

* feat bitfield distribution part 2

* pair programming with rustc & cargo

* lets go

* move bitfield-distribution to the node/network folder

* shape shifting

* lunchtime

* ignore the two fn recursion for now

* step by step

* triplesteps

* bandaid commit

* unordered futures magic

* chore

* reword markdown

* clarify

* lacks abortable processing impl details

* slimify

* fix: warnings and avoid ctx.clone() improve comments

* review comments

* fix details

* make sure outgoing messages are tracked

* fix name

* fix subsystem

* partial test impl

* relax context bounds

* test

* X

* X

* initial test

* fix relay_message not tracked when origin is self

* fix/guide: grammar

Co-authored-by: Robert Habermeier <[email protected]>

* work around missing Eq+PartialEq

* fix: add missing message to provisioner

* unify per_job to job_data

* fix/review: part one

* fix/review: more grumbles

* fix/review: track incoming messages per peer

* fix/review: extract fn, avoid nested matches

* fix/review: more tests, simplify test

* fix/review: extend tests to cover more cases

* chore/rename: Tracker -> ProtocolState

* chore check and comment rewording

* feat test: invalid peer message

* remove ignored test cases and unused macros

* fix master merge fallout + warnings

Co-authored-by: Robert Habermeier <[email protected]>

* get conclude signal working properly; don't allocate a vector

* wip: add test suite / example / explanation for using utility subsystem

Unfortunately, the test fails right now for reasons which seem
very odd. Just have to keep poking at it.

* explicitly import everything

* fix subsystem-util test

The root problem here was two-fold:

- there was a circular dependency from subsystem -> test-helpers/subsystem ->
  subsystem
- cfg(test) doesn't propagate between crates

The solution: move the subsystem test helpers into a sub-module
within subsystem. Publicly export them from the previous location
so no other code breaks.

Doing this has an additional benefit: it ensures that no production
code can ever accidentally use the subsystem helpers, as they are compile-
gated on cfg(test).

* fully commit to moving test helpers into a subsystem module

* add some more tests

* get rid of log tests in favor of real error forwarding

It's not obvious whether we'll ever really want to chase down
these errors outside a testing context, but having the capability
won't hurt.

* fix issue which caused test to hang on osx

* only require that job errors are PartialEq when testing

also fix polkadot-node-core-backing tests

* get rid of any notion of partialeq

* rethink testing

Combine tests of starting and stopping job: leaving a test executor
with a job running was pretty clearly the cause of the sometimes-hang.

Also, add a timeout so tests _can't_ hang anymore; they just fail
after a while.

* rename fwd_errors -> forward_errors

* warn on error propagation failure

* fix unused import leftover from merge

* derive eq for subsystemerror

* Use SpawnNamed instead of Spawn in Overseer

* reexport SpawnNamed and fix doc tests

* Fix deps

* introduce candidatedescriptor type

* add PoVDistribution message type

* loosen bound on PoV Distribution to account for equivocations

* re-export some types from the messages module

* begin PoV Distribution subsystem

* remove redundant index from PoV distribution

* define state machine for pov distribution

* handle overseer signals

* set up control flow

* remove `ValidatorStatement` section

* implement PoV fetching

* implement distribution logic

* add missing `

* implement some network bridge event handlers

* stub for message processing, handle our view change

* control flow for handling messages

* handle `awaiting` message

* handle any incoming PoVs and redistribute

* actually provide a subsystem implementation

* remove set-builder notation

* begin testing PoV distribution

* test that we send awaiting messages only to peers with same view

* ensure we distribute awaited PoVs to peers on view changes

* test that peers can complete fetch and are rewarded

* test some reporting logic

* ensure peer is reported for flooding

* test punishing peers diverging from awaited protocol

* test that we eagerly complete peers' awaited PoVs based on what we receive

* test that we prune the awaited set after receiving

* expand pov-distribution in guide to match a change I made

* remove unneeded import

* network bridge skeleton

* move some primitives around and add debug impls

* protocol registration glue & abstract network interface

* add send_msgs to subsystemctx

* select logic

* transform different events into actions and handle

* implement remaining network bridge state machine

* start test skeleton

* make network methods asynchronous

* extract subsystem out to subsystem crate

* port over overseer to subsystem context trait

* fix minimal example

* fix overseer doc test

* update network-bridge crate

* write a subsystem test-helpers crate

* write a network test helper for network-bridge

* set up (broken) view test

* Revamp network to be more async-friendly and not require Sync

* fix spacing

* fix test compilation

* insert side-channel for actions

* Add some more message types to AllMessages

* introduce a test harness

* add some tests

* ensure service compiles and passes tests

* fix typo

* fix service-new compilation

* Subsystem test helpers send messages synchronously

* remove smelly action inspector

* remove superfluous let binding

* fix warnings

* Update node/network/bridge/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* fix compilation

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>