* create a v1 primitives module

* Improve guide on availability types

* punctuate

* new parachains runtime uses new primitives

* tests of new runtime now use new primitives

* add ErasureChunk to guide

* export erasure chunk from v1 primitives

* subsystem crate uses v1 primitives

* node-primitives uses new v1 primitives

* port overseer to new primitives

* new-proposer uses v1 primitives (no ParachainHost anymore)

* fix no-std compilation for primitives

* service-new uses v1 primitives

* network-bridge uses new primitives

* statement distribution uses v1 primitives

* PoV distribution uses v1 primitives; add PoV::hash fn

* move parachain to v0

* remove inclusion_inherent module and place into v1

* remove everything from primitives crate root

* remove some unused old types from v0 primitives

* point everything else at primitives::v0

* squanch some warns up

* add RuntimeDebug import to no-std as well

* port over statement-table and validation

* fix final errors in validation and node-primitives

* add dummy Ord impl to committed candidate receipt

* guide: update CandidateValidationMessage

* add primitive for validationoutputs

* expand CandidateValidationMessage further

* bikeshed

* add some impls to omitted-validation-data and available-data

* expand CandidateValidationMessage

* make erasure-coding generic over v1/v0

* update usages of erasure-coding

* implement commitments.hash()

* use Arc<Pov> for CandidateValidation

* improve new erasure-coding method names

* fix up candidate backing

* update docs a bit

* fix most tests and add short-circuiting to make_pov_available

* fix remainder of candidate backing tests

* squanching warns

* squanch it up

* some fallout

* overseer fallout

* free from polkadot-test-service hell

* overseer: introduce a utility typemap

* it's ugly but it compiles

* move DummySubsystem to subsystem crate

* fix tests fallout

* use a struct for all subsystems

* more tests fallout

* add missing pov_distribution subsystem

* remove unused imports and bounds

* fix minimal-example

* Updates guide for CandidateBacking

* Move assignment types to primitives

* Initial implementation.

* More functionality

* use assert_matches

* Changes to report misbehaviors

* Some fixes after a review

* Remove a blank line

* Update guide and some types

* Adds run_job function

* Some comments and refactorings

* Fix review

* Remove warnings

* Use summary in kicking off validation

* Parallelize requests

* Validation provides local and global validation params

* Test issued validity tracking

* Nits from review

* Switch branch

* Fix service creation

* Revert "Switch branch"

This reverts commit a7063358.

* Switch branch again

* Revert "Switch branch again"

This reverts commit 04afc509.

* Update substrate branch

Fixes #833

* introduce candidatedescriptor type

* add PoVDistribution message type

* loosen bound on PoV Distribution to account for equivocations

* re-export some types from the messages module

* begin PoV Distribution subsystem

* remove redundant index from PoV distribution

* define state machine for pov distribution

* handle overseer signals

* set up control flow

* remove `ValidatorStatement` section

* implement PoV fetching

* implement distribution logic

* add missing `

* implement some network bridge event handlers

* stub for message processing, handle our view change

* control flow for handling messages

* handle `awaiting` message

* handle any incoming PoVs and redistribute

* actually provide a subsystem implementation

* remove set-builder notation

* begin testing PoV distribution

* test that we send awaiting messages only to peers with same view

* ensure we distribute awaited PoVs to peers on view changes

* test that peers can complete fetch and are rewarded

* test some reporting logic

* ensure peer is reported for flooding

* test punishing peers diverging from awaited protocol

* test that we eagerly complete peers' awaited PoVs based on what we receive

* test that we prune the awaited set after receiving

* expand pov-distribution in guide to match a change I made

* remove unneeded import

* Companion for #6599

https://github.com/paritytech/substrate/pull/6599

* Update Substrate ref

* rename implementor's guide to implementer's guide

* fix typos in more places

Closes #1339.

* set up data types and control flow for statement distribution

* add some set-like methods to View

* implement sending to peers

* start fixing equivocation handling

* Add a section to the statement distribution subsystem on equivocations and flood protection

* fix typo and amend wording

* implement flood protection

* have peer knowledge tracker follow when peer first learns about a candidate

* send dependents after circulating

* add another TODO

* trigger send in one more place

* refactors from review

* send new statements to candidate backing

* instantiate active head data with runtime API values

* track our view changes and peer view changes

* apply a benefit to peers who send us statements we want

* remove unneeded TODO

* add some comments and improve Hash implementation

* start tests and fix `note_statement`

* test active_head seconding logic

* test that the per-peer tracking logic works

* test per-peer knowledge tracker

* test that peer view updates lead to messages being sent

* test statement circulation

* address review comments

* have view set methods return references

* network bridge skeleton

* move some primitives around and add debug impls

* protocol registration glue & abstract network interface

* add send_msgs to subsystemctx

* select logic

* transform different events into actions and handle

* implement remaining network bridge state machine

* start test skeleton

* make network methods asynchronous

* extract subsystem out to subsystem crate

* port over overseer to subsystem context trait

* fix minimal example

* fix overseer doc test

* update network-bridge crate

* write a subsystem test-helpers crate

* write a network test helper for network-bridge

* set up (broken) view test

* Revamp network to be more async-friendly and not require Sync

* fix spacing

* fix test compilation

* insert side-channel for actions

* Add some more message types to AllMessages

* introduce a test harness

* impl ProvideInherent for InclusionInherent

* reduce import churn; correct expect message

* move inclusion inherent identifier into primitives

It's not clear precisely why this is desired, but it's a pattern
I've seen in several places, so I'm going this to be on the
safe side. Worst case, we can revert this commit pretty easily.

* bump kusama spec_version to placate CI

* copy sc_basic_authorship::{ProposerFactory, Proposer}

We have from the problem description:

> This Proposer will require an OverseerHandle to make requests via.

That's next on the plate.

* use polkadot custom proposer instead of basic-authorship one

* add some tests

* ensure service compiles and passes tests

* fix typo

* fix service-new compilation

* Subsystem test helpers send messages synchronously

* remove smelly action inspector

* remove superfluous let binding

* fix warnings

* add license header

* empty commit; maybe github will notice the one with changes

* Update node/network/bridge/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* add sanity check to only include valid inherents

* stub: encapsulate block production mechanics instead of copying them

The goal is to end up with something like what's in
validation::block_production::*, which encapsulates
basic block production mechanics. This is a better idea than
just straight-up copying those mechanics.

* partial implementation of propose fn

Doesn't actually compile yet; need to bring in some other
commits to ensure ProvisionerMessage is a thing, and also
figure out how to get the block hash given the current
context.

* fix compilation

* clear a few more compile errors

* finish fn propose

* broken: add timeout to proposal

* add timeout to proposal

* guide: provisioner is responsible for selecting parachain candidates

* implement ProvisionerMessage::RequestInherentData & update fn propose

* impl CreateProposer::init; clean up

* impl std::error::Error for Error

* document error-handling rationale

* cause polkadot-service-new to compile correctly

* Move potentially-blocking call from fn init -> fn propose

This means that we can wrap the delayed call into the same
timeout check used elsewhere.

* document struct Proposer

* extract provisioner data fetch

This satisfies two requirements:

- only applies the timeout to actually fetching the provisioner data,
  not to constructing the block after
- simplifies the problem of injecting default data if we could not
  get the real provisioner data in time.

Co-authored-by: Robert Habermeier <[email protected]>
Co-authored-by: Gavin Wood <[email protected]>

* enable BABE equivocation reporting on all runtimes

* runtime: fix parachains mock test runtime

* cargo update -p sp-io

Co-authored-by: Shawn Tabrizi <[email protected]>

* Switch branch

* Return chain ops parts in new_chain_ops

* Remove where param from new_chain_ops

* Add task manager to new_chain_ops return

* Revert branch switch

* Revert "Revert branch switch"

This reverts commit 7c7900c0

.

* network/test/src/lib: Adjust network worker polling

Companion for https://github.com/paritytech/substrate/pull/6552.

* Fix adder parachain

* Fix collator tests

* Revert branch switch

* Bump everything

- Remove old migration code
- Reduce deposit requried for westend

* Reapply fixes

* Bump locl

* Fix for #6550

* Fix message

Co-authored-by: Ashley Ruglys <[email protected]>
Co-authored-by: Max Inden <[email protected]>

* Subsystem::start takes self by-value

* fix doc-test compilation

* network bridge skeleton

* move some primitives around and add debug impls

* protocol registration glue & abstract network interface

* add send_msgs to subsystemctx

* select logic

* transform different events into actions and handle

* implement remaining network bridge state machine

* start test skeleton

* make network methods asynchronous

* extract subsystem out to subsystem crate

* port over overseer to subsystem context trait

* fix minimal example

* fix overseer doc test

* update network-bridge crate

* write a subsystem test-helpers crate

* write a network test helper for network-bridge

* set up (broken) view test

* Revamp network to be more async-friendly and not require Sync

* fix spacing

* fix test compilation

* insert side-channel for actions

* Add some more message types to AllMessages

* introduce a test harness

* add some tests

* ensure service compiles and passes tests

* fix typo

* fix service-new compilation

* Subsystem test helpers send messages synchronously

* remove smelly action inspector

* remove superfluous let binding

* fix warnings

* Update node/network/bridge/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* fix compilation

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

Companion PR for `Remove the service, replacing it with a struct of individual chain components` (#1288)

* Switch branch

* Fix service things

* Fix browser node compilation

* Update branch

* fixed new service

* Update for new branch

* Fix browser node

* Update branch

* Revert "Switch branch"

This reverts commit 3623adff

.

* Update cargo.lock

Co-authored-by: Bastian Köcher <[email protected]>

 Companion PR for `Fix the browser node and ensure it doesn't colour the informant output` #6457 (#1307)

* Companion for Substrate #6269 (nested storage tx)

* Adjust to PR changes

* Bump to newest substrate

* Bump spec

* Switch branch

* Remove redundant informant_prefix

* Fix collator tests

* Update branch

* Revert "Switch branch"

This reverts commit bcce560c

.

Co-authored-by: Alexander Theißen <[email protected]>

* node: spawn babe and grandpa as blocking tasks

* update to latest substrate

* add signed wrapper, typedef SignedStatement

* typedef SignedAvailabilityBitfield

* implement Signed wrapper

This is strictly an addition as of this commit; nothing is yet
changed in existing behavior.

* inline getters, remove review comment

* move EncodeAs, Signed from node::primitives to primitives::parachain

* Refactor SignedAvailabilityBitfield to use Signed

* don't double-encode real payload

This isn't an ideal solution, because it depends on the
implementation details of how SCALE encodes tuples, but OTOH
that behavior seems unlikely to change anytime soon.

* fix build errors

* cause the runtime to build properly with the new changes

Not sure why cargo check didn't catch this earlier; oh well.

* fix runtime tests and separate SignedStatement from SignedFullStatement

* better explain why CompactStatement exists

Co-authored-by: Robert Habermeier <[email protected]>

Co-authored-by: Robert Habermeier <[email protected]>

* introduce polkadot-node-primitives

* guide: change statement distribution message types

* guide: remove variant from `CandidateSelectionMessage`

* add a few more message types

* add TODOs

* Almost all messages

* NewBackedCandidate notification

* Formatting

* Use AttestedCandidate as BackedCandidate

* Update node/primitives/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* Fix the tests

* Bring in types from #1242

* Adds network bridge messages

* More message types from doc

* use fn pointer type

* Fixes from the review

* Add missing Runtime subsystem message

* rename to CandidateValidationMessage and fix tests

Co-authored-by: Fedor Sakharov <[email protected]>
Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* New service initial commit

* More separation of the new and old services

* Fix review comments

* Adds polkadot.json

* Fix browser build

* Remove unused import

* Update node/service/src/lib.rs

Co-authored-by: Fedor Sakharov <[email protected]>

* Remove duplicate json files

Co-authored-by: Robert Habermeier <[email protected]>

* establish new `node` folder for overseer, messages, and subsystems

* extract message types from overseer crate

* remove doc links