* some fixes to please cargo-spellcheck

* some (not all) fixes for the impl guide

* fix

* CI: add spellcheck

* revert me

* CI: explicit command for spellchecker

* spellcheck: edit misspells

* CI: run spellcheck on diff

* spellcheck: edits

* spellcheck: edit misspells

* spellcheck: add rules

* spellcheck: mv configs

* spellcheck: more edits

* spellcheck: chore

* spellcheck: one more thing

* spellcheck: and another one

* spellcheck: seems like it doesn't get to an end

* spellcheck: new words after rebase

* spellcheck: new words appearing out of nowhere

* chore

* review edits

* more review edits

* more edits

* wonky behavior

* wonky behavior 2

* wonky behavior 3

* change git behavior

* spellcheck: another bunch of new edits

* spellcheck: new words are koming out of nowhere

* CI: finding the master

* CI: fetching master implicitly

* CI: undebug

* new errors

* a bunch of new edits

* and some more

* Update node/core/approval-voting/src/approval_db/v1/mod.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Update xcm/xcm-executor/src/assets.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Apply suggestions from code review

Co-authored-by: Andronik Ordian <write@reusable.software>

* Suggestions from the code review

* CI: scan only changed files

Co-authored-by: Andronik Ordian <write@reusable.software>

* disputes: Allow batch queries in dispute-coordinator

This commit moves to batch queries when responding to QueryCandidateVotes
messages. This simplifies the code in the provisioner and dispute-coordinator
by no longer requiring to make use of a FuturesOrdered when awaiting multiple
quries. Instead, the provisioner need only request the batch itself.

* node/approval-voting: Address Feedback to fail on query element missing.

* Address feedback

* Fix implementer's guide

* add a from_backing_statement to SignedDisputeStatement

* inform dispute coordinator of all backing statements

* add dispute coordinator message to backing tests

* send positive dispute statement with every approval

* issue disputes when encountering invalid candidates.

* try to fix flaky test for CI (passed locally)

* guide: keep track of concluded-positive disputes until pruned

* guide: block implications

* guide: new dispute inherent flow

* mostly implement recency changes for dispute coordinator

* add a clock to dispute coordinator

* adjust DB tests

* fix and add new dispute coordinator tests

* provisioner: select disputes

* import all validators' approvals

* address nit: refactor backing statement submission

* gracefully handle disconnected dispute coordinator

* remove `review` comment

* fix up old_tests

* fix approval-voting compilation

* fix backing compilation

* use known-leaves in WaitForActivation

* follow-up test fixing

* add back allow(dead_code)

* Guide updates for disputes.

* Working availability recovery flood protection.

* More fixes.

* Formatting.

* Fix.

* Update roadmap/implementers-guide/src/node/disputes/dispute-participation.md

Co-authored-by: Sergei Shulepov <sergei@parity.io>

* Review remarks.

Co-authored-by: Sergei Shulepov <sergei@parity.io>

* Only send one collation per relay parent at a time to validators

This changes the way we are sending collations to validators. Before we
answered every collation request immediatley. Now we only answer one
pov request at a time per relay parent. This should bring down the
bandwidth requirements and should help parachains to include bigger
blocks more easily.

* Guide updates

* Review feedback.

* av-store: use determine_new_blocks

* fix tests

* update the guide

* rename KnownBlocks

* fix iteration order

* add a test

* Dispute distribution initial design.

* WIP.

* Dispute distribution guide update.

* Make invalid statement include `InvalidStatementKind`.

* Clarify the scope of disputes.

* A few fixes + introduced back pressure oneshot.

* Fixes and spam protection WIP.

* More spam considerations.

* More fixes.

* Fixes + add note about not dispute participating nodes.

* guide: ChainApiMessage::BlockWeight

* node: BlockWeight ChainAPI

* fix compile issue

* implement ChainApi::BlockWeight

* add test for ChainApi::BlockWeight

* update substrate

Co-authored-by: André Silva <andrerfosilva@gmail.com>

* gossip-support: gossip topology

* some fixes

* handle view update for newly added gossip peers

* fix neighbors calculation

* fix test

* resolve TODOs

* typo

* guide updates

* spaces in the guide

* sneaky spaces

* hash randomness

* address some review nits

* use unbounded in bridge for subsystem msg

* high-level discussion of fork-choice and chain selection

* notes on chain-selection subsystem

* send `Approve` messages from approval-voting

* remove references to candidate-selection

* adjust grandpa voting rule docs

* Update roadmap/implementers-guide/src/node/grandpa-voting-rule.md

Co-authored-by: Lldenaurois <ljdenaurois@gmail.com>

* Update roadmap/implementers-guide/src/protocol-chain-selection.md

Co-authored-by: Lldenaurois <ljdenaurois@gmail.com>

* Update roadmap/implementers-guide/src/protocol-chain-selection.md

Co-authored-by: Lldenaurois <ljdenaurois@gmail.com>

Co-authored-by: Lldenaurois <ljdenaurois@gmail.com>

* implement dispute participation subsystem

* guide: minor fix in dispute participation

* Update node/core/dispute-participation/src/lib.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* dispute: add comments to participation logic

* dispute-coordinator: fix test compilation

* implementers-guide: update dispute participation

* dispute-participation: add error for missing validation code

* dispute-participation: add tests

* Update node/core/dispute-participation/src/lib.rs

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

* guide: update overseer protocol dispute participation message

* dispute-participation: remove duplication in tests

Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: Robert Habermeier <rphmeier@gmail.com>

* skeleton for dispute-coordinator

* add coordinator and participation message types

* begin dispute-coordinator DB

* functions for loading

* implement strongly-typed DB transaction

* add some tests for DB transaction

* core logic for pruning

* guide: update candidate-votes key for coordinator

* update candidate-votes key

* use big-endian encoding for session, and implement upper bound generator

* finish implementing pruning

* add a test for note_current_session

* define state of the subsystem itself

* barebones subsystem definition

* control flow

* more control flow

* implement session-updating logic

* trace

* control flow for message handling

* Update node/core/dispute-coordinator/src/lib.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* Update node/subsystem/src/messages.rs

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>

* some more control flow

* guide: remove overlay

* more control flow

* implement some DB getters

* make progress on importing statements

* add SignedDisputeStatement struct

* move ApprovalVote to shared primitives

* add a signing-payload API to explicit dispute statements

* add signing-payload to CompactStatement

* add relay-parent hash to seconded/valid dispute variatns

* correct import

* type-safe wrapper around dispute statements

* use checked dispute statement in message type

* extract rolling session window cache to subsystem-util

* extract session window tests

* approval-voting: use rolling session info cache

* reduce dispute window to match runtime in practice

* add byzantine_threshold and supermajority_threshold utilities to primitives

* integrate rolling session window

* Add PartialOrd to CandidateHash

* add Ord to CandidateHash

* implement active dispute update

* add dispute messages to AllMessages

* add dispute stubs to overseer

* inform dispute participation to participate

* implement issue_local_statement

* implement `determine_undisputed_chain`

* fix warnings

* test harness for dispute coordinator tests

* add more helpers to test harness

* add some more helpers

* some tests for dispute coordinator

* ignore wrong validator indices

* test finality voting rule constraint

* add more tests

* add variants to network bridge

* fix test compilation

* remove most dispute coordinator functionality

as of #3222 we can do most of the work within the approval voting subsystem

* Revert "remove most dispute coordinator functionality"

This reverts commit 9cd615e8

.

* Use thiserror

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* Update node/core/dispute-coordinator/src/lib.rs

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* extract tests to separate module

* address nit

* adjust run_iteration API

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* replace HistoricalValidationCode usages with ValidationCodeByHash

* runtime-api: tabify tests file

* update implementers guide

* Create validator_side module

* Subsume Candidate Selection

* Add test to ensure candidate backing logic is correct

* Ensure secondings are adequately cleaned up and address test flakyness

* Address Feedback

* approval-distribution: send all messages on unify

* update the guide

* guide: reversion safety

* guide: manage reversion safety in subsystems

* add leaf status to ActivatedLeaf

* add an LRU-cache to overseer for staleness detection

* update ActivatedLeaf usages in tests to contain status field

* add variant where missed accidentally

* add some helpers to LeafStatus

* address grumbles

* Guide updates WIP

* Add note.

* Statement distribution guide update.

* Small fixes.

* Update branch

* Make it compile

* Compile

* gate approval-checking logic (#2470)

* Fix build

* Updates

* Fix merge

* Adds missing crate

* Companion for Substrate#8386

https://github.com/paritytech/substrate/pull/8386



* Fix fix fix

* Fix

* Fix compilation

* Rewrite to `ParachainsInherentDataProvider`

* Make it compile

* Renamings

* Revert stuff

* Remove stale file

* Guide updates

* Update node/core/parachains-inherent/src/lib.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Update node/core/parachains-inherent/src/lib.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* Apply suggestions from code review

* Reset accidental changes

* More

* Remove stale file

* update Substrate

Co-authored-by: Robert Habermeier <rphmeier@gmail.com>
Co-authored-by: Andronik Ordian <write@reusable.software>
Co-authored-by: parity-processbot <>

* distribution: handle sqrt peer view updates

* someone please put rustc into my brain

* guide updates

* guide: declare one para as a collator

* add ParaId to Declare messages and clean up

* fix build

* fix the testerinos

* begin adding keystore to collator-protocol

* remove request_x_ctx

* add core_for_group

* add bump_rotation

* add some more helpers to subsystem-util

* change signing_key API to take ref

* determine current and next para assignments

* disconnect collators who are not on current or next para

* add collator peer count metric

* notes for later

* some fixes

* add data & keystore to test state

* add a test utility for answering runtime API requests

* fix existing collator tests

* add new tests

* remove sc_keystore

* update cargo lock

Co-authored-by: Andronik Ordian <write@reusable.software>

* approval-distribution: split peer knowledge into sent and received

* guide updates

* fixes

* revert doc changes

* extract database from av-store itself

* generalize approval-voting over database type

* modes (without handling) and pruning old wakeups

* rework approval importing

* add our_approval_sig to ApprovalEntry

* import assignment

* guide updates for check-full-approval changes

* some aux functions

* send messages when becoming active.

* guide: network bridge sends view updates only when done syncing

* network bridge: send view updates only when done syncing

* tests for new network-bridge behavior

* add a test for updating approval entry with sig

* fix some warnings

* test load-all-blocks

* instantiate new parachains DB

* fix network-bridge empty view updates

* tweak

* fix wasm build, i think

* Update node/core/approval-voting/src/lib.rs

Co-authored-by: Andronik Ordian <write@reusable.software>

* add some versioning to parachains_db

* warnings

* fix merge changes

* remove versioning again

Co-authored-by: Andronik Ordian <write@reusable.software>

* gossip: do not issue a connection request if we are not a validator

* guide updates

* use all relevant authorities when issuing a request

* use AuthorityDiscoveryApi instead

* update comments to the status quo

* guide: don't request availability data from ourselves

* add QueryAllChunks message

* implement QueryAllChunks

* remove unused relay_parent from StoreChunk

* test QueryAllChunks

* fast paths make short roads

* test early exit behavior

* dispute subsystem files

* rename

* fix linkcheck

* flesh out section README

* coordinator schema

* DisputeCoordinatorMessage

* stub & coordinator protocol

* dispute coordinator

* add some more message fields

* move links to bottom

* dispute participation

* Cleen It Up !

* runtime: store candidate receipts in dispute state

yeah, this is a little heavier. why are you reading this?

* Revert "runtime: store candidate receipts in dispute state"

This reverts commit 51c10bfd.

* add dispute availability statement type and prepare for availability

* add 'spam slots' to disputes runtmie

* return Spam Slots info from runtime

* rework `ImportStatement` to `ImportStatements`

* some more methods for dispute coordinator

* candidates-included runtime API

* algo for providing disputes to runtime.

* handle signing with coordinator

* dispute coordinator chain ops

* remove dead file

* remove keystore from dispute participation

* adjust ApprovedAncestor to return the necssary data

* discuss how approved ancestor and determine undisputed chain are used together

* add TODO

* initiate disputes from approval voting

* route statements from candidate backing and approval voting

* fix guide build

* onto the front

* fix early exit for waiting for requests

* add logging back

* guide updates

* keep interactions alive until receivers drop

* retry indefinitely

* cancel approval tasks on finality

* use swap_remove instead of remove

* approval-distribution: limit the amount of packets on unify

* guide: fix a typo

* compilation fix

* grammar

* Update roadmap/implementers-guide/src/node/approval/approval-distribution.md

Co-authored-by: David <dvdplm@gmail.com>

* more grammar

* propagate only local assignments/approvals after a certain depth

* increase the threshold

* guides update

Co-authored-by: David <dvdplm@gmail.com>

* Indentation fix.

* Prepare request-response for PoV fetching.

* Drop old PoV distribution.

* WIP: Fetch PoV directly from backing.

* Backing compiles.

* Runtime access and connection management for PoV distribution.

* Get rid of seemingly dead code.

* Implement PoV fetching.

Backing does not yet use it.

* Don't send `ConnectToValidators` for empty list.

* Even better - no need to check over and over again.

* PoV fetching implemented.

+ Typechecks
+ Should work

Missing:

- Guide
- Tests
- Do fallback fetching in case fetching from seconding validator fails.

* Check PoV hash upon reception.

* Implement retry of PoV fetching in backing.

* Avoid pointless validation spawning.

* Add jaeger span to pov requesting.

* Add back tracing.

* Review remarks.

* Whitespace.

* Whitespace again.

* Cleanup + fix tests.

* Log to log target in overseer.

* Fix more tests.

* Don't fail if group cannot be found.

* Simple test for PoV fetcher.

* Handle missing group membership better.

* Add test for retry functionality.

* Fix flaky test.

* Spaces again.

* Guide updates.

* Spaces.

* add AvailableDataFetchingRequest

* rename AvailabilityFetchingRequest to ChunkFetchingRequest

* rename AvailabilityFetchingResponse to Chunk_

* add AvailableDataFetching request

* add available data fetching request to availability recovery message

* remove availability recovery message

* fix

* update network bridge

* port availability recovery to request/response

* use validators.len(), not shuffling

* fix availability recovery tests

* update guide

* Update node/network/availability-recovery/src/lib.rs

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>

* Update node/network/availability-recovery/src/lib.rs

Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>

* remove println

Co-authored-by: Bernhard Schuster <bernhard@ahoi.io>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>

* f+1 always approves

* guide

* grumbles

* grumbles

* fix test

* fix tests

* Update roadmap/implementers-guide/src/node/approval/approval-voting.md

Co-authored-by: Sergei Shulepov <sergei@parity.io>

Co-authored-by: Sergei Shulepov <sergei@parity.io>

* malicious reputation cost is fatal

* make ReportBad a malicious cost

* futures control-flow for cleaning up inactive collator peers

* guide: network bridge updates

* add `PeerDisconnected` message

* guide: update

* reverse order

* remember to match

* implement disconnect peer in network bridge

* implement disconnect_inactive_peers

* test

* remove println

* don't hardcore policy

* add fuse outside of loop

* use default eviction policy

* Introduce collation fetching protocol

also move to mod.rs

* Allow `PeerId`s in requests to network bridge.

* Fix availability distribution tests.

* Move CompressedPoV to primitives.

* Request based collator protocol: validator side

- Missing: tests
- Collator side
- don't connect, if not connected

* Fixes.

* Basic request based collator side.

* Minor fix on collator side.

* Don't connect in requests in collation protocol.

Also some cleanup.

* Fix PoV distribution

* Bump substrate

* Add back metrics + whitespace fixes.

* Add back missing spans.

* More cleanup.

* Guide update.

* Fix tests

* Handle results in tests.

* Fix weird compilation issue.

* Add missing )

* Get rid of dead code.

* Get rid of redundant import.

* Fix runtime build.

* Cleanup.

* Fix wasm build.

* Format fixes.

Thanks @andronik !

* Statement distribution is now validator only.

* Avoid Arc creation where it is not necessarily needed.

* Guide update for availability distribution.

* Link fix.

* Hopefully fixed implementers guide build.

* add tracing to approval voting

* notify if session info is not working

* add dispute period to chain specs

* propagate genesis session to parachains runtime

* use `on_genesis_session`

* protect against zero cores in computation

* tweak voting rule to be based off of best and add logs

* genesis configuration should use VRF slots only

* swallow more keystore errors

* add some docs

* make validation-worker args non-optional and update clap

* better tracing for bitfield signing and provisioner

* pass amount of bits in bitfields to inclusion instead of recomputing

* debug -> warn for some logs

* better tracing for availability recovery

* a little av-store tracing

* bridge: forward availability recovery messages

* add missing try_from impl

* some more tracing

* improve approval distribution tracing

* guide: hold onto pending approval messages until NewBlocks

* Hold onto pending approval messages until NewBlocks

* guide: adjust comment

* process all actions for one wakeup at a time

* vec

* fix network bridge test

* replace randomness-collective-flip with Babe

* remove PairNotFound

* guide changes for a fast-path requesting from backing validators

* add backing group to availability recovery message

* add new phase to interaction

* typos

* add full data messages

* handle new network messages

* dispatch full data requests

* cleanup

* check chunk index

* test for invalid recovery

* tests

* Typos.

* fix some grumbles

* be more explicit about error handling and control flow

* fast-path param

* use with_chunks_only in Service

Co-authored-by: Robert Klotzner <robert.klotzner@gmx.at>