* node/approval-voting: extract and test count_no_shows method

This commit extracts no_show computation into a pure function so that it can be
extensively unit tested.

* node/approval-voting: ignore invalid validator indexes in count_no_show

Previously indexes that were past the length of the approvals bitvector
would contribute to the no_show count or the next_no_show value. This
commit changes the behavior to ignore garbage values.

* node/approval-voting: add comment for next_no_show adding clock_drift

* cleanup more tests and spaces

* oops

* extract determine_new_blocks into a separate utility

* rework docs

* node/approval_checking: break out filled_tranch_iterator method

In the subsequent commit, we will begin to test this method in
isolation.

* node/approval-voting: fix tranche back-filling algorithm

Previously, this algorithm would generate duplicate, empty entries for
tranches (1..pre_end). This is caused because the initial value (0) for
gap_end is treated as the end of a prior tranche that wasn't actually
processed. The first pass thus would add (1..tranche) empty entries, in
addition to the (0..pre_end) empty entries chained at the end of the
method.

This is fixed by using the current tranche as the gap_start for the
first iteration, ensuring that the approval_entries_filled only produces
entries in the range (pre_end..post_start).

* Address feedback

* Drop guard for detecting stale approvals

* address nits with different API

* update apis

* fix api

* rename to `KillStorageResult`

* update Substrate

Co-authored-by: parity-processbot <>

* implement dispute participation subsystem

* guide: minor fix in dispute participation

* Update node/core/dispute-participation/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

* dispute: add comments to participation logic

* dispute-coordinator: fix test compilation

* implementers-guide: update dispute participation

* dispute-participation: add error for missing validation code

* dispute-participation: add tests

* Update node/core/dispute-participation/src/lib.rs

Co-authored-by: Robert Habermeier <[email protected]>

* guide: update overseer protocol dispute participation message

* dispute-participation: remove duplication in tests

Co-authored-by: Andronik Ordian <[email protected]>
Co-authored-by: Robert Habermeier <[email protected]>

* add more information about what's happening during approval work

* record assignment tranches in histogram

* approval-voting: logs for invalid votes

* proper errors for assignment checks

* proper errors for approval checks

* skeleton for dispute-coordinator

* add coordinator and participation message types

* begin dispute-coordinator DB

* functions for loading

* implement strongly-typed DB transaction

* add some tests for DB transaction

* core logic for pruning

* guide: update candidate-votes key for coordinator

* update candidate-votes key

* use big-endian encoding for session, and implement upper bound generator

* finish implementing pruning

* add a test for note_current_session

* define state of the subsystem itself

* barebones subsystem definition

* control flow

* more control flow

* implement session-updating logic

* trace

* control flow for message handling

* Update node/core/dispute-coordinator/src/lib.rs

Co-authored-by: André Silva <[email protected]>

* Update node/subsystem/src/messages.rs

Co-authored-by: André Silva <[email protected]>

* some more control flow

* guide: remove overlay

* more control flow

* implement some DB getters

* make progress on importing statements

* add SignedDisputeStatement struct

* move ApprovalVote to shared primitives

* add a signing-payload API to explicit dispute statements

* add signing-payload to CompactStatement

* add relay-parent hash to seconded/valid dispute variatns

* correct import

* type-safe wrapper around dispute statements

* use checked dispute statement in message type

* extract rolling session window cache to subsystem-util

* extract session window tests

* approval-voting: use rolling session info cache

* reduce dispute window to match runtime in practice

* add byzantine_threshold and supermajority_threshold utilities to primitives

* integrate rolling session window

* Add PartialOrd to CandidateHash

* add Ord to CandidateHash

* implement active dispute update

* add dispute messages to AllMessages

* add dispute stubs to overseer

* inform dispute participation to participate

* implement issue_local_statement

* implement `determine_undisputed_chain`

* fix warnings

* test harness for dispute coordinator tests

* add more helpers to test harness

* add some more helpers

* some tests for dispute coordinator

* ignore wrong validator indices

* test finality voting rule constraint

* add more tests

* add variants to network bridge

* fix test compilation

* remove most dispute coordinator functionality

as of #3222 we can do most of the work within the approval voting subsystem

* Revert "remove most dispute coordinator functionality"

This reverts commit 9cd615e8eb6ca0b382cbaff525d813e753d6004e.

* Use thiserror

Co-authored-by: Bernhard Schuster <[email protected]>

* Update node/core/dispute-coordinator/src/lib.rs

Co-authored-by: Bernhard Schuster <[email protected]>

* extract tests to separate module

* address nit

* adjust run_iteration API

Co-authored-by: André Silva <[email protected]>
Co-authored-by: Bernhard Schuster <[email protected]>

* create newtype for ValidationCodeHash

* pvf: fix tests compilation

* primitives: fix test compilation

* tabify tests

* move mod tests; up

* replace HistoricalValidationCode usages with ValidationCodeByHash

* runtime-api: tabify tests file

* update implementers guide

Bumps [pin-project](https://github.com/taiki-e/pin-project) from 1.0.4 to 1.0.7.
- [Release notes](https://github.com/taiki-e/pin-project/releases)
- [Changelog](https://github.com/taiki-e/pin-project/blob/main/CHANGELOG.md)
- [Commits](https://github.com/taiki-e/pin-project/compare/v1.0.4...v1.0.7

)

---
updated-dependencies:
- dependency-name: pin-project
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Moves tests into separate files
in order to limit the loc per file.

Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.14 to 0.3.15.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.14...0.3.15

)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* remove tracing::intrument annotations

* remove unused param and leftover

* more leftovers

* Create validator_side module

* Subsume Candidate Selection

* Add test to ensure candidate backing logic is correct

* Ensure secondings are adequately cleaned up and address test flakyness

* Address Feedback

* Companion for PR 8913

polkadot companion: https://github.com/paritytech/substrate/pull/8913

* update Substrate

Co-authored-by: parity-processbot <>

* guide: reversion safety

* guide: manage reversion safety in subsystems

* add leaf status to ActivatedLeaf

* add an LRU-cache to overseer for staleness detection

* update ActivatedLeaf usages in tests to contain status field

* add variant where missed accidentally

* add some helpers to LeafStatus

* address grumbles

Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.25 to 0.1.26.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.25...tracing-0.1.26

)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* log validator index when importing statement

* log validation failures in debug

* Put WIP artifacts next to ready ones

Fixes #3044

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>

* Register ReadRuntimeVersionExt

* Update Substrate

This includes the fix of compilation for macOS platforms.

* issue with ArtifactId::path

* fix artifacts path

* Wrap an overly long line

Co-authored-by: Sergei Shulepov <[email protected]>

* Remove signature verification in backing.

`SignedFullStatement` now signals that the signature has already been
checked.

* Remove unused check_payload function.

* Introduced unchecked signed variants.

* Fix inclusion to use unchecked variant.

* More unchecked variants.

* Use unchecked variants in protocols.

* Start fixing statement-distribution.

* Fixup statement distribution.

* Fix inclusion.

* Fix warning.

* Fix backing properly.

* Fix bitfield distribution.

* Make crypto store optional for `RuntimeInfo`.

* Factor out utility functions.

* get_group_rotation_info

* WIP: Collator cleanup + check signatures.

* Convenience signature checking functions.

* Check signature on collator-side.

* Fix warnings.

* Fix collator side tests.

* Get rid of warnings.

* Better Signed/UncheckedSigned implementation.

Also get rid of Encode/Decode for Signed! *party*

* Get rid of dead code.

* Move Signed in its own module.

* into_checked -> try_into_checked

* Fix merge.

* Update branch

* Make it compile

* Compile

* gate approval-checking logic (#2470)

* Fix build

* Updates

* Fix merge

* Adds missing crate

* Companion for Substrate#8386

https://github.com/paritytech/substrate/pull/8386



* Fix fix fix

* Fix

* Fix compilation

* Rewrite to `ParachainsInherentDataProvider`

* Make it compile

* Renamings

* Revert stuff

* Remove stale file

* Guide updates

* Update node/core/parachains-inherent/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Update node/core/parachains-inherent/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Apply suggestions from code review

* Reset accidental changes

* More

* Remove stale file

* update Substrate

Co-authored-by: Robert Habermeier <[email protected]>
Co-authored-by: Andronik Ordian <[email protected]>
Co-authored-by: parity-processbot <>

* Simplify some Option / Result / ? operator patterns

When they identically match a combinator on those types.

Tool-aided by [comby-rust](https://github.com/huitseeker/comby-rust

).

* adjust review comments

Co-authored-by: Shawn Tabrizi <[email protected]>

* Factor out runtime module into utils.

* Add maybe_authority information to `PeerConnected` event.

We already gather this information in authority discovery, so we might
as well share it with others.

This opens up an easy path to trigger validators differently from normal
nodes, e.g. for prioritization. This change has become more important
now, that we just connect to all validators and therefore just have a
long peer list without any information about those nodes.

* Test fix.

* Companion for Substrate#8588

https://github.com/paritytech/substrate/pull/8588

* update Substrate

Co-authored-by: parity-processbot <>

* Wip

* Increase proposer timeout.

* WIP.

* Better timeout values now that we are going to be connected to all nodes. (#2778)

* Better timeout values.

* Fix typo.

* Fix validator bandwidth.

* Fix compilation.

* Better and more consistent sizes.

Most importantly code size is now 5 Meg, which is the limit we currently
want to support in statement distribution.

* Introduce statement fetching request.

* WIP

* Statement cache retrieval logic.

* Review remarks by @rphmeier

* Fixes.

* Better requester logic.

* WIP: Handle requester messages.

* Missing dep.

* Fix request launching logic.

* Finish fetching logic.

* Sending logic.

* Redo code size calculations.

Now that max code size is compressed size.

* Update Cargo.lock (new dep)

* Get request receiver to statement distribution.

* Expose new functionality for responding to requests.

* Cleanup.

* Responder logic.

* Fixes + Cleanup.

* Cargo.lock

* Whitespace.

* Add lost copyright.

* Launch responder task.

* Typo.

* info -> warn

* Typo.

* Fix.

* Fix.

* Update comment.

* Doc fix.

* Better large statement heuristics.

* Fix tests.

* Fix network bridge tests.

* Add test for size estimate.

* Very simple tests that checks we get LargeStatement.

* Basic check, that fetching of large candidates is performed.

* More tests.

* Basic metrics for responder.

* More metrics.

* Use Encode::encoded_size().

* Some useful spans.

* Get rid of redundant metrics.

* Don't add peer on duplicate.

* Properly check hash

instead of relying on signatures alone.

* Preserve ordering + better flood protection.

* Get rid of redundant clone.

* Don't shutdown responder on failed query.

And add test for this.

* Smaller fixes.

* Quotes.

* Better queue size calculation.

* A bit saner response sizes.

* Fixes.

shows what I get for being hasty last time. i added a test this time, which would have caught this issue last time.

with the MaxValidators parameter set, this meant that only validators participating in parachain consensus would be able to author relay-chain blocks containing backed candidates