* choose the straightforward candidate selection algorithm for now

* add draft implementation of candidate selection

* fix typo in summary

* more properly report misbehaving collators

* describe how CandidateSelection subsystem becomes aware of candidates

* revise candidate selection / collator protocol interaction pattern

* implement rest of candidate selection per the guide

* review: resolve nits

* start writing test suite, harness

* implement first test

* add second test

* implement third test

Co-authored-by: Bernhard Schuster <[email protected]>

Limit the number of channels a para can have.

Split HrmpTransientValidationData and add stuff.

* Companion PR

* Implement traits for Client

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Multiple fix ups

* Remove missed HrmpOpenChannelRequestCount

* Fix the invariant string for HrmpOpenChannelRequestCount

* Update roadmap/implementers-guide/src/types/candidate.md

Co-authored-by: Bernhard Schuster <[email protected]>

* Add small clarification

* Fix another small typo

* Add a clarification to InboundHrmpMessage.sent_at

Co-authored-by: Bernhard Schuster <[email protected]>

* Add a note about time of collection of TransientValidationData

* Add HRMP digest and dmq length to TransientValidationData

* Add a note that the vector in hrmp digest is never empty

* Add hrmp watermark to TransientValidationData

* Add HRMP egress limits

* Incorporate the latest dispatchable upward messages changes.

* Update candidate.md

* Update candidate.md docs

* Fix wording

* Delete assignmets.md

* Update roadmap/implementers-guide/src/types/candidate.md

Co-authored-by: Bastian Köcher <[email protected]>

* Extract HrmpTransientValidationData and add additional data.

* Some clarifications.

* Introduce HrmpAbridgedOpenChannelRequest

* Update roadmap/implementers-guide/src/types/candidate.md

Co-authored-by: Bernhard Schuster <[email protected]>

* Fix typo: egress->ingress

* A note about sorting

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Bernhard Schuster <[email protected]>

* move validity module to disputes module

* prepare a section on approvals

* extract approval/assignments text to an overview section

* Apply suggestions from code review

Spelling

Co-authored-by: Fedor Sakharov <[email protected]>

* update kvdb-rocksdb to 0.9.1

* update Cargo.lock

* "Update Substrate"

* resolve merge conflicts

* resolve merge conflicts t. 2

Co-authored-by: parity-processbot <>

* node/network/bridge: Define protocol names as str

* "Update Substrate"

Co-authored-by: parity-processbot <>

* add and use generated weights for collective pallet

* correct pallet name

* MaxMembers

* add weight to technical collective

* Fix const assert

* update weights

* update import

* "Update Substrate"

* bump spec_version

Co-authored-by: Shawn Tabrizi <[email protected]>
Co-authored-by: parity-processbot <>

* Allow manual action for building a PR binary

* Allow test-refs for PR manual build

* Test PR build in github actions

* Use steps in build

* Removing build via github actions

* bump version to v0.8.23
* bump substrate

* Add release checklist

* add space to checkbox

* polkadot-js-apps

* Apply suggestions from code review

Co-authored-by: joe petrowski <[email protected]>

* add notes for each release item

* Apply suggestions from code review

Co-authored-by: s3krit <[email protected]>

* Apply suggestions from code review

Co-authored-by: Pierre Krieger <[email protected]>
Co-authored-by: Gavin Wood <[email protected]>

* Update RELEASE.md

Co-authored-by: Bastian Köcher <[email protected]>

* Update RELEASE.md - rearrange sections

Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: joepetrowski <[email protected]>
Co-authored-by: s3krit <[email protected]>
Co-authored-by: Pierre Krieger <[email protected]>
Co-authored-by: Gavin Wood <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* Update Cargo.toml

Fix version number

* Update Cargo.lock

* Update for substrate changes

* cargo update -p sp-io

* cargo update -p sp-io

* clean out v0 consensus crates

* remove service dependencies on old consensus code

* fix cli

* kill adder-collator

* bump Cargo.lock

* Update Kusama Council seats & Runners Up number

Update Kusama council seats to 19 and DesiredRunnersUp to 19 as well, increase representation of passive stake holders in the council and giving more visibility to runners up in Kusama Network (from 7 to 19): this will be more visibility to new candidates to expand options for stakeholders when voting.

* Update lib.rs

* Update lib.rs

* Fix up proxy

* Update weights

* Fixes

* rename 'proxy_announced' -> 'announced_proxy'

* flip rename

* update weight formulas

* Fixes

* fix merge

* Fix runtime

* Fix runtimes

* "Update Substrate"

Co-authored-by: Shawn Tabrizi <[email protected]>
Co-authored-by: parity-processbot <>

* jobs: don't early exit when there are no jobs

* utils: fix merged test

* utils: less verbose

* utils: add an assert subsystem is running

* utils: use TimeoutExt from test-helpers

* test-helpers: use TimeoutExt

* utils: handle race condition gracefully

* utils: add a test

* update Cargo.lock

* utils: remove a warning

* utils: init logger in tests

* utils: update the outdated comment

* util: wait for both subsystem and test_future to finish

* Revert "util: wait for both subsystem and test_future to finish"

This reverts commit 075b3924.

* annotated parameter names for events

* Update runtime/parachains/src/inclusion.rs

Co-authored-by: Bastian Köcher <[email protected]>

* overseer: add ExternalRequest to Event

* proposer: wait for the hash to be activated

* update comments

* overseer: handle unbounded growth of listeners map

* overseer: fix compilation

* overseer: clean up dead listeners

* overseer: cosmetic changes

* overseer: cosmetic changes t.2

* overseer: add debug_assertions

* overseer: fix formatting

* Enable Dynamic Whitelisting

* "Update Substrate"

Co-authored-by: parity-processbot <>

* extract collator assignment from cores

* check required collator

* test and fix checks

* Initial guide modifications for approvals

* Split approval assignments keys and approval votes keys

This should avoid the political problems with validator operators
wnting everything to be a remote signer.

* Update roadmap/implementers-guide/src/runtime/validity.md

Co-authored-by: Robert Habermeier <[email protected]>

* Update roadmap/implementers-guide/src/node/validity/assignments.md

Co-authored-by: Robert Habermeier <[email protected]>

* Update roadmap/implementers-guide/src/node/validity/assignments.md

Co-authored-by: Robert Habermeier <[email protected]>

* Update roadmap/implementers-guide/src/node/validity/assignments.md

Co-authored-by: Robert Habermeier <[email protected]>

* Update roadmap/implementers-guide/src/node/validity/assignments.md

Co-authored-by: Robert Habermeier <[email protected]>

* Update roadmap/implementers-guide/src/node/validity/assignments.md

Co-authored-by: Robert Habermeier <[email protected]>

* Update roadmap/implementers-guide/src/node/validity/assignments.md

Co-authored-by: Robert Habermeier <[email protected]>

* Update roadmap/implementers-guide/src/node/validity/assignments.md

Co-authored-by: Robert Habermeier <[email protected]>

* Update roadmap/implementers-guide/src/node/validity/approvals.md

Co-authored-by: hndnklnc <[email protected]>

* Update roadmap/implementers-guide/src/node/validity/approvals.md

Co-authored-by: hndnklnc <[email protected]>

* Update roadmap/implementers-guide/src/runtime/validity.md

Co-authored-by: Robert Habermeier <[email protected]>

* Not sure where these go..

* Update roadmap/implementers-guide/src/node/validity/approvals.md

Co-authored-by: Robert Habermeier <[email protected]>

* Local clock leniency

* Explain HSM blabla

* double word

* Clarify okasing things on-chain

* Assignments integration with relative time

* minor tweaks

* Clarify structure and work savings

* Use included more

* Cleanup and paramaters

See https://github.com/w3f/research-internal/issues/515



* More on GRANDPA

* Explain relay equivocation relevance

* Address postponement and on-chain verification somewhat

* small bit more explination

* just use ed25519 here: could -> should

* spelling

Co-authored-by: Robert Habermeier <[email protected]>
Co-authored-by: hndnklnc <[email protected]>

* update primitives

* correct parent_head field

* make hrmp field pub

* refactor validation data: runtime

* refactor validation data: messages

* add arguments to full_validation_data runtime API

* port runtime API

* mostly port over candidate validation

* remove some parameters from ValidationParams

* guide: update candidate validation

* update candidate outputs

* update ValidationOutputs in primitives

* port over candidate validation

* add a new test for no-transient behavior

* update util runtime API wrappers

* candidate backing

* fix missing imports

* change some fields of validation data around

* runtime API impl

* update candidate validation

* fix backing tests

* grumbles from review

* fix av-store tests

* fix some more crates

* fix provisioner tests

* fix availability distribution tests

* port collation-generation to new validation data

* fix overseer tests

* Update roadmap/implementers-guide/src/node/utility/candidate-validation.md

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* Draft HRMP related message types

* Make HRMP use upward message kinds

* Incorporate changes into messaging.md

* Make docs a bit more clear

* Clarify remove "D"

* Update roadmap/implementers-guide/src/messaging.md

Co-authored-by: Bernhard Schuster <[email protected]>

* Update roadmap/implementers-guide/src/runtime/router.md

Co-authored-by: Bernhard Schuster <[email protected]>

* Update router.md

* Update roadmap/implementers-guide/src/runtime/router.md

Co-authored-by: Robert Habermeier <[email protected]>

Co-authored-by: Bernhard Schuster <[email protected]>
Co-authored-by: Robert Habermeier <[email protected]>

* service-new: cosmetic changes

* overseer: draft of prometheus metrics

* metrics: update active_leaves metrics

* metrics: extract into functions

* metrics: resolve XXX

* metrics: it's ugly, but it works

* Bump Substrate

* metrics: move a bunch of code around

* Bumb substrate again

* metrics: fix a warning

* fix a warning in runtime

* metrics: statements signed

* metrics: statements impl RegisterMetrics

* metrics: refactor Metrics trait

* metrics: add Metrics assoc type to JobTrait

* metrics: move Metrics trait to util

* metrics: fix overseer

* metrics: fix backing

* metrics: fix candidate validation

* metrics: derive Default

* metrics: docs

* metrics: add stubs for other subsystems

* metrics: add more stubs and fix compilation

* metrics: fix doctest

* metrics: move to subsystem

* metrics: fix candidate validation

* metrics: bitfield signing

* metrics: av store

* metrics: chain API

* metrics: runtime API

* metrics: stub for avad

* metrics: candidates seconded

* metrics: ok I gave up

* metrics: provisioner

* metrics: remove a clone by requiring Metrics: Sync

* metrics: YAGNI

* metrics: remove another TODO

* metrics: for later

* metrics: add parachain_ prefix

* metrics: s/signed_statement/signed_statements

* utils: add a comment for job metrics

* metrics: address review comments

* metrics: oops

* metrics: make sure to save files before commit 😅



* use _total suffix for requests metrics

Co-authored-by: Max Inden <[email protected]>

* metrics: add tests for overseer

* update Cargo.lock

* overseer: add a test for CollationGeneration

* collation-generation: impl metrics

* collation-generation: use kebab-case for name

* collation-generation: add a constructor

Co-authored-by: Gav Wood <[email protected]>
Co-authored-by: Ashley Ruglys <[email protected]>
Co-authored-by: Max Inden <[email protected]>

* fix compile warning

* remove tests for weights

* add custom weights

* "Update Substrate"

Co-authored-by: parity-processbot <>

* overseer: add a test for CollationGeneration

* fix build

* Companion PR for #6862

* Check in Cargo.lock for CI

* "Update Substrate"

* update substrate

Co-authored-by: parity-processbot <>
Co-authored-by: André Silva <[email protected]>

* start sketching out a collation generation subsystem

* invent a basic strategy for double initialization

* clean up warnings

* impl util requests from runtime assuming a context instead of a FromJob sender

* implement collation generation algorithm from guide

* update AllMessages in tests

* fix trivial review comments

* remove another redundant declaration from merge

* filter availability cores by para_id

* handle new activations each in their own async task

* update guide according to the actual current implementation

* add initialization to guide

* add general-purpose subsystem_test_harness helper

* write first handle_new_activations test

* add test that handle_new_activations filters local_validation_data requests

* add (failing) test of collation distribution message sending

* rustfmt

* broken: work on fixing sender test

Unfortunately, for reasons that are not yet clear, despite the public key
and checked data being identical, the signer is not producing an identical
signature. This commit produces this output (among more):

signing with  Public(c4733ab0bbe3ba4c096685d1737a7f498cdbdd167a767d04a21dc7df12b8c858 (5GWHUNm5...))
checking with Public(c4733ab0bbe3ba4c096685d1737a7f498cdbdd167a767d04a21dc7df12b8c858 (5GWHUNm5...))
signed payload:  [4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 10, 0, 0, 0, c7, e5, c0, 64, 7a, db, fe, 44, 81, e5, 51, 11, 79, 9f, a5, 63, 93, 94, 3c, c4, 36, c6, 30, 36, c2, c5, 44, a2, 1b, db, b7, 82, 3, 17, a, 2e, 75, 97, b7, b7, e3, d8, 4c, 5, 39, 1d, 13, 9a, 62, b1, 57, e7, 87, 86, d8, c0, 82, f2, 9d, cf, 4c, 11, 13, 14]
checked payload: [4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 10, 0, 0, 0, c7, e5, c0, 64, 7a, db, fe, 44, 81, e5, 51, 11, 79, 9f, a5, 63, 93, 94, 3c, c4, 36, c6, 30, 36, c2, c5, 44, a2, 1b, db, b7, 82, 3, 17, a, 2e, 75, 97, b7, b7, e3, d8, 4c, 5, 39, 1d, 13, 9a, 62, b1, 57, e7, 87, 86, d8, c0, 82, f2, 9d, cf, 4c, 11, 13, 14]

* fix broken test

* collation function returns commitments hash

It doesn't look like we use the actual commitments data anywhere, and
it's not obvious if there are any fields of `CandidateCommitments`
not available to the collator, so this commit just assigns them the
entire responsibility of generating the hash.

* add missing overseer impls

* calculating erasure coding is polkadot's responsibility, not cumulus

* concurrentize per-relay_parent requests