* extract database from av-store itself

* generalize approval-voting over database type

* modes (without handling) and pruning old wakeups

* rework approval importing

* add our_approval_sig to ApprovalEntry

* import assignment

* guide updates for check-full-approval changes

* some aux functions

* send messages when becoming active.

* guide: network bridge sends view updates only when done syncing

* network bridge: send view updates only when done syncing

* tests for new network-bridge behavior

* add a test for updating approval entry with sig

* fix some warnings

* test load-all-blocks

* instantiate new parachains DB

* fix network-bridge empty view updates

* tweak

* fix wasm build, i think

* Update node/core/approval-voting/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

* add some versioning to parachains_db

* warnings

* fix merge changes

* remove versioning again

Co-authored-by: Andronik Ordian <[email protected]>

* dispute types

* add Debug to dispute primitives in std and InherentData

* use ParachainsInherentData on node-side

* change inclusion_inherent to paras_inherent

* RuntimeDebug

* add type parameter to PersistedValidationData users

* fix test client

* spaces

* fix collation-generation test

* fix provisioner tests

* remove references to inclusion inherent

* guide: don't request availability data from ourselves

* add QueryAllChunks message

* implement QueryAllChunks

* remove unused relay_parent from StoreChunk

* test QueryAllChunks

* fast paths make short roads

* test early exit behavior

* guide updates

* keep interactions alive until receivers drop

* retry indefinitely

* cancel approval tasks on finality

* use swap_remove instead of remove

* reduce signal channel capacity

* more tracing for approval-voting

* Remove stuff out of the runtime that does not belong there.

There might be more, but it is a start.

* White space fixes.

* Fix tests.

* Leave whitespace in ui tests alone.

* Add back zstd for no reason.

* Fix browser wasm (hopefully)

* overseer: pass messages directly between subsystems

* test that message is held on to

* Update node/overseer/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* give every subsystem an unbounded sender too

* remove metered_channel::name

1. we don't provide good names
2. these names are never used anywhere

* unused mut

* remove unnecessary &mut

* subsystem unbounded_send

* remove unused MaybeTimer

We have channel size metrics that serve the same purpose better now and the implementation of message timing was pretty ugly.

* remove comment

* split up senders and receivers

* update metrics

* fix tests

* fix test subsystem context

* use SubsystemSender in jobs system now

* refactor of awful jobs code

* expose public `run` on JobSubsystem

* update candidate backing to new jobs & use unbounded

* bitfield signing

* candidate-selection

* provisioner

* approval voting: send unbounded for assignment/approvals

* async not needed

* begin bridge split

* split up network tasks into background worker

* port over network bridge

* Update node/network/bridge/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

* rename ValidationWorkerNotifications

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>
Co-authored-by: Andronik Ordian <[email protected]>

* approval-voting: more spans and metrics

* s/db/approval db

* Indentation fix.

* Prepare request-response for PoV fetching.

* Drop old PoV distribution.

* WIP: Fetch PoV directly from backing.

* Backing compiles.

* Runtime access and connection management for PoV distribution.

* Get rid of seemingly dead code.

* Implement PoV fetching.

Backing does not yet use it.

* Don't send `ConnectToValidators` for empty list.

* Even better - no need to check over and over again.

* PoV fetching implemented.

+ Typechecks
+ Should work

Missing:

- Guide
- Tests
- Do fallback fetching in case fetching from seconding validator fails.

* Check PoV hash upon reception.

* Implement retry of PoV fetching in backing.

* Avoid pointless validation spawning.

* Add jaeger span to pov requesting.

* Add back tracing.

* Review remarks.

* Whitespace.

* Whitespace again.

* Cleanup + fix tests.

* Log to log target in overseer.

* Fix more tests.

* Don't fail if group cannot be found.

* Simple test for PoV fetcher.

* Handle missing group membership better.

* Add test for retry functionality.

* Fix flaky test.

* Spaces again.

* Guide updates.

* Spaces.

* add consensus log type

* origin and issue force_approve

* add origin in runtimes

* ref API

* scrape force_approve digest from header

* add parent_hash to BlockEntry

* add block_number to block entry and force_approve skeleton

* implement and plug in force-approve

* test force_approve

* test force_approve extraction

* westend runtime

* Update node/core/approval-voting/src/approval_db/v1/mod.rs

Co-authored-by: Bastian Köcher <[email protected]>

* rename

* Update runtime/parachains/src/initializer.rs

Co-authored-by: André Silva <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: André Silva <[email protected]>

* add number to `ActivatedLeavesUpdate`

* update subsystem util and overseer

* use new ActivatedLeaf everywhere

* sort view

* sorted and limited view in network bridge

* use live block hash only if it's newer

* grumples

* f+1 always approves

* guide

* grumbles

* grumbles

* fix test

* fix tests

* Update roadmap/implementers-guide/src/node/approval/approval-voting.md

Co-authored-by: Sergei Shulepov <[email protected]>

Co-authored-by: Sergei Shulepov <[email protected]>

* improve tracing for approval voting

* assignment criteria tracing

* new syntax

* Improve logging

* Review feedback

* Fix some warning and some further logging changes

* more approval voting instrumentation

* fix `unapproved_candidates`

* Update node/core/approval-voting/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* improve backing/provisioner spans

* span for collation requests

* add para_id to unbacked candidate spans

* differentiate validation-construction and find-assignment in selection

* better find-assignment spans

* organize unbacked-candidate spans directly under job root

* Update node/core/provisioner/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

Co-authored-by: Andronik Ordian <[email protected]>

* some more metrics for approval voting

* fix tests

Co-authored-by: Andronik Ordian <[email protected]>

* more tracing for av-store

* Update node/core/av-store/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/core/av-store/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/core/av-store/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update tracing everywhere

* Fix build

* More fixes

* Push cargo.lock

* Update

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Companion for https://github.com/paritytech/substrate/pull/8143

* make approval voting resilient to dropped requests

* some more

* skip whole chain if encountering spurious error

* prefix parachain log targets with parachain::

* even more consistent

* Add a config field to babe epochs

* Fix test

* Add BABE_GENESIS_EPOCH_CONFIG consts

* Use PrimaryAndSecondaryVRFSlots and remove newlines

* Make epoch_configs Some

* Fix tests

* Fix test service tests

* Add a BabeEpochConfigMigrations OnRuntimeUpgrade

* Apply suggestions

* Use PrimaryAndSecondaryPlainSlots in kusama

* Remove migration from test runtime and rococo

* Add HasPalletPrefix

* Rename to BabePalletPrefix and change BabeApi -> Babe

* "Update Substrate"

* Update substrate

* Resolve parantheses errors

Co-authored-by: parity-processbot <>

* approval-voting metrics

* metric: approvals produced

* add a magic number to backing statements encoded

* fix fallout in statement table

* fix some fallout in backing

* add magic to approval votes

* remove last references to Candidate variant

* update size-hint

* Statement distribution is now validator only.

* Avoid Arc creation where it is not necessarily needed.

* WIP

* availability distribution, still very wip.

Work on the requesting side of things.

* Some docs on what I intend to do.

* Checkpoint of session cache implementation

as I will likely replace it with something smarter.

* More work, mostly on cache

and getting things to type check.

* Only derive MallocSizeOf and Debug for std.

* availability-distribution: Cache feature complete.

* Sketch out logic in `FetchTask` for actual fetching.

- Compile fixes.
- Cleanup.

* Format cleanup.

* More format fixes.

* Almost feature complete `fetch_task`.

Missing:

- Check for cancel
- Actual querying of peer ids.

* Finish FetchTask so far.

* Directly use AuthorityDiscoveryId in protocol and cache.

* Resolve `AuthorityDiscoveryId` on sending requests.

* Rework fetch_task

- also make it impossible to check the wrong chunk index.
- Export needed function in validator_discovery.

* From<u32> implementation for `ValidatorIndex`.

* Fixes and more integration work.

* Make session cache proper lru cache.

* Use proper lru cache.

* Requester finished.

* ProtocolState -> Requester

Also make sure to not fetch our own chunk.

* Cleanup + fixes.

* Remove unused functions

- FetchTask::is_finished
- SessionCache::fetch_session_info

* availability-distribution responding side.

* Cleanup + Fixes.

* More fixes.

* More fixes.

adder-collator is running!

* Some docs.

* Docs.

* Fix reporting of bad guys.

* Fix tests

* Make all tests compile.

* Fix test.

* Cleanup + get rid of some warnings.

* state -> requester

* Mostly doc fixes.

* Fix test suite.

* Get rid of now redundant message types.

* WIP

* Rob's review remarks.

* Fix test suite.

* core.relay_parent -> leaf for session request.

* Style fix.

* Decrease request timeout.

* Cleanup obsolete errors.

* Metrics + don't fail on non fatal errors.

* requester.rs -> requester/mod.rs

* Panic on invalid BadValidator report.

* Fix indentation.

* Use typed default timeout constant.

* Make channel size 0, as each sender gets one slot anyways.

* Fix incorrect metrics initialization.

* Fix build after merge.

* More fixes.

* Hopefully valid metrics names.

* Better metrics names.

* Some tests that already work.

* Slightly better docs.

* Some more tests.

* Fix network bridge test.

* some initial spans for approval voting

* add stage earlier

* Companion for Substrate #8185

https://github.com/paritytech/substrate/pull/8185

* "Update Substrate"

Co-authored-by: parity-processbot <>

* add tracing to approval voting

* notify if session info is not working

* add dispute period to chain specs

* propagate genesis session to parachains runtime

* use `on_genesis_session`

* protect against zero cores in computation

* tweak voting rule to be based off of best and add logs

* genesis configuration should use VRF slots only

* swallow more keystore errors

* add some docs

* make validation-worker args non-optional and update clap

* better tracing for bitfield signing and provisioner

* pass amount of bits in bitfields to inclusion instead of recomputing

* debug -> warn for some logs

* better tracing for availability recovery

* a little av-store tracing

* bridge: forward availability recovery messages

* add missing try_from impl

* some more tracing

* improve approval distribution tracing

* guide: hold onto pending approval messages until NewBlocks

* Hold onto pending approval messages until NewBlocks

* guide: adjust comment

* process all actions for one wakeup at a time

* vec

* fix network bridge test

* replace randomness-collective-flip with Babe

* remove PairNotFound

* Substrate companion for #8163

https://github.com/paritytech/substrate/pull/8163

* "Update Substrate"

Co-authored-by: parity-processbot <>

* feat/jaeger: more spans, more stages

Stage numbers are still arbitrarily picked.

* feat/jaeger: additional spans

* chore/spellcheck: improve the dictionary

* fix/jaeger JaegerSpan -> jaeger::Span

* Use KeyValueDB in approval-voting

* use KVDB instead of AuxStore

* add rocksdb to cargo toml

* add a Config struct

* create new DB in service

* fix dep for regular node

* make optional

* post merge fix

Co-authored-by: Andronik Ordian <[email protected]>