* availability distribution: don't early return on runtime errors

* log error

* extract runtime api error from Error

* uh

* oh

* some more metrics for approval voting

* fix tests

Co-authored-by: Andronik Ordian <[email protected]>

* more tracing for av-store

* Update node/core/av-store/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/core/av-store/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/core/av-store/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update tracing everywhere

* Fix build

* More fixes

* Push cargo.lock

* Update

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Companion for https://github.com/paritytech/substrate/pull/8143

* make approval voting resilient to dropped requests

* some more

* skip whole chain if encountering spurious error

* Switch branch

* Fix compilation.

* Revert "Switch branch"

This reverts commit a3bd3621772ae98618c46102d03501856b7950ba.

* Update Cargo.lock.

* Add telemetry span back.

* Fix telemetry_span issues.

* Update substrate.

* prefix parachain log targets with parachain::

* even more consistent

* Add a config field to babe epochs

* Fix test

* Add BABE_GENESIS_EPOCH_CONFIG consts

* Use PrimaryAndSecondaryVRFSlots and remove newlines

* Make epoch_configs Some

* Fix tests

* Fix test service tests

* Add a BabeEpochConfigMigrations OnRuntimeUpgrade

* Apply suggestions

* Use PrimaryAndSecondaryPlainSlots in kusama

* Remove migration from test runtime and rococo

* Add HasPalletPrefix

* Rename to BabePalletPrefix and change BabeApi -> Babe

* "Update Substrate"

* Update substrate

* Resolve parantheses errors

Co-authored-by: parity-processbot <>

* approval-voting metrics

* metric: approvals produced

* Substrate companion #8280

https://github.com/paritytech/substrate/pull/8280

* "Update Substrate"

Co-authored-by: parity-processbot <>

* add a magic number to backing statements encoded

* fix fallout in statement table

* fix some fallout in backing

* add magic to approval votes

* remove last references to Candidate variant

* update size-hint

* Statement distribution is now validator only.

* Avoid Arc creation where it is not necessarily needed.

* Substrate companion #8275

https://github.com/paritytech/substrate/pull/8275

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Print an error if a collator connects to a node without real-overseer

* Update node/service/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>

* Logging functionality for spans.

* Jaeger spans for availability distribution.

* Fix instrumentation to use log target properly.

* Add some tracing instrumentation macros.

* Use int_tags instead of logs.

* Add span per iteration.

* Remove span::log functionality.

* Fix instrumentation log target for real.

* Add jaeger span to responding side as well.

* Revert "Fix instrumentation log target for real."

This reverts commit e1c2a2e6ff6f257e702f07d8a77c2668af92b0ef.

* Revert "Fix instrumentation to use log target properly."

This reverts commit 7caa0bd1acc6fe9727bb3a91851560d756c40ab8.

* target -> subsystem in instrumentatio macro

target is not correct either, and the correct way of using a top level
target = LOG_TARGET does not work, as the macro expects a string literal
and gets confused by the constant `LOG_TARGET`.

* Use kebab-case for spa names.

Co-authored-by: Andronik Ordian <[email protected]>

Co-authored-by: Andronik Ordian <[email protected]>

* WIP: Whole subsystem test.

* New tests compile.

* Avoid needless runtime queries for no validator nodes.

* Make tx and rx publicly accessible in virtual overseer.

This simplifies mocking in some cases, as tx can be cloned, but rx can
not.

* Whole subsystem test working.

* Update node/network/availability-distribution/src/session_cache.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Update node/network/availability-distribution/src/session_cache.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Document better what `None` return value means.

* Get rid of BitVec dependency.

* Update Cargo.lock

* Hopefully fixed implementers guide build.

Co-authored-by: Andronik Ordian <[email protected]>

This fixes a bug in block import not reporting to the telemetry.

* initial subsystem for issuing connection requests

* finish the initial impl

* integrate with the overseer

* rename to gossip-support

* fix renamings leftover

* remove run_inner

* fix compilation

* random subset of sqrt

* WIP

* availability distribution, still very wip.

Work on the requesting side of things.

* Some docs on what I intend to do.

* Checkpoint of session cache implementation

as I will likely replace it with something smarter.

* More work, mostly on cache

and getting things to type check.

* Only derive MallocSizeOf and Debug for std.

* availability-distribution: Cache feature complete.

* Sketch out logic in `FetchTask` for actual fetching.

- Compile fixes.
- Cleanup.

* Format cleanup.

* More format fixes.

* Almost feature complete `fetch_task`.

Missing:

- Check for cancel
- Actual querying of peer ids.

* Finish FetchTask so far.

* Directly use AuthorityDiscoveryId in protocol and cache.

* Resolve `AuthorityDiscoveryId` on sending requests.

* Rework fetch_task

- also make it impossible to check the wrong chunk index.
- Export needed function in validator_discovery.

* From<u32> implementation for `ValidatorIndex`.

* Fixes and more integration work.

* Make session cache proper lru cache.

* Use proper lru cache.

* Requester finished.

* ProtocolState -> Requester

Also make sure to not fetch our own chunk.

* Cleanup + fixes.

* Remove unused functions

- FetchTask::is_finished
- SessionCache::fetch_session_info

* availability-distribution responding side.

* Cleanup + Fixes.

* More fixes.

* More fixes.

adder-collator is running!

* Some docs.

* Docs.

* Fix reporting of bad guys.

* Fix tests

* Make all tests compile.

* Fix test.

* Cleanup + get rid of some warnings.

* state -> requester

* Mostly doc fixes.

* Fix test suite.

* Get rid of now redundant message types.

* WIP

* Rob's review remarks.

* Fix test suite.

* core.relay_parent -> leaf for session request.

* Style fix.

* Decrease request timeout.

* Cleanup obsolete errors.

* Metrics + don't fail on non fatal errors.

* requester.rs -> requester/mod.rs

* Panic on invalid BadValidator report.

* Fix indentation.

* Use typed default timeout constant.

* Make channel size 0, as each sender gets one slot anyways.

* Fix incorrect metrics initialization.

* Fix build after merge.

* More fixes.

* Hopefully valid metrics names.

* Better metrics names.

* Some tests that already work.

* Slightly better docs.

* Some more tests.

* Fix network bridge test.

* test: add unit test to catch missing distribution to subsystems faster

* add a simple count

* introduce proc macro to generate dispatch type

* refactor

* refactor

* chore: add license

* fixup unit test

* fixup merge

* better errors

* better fmt

* fix error spans

* better docs

* better error messages

* ui test foo

* Update node/subsystem/dispatch-gen/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/network/bridge/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/subsystem/Cargo.toml

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/subsystem/dispatch-gen/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/subsystem/dispatch-gen/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/network/bridge/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

* fix compilation

* use find_map

* drop the silly 2, use _inner instead

* Update node/network/bridge/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Update node/subsystem/dispatch-gen/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* nail deps down

* more into()

* flatten

* missing use statement

* fix messages order

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Andronik Ordian <[email protected]>

* some initial spans for approval voting

* add stage earlier

* Pass shared_authority_set to request_response_config_for_chain

* "Update Substrate"

Co-authored-by: Ashley Ruglys <[email protected]>
Co-authored-by: parity-processbot <>

* Companion for Substrate #8185

https://github.com/paritytech/substrate/pull/8185

* "Update Substrate"

Co-authored-by: parity-processbot <>

* spawn availability store and approval voting subsystems as blocking tasks

* refactor

Currently the first last timestamp is `0` and that leads to the first
slot being `0` which is reserved for genesis. There is some debug assert
in BABE that complains about this in Cumulus :D

* bump version and substrate

* update weights

* bump substrate

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=frame_system --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/

* cargo run --release --features=runti...

* Template

* Midway..

* Make all 3 runtimes work

* Update node/service/Cargo.toml

* undo brach updates

* fix with latest version

* Update runtime/kusama/src/lib.rs

Co-authored-by: Shawn Tabrizi <[email protected]>

* Update runtime/rococo/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Final nits

* Update runtime/kusama/src/lib.rs

Co-authored-by: Guillaume Thiolliere <[email protected]>

Co-authored-by: Shawn Tabrizi <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Guillaume Thiolliere <[email protected]>

* add tracing to approval voting

* notify if session info is not working

* add dispute period to chain specs

* propagate genesis session to parachains runtime

* use `on_genesis_session`

* protect against zero cores in computation

* tweak voting rule to be based off of best and add logs

* genesis configuration should use VRF slots only

* swallow more keystore errors

* add some docs

* make validation-worker args non-optional and update clap

* better tracing for bitfield signing and provisioner

* pass amount of bits in bitfields to inclusion instead of recomputing

* debug -> warn for some logs

* better tracing for availability recovery

* a little av-store tracing

* bridge: forward availability recovery messages

* add missing try_from impl

* some more tracing

* improve approval distribution tracing

* guide: hold onto pending approval messages until NewBlocks

* Hold onto pending approval messages until NewBlocks

* guide: adjust comment

* process all actions for one wakeup at a time

* vec

* fix network bridge test

* replace randomness-collective-flip with Babe

* remove PairNotFound

* feat/view: assure heads in a view are sorted

Allows O(n) comparisons, adds an alternate equiv relation
which takes O(n^2) for integrity verification.

Ref #2133

* revert: remove custom PartialEq impl, there are no duplicates

* fix: do not sort the live_heads, that alters the local view

* refactor/view: heads should not be public

* chore/spellcheck: add unfinalized

* fix/view: add missing len() and is_empty() fns

* quirk

* vec is not view

* Update node/network/approval-distribution/src/tests.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Update node/network/bridge/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Update node/network/protocol/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

* fixup comment

* fix botched test

Co-authored-by: Andronik Ordian <[email protected]>