* Companion for Taskmanager: Remove `clean_shutdown`

https://github.com/paritytech/substrate/pull/10314

* Update Substrate

* Remove warning

* impl prefered items

Closes #4330

* do not stop attempting to select, just because one did not fit

* doc

* prefered -> preferred

* missing usage of the preferred indices

* sigh

* shuffle is not available for chacha

* remove duplicate weight addition

* ref vs no ref

* chore/provisioner: move metrics to a separate module

* avoid the duplicate names

* reduce all checks

* fixup tests

* Update node/core/provisioner/src/lib.rs

Co-authored-by: Zeke Mostov <[email protected]>

* chore: fmt

* chore: spellcheck

* doc

* remove the enum anti-pattern

* guide update - remove all the responsibilities

* add another trivial check

* Update node/core/provisioner/src/metrics.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Update roadmap/implementers-guide/src/node/utility/provisioner.md

Co-authored-by: Andronik Ordian <[email protected]>

* Update node/core/provisioner/src/metrics.rs

Co-authored-by: Andronik Ordian <[email protected]>

Co-authored-by: Zeke Mostov <[email protected]>
Co-authored-by: Andronik Ordian <[email protected]>

* Mostly notes.

* Better error messages.

* Introduce Fatal/NonFatal + drop back channel participation

- Fatal/NonFatal - in order to make it easier to use utility functions.
- We drop the back channel in dispute participation as it won't be
needed any more.

* Better error messages.

* Utility function for receiving `CandidateEvent`s.

* Ordering module typechecks.

* cargo fmt

* Prepare spam slots module.

* Implement SpamSlots mechanism.

* Implement queues.

* cargo fmt

* Participation.

* Participation taking shape.

* Finish participation.

* cargo fmt

* Cleanup.

* WIP: Cleanup + Integration.

* Make `RollingSessionWindow` initialized by default.

* Make approval voting typecheck.

* Get rid of lazy_static & fix approval voting tests

* Move `SessionWindowSize` to node primitives.

* Implement dispute coordinator initialization.

* cargo fmt

* Make queues return error instead of boolean.

* Initialized: WIP

* Introduce chain api for getting finalized block.

* Fix ordering to only prune candidates on finalized events.

* Pruning of old sessions in spam slots.

* New import logic.

* Make everything typecheck.

* Fix warnings.

* Get rid of obsolete dispute-participation.

* Fixes.

* Add back accidentelly deleted Cargo.lock

* Deliver disputes in an ordered fashion.

* Add module docs for errors

* Use type synonym.

* hidden docs.

* Fix overseer tests.

* Ordering provider taking `CandidateReceipt`.

... To be kicked on one next commit.

* Fix ordering to use relay_parent

as included block is not unique per candidate.

* Add comment in ordering.rs.

* Take care of duplicate entries in queues.

* Better spam slots.

* Review remarks + docs.

* Fix db tests.

* Participation tests.

* Also scrape votes on first leaf for good measure.

* Make tests typecheck.

* Spelling.

* Only participate in actual disputes, not on every import.

* Don't account backing votes to spam slots.

* Fix more tests.

* Don't participate if we don't have keys.

* Fix tests, typos and warnings.

* Fix merge error.

* Spelling fixes.

* Add missing docs.

* Queue tests.

* More tests.

* Add metrics + don't short circuit import.

* Basic test for ordering provider.

* Import fix.

* Remove dead link.

* One more dead link.

Co-authored-by: Lldenaurois <[email protected]>

* export hrmp config

* fix build

* Add missing license header

* Update runtime/parachains/src/builder.rs

Co-authored-by: Andronik Ordian <[email protected]>

Co-authored-by: Andronik Ordian <[email protected]>

* prepare worker: Catch unexpected unwinds

* Use more specific wording for unknown panic payload

* enable beefy explicitly

* update beefy cli flag docs

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.70 to 1.0.71.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.70...v1.0.71

)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [strum](https://github.com/Peternator7/strum) from 0.22.0 to 0.23.0.
- [Release notes](https://github.com/Peternator7/strum/releases)
- [Changelog](https://github.com/Peternator7/strum/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Peternator7/strum/commits

)

---
updated-dependencies:
- dependency-name: strum
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.13.0 to 1.14.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/commits

)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Substrate companion: Authority discovery multiple peer ids

Authority discovery before had a fixed mapping from `PeerId` to
`AuthorityId`. This wasn't correct, as a `PeerId` can actually map to
multiple `AuthorityId`s. The linked Substrate pr fixes this.

https://github.com/paritytech/substrate/pull/10259

* Update node/network/availability-distribution/src/requester/mod.rs

* Update node/network/collator-protocol/src/validator_side/mod.rs

* Update node/network/statement-distribution/src/tests.rs

* Update guide

* Adapt to Substrate pr

* Update Substrate

* Companion for substrate#9878

Signed-off-by: koushiro <[email protected]>

* update substrate

Co-authored-by: thiolliere <[email protected]>

* move things around, add filter methods

* validator keys, modify availability bitfields according to disputes

* simplify, keep the filter -> sanitize generic for both usecases

* minor

* assure tests still work, reduce changeset

* integration

* start entropy passing

* fixins

* compile, 1 failing test

* filter with coverage

* fixins

* Update runtime/parachains/src/paras_inherent.rs

Co-authored-by: Robert Habermeier <[email protected]>

* slip of the pen

* improve test cases

* misc

* fix

* fixins

* test avoid extra into() calls in assert_noop!

* chores

* ff

* test fixup superfluous into call

* chore: pfmt

* improve apply_block_weight_limit to try to maximize the number of sufficiently backed

blocks and add extra bitfields in a round-robin fashion

* new code treats the lack of backed candidates as ok

* Use vrf based entropy

* fixup vrf random

* add warn

* slip of the pen

* fixup

* assure ordering

* rethink apply_weights

* mock

* use a closure as predicate check

* extract and use DisputedBitfield

* chore: simplify

* remove stray dbg

* chore: fmt

* address feedback

* fix test, halfway there

* stage1

* dbg stuff

* make group selection align

* fix session index

* fix wrongly returned candidates

* cleanup

* chore fmt

* fix ensure check

* make good case test work

* more tests for bitfields

* create sanitize_backed_candidates

* fixup tests

* update guide

* add check referenced in the guide

* improve weights code

* fmt

* fixins

* Update roadmap/implementers-guide/src/runtime/inclusion.md

Co-authored-by: Zeke Mostov <[email protected]>

* compiling + address review

* add comments

* fix weight calc

* address review comments and test failure

* fix

* fix: condition

* Fix random_sel function

* Fix overlength block check

* Zeke + Ladi commit for disputes filtering + integration test builder + runtime benchmarks + integration tests

* Add benchmarks for code upgrades

* Code upgrade bench; Feature gate TestWeightInfo

* Try and make CI happier

* Feature gate enter test to not(benchmarks)

* Make sure no unused imports/fn

* refactor, re-use, the beginning

* Fix issue with frame benchmarking dep compilation

* More precise feature gating for some derives

* integrate piece-wise

* foo

* fixins

* chore fmt

* fixins

* rename const generic

* Update runtime/parachains/src/paras_inherent.rs

Co-authored-by: Zeke Mostov <[email protected]>

* Fix compilation

* limit to test

* remove unused spam slots

* spellcheck

* remove a tick, fix a typo

* Add Code upgrade weights

* comment improvements + >=

Co-authored-by: Zeke Mostov <[email protected]>

* remove another tick

* Update runtime/parachains/src/paras_inherent/benchmarking.rs

Co-authored-by: Zeke Mostov <[email protected]>

* saturating fixins + some spaces

* fix

* benchmarking - preliminary results

* Add training wheels

* Refactor some early exit logic for enter

* Gracefully handle filtering bitfields & candidates (#4280)

This updates the logic for sanitize_bitfields and sanitize_backed_candidates to never error when there is an issue, but instead to simply skip the problematic items.

* Refactor inherent data weight limiting logic (#4287)

* Apply suggestions from code review

* Update runtime/parachains/src/builder.rs

Co-authored-by: Zeke Mostov <[email protected]>

* Update runtime/parachains/src/builder.rs

* Update runtime/parachains/src/paras_inherent.rs

* final pass

* Run cargo +nightly-2021-10-29 fmt

* Update implementors guide with `sanitize_*` & `enter` (#4294)

* Make spell check happier

* Make wasm runtimes compile with benchmarks enabled (#4303)

* comment stuff out, use old toml

* Seems to be working?

* Remove feature gating from builder

* Remove commented out stuff

* Remove generic from digest

* Update weight files for runtime

Co-authored-by: Robert Habermeier <[email protected]>
Co-authored-by: Zeke Mostov <[email protected]>
Co-authored-by: Lldenaurois <[email protected]>
Co-authored-by: Zeke Mostov <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Bumps [async-process](https://github.com/smol-rs/async-process) from 1.1.0 to 1.3.0.
- [Release notes](https://github.com/smol-rs/async-process/releases)
- [Changelog](https://github.com/smol-rs/async-process/blob/master/CHANGELOG.md)
- [Commits](https://github.com/smol-rs/async-process/compare/v1.1.0...v1.3.0

)

---
updated-dependencies:
- dependency-name: async-process
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.69 to 1.0.70.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.69...v1.0.70

)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* update event variants

* fmt

* fixes

* update substrate

Co-authored-by: thiolliere <[email protected]>

* tweaks to xcm stuff for benchmarking

* Update xcm/xcm-executor/src/lib.rs

* Increase preparation-timeout to 60 seconds

* Adapt `pvf_preparation_time` metric to the new value

Bumps [hex-literal](https://github.com/RustCrypto/utils) from 0.3.3 to 0.3.4.
- [Release notes](https://github.com/RustCrypto/utils/releases)
- [Commits](https://github.com/RustCrypto/utils/compare/hex-literal-v0.3.3...hex-literal-v0.3.4

)

---
updated-dependencies:
- dependency-name: hex-literal
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* pvf host: store only compiled artifacts on disk

* Correctly handle failed artifacts

* Serialize result of PVF preparation uniquely

* Set the artifact state depending on the result

* Return the result of PVF preparation directly

* Move PrepareError to the error module

* Update doc comments

* Update misleading comment

* pvf host: turn off parallel compilation

* pvf host: implement precheck requests

* Fix warnings

* Unnecessary clone

* Add a note about timed out outcome

* Revert the pool outcome handling behavior

* Move the prepare result type into error mod

* Test prepare done

* fmt

* Add an explanation to wasmtime config

* Split pvf host test

* Add precheck to dictionary

Co-authored-by: Sergei Shulepov <[email protected]>

* Limit the number of PVF workers

In particular, limit the number of preparation workers to 1 (soft &
hard) and limit the number of execution workers to 2.

The reason why we are doing this is that it seems many workers launched
at the same time can cause problems. I.e. if there are more than 2
preparation workers, the time for preparation rises significantly to the
point of reaching the timeout.

This was mostly observed with parallel_compilation=true, so each worker
used `numcpu` threads and now we are looking to flip that parameter to
`false`. That said, we want to err on the safe side here and gradually
enable it later if our measurements show that we can do that safely.

* Adjust the test to accomodate the changed config value

* CI: revert me [skip ci]

* CI: revert me too

* fmt with latest nightly

* CI: undebug

Co-authored-by: Andronik Ordian <[email protected]>

Because it's just too aggravating having to fetch the container every time to
find out the version of cargo +nightly fmt.

* req/resp: use IfDisconnected::ImmediateError

* remove outdated comments

* fmt

* Remove light client companion

* Update substrate

* cargo fmt

* Fixed benches

* fmt

* collator-protocol: do not connect to the next group

* fmt

* SubsystemContext: add subsystem name str

Signed-off-by: Andrei Sandu <[email protected]>

* Overseer builder proc macro changes

* initilize SubsystemContext name field.
* Add subsystem name in TaskKind::launch_task()

Signed-off-by: Andrei Sandu <[email protected]>

* Update ToOverseer enum

Signed-off-by: Andrei Sandu <[email protected]>

* Assign subsystem names to orphan tasks

Signed-off-by: Andrei Sandu <[email protected]>

* cargo fmt

Signed-off-by: Andrei Sandu <[email protected]>

* SubsystemContext: add subsystem name str

Signed-off-by: Andrei Sandu <[email protected]>

* Overseer builder proc macro changes

* initilize SubsystemContext name field.
* Add subsystem name in TaskKind::launch_task()

Signed-off-by: Andrei Sandu <[email protected]>

* Update ToOverseer enum

Signed-off-by: Andrei Sandu <[email protected]>

* Assign subsystem names to orphan tasks

Signed-off-by: Andrei Sandu <[email protected]>

* cargo fmt

Signed-off-by: Andrei Sandu <[email protected]>

* Rebase changes for new spawn() group param

Signed-off-by: Andrei Sandu <[email protected]>

* Add subsystem constat in JobTrait

Signed-off-by: Andrei Sandu <[email protected]>

* Add subsystem string

Signed-off-by: Andrei Sandu <[email protected]>

* Fix tests

Signed-off-by: Andrei Sandu <[email protected]>

* Fix spawn() calls

Signed-off-by: Andrei Sandu <[email protected]>

* cargo fmt

Signed-off-by: Andrei Sandu <[email protected]>

* Fix

Signed-off-by: Andrei Sandu <[email protected]>

* Fix tests

Signed-off-by: Andrei Sandu <[email protected]>

* fix

Signed-off-by: Andrei Sandu <[email protected]>

* Fix more tests

Signed-off-by: Andrei Sandu <[email protected]>

* Address PR review feedback #1

Signed-off-by: Andrei Sandu <[email protected]>

* Address PR review round 2

Signed-off-by: Andrei Sandu <[email protected]>

* Fixes
- remove JobTrait::Subsystem
- fix tests

Signed-off-by: Andrei Sandu <[email protected]>

* update Cargo.lock

Co-authored-by: Andronik Ordian <[email protected]>

* CI: chore

* CI: lsof and jq are installed

* CI: generate docs with deps

* CI: remove check width, fmt checks it

* CI: simplify rules

* CI: make CI image variable

* CI: more chore

* rm check_line_width.sh

* CI: fix spellcheck; fix check-transaction

* CI: return installing jq

* lint: remove dupes

* Update staking-miner

* fmt

* lockfile

* Point to substrate companion branch

* Revert "Point to substrate companion branch"

This reverts commit 3f8afc5d3137614776f54c7dac3c9077268c5fe5.

* Revert 3f8afc5d3137614776f54c7dac3c9077268c5fe5

* Remove `params` macro in favour of `rpc_macro` from jsonrpsee

* update Cargo.lock

* remove unused import

Co-authored-by: Andronik Ordian <[email protected]>