* cargo +nightly fmt

* add cargo-fmt check to ci

* update ci

* fmt

* fmt

* skip macro

* ignore bridges

* guide: filter_multi_dispute_data

* guide: elaborate

* Implementation of dispute data filtering

* tests for filtering

* don't use std, you fool!

* use swap_remove

* Update runtime/parachains/src/disputes.rs

Co-authored-by: Andronik Ordian <[email protected]>

* use btreeste

* address API nit

Co-authored-by: Andronik Ordian <[email protected]>

* disputes module skeleton and storage

* implement dispute module initialization logic

* implement disputes session change logic

* provide dispute skeletons

* deduplication & ancient check

* fix a couple of warnings

* begin provide_dispute_data impl

* flesh out statement set import somewhat

* move ApprovalVote to shared primitives

* add a signing-payload API to explicit dispute statements

* implement statement signature checking

* some bitflags glue for observing changes in disputes

* implement dispute vote import logic

* flesh out everything except slashing

* guide: tweaks

* declare and use punishment trait

* punish validators for inconclusive disputes

* guide: tiny fix

* guide: update docs

* add disputes getter fn

* guide: small change to spam slots handling

* improve spam slots handling and fix some bugs

* finish API of disputes runtime

* define and deposit `RevertTo` log

* begin integrating disputes into para_inherent

* use precomputed slash_for/against

* return candidate hash from process_bitfields

* implement inclusion::collect_disputed

* finish integration into rest of runtime

* add Disputes to initializer

* address suggestions

* use pallet macro

* fix typo

* Update runtime/parachains/src/disputes.rs

* add test: fix pruning

* document specific behavior

* deposit events on dispute changes

* add an allow(unused) on fn disputes

* add a dummy PunishValidators implementation

* add disputes module to Rococo

* add disputes module to westend runtime

* add disputes module to test runtime

* add disputes module to kusama runtime

* guide: prepare for runtime API for checking frozenness

* remove revert digests in favor of state variable

* merge reversions

* Update runtime/parachains/src/disputes.rs

Co-authored-by: André Silva <[email protected]>

* Update runtime/parachains/src/disputes.rs

Co-authored-by: André Silva <[email protected]>

* Update runtime/parachains/src/disputes.rs

Co-authored-by: André Silva <[email protected]>

* add byzantine_threshold and supermajority_threshold utilities to primitives

* use primitive helpers

* deposit revert event when freezing chain

* deposit revert log when freezing chain

* test revert event and log are generated when freezing

* add trait to decouple disputes handling from paras inherent handling

* runtime: fix compilation and setup dispute handler

* disputes: add hook for filtering out dispute statements

* disputes: add initializer hooks to DisputesHandler

* runtime: remove disputes pallet from all runtimes

* tag TODOs

* don't import any dispute statements just yet...

* address grumbles

* fix spellcheck, hopefully

* maybe now?

* last spellcheck round

* fix runtime tests

* fix test-runtime

Co-authored-by: thiolliere <[email protected]>
Co-authored-by: André Silva <[email protected]>
Co-authored-by: André Silva <[email protected]>

* CI: add spellcheck

* revert me

* CI: explicit command for spellchecker

* spellcheck: edit misspells

* CI: run spellcheck on diff

* spellcheck: edits

* spellcheck: edit misspells

* spellcheck: add rules

* spellcheck: mv configs

* spellcheck: more edits

* spellcheck: chore

* spellcheck: one more thing

* spellcheck: and another one

* spellcheck: seems like it doesn't get to an end

* spellcheck: new words after rebase

* spellcheck: new words appearing out of nowhere

* chore

* review edits

* more review edits

* more edits

* wonky behavior

* wonky behavior 2

* wonky behavior 3

* change git behavior

* spellcheck: another bunch of new edits

* spellcheck: new words are koming out of nowhere

* CI: finding the master

* CI: fetching master implicitly

* CI: undebug

* new errors

* a bunch of new edits

* and some more

* Update node/core/approval-voting/src/approval_db/v1/mod.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Update xcm/xcm-executor/src/assets.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Apply suggestions from code review

Co-authored-by: Andronik Ordian <[email protected]>

* Suggestions from the code review

* CI: scan only changed files

Co-authored-by: Andronik Ordian <[email protected]>

* Companion

* Update branch

* Update to correspond to substrate fixes

* Update cargo.lock

* Remove patches

* update Substrate

Co-authored-by: parity-processbot <>

* use `configuration::config()` for max bytes

* Update integration_tests.rs

* Update paras_registrar.rs

* remove consts

* add asserts for non-zero

* more const clean up

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_common::paras_registrar --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_common_paras_registrar.rs

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_common::paras_registrar --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_common_paras_registrar.rs

* add checks to `MAX_CODE_SIZE`

* re-pot MAX_POV_SIZE

* check pov limit in runtime

* POV_BOMB_LIMIT multiplier

* fix compile

* Update configuration.rs

* Update node/primitives/src/lib.rs

* fix test

Co-authored-by: Parity Bot <[email protected]>

* skeleton for dispute-coordinator

* add coordinator and participation message types

* begin dispute-coordinator DB

* functions for loading

* implement strongly-typed DB transaction

* add some tests for DB transaction

* core logic for pruning

* guide: update candidate-votes key for coordinator

* update candidate-votes key

* use big-endian encoding for session, and implement upper bound generator

* finish implementing pruning

* add a test for note_current_session

* define state of the subsystem itself

* barebones subsystem definition

* control flow

* more control flow

* implement session-updating logic

* trace

* control flow for message handling

* Update node/core/dispute-coordinator/src/lib.rs

Co-authored-by: André Silva <[email protected]>

* Update node/subsystem/src/messages.rs

Co-authored-by: André Silva <[email protected]>

* some more control flow

* guide: remove overlay

* more control flow

* implement some DB getters

* make progress on importing statements

* add SignedDisputeStatement struct

* move ApprovalVote to shared primitives

* add a signing-payload API to explicit dispute statements

* add signing-payload to CompactStatement

* add relay-parent hash to seconded/valid dispute variatns

* correct import

* type-safe wrapper around dispute statements

* use checked dispute statement in message type

* extract rolling session window cache to subsystem-util

* extract session window tests

* approval-voting: use rolling session info cache

* reduce dispute window to match runtime in practice

* add byzantine_threshold and supermajority_threshold utilities to primitives

* integrate rolling session window

* Add PartialOrd to CandidateHash

* add Ord to CandidateHash

* implement active dispute update

* add dispute messages to AllMessages

* add dispute stubs to overseer

* inform dispute participation to participate

* implement issue_local_statement

* implement `determine_undisputed_chain`

* fix warnings

* test harness for dispute coordinator tests

* add more helpers to test harness

* add some more helpers

* some tests for dispute coordinator

* ignore wrong validator indices

* test finality voting rule constraint

* add more tests

* add variants to network bridge

* fix test compilation

* remove most dispute coordinator functionality

as of #3222 we can do most of the work within the approval voting subsystem

* Revert "remove most dispute coordinator functionality"

This reverts commit 9cd615e8eb6ca0b382cbaff525d813e753d6004e.

* Use thiserror

Co-authored-by: Bernhard Schuster <[email protected]>

* Update node/core/dispute-coordinator/src/lib.rs

Co-authored-by: Bernhard Schuster <[email protected]>

* extract tests to separate module

* address nit

* adjust run_iteration API

Co-authored-by: André Silva <[email protected]>
Co-authored-by: Bernhard Schuster <[email protected]>

* create newtype for ValidationCodeHash

* pvf: fix tests compilation

* primitives: fix test compilation

* replace HistoricalValidationCode usages with ValidationCodeByHash

* runtime-api: tabify tests file

* update implementers guide

* Make ParaId value of public paras non-discretionary

* Fixes

* Fixes

* fix tests

* fix benchmark tests

* dont use hardcoded number

Co-authored-by: Shawn Tabrizi <[email protected]>

* Remove signature verification in backing.

`SignedFullStatement` now signals that the signature has already been
checked.

* Remove unused check_payload function.

* Introduced unchecked signed variants.

* Fix inclusion to use unchecked variant.

* More unchecked variants.

* Use unchecked variants in protocols.

* Start fixing statement-distribution.

* Fixup statement distribution.

* Fix inclusion.

* Fix warning.

* Fix backing properly.

* Fix bitfield distribution.

* Make crypto store optional for `RuntimeInfo`.

* Factor out utility functions.

* get_group_rotation_info

* WIP: Collator cleanup + check signatures.

* Convenience signature checking functions.

* Check signature on collator-side.

* Fix warnings.

* Fix collator side tests.

* Get rid of warnings.

* Better Signed/UncheckedSigned implementation.

Also get rid of Encode/Decode for Signed! *party*

* Get rid of dead code.

* Move Signed in its own module.

* into_checked -> try_into_checked

* Fix merge.

* Wip

* Increase proposer timeout.

* WIP.

* Better timeout values now that we are going to be connected to all nodes. (#2778)

* Better timeout values.

* Fix typo.

* Fix validator bandwidth.

* Fix compilation.

* Better and more consistent sizes.

Most importantly code size is now 5 Meg, which is the limit we currently
want to support in statement distribution.

* Introduce statement fetching request.

* WIP

* Statement cache retrieval logic.

* Review remarks by @rphmeier

* Fixes.

* Better requester logic.

* WIP: Handle requester messages.

* Missing dep.

* Fix request launching logic.

* Finish fetching logic.

* Sending logic.

* Redo code size calculations.

Now that max code size is compressed size.

* Update Cargo.lock (new dep)

* Get request receiver to statement distribution.

* Expose new functionality for responding to requests.

* Cleanup.

* Responder logic.

* Fixes + Cleanup.

* Cargo.lock

* Whitespace.

* Add lost copyright.

* Launch responder task.

* Typo.

* info -> warn

* Typo.

* Fix.

* Fix.

* Update comment.

* Doc fix.

* Better large statement heuristics.

* Fix tests.

* Fix network bridge tests.

* Add test for size estimate.

* Very simple tests that checks we get LargeStatement.

* Basic check, that fetching of large candidates is performed.

* More tests.

* Basic metrics for responder.

* More metrics.

* Use Encode::encoded_size().

* Some useful spans.

* Get rid of redundant metrics.

* Don't add peer on duplicate.

* Properly check hash

instead of relying on signatures alone.

* Preserve ordering + better flood protection.

* Get rid of redundant clone.

* Don't shutdown responder on failed query.

And add test for this.

* Smaller fixes.

* Quotes.

* Better queue size calculation.

* A bit saner response sizes.

* Fixes.

* guide: declare one para as a collator

* add ParaId to Declare messages and clean up

* fix build

* fix the testerinos

* begin adding keystore to collator-protocol

* remove request_x_ctx

* add core_for_group

* add bump_rotation

* add some more helpers to subsystem-util

* change signing_key API to take ref

* determine current and next para assignments

* disconnect collators who are not on current or next para

* add collator peer count metric

* notes for later

* some fixes

* add data & keystore to test state

* add a test utility for answering runtime API requests

* fix existing collator tests

* add new tests

* remove sc_keystore

* update cargo lock

Co-authored-by: Andronik Ordian <[email protected]>

* code stored in para + modify CandidateDescriptor.

* WIP: digest + some more impl

* validation_code_hash in payload + check in inclusion

* check in client + refator

* tests

* fix encoding indices

* remove old todos

* fix test

* fix test

* add test

* fetch validation code inside collation-generation from the relay-chain

* HashMismatch -> PoVHashMismatch + miscompilation

* refactor, store hash when needed

* storage rename: more specific but slightly too verbose

* do not hash on candidate validation, fetch hash instead

* better test

* fix test

* guide updates

* don't panic in runtime

Co-authored-by: Robert Habermeier <[email protected]>

* dispute types

* add Debug to dispute primitives in std and InherentData

* use ParachainsInherentData on node-side

* change inclusion_inherent to paras_inherent

* RuntimeDebug

* add type parameter to PersistedValidationData users

* fix test client

* spaces

* fix collation-generation test

* fix provisioner tests

* remove references to inclusion inherent

* Remove stuff out of the runtime that does not belong there.

There might be more, but it is a start.

* White space fixes.

* Fix tests.

* Leave whitespace in ui tests alone.

* Add back zstd for no reason.

* Fix browser wasm (hopefully)

* add consensus log type

* origin and issue force_approve

* add origin in runtimes

* ref API

* scrape force_approve digest from header

* add parent_hash to BlockEntry

* add block_number to block entry and force_approve skeleton

* implement and plug in force-approve

* test force_approve

* test force_approve extraction

* westend runtime

* Update node/core/approval-voting/src/approval_db/v1/mod.rs

Co-authored-by: Bastian Köcher <[email protected]>

* rename

* Update runtime/parachains/src/initializer.rs

Co-authored-by: André Silva <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: André Silva <[email protected]>

* Unify maximum supported PoV size a bit.

* Use MAX_POV_SIZE also in `HostConfiguration`.

* Fix types.

* Additional logging for polkadot network protocols

* Additional log

* Update node/network/bitfield-distribution/src/lib.rs

Co-authored-by: Robert Habermeier <[email protected]>

* Update node/network/availability-distribution/src/responder.rs

* Added additional chunk info

* Added additional peer info

Co-authored-by: Robert Habermeier <[email protected]>

* Introduce collation fetching protocol

also move to mod.rs

* Allow `PeerId`s in requests to network bridge.

* Fix availability distribution tests.

* Move CompressedPoV to primitives.

* Request based collator protocol: validator side

- Missing: tests
- Collator side
- don't connect, if not connected

* Fixes.

* Basic request based collator side.

* Minor fix on collator side.

* Don't connect in requests in collation protocol.

Also some cleanup.

* Fix PoV distribution

* Bump substrate

* Add back metrics + whitespace fixes.

* Add back missing spans.

* More cleanup.

* Guide update.

* Fix tests

* Handle results in tests.

* Fix weird compilation issue.

* Add missing )

* Get rid of dead code.

* Get rid of redundant import.

* Fix runtime build.

* Cleanup.

* Fix wasm build.

* Format fixes.

Thanks @andronik !

* add a magic number to backing statements encoded

* fix fallout in statement table

* fix some fallout in backing

* add magic to approval votes

* remove last references to Candidate variant

* update size-hint

* WIP

* availability distribution, still very wip.

Work on the requesting side of things.

* Some docs on what I intend to do.

* Checkpoint of session cache implementation

as I will likely replace it with something smarter.

* More work, mostly on cache

and getting things to type check.

* Only derive MallocSizeOf and Debug for std.

* availability-distribution: Cache feature complete.

* Sketch out logic in `FetchTask` for actual fetching.

- Compile fixes.
- Cleanup.

* Format cleanup.

* More format fixes.

* Almost feature complete `fetch_task`.

Missing:

- Check for cancel
- Actual querying of peer ids.

* Finish FetchTask so far.

* Directly use AuthorityDiscoveryId in protocol and cache.

* Resolve `AuthorityDiscoveryId` on sending requests.

* Rework fetch_task

- also make it impossible to check the wrong chunk index.
- Export needed function in validator_discovery.

* From<u32> implementation for `ValidatorIndex`.

* Fixes and more integration work.

* Make session cache proper lru cache.

* Use proper lru cache.

* Requester finished.

* ProtocolState -> Requester

Also make sure to not fetch our own chunk.

* Cleanup + fixes.

* Remove unused functions

- FetchTask::is_finished
- SessionCache::fetch_session_info

* availability-distribution responding side.

* Cleanup + Fixes.

* More fixes.

* More fixes.

adder-collator is running!

* Some docs.

* Docs.

* Fix reporting of bad guys.

* Fix tests

* Make all tests compile.

* Fix test.

* Cleanup + get rid of some warnings.

* state -> requester

* Mostly doc fixes.

* Fix test suite.

* Get rid of now redundant message types.

* WIP

* Rob's review remarks.

* Fix test suite.

* core.relay_parent -> leaf for session request.

* Style fix.

* Decrease request timeout.

* Cleanup obsolete errors.

* Metrics + don't fail on non fatal errors.

* requester.rs -> requester/mod.rs

* Panic on invalid BadValidator report.

* Fix indentation.

* Use typed default timeout constant.

* Make channel size 0, as each sender gets one slot anyways.

* Fix incorrect metrics initialization.

* Fix build after merge.

* More fixes.

* Hopefully valid metrics names.

* Better metrics names.

* Some tests that already work.

* Slightly better docs.

* Some more tests.

* Fix network bridge test.

* Substrate companion for #8163

https://github.com/paritytech/substrate/pull/8163

* "Update Substrate"

Co-authored-by: parity-processbot <>

* add full data messages

* handle new messages

* flip condition

* skeleton

* skeleton aux-schema module

* start approval types

* start aux schema with aux store

* doc

* finish basic types

* start approval types

* doc

* finish basic types

* write out schema types

* add debug and codec impls to approval types

* add debug and codec impls to approval types

also add some key computation

* add debug and codec impls to approval types

* getters for block and candidate entries

* grumbles

* remove unused AssignmentId

* load_decode utility

* implement DB clearing

* function for adding new block entry to aux store

* start `canonicalize` implementation

* more skeleton

* finish implementing canonicalize

* tag TODO

* implement a test AuxStore

* add allow(unused)

* basic loading and deleting test

* block_entry test function

* add a test for `add_block_entry`

* ensure range is exclusive at end

* test clear()

* test that add_block sets children

* add a test for canonicalize

* extract Pre-digest from header

* utilities for extracting RelayVRFStory from the header-chain

* add approval voting message types

* approval distribution message type

* subsystem skeleton

* state struct

* add futures-timer

* prepare service for babe slot duration

* more skeleton

* better integrate AuxStore

* RelayVRF -> RelayVRFStory

* canonicalize

* implement some tick functionality

* guide: tweaks

* check_approval

* more tweaks and helpers

* guide: add core index to candidate event

* primitives: add core index to candidate event

* runtime: add core index to candidate events

* head handling (session window)

* implement `determine_new_blocks`

* add TODO

* change error type on functions

* compute RelayVRFModulo assignments

* compute RelayVRFDelay assignments

* fix delay tranche calc

* assignment checking

* pluralize

* some dummy code for fetching assignments

* guide: add babe epoch runtime API

* implement a current_epoch() runtime API

* compute assignments

* candidate events get backing group

* import blocks and assignments into DB

* push block approval meta

* add message types, no overseer integration yet

* notify approval distribution of new blocks

* refactor import into separate functions

* impl tranches_to_approve

* guide: improve function signatures

* guide: remove Tick from ApprovalEntry

* trigger and broadcast assignment

* most of approval launching

* remove byteorder crate

* load blocks back to finality, except on startup

* check unchecked assignments

* add claimed core to approval voting message

* fix checks

* assign only to backing group

* remove import_checked_assignment from guide

* newline

* import assignments

* abstract out a bit

* check and import approvals

* check full approvals from assignment import too

* comment

* create a Transaction utility

* must_use

* use transaction in `check_full_approvals`

* wire up wakeups

* add Ord to CandidateHash

* wakeup refactoring

* return candidate info from add_block_entry

* schedule wakeups

* background task: do candidate validation

* forward candidate validation requests

* issue approval votes when requested

* clean up a couple TODOs

* fix up session caching

* clean up last unimplemented!() items

* fix remaining warnings

* remove TODO

* implement handle_approved_ancestor

* update Cargo.lock

* fix runtime API tests

* guide: cleanup assignment checking

* use claimed candidate index instead of core

* extract time to a trait

* tests module

* write a mock clock for testing

* allow swapping out the clock

* make abstract over assignment criteria

* add some skeleton tests and simplify params

* fix backing group check

* do backing group check inside check_assignment_cert

* write some empty test functions to implement

* add a test for non-backing

* test that produced checks pass

* some empty test ideas

* runtime/inclusion: remove outdated TODO

* fix compilation

* av-store: fix tests

* dummy cert

* criteria tests

* move `TestStore` to main tests file

* fix unused warning

* test harness beginnings

* resolve slots renaming fallout

* more compilation fixes

* wip: extract pure data into a separate module

* wip: extract pure data into a separate module

* move types completely to v1

* add persisted_entries

* add conversion trait impls

* clean up some warnings

* extract import logic to own module

* schedule wakeups

* experiment with Actions

* uncomment approval-checking

* separate module for approval checking utilities

* port more code to use actions

* get approval pipeline using actions

* all logic is uncommented

* main loop processes actions

* all loop logic uncommented

* separate function for handling actions

* remove last unimplemented item

* clean up warnings

* State gives read-only access to underlying DB

* tests for approval checking

* tests for approval criteria

* skeleton test module for import

* list of import tests to do

* some test glue code

* test reject bad assignment

* test slot too far in future

* test reject assignment with unknown candidate

* remove loads_blocks tests

* determine_new_blocks back to finalized & harness

* more coverage for determining new blocks

* make `imported_block_info` have less reliance on State

* candidate_info tests

* tests for session caching

* remove println

* extricate DB and main TestStores

* rewrite approval checking logic to counteract early delays

* move state out of function

* update approval-checking tests

* tweak wakeups & scheduling logic

* rename check_full_approvals

* test that assignment import updates candidate

* some approval import tests

* some tests for check_and_apply_approval

* add 'full' qualifier to avoid confusion

* extract should-trigger logic to separate function

* some tests for all triggering

* tests for when we trigger assignments

* test wakeups

* add block utilities for testing

* some more tests for approval updates

* approved_ancestor tests

* new action type for launch approval

* process-wakeup tests

* clean up some warnings

* fix in_future test

* approval checking tests

* tighten up too-far-in-future

* special-case genesis when caching sessions

* fix bitfield len

Co-authored-by: Andronik Ordian <[email protected]>

* PVD: `block_number`->`relay_parent_number`

* ValidationParams: `relay_chain_height`->`relay_parent_number`

* Expose DMQ MQC hash as a well-known-key

This way the relay storage merkle proofs will be able to obtain the DMQ
MQC hash and we will be able to remove the it from the
PersistedValidationData struct.

* PersistedValidationData: Remove HRMP MQC heads

* PersistedValidationData: Remove `dmq_mqc_head`

* Expose the HRMP ingress channel index as a well-known-key

This way a parachain (PVF and collator) can find all the parachains that
have an outbound channel to the given one. That allows in turn to find
all the inbound channels for the given para.

Having access to that allows the parachain to get the same information
as the hrmp_mqc_heads now provide.

* Rename `relay_storage_root` to `relay_parent_storage_root`

* upgrade codec and bitvec

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Add memory-lru cache to runtime-api

* Add cache.rs

* Adds MallocSizeOf

* Review nits

* Add a cached requests metric

* More review nits

* Some more review nits

* initial impl approval distribution

* initial tests and fixes

* batching seems difficult: different peers have different needs

* bridge: fix test after merge

* some guide updates

* only send assignments to peers who know about the block

* fix a test, add approvals test

* simplify

* do not send assignment to peers for finalized blocks

* guide: protocol input and output

* one more test

* more comments, logs, initial metrics

* fix a typo

* one more thing: early return when reimporting a thing locally

* Add a `convert_payload` function to `Signed`

* Review feedback

* Put parachain head hash into `CandidateDescriptor`

* Update guide

* Add some checks

* collation-generation: use persisted validation data

* node: remote FullValidationData API

* runtime: remove FullValidationData API

* backing tests: use persisted validation data

* FullCandidateReceipt: use persisted validation data

This is not a big change since this type is not used anywhere

* Remove ValidationData and TransientValidationData

Also update the guide

* scheduler: handle re-scheduling around finalization correctly

* also make sure parathreads get cleaned

* run scheduling in finalization

* Remove stray println!

* Update the schedule call site in inclusion inherent

* Clarify subtlety around SessionStartBlock

* Remove double semi-colon

* reschedule prior to `availability_cores` and in on-initialize

* improve docs

* fix line

* more doc reformat

* remove unneeded call

* avoid unnecessary scheduling on initialize

* split `clear` and `schedule

* Update runtime/parachains/src/scheduler.rs

Co-authored-by: Sergei Shulepov <[email protected]>

Co-authored-by: Sergei Shulepov <[email protected]>

* don't modify inherent data on heavy block

* write up current thinking on block weight detection

* extract inherent inclusion check into its own function

* put heavy block check into runtime

* the `inclusion` inherent call is Operational, not Mandatory

This resolves a lot of the trickiness about this issue, because
we no longer need to override or supplant any existing proposer
logic; the existing logic should exhibit these behaviors:

- the `inclusion` inherent is prioritized over standard transactions
- but if it's too heavy, i.e. in case of runtime upgrade, it'll be
  dropped in favor of that.

It is my belief that allowing the proposer to just not include
this data won't have any adverse effects: it's equivalent to replacing
them with empty versions of themselves, which the `ProvideInherent`
impl already does.

* Revert "the `inclusion` inherent call is Operational, not Mandatory"

This reverts commit e58858d109b18b84e7af3ac47981c6900b2d9a3e.

* Revert "write up current thinking on block weight detection"

This reverts commit fd587b80c46761b2a2b62448193348237863f99f.

* Revert "don't modify inherent data on heavy block"

This reverts commit 38299d3c23e9efb5a354d8cfa658e62a5c8c7ddf.

* add backed candidate block weight assumption to configuration

* Limit backed candidates according to a candidate weight heuristic.

This approach replaces making the inclusion inherent non-mandatory.
It's still not ideal in that we have to configure a heuristic for
how much each backed candidate 'weighs', instead of directly
measuring it somehow.

This approach also never truncates the signed bitfields. The
rationale for that depends on some assumptions:

- processing the signed bitfields is cheap compared to the
  backed candidates
- it is beneficial to the progress of the relay chain
  to update the signed bitfields even if not all backed candidates
  are updated

* simplify limit_backed_candidates and weight assumption

* don't trust the provisioner to fairly distribute candidates

* use saturating subtraction

* empty commit to restart ci

* use new mechanism for getting max block weight

* apply weight refunds to the inclusion inherent

This makes some assumptions about fundamental weights, which are
encapsulated as constants. From there, it lets Substrate know
what the actual computed weight of the inherent is.

* use a correct fixed weight for the inclusion inherent

Co-authored-by: Guillaume Thiolliere <[email protected]>

* use dynamic inclusion weight so we reduce calculated weight when excluding candidates

* don't double-count this intrinsic's weight in the block weight

* add unit tests of fn limit_backed_candidates

* add tests that the inclusion inherent's weight correctly updates

Co-authored-by: Guillaume Thiolliere <[email protected]>

* Add well_known_keys

* Reorder HrmpChannel and HostConfiguration members

* abridged versions and well known keys tests

* Add some comments

* Add a note on generation of the prefixes

and other magic values

* Recommend accessing the well known values through abridged structs