* Lol

* Add the features

* Remove some more runtime logger init calls

* Make companion check work

* Revert "Make companion check work"

This reverts commit a255c798076466c0fa20a4db713fc712772c2b4d.

* Update Substrate

* Add Xcm sender.

* Rename XCM sender and add description.

* Update copyright header.

Co-authored-by: Shawn Tabrizi <[email protected]>

* Add events for hrmp pallet.

* Apply review suggestions.

* WIP

* availability distribution, still very wip.

Work on the requesting side of things.

* Some docs on what I intend to do.

* Checkpoint of session cache implementation

as I will likely replace it with something smarter.

* More work, mostly on cache

and getting things to type check.

* Only derive MallocSizeOf and Debug for std.

* availability-distribution: Cache feature complete.

* Sketch out logic in `FetchTask` for actual fetching.

- Compile fixes.
- Cleanup.

* Format cleanup.

* More format fixes.

* Almost feature complete `fetch_task`.

Missing:

- Check for cancel
- Actual querying of peer ids.

* Finish FetchTask so far.

* Directly use AuthorityDiscoveryId in protocol and cache.

* Resolve `AuthorityDiscoveryId` on sending requests.

* Rework fetch_task

- also make it impossible to check the wrong chunk index.
- Export needed function in validator_discovery.

* From<u32> implementation for `ValidatorIndex`.

* Fixes and more integration work.

* Make session cache proper lru cache.

* Use proper lru cache.

* Requester finished.

* ProtocolState -> Requester

Also make sure to not fetch our own chunk.

* Cleanup + fixes.

* Remove unused functions

- FetchTask::is_finished
- SessionCache::fetch_session_info

* availability-distribution responding side.

* Cleanup + Fixes.

* More fixes.

* More fixes.

adder-collator is running!

* Some docs.

* Docs.

* Fix reporting of bad guys.

* Fix tests

* Make all tests compile.

* Fix test.

* Cleanup + get rid of some warnings.

* state -> requester

* Mostly doc fixes.

* Fix test suite.

* Get rid of now redundant message types.

* WIP

* Rob's review remarks.

* Fix test suite.

* core.relay_parent -> leaf for session request.

* Style fix.

* Decrease request timeout.

* Cleanup obsolete errors.

* Metrics + don't fail on non fatal errors.

* requester.rs -> requester/mod.rs

* Panic on invalid BadValidator report.

* Fix indentation.

* Use typed default timeout constant.

* Make channel size 0, as each sender gets one slot anyways.

* Fix incorrect metrics initialization.

* Fix build after merge.

* More fixes.

* Hopefully valid metrics names.

* Better metrics names.

* Some tests that already work.

* Slightly better docs.

* Some more tests.

* Fix network bridge test.

* add tracing to approval voting

* notify if session info is not working

* add dispute period to chain specs

* propagate genesis session to parachains runtime

* use `on_genesis_session`

* protect against zero cores in computation

* tweak voting rule to be based off of best and add logs

* genesis configuration should use VRF slots only

* swallow more keystore errors

* add some docs

* make validation-worker args non-optional and update clap

* better tracing for bitfield signing and provisioner

* pass amount of bits in bitfields to inclusion instead of recomputing

* debug -> warn for some logs

* better tracing for availability recovery

* a little av-store tracing

* bridge: forward availability recovery messages

* add missing try_from impl

* some more tracing

* improve approval distribution tracing

* guide: hold onto pending approval messages until NewBlocks

* Hold onto pending approval messages until NewBlocks

* guide: adjust comment

* process all actions for one wakeup at a time

* vec

* fix network bridge test

* replace randomness-collective-flip with Babe

* remove PairNotFound

* override authorities runtime call

* addressing feedback

* addressing feedback and restoring semantics

* updated helper function

* naming convention + comment

Co-authored-by: Parity <[email protected]>

* Substrate companion for #8163

https://github.com/paritytech/substrate/pull/8163

* "Update Substrate"

Co-authored-by: parity-processbot <>

* initial implementation of lifecycles and upgrades

* clean up a bit

* fix doc comment

* more rigid lifecycle checks

* include paras which are transitioning, and lifecycle query

* format guide

* update api

* update guide

* explicit outgoing state, fix genesis

* handle outgoing with transitioning paras

* do not include transitioning paras in identifier

* Update roadmap/implementers-guide/src/runtime/paras.md

* Update roadmap/implementers-guide/src/runtime/paras.md

* Update roadmap/implementers-guide/src/runtime/paras.md

* Apply suggestions from code review

* Use matches macro

* Correct terms

* Apply suggestions from code review

* actions queue

* Revert "actions queue"

This reverts commit b2e9011ec8937d6c73e99292416c9692aeb30f73.

* collapse onboarding state

* starting actions queue

* consolidate actions queue

* schedule para initialize result

* more actions queue for upgrade/downgrade

* clean up with fully implemented actions queue

* fix tests

* fix scheduler tests

* fix hrmp tests

* fix test

* doc fixes

* fix hrmp test w/ valid para

* Update paras.md

* fix paras registrar

* Update propose_parachain.rs

* fix merge

* Introduce "shared" module

* fix rococo build

* fix up and use shared

* guide updates

* add shared config to common tests

* add shared to test-runtime

* remove println

* fix note

Co-authored-by: Gavin Wood <[email protected]>

* guide: max one candidate with upgrade per block

* max one candidate with upgrade per block

* ignore on runtime level too

* initial hack in

* finish up

* use notification to pass outgoing paras

* move outgoing paras from notifications

* missing comma

* update guides

* clean up

* skeleton

* skeleton aux-schema module

* start approval types

* start aux schema with aux store

* doc

* finish basic types

* start approval types

* doc

* finish basic types

* write out schema types

* add debug and codec impls to approval types

* add debug and codec impls to approval types

also add some key computation

* add debug and codec impls to approval types

* getters for block and candidate entries

* grumbles

* remove unused AssignmentId

* load_decode utility

* implement DB clearing

* function for adding new block entry to aux store

* start `canonicalize` implementation

* more skeleton

* finish implementing canonicalize

* tag TODO

* implement a test AuxStore

* add allow(unused)

* basic loading and deleting test

* block_entry test function

* add a test for `add_block_entry`

* ensure range is exclusive at end

* test clear()

* test that add_block sets children

* add a test for canonicalize

* extract Pre-digest from header

* utilities for extracting RelayVRFStory from the header-chain

* add approval voting message types

* approval distribution message type

* subsystem skeleton

* state struct

* add futures-timer

* prepare service for babe slot duration

* more skeleton

* better integrate AuxStore

* RelayVRF -> RelayVRFStory

* canonicalize

* implement some tick functionality

* guide: tweaks

* check_approval

* more tweaks and helpers

* guide: add core index to candidate event

* primitives: add core index to candidate event

* runtime: add core index to candidate events

* head handling (session window)

* implement `determine_new_blocks`

* add TODO

* change error type on functions

* compute RelayVRFModulo assignments

* compute RelayVRFDelay assignments

* fix delay tranche calc

* assignment checking

* pluralize

* some dummy code for fetching assignments

* guide: add babe epoch runtime API

* implement a current_epoch() runtime API

* compute assignments

* candidate events get backing group

* import blocks and assignments into DB

* push block approval meta

* add message types, no overseer integration yet

* notify approval distribution of new blocks

* refactor import into separate functions

* impl tranches_to_approve

* guide: improve function signatures

* guide: remove Tick from ApprovalEntry

* trigger and broadcast assignment

* most of approval launching

* remove byteorder crate

* load blocks back to finality, except on startup

* check unchecked assignments

* add claimed core to approval voting message

* fix checks

* assign only to backing group

* remove import_checked_assignment from guide

* newline

* import assignments

* abstract out a bit

* check and import approvals

* check full approvals from assignment import too

* comment

* create a Transaction utility

* must_use

* use transaction in `check_full_approvals`

* wire up wakeups

* add Ord to CandidateHash

* wakeup refactoring

* return candidate info from add_block_entry

* schedule wakeups

* background task: do candidate validation

* forward candidate validation requests

* issue approval votes when requested

* clean up a couple TODOs

* fix up session caching

* clean up last unimplemented!() items

* fix remaining warnings

* remove TODO

* implement handle_approved_ancestor

* update Cargo.lock

* fix runtime API tests

* guide: cleanup assignment checking

* use claimed candidate index instead of core

* extract time to a trait

* tests module

* write a mock clock for testing

* allow swapping out the clock

* make abstract over assignment criteria

* add some skeleton tests and simplify params

* fix backing group check

* do backing group check inside check_assignment_cert

* write some empty test functions to implement

* add a test for non-backing

* test that produced checks pass

* some empty test ideas

* runtime/inclusion: remove outdated TODO

* fix compilation

* av-store: fix tests

* dummy cert

* criteria tests

* move `TestStore` to main tests file

* fix unused warning

* test harness beginnings

* resolve slots renaming fallout

* more compilation fixes

* wip: extract pure data into a separate module

* wip: extract pure data into a separate module

* move types completely to v1

* add persisted_entries

* add conversion trait impls

* clean up some warnings

* extract import logic to own module

* schedule wakeups

* experiment with Actions

* uncomment approval-checking

* separate module for approval checking utilities

* port more code to use actions

* get approval pipeline using actions

* all logic is uncommented

* main loop processes actions

* all loop logic uncommented

* separate function for handling actions

* remove last unimplemented item

* clean up warnings

* State gives read-only access to underlying DB

* tests for approval checking

* tests for approval criteria

* skeleton test module for import

* list of import tests to do

* some test glue code

* test reject bad assignment

* test slot too far in future

* test reject assignment with unknown candidate

* remove loads_blocks tests

* determine_new_blocks back to finalized & harness

* more coverage for determining new blocks

* make `imported_block_info` have less reliance on State

* candidate_info tests

* tests for session caching

* remove println

* extricate DB and main TestStores

* rewrite approval checking logic to counteract early delays

* move state out of function

* update approval-checking tests

* tweak wakeups & scheduling logic

* rename check_full_approvals

* test that assignment import updates candidate

* some approval import tests

* some tests for check_and_apply_approval

* add 'full' qualifier to avoid confusion

* extract should-trigger logic to separate function

* some tests for all triggering

* tests for when we trigger assignments

* test wakeups

* add block utilities for testing

* some more tests for approval updates

* approved_ancestor tests

* new action type for launch approval

* process-wakeup tests

* clean up some warnings

* fix in_future test

* approval checking tests

* tighten up too-far-in-future

* special-case genesis when caching sessions

* fix bitfield len

Co-authored-by: Andronik Ordian <[email protected]>

* Claims

* Crowdloan

* Runtime common

* Parachains registrar

* Impls

* Purchase

* Slots

* parachains runtime mock

* Use MockGenesisConfig in tests

* Update runtime/common/src/claims.rs

Co-authored-by: Guillaume Thiolliere <[email protected]>

Co-authored-by: Guillaume Thiolliere <[email protected]>

* initial implementation of lifecycles and upgrades

* clean up a bit

* fix doc comment

* more rigid lifecycle checks

* include paras which are transitioning, and lifecycle query

* format guide

* update api

* update guide

* explicit outgoing state, fix genesis

* handle outgoing with transitioning paras

* do not include transitioning paras in identifier

* Update roadmap/implementers-guide/src/runtime/paras.md

* Update roadmap/implementers-guide/src/runtime/paras.md

* Update roadmap/implementers-guide/src/runtime/paras.md

* Apply suggestions from code review

* Use matches macro

* Correct terms

* Apply suggestions from code review

* actions queue

* Revert "actions queue"

This reverts commit b2e9011ec8937d6c73e99292416c9692aeb30f73.

* collapse onboarding state

Co-authored-by: Gavin Wood <[email protected]>

* Conpanion for Substrate#7127

* Use sp_session::OneSessionHandler

* .

* Fix pallet_session::OneSessionHandler

* OneSessionHandler is in frame_support now

* "Update Substrate"

Co-authored-by: parity-processbot <>

* PVD: `block_number`->`relay_parent_number`

* ValidationParams: `relay_chain_height`->`relay_parent_number`

* Expose DMQ MQC hash as a well-known-key

This way the relay storage merkle proofs will be able to obtain the DMQ
MQC hash and we will be able to remove the it from the
PersistedValidationData struct.

* PersistedValidationData: Remove HRMP MQC heads

* PersistedValidationData: Remove `dmq_mqc_head`

* Expose the HRMP ingress channel index as a well-known-key

This way a parachain (PVF and collator) can find all the parachains that
have an outbound channel to the given one. That allows in turn to find
all the inbound channels for the given para.

Having access to that allows the parachain to get the same information
as the hrmp_mqc_heads now provide.

* Rename `relay_storage_root` to `relay_parent_storage_root`

* DMP

* HRMP

* Update runtime/parachains/src/dmp.rs

Co-authored-by: Bastian Köcher <[email protected]>

* collation-generation: use persisted validation data

* node: remote FullValidationData API

* runtime: remove FullValidationData API

* backing tests: use persisted validation data

* FullCandidateReceipt: use persisted validation data

This is not a big change since this type is not used anywhere

* Remove ValidationData and TransientValidationData

Also update the guide

* Add warning when dropping inclusion inherent

* improve warning when dropping inclusion inherent

- use the actual runtime `warn!` macro
- do not speculate about why the error occurred
- show the actual error returned

* BROKEN: add note if the current block matches the session start block

This doesn't compile for a variety of reasons; this commit demonstrates
what the issues are. I don't think this is an essential feature, so I'm
stopping work on it here instead of pushing through it.

* Revert "BROKEN: add note if the current block matches the session start block"

This reverts commit eeb79ab7708ec94224fa60c9e1a46e12c18d52fd.

* ensure a runtime logger exists before attempting to log a warning to it

* session_info: fix authorities docstring

* overseer: more consistent metrics naming

* session_info: mention ordering

* use correct bucket sizes

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* scheduler: handle re-scheduling around finalization correctly

* also make sure parathreads get cleaned

* run scheduling in finalization

* Remove stray println!

* Update the schedule call site in inclusion inherent

* Clarify subtlety around SessionStartBlock

* Remove double semi-colon

* reschedule prior to `availability_cores` and in on-initialize

* improve docs

* fix line

* more doc reformat

* remove unneeded call

* avoid unnecessary scheduling on initialize

* split `clear` and `schedule

* Update runtime/parachains/src/scheduler.rs

Co-authored-by: Sergei Shulepov <[email protected]>

Co-authored-by: Sergei Shulepov <[email protected]>

* session_info: use correct authorities set

* bump rococo spec_version to 19

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Drive by fixes

The visibility modifiers are remnants of the previous structure where
HRMP wasn't a standalone module, by rather a submodule of the router
module.

* Add Currency assoc type to Config

This would allow us to reserve balance for deposits. This commit also
integrates the HRMP module in rococo, test-runtime and mocks to use the
balances pallet.

* Fix a bug that doesn't increment the age

In case the request is not confirmed, the age would be incremented but
not persisted.

* Fix cleaning the indexes

Before that change, the cleaning of the channel indexes was wrong, because it
naively removed entire rows that was pertaining to the para we delete.
This approach is flawed because it doesn't account for the rows that are
pertaining to other paras that contain the outgoing one.

This clearly violates the invariant imposed on the indexes, that all
the index rows must contain alive paras, but apart from that it also
lead to the situation where ingress index would contain the a different
set of channels that an egress have.

* Reserve currency for opening the channels

Note the ugly `unique_saturated_into` calls. The reason for them is the
currency trait accepts and defines the `Balance` associated type and the
deposit values are coming from the `HostConfiguration` where they are
defined using the `Balance`.

I figured that parameterising `HostConfiguration` would be annoying. On
the other hand, I don't expect these `unique_saturated_into` calls to
give us problems since it seems to be a reasonable assumption that this
module will be instantiated within a runtime where the Currency provided
will have a Balance that matches the one used in the configuration.

* Tests: Adapt `run_to_block` so that it submits a proper config

* Tests: exercise the deposit logic

Closes #2241

* Adds propose parachain pallet

* Update runtime/rococo/src/propose_parachain.rs

Co-authored-by: Shawn Tabrizi <[email protected]>

* Fix runtime benchmarks

* Get rid of staking

* Fix benchmarking feature..

* Remove accidentally added crate

* Bump Rococo spec_version

Co-authored-by: Shawn Tabrizi <[email protected]>

* don't modify inherent data on heavy block

* write up current thinking on block weight detection

* extract inherent inclusion check into its own function

* put heavy block check into runtime

* the `inclusion` inherent call is Operational, not Mandatory

This resolves a lot of the trickiness about this issue, because
we no longer need to override or supplant any existing proposer
logic; the existing logic should exhibit these behaviors:

- the `inclusion` inherent is prioritized over standard transactions
- but if it's too heavy, i.e. in case of runtime upgrade, it'll be
  dropped in favor of that.

It is my belief that allowing the proposer to just not include
this data won't have any adverse effects: it's equivalent to replacing
them with empty versions of themselves, which the `ProvideInherent`
impl already does.

* Revert "the `inclusion` inherent call is Operational, not Mandatory"

This reverts commit e58858d109b18b84e7af3ac47981c6900b2d9a3e.

* Revert "write up current thinking on block weight detection"

This reverts commit fd587b80c46761b2a2b62448193348237863f99f.

* Revert "don't modify inherent data on heavy block"

This reverts commit 38299d3c23e9efb5a354d8cfa658e62a5c8c7ddf.

* add backed candidate block weight assumption to configuration

* Limit backed candidates according to a candidate weight heuristic.

This approach replaces making the inclusion inherent non-mandatory.
It's still not ideal in that we have to configure a heuristic for
how much each backed candidate 'weighs', instead of directly
measuring it somehow.

This approach also never truncates the signed bitfields. The
rationale for that depends on some assumptions:

- processing the signed bitfields is cheap compared to the
  backed candidates
- it is beneficial to the progress of the relay chain
  to update the signed bitfields even if not all backed candidates
  are updated

* simplify limit_backed_candidates and weight assumption

* don't trust the provisioner to fairly distribute candidates

* use saturating subtraction

* empty commit to restart ci

* use new mechanism for getting max block weight

* apply weight refunds to the inclusion inherent

This makes some assumptions about fundamental weights, which are
encapsulated as constants. From there, it lets Substrate know
what the actual computed weight of the inherent is.

* use a correct fixed weight for the inclusion inherent

Co-authored-by: Guillaume Thiolliere <[email protected]>

* use dynamic inclusion weight so we reduce calculated weight when excluding candidates

* don't double-count this intrinsic's weight in the block weight

* add unit tests of fn limit_backed_candidates

* add tests that the inclusion inherent's weight correctly updates

Co-authored-by: Guillaume Thiolliere <[email protected]>

* Add well_known_keys

* Reorder HrmpChannel and HostConfiguration members

* abridged versions and well known keys tests

* Add some comments

* Add a note on generation of the prefixes

and other magic values

* Recommend accessing the well known values through abridged structs

* Cont.: Implement the state root obtaining during inclusion

During inclusion now we obtain the storage root by passing it through
the inclusion_inherent.

* Fix tests

* Bump rococo spec version

* Reorder the parent header into the end

of the inclusion inherent.

When the parent header is in the beginning, it shifts the other two
fields, so that a previous version won't be able to decode that. If
we put the parent header in the end, the other two fields will stay
at their positions, thus make it possible to decode with the previous
version.

That allows us to perform upgrade of rococo runtime without needing of
simultanuous upgrade of nodes and runtime, or restart of the network.

* Squash a stray tab

* Companion for #7810

* Added missing trait items for tests

* Add another missing trait item

* fixup

* "Update Substrate"

Co-authored-by: parity-processbot <>

* guide: ensure max-per-core leads to creation of extra, semi-useless cores

* alter behavior of max_validators_per_core

* guide fixes

* Don't initialize block when calling runtime APIs

* Adapt check_validation_outputs

We split the code path for the inclusion and for the commitments checking.

* Slap #[skip_initialize_block] on safe runtime APIs

That is, those that should not be affected by this attribute

* Make `Scheduled` not ephemeral

So that it is persisted in the storage and ready to be inspected
by the runtime APIs. This is in contrast to what was before, where we
would remove the storage entry and then rely on the scheduling performed
by `on_initialize` again.

* Add a big fat comment

* Typos

Co-authored-by: Robert Habermeier <[email protected]>

* Move session change to the end of the current block

Previously, it was the beginning of the next block. This allows us to
put #[skip_initialize_block]

* Update tests

* Fix a test in paras registrar

Also refactor it a bit so the next time there are more chances this kind
of issue is diagnosed quicker.

* Add for_runtime_api to inclusion's check_validation_outputs

Co-authored-by: Robert Habermeier <[email protected]>

* guide: add candidate information to OccupiedCore

* add descriptor and hash to occupied core type

* guide: add candidate hash to inclusion

* runtime: return candidate info in core state

* bitfield signing: stop querying runtime as much

* minimize going to runtime in availability distribution

* fix availability distribution tests

* guide: remove para ID from Occupied core

* get all crates compiling

* refactor View to include finalized_number

* guide: update the NetworkBridge on BlockFinalized

* av-store: fix the tests

* actually fix tests

* grumbles

* ignore macro doctest

* use Hash::repeat_bytes more consistently

* broadcast empty leaves updates as well

* fix issuing view updates on empty leaves updates

* plumbing for rewarding backers

* give validators reward points for participating

* fix tests

* add bitfield rewarding

* add mocks for backing rewards

* add testing for backing & availability rewards

* implement RewardValidators on top of staking

* add to test-runtime and rococo

* add to test-runtime & rococo

* point to source on rewards values

* fix common tests

* do not reward availability anymore

* basic setup

* fix compile

* add back xcmsink

* Update primitives.rs

* add downward xcm for sudo

* Update paras_sudo_wrapper.rs

* Update Cargo.lock

* some cleanups

* Add error log

Co-authored-by: Sergei Shulepov <[email protected]>

* guide: merge backing and approval keys

* bump substrate master & update primitives

* use new SessionInfo struct in session_info

* session keys upgrade for Polkadot

* kusama & westend runtimes

* bump westend, kusama, and polkadot versions

* add session key to rococo & test-runtime

* update prepare-test-net to latest subkey

* update chain specs to support new para_assignment session key

* get cargo.lock from master

* formatting

* update kill_storage based on substrate master

* fix test-service

* assgn -> asgn

* use session info module for assignment session key

* session_info: use proper timeout in test

* Revert "fix off-by-one error"

This reverts commit 35cb56305a19134acd8a8f881f3aabf999a09d74.

* session_info: use correct EarliestStoredSession when introduced on a live chain

* use saturating_sub

Co-authored-by: Bastian Köcher <[email protected]>

* session_info: revert the timeout test

* session_info: rust is dumb

Co-authored-by: Bastian Köcher <[email protected]>

* session_info: store the assigment ids

* how about this?

* session_info: a heavy loop test

* session_info: fix a typo

* session_info: fix heavy loop

* session_info: crank the iterations all the way up