* update futures to 0.3.9

* companion changes for generic error types

* improve errors, keep error sources

* command compile fix

* use the correct Error type, add annotation

* avoid async { fut.await.map_err(E::Variant) } with TryFuture

* fix cargo lock, bump futures to 0.3.10

futures 0.3.9 got yanked

* chore cargo update -p sp-io

* minor quirk

* make rustc 1.48 happy by bouning the error more tightly

* be more explicit

* cargo lock futures

* session_info: use correct authorities set

* bump rococo spec_version to 19

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Drive by fixes

The visibility modifiers are remnants of the previous structure where
HRMP wasn't a standalone module, by rather a submodule of the router
module.

* Add Currency assoc type to Config

This would allow us to reserve balance for deposits. This commit also
integrates the HRMP module in rococo, test-runtime and mocks to use the
balances pallet.

* Fix a bug that doesn't increment the age

In case the request is not confirmed, the age would be incremented but
not persisted.

* Fix cleaning the indexes

Before that change, the cleaning of the channel indexes was wrong, because it
naively removed entire rows that was pertaining to the para we delete.
This approach is flawed because it doesn't account for the rows that are
pertaining to other paras that contain the outgoing one.

This clearly violates the invariant imposed on the indexes, that all
the index rows must contain alive paras, but apart from that it also
lead to the situation where ingress index would contain the a different
set of channels that an egress have.

* Reserve currency for opening the channels

Note the ugly `unique_saturated_into` calls. The reason for them is the
currency trait accepts and defines the `Balance` associated type and the
deposit values are coming from the `HostConfiguration` where they are
defined using the `Balance`.

I figured that parameterising `HostConfiguration` would be annoying. On
the other hand, I don't expect these `unique_saturated_into` calls to
give us problems since it seems to be a reasonable assumption that this
module will be instantiated within a runtime where the Currency provided
will have a Balance that matches the one used in the configuration.

* Tests: Adapt `run_to_block` so that it submits a proper config

* Tests: exercise the deposit logic

Closes #2241

* Add bad block to rococo

* Bump rococo spec_version

* Revert "Add bad block to rococo"

This reverts commit 56664ce36f4fd9ea570b63e7b6978a6d2ff2aace.

* CI: add rococo build and conteinerization

* CI: overhaul rules

* CI: two new conditional jobs to build rococo, CI overhaul, npm security fix

* CI: remove web-wasm publishing

* Adds propose parachain pallet

* Update runtime/rococo/src/propose_parachain.rs

Co-authored-by: Shawn Tabrizi <[email protected]>

* Fix runtime benchmarks

* Get rid of staking

* Fix benchmarking feature..

* Remove accidentally added crate

* Bump Rococo spec_version

Co-authored-by: Shawn Tabrizi <[email protected]>

Bumps [log](https://github.com/rust-lang/log) from 0.4.11 to 0.4.13.
- [Release notes](https://github.com/rust-lang/log/releases)
- [Changelog](https://github.com/rust-lang/log/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/log/compare/0.4.11...0.4.13)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [tempfile](https://github.com/Stebalien/tempfile) from 3.1.0 to 3.2.0.
- [Release notes](https://github.com/Stebalien/tempfile/releases)
- [Changelog](https://github.com/Stebalien/tempfile/blob/master/NEWS)
- [Commits](https://github.com/Stebalien/tempfile/commits)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Rename crowdfund -> crowdloan

* allow contribution on behalf of another user

* starting some benchmarks

* optimization: use append api

* Use on_initialize instead of on_finalize

* More benchmarks

* try to implement partial child storage removal

* partial dissolve test

* onboard benchmark

* begin retirement

* on_initialize

* remove _ { }

* Revert "allow contribution on behalf of another user"

This reverts commit b7dd7d1ec751495cee3ddb57133a13c390b020e5.

* finish undo of "allow contribution on behalf of another user"

* Allow any user to trigger withdraw on closed crowdloan

* use transfer instead of withdraw/create pattern

* unused warning

* Update runtime/common/src/crowdloan.rs

Co-authored-by: Kian Paimani <[email protected]>

* dont need to assign to empty variable

Co-authored-by: Kian Paimani <[email protected]>

Prior this commit, a PVF wasn't able to access this property.

* alternate availability store schema

* improvements

* tweaks

* new DB schema and skeleton

* expand skeleton and tweaks

* handle backing and inclusion

* let finality be handled later

* handle finalized blocks

* implement query methods

* implement chunk storing

* StoreAvailableData

* fix an off-by-one

* implement pruning

* reinstate subsystem trait impl

* reinstate metrics

* fix warnings

* remove chunks_cache

* oops

* actually store the available data

* mockable pruning interval

* fix tests

* spacing

* fix code grumbles

* guide improvements

* make time mockable

* implement a mocked clock for testing

* return DB errors

* Update node/core/av-store/Cargo.toml

Co-authored-by: Bastian Köcher <[email protected]>

* Update roadmap/implementers-guide/src/node/utility/availability-store.md

Co-authored-by: Bastian Köcher <[email protected]>

* Update roadmap/implementers-guide/src/node/utility/availability-store.md

Co-authored-by: Bastian Köcher <[email protected]>

* review grumbles & clarity

* fix review grumbles

* Add docs

Co-authored-by: Andronik Ordian <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Andronik Ordian <[email protected]>

Bumps [pin-project](https://github.com/taiki-e/pin-project) from 1.0.3 to 1.0.4.
- [Release notes](https://github.com/taiki-e/pin-project/releases)
- [Changelog](https://github.com/taiki-e/pin-project/blob/master/CHANGELOG.md)
- [Commits](https://github.com/taiki-e/pin-project/compare/v1.0.3...v1.0.4)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [smallvec](https://github.com/servo/rust-smallvec) from 1.6.0 to 1.6.1.
- [Release notes](https://github.com/servo/rust-smallvec/releases)
- [Commits](https://github.com/servo/rust-smallvec/compare/v1.6.0...v1.6.1)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* subsystems have an unbounded channel to the overseer

* Update node/overseer/src/lib.rs

Co-authored-by: Bernhard Schuster <[email protected]>

* bump Cargo.lock

Co-authored-by: Bernhard Schuster <[email protected]>

* Do not send empty view updates to peers

It happened that we send empty view updates to our peers, because we
only updated our finalized block. This could lead to situations where we
overwhelmed sub systems with too many messages. On Rococo this lead to
constant restarts of our nodes, because some node apparently was
finalizing a lot of blocks.

To prevent this, the pr is doing the following:

1. If a peer sends us an empty view, we report this peer and decrease it
reputation.

2. We ensure that we only send a view update when the `heads` changed
and not only the `finalized_number`.

3. We do not send empty `ActiveLeavesUpdates` from the overseer, as this
makes no sense to send these empty updates. If some subsystem is relying
on the finalized block, it needs to listen for the overseer signal.

* Update node/network/bridge/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* Don't work if they're are no added heads

* Fix test

* Ahhh

* More fixes

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* Fuse receive stream in Context

* Revert "Fuse receive stream in Context"

This reverts commit ddd26fa98f0ca1afbc22064e93010e4193a058b2.

* Exit on node shutdown from av-store loop

* Filter only context error

Batch messages to network bridge and introduce a timeout to `SubsystemContext::send_message` (#2197)

* guide: batch network messages

* bridge: batch

* av-dist: batch outgoing messages

* time-out message sends in subsystem context

* Update node/subsystem/src/messages.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Revert "time-out message sends in subsystem context"

This reverts commit d49be62557ec37c8a350b93718acad723df704ef.

* Update node/network/availability-distribution/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Store all chunks and in a single transaction

* Adds chunks LRU to store

* Add pruning records metrics

* Use honest cache instead of LRU

* Remove unnecessary optional cache

* Fix review nits that are fixable

* Switch to latest Jaeger and improve the spans

* Update node/jaeger/src/lib.rs

Co-authored-by: Robert Habermeier <[email protected]>

* Use better span in bitfield signing

* Update node/core/bitfield-signing/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

Co-authored-by: Robert Habermeier <[email protected]>
Co-authored-by: Andronik Ordian <[email protected]>

* Companion PR for refactoring priority groups

* Fix non reserved node

* Try fix tests

* Missing import

* Fix warning

* Change protocols order

* Fix test

* Renames

* Update syn dependency to make it compile again after merging master

* "Update Substrate"

Co-authored-by: parity-processbot <>

* upgrade a few dependencies

* "Update Substrate"

Co-authored-by: parity-processbot <>

* added new bootnode to chain spec's

* remove trailing newline

* add timing setup to OverseerSubsystemContext

* figure out how to initialize the rng

* attach a timer to a portion of the messages traveling to the Overseer

This timer only exists / logs a fraction of the time (configurable
by `MESSAGE_TIMER_METRIC_CAPTURE_RATE`). When it exists, it tracks
the span between the `OverSubsystemContext` receiving the message
and its receipt in `Overseer::run`.

* propagate message timing to the start of route_message

This should be more accurate; it ensures that the timer runs
at least as long as that function. As `route_message` is async,
it may not actually run for some time after it is called (or ever).

* fix failing test

* rand_chacha apparently implicitly has getrandom feature

* change rng initialization

The previous impl using `from_entropy` depends on the `getrandom`
crate, which uses the system entropy source, and which does not
work on `wasm32-unknown-unknown` because it wants to fall back to
a JS implementation which we can't assume exists.

This impl depends only on `rand::thread_rng`, which has no documentation
stating that it's similarly limited.

* remove randomness in favor of a simpler 1 of N procedure

This deserves a bit of explanation, as the motivating issue explicitly
requested randomness. In short, it's hard to get randomness to compile
for `wasm32-unknown-unknown` because that is explicitly intended to be
as deterministic as practical. Additionally, even though it would never
be used for consensus purposes, it still felt offputting to intentionally
introduce randomness into a node's operations. Except, it wasn't really
random, either: it was a deterministic PRNG varying only in its state,
and getting the state to work right for that target would have required
initializing from a constant.

Given that it was a deterministic sequence anyway, it seemed much simpler
and more explicit to simply select one of each N messages instead of
attempting any kind of realistic randomness.

* reinstate randomness for better statistical properties

This partially reverts commit 0ab8594c328b3f9ce1f696fe405556d4000630e9.

`oorandom` is much lighter than the previous `rand`-based implementation,
which makes this easier to work with.

This implementation gives each subsystem and each child RNG a distinct
increment, which should ensure they produce distinct streams of values.

Bumps [trie-db](https://github.com/paritytech/trie) from 0.22.1 to 0.22.2.
- [Release notes](https://github.com/paritytech/trie/releases)
- [Commits](https://github.com/paritytech/trie/compare/trie-db-v0.22.1...trie-db-v0.22.2)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* runtime: add Babe::next_epoch runtime api method

* "Update Substrate"

Co-authored-by: parity-processbot <>

Bumps [pin-project](https://github.com/taiki-e/pin-project) from 1.0.2 to 1.0.3.
- [Release notes](https://github.com/taiki-e/pin-project/releases)
- [Changelog](https://github.com/taiki-e/pin-project/blob/master/CHANGELOG.md)
- [Commits](https://github.com/taiki-e/pin-project/compare/v1.0.2...v1.0.3)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Remove `_{}` from benchmarks macro

* "Update Substrate"

Co-authored-by: parity-processbot <>

This ensures that we also record the first `import_statement` that will
lead to the creation of the `unbacked-span`.

* don't modify inherent data on heavy block

* write up current thinking on block weight detection

* extract inherent inclusion check into its own function

* put heavy block check into runtime

* the `inclusion` inherent call is Operational, not Mandatory

This resolves a lot of the trickiness about this issue, because
we no longer need to override or supplant any existing proposer
logic; the existing logic should exhibit these behaviors:

- the `inclusion` inherent is prioritized over standard transactions
- but if it's too heavy, i.e. in case of runtime upgrade, it'll be
  dropped in favor of that.

It is my belief that allowing the proposer to just not include
this data won't have any adverse effects: it's equivalent to replacing
them with empty versions of themselves, which the `ProvideInherent`
impl already does.

* Revert "the `inclusion` inherent call is Operational, not Mandatory"

This reverts commit e58858d109b18b84e7af3ac47981c6900b2d9a3e.

* Revert "write up current thinking on block weight detection"

This reverts commit fd587b80c46761b2a2b62448193348237863f99f.

* Revert "don't modify inherent data on heavy block"

This reverts commit 38299d3c23e9efb5a354d8cfa658e62a5c8c7ddf.

* add backed candidate block weight assumption to configuration

* Limit backed candidates according to a candidate weight heuristic.

This approach replaces making the inclusion inherent non-mandatory.
It's still not ideal in that we have to configure a heuristic for
how much each backed candidate 'weighs', instead of directly
measuring it somehow.

This approach also never truncates the signed bitfields. The
rationale for that depends on some assumptions:

- processing the signed bitfields is cheap compared to the
  backed candidates
- it is beneficial to the progress of the relay chain
  to update the signed bitfields even if not all backed candidates
  are updated

* simplify limit_backed_candidates and weight assumption

* don't trust the provisioner to fairly distribute candidates

* use saturating subtraction

* empty commit to restart ci

* use new mechanism for getting max block weight

* apply weight refunds to the inclusion inherent

This makes some assumptions about fundamental weights, which are
encapsulated as constants. From there, it lets Substrate know
what the actual computed weight of the inherent is.

* use a correct fixed weight for the inclusion inherent

Co-authored-by: Guillaume Thiolliere <[email protected]>

* use dynamic inclusion weight so we reduce calculated weight when excluding candidates

* don't double-count this intrinsic's weight in the block weight

* add unit tests of fn limit_backed_candidates

* add tests that the inclusion inherent's weight correctly updates

Co-authored-by: Guillaume Thiolliere <[email protected]>