* WIP: add initial check_extrinsics_ordering.sh script

* iterate through runtimes, add gitlab job

* move job to publish

* temp force build-linux-release to run

* update check_extrinsics_ordering.sh

* maybe we have to fetch release

* use node docker image

* revert before opening pr: force bad extrinsic ordering

* revert commits to prepare for PR

* move job to build stage, use bin from test-linux-release

* remove FIXME

* fix PR nags

* Revert "Restrict collating_using_adder_collator test on feature real-overseer (#1947)"

This reverts commit 75da335d.

* Revert "Revert "Restrict collating_using_adder_collator test on feature real-overseer (#1947)""

This reverts commit 4e92cdfb1046b0e3da7627297050ae71ae3dd6e3.

* Test something

This test can only pass given that feature, and runs forever otherwise.

This change ensure that a careless `cargo test --all` doesn't run forever.

* babe: backoff authoring blocks when finality lags

* service: use default constructor for backoff authoring strat

* slots: update to use renamed backoff strategy name

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Improve diagnostics for acceptance criteria failures during inclusion

* Initialize the runtime logger just before logging during inclusion

* Formatting suggestions

Co-authored-by: Bastian Köcher <[email protected]>

* Missed one suggestion

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>

* Don't just swallow error in dummy-subsystem, but log

* Add target to log.

* Rename ExecutionMode to IsolationStrategy

Execution mode is too generic name and can imply a lot of different
aspects of execution. The notion of isolation better describes the
meant aspect.

And while I am at it, I also renamed mode -> strategy cause it seems a
bit more appropriate, although that is way more subjective.

* Fix compilation in wasm_executor tests.

* Add a comment to IsolationStrategy

* Update comments on IsolationStrategy

* Update node/core/candidate-validation/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Accomodate the point on interruption

* Update parachain/src/wasm_executor/mod.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Naming nits

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Andronik Ordian <[email protected]>

* Introduce CollatorFn type alias

* Make test-runtime imports consistent with rococo-runtime

* Update node/primitives/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* fix warnings

Co-authored-by: Bastian Köcher <[email protected]>

* Adds integration test based on adder collator

This adds an integration test for parachains that uses the adder
collator. The test will start two relay chain nodes and one collator and
waits until 4 blocks are build and enacted by the parachain.

* Make sure the integration test is run in CI

* Fix wasm compilation

* Update parachain/test-parachains/adder/collator/src/lib.rs

Co-authored-by: Sergei Shulepov <[email protected]>

* Update cli/src/command.rs

Co-authored-by: Sergei Shulepov <[email protected]>

* HRMP: Update the impl guide

* HRMP: Incorporate the channel notifications into the guide

* HRMP: Renaming in the impl guide

* HRMP: Constrain the maximum number of HRMP messages per candidate

This commit addresses the HRMP part of https://github.com/paritytech/polkadot/issues/1869

* XCM: Introduce HRMP related message types

* HRMP: Data structures and plumbing

* HRMP: Configuration

* HRMP: Data layout

* HRMP: Acceptance & Enactment

* HRMP: Test base logic

* Update adder collator

* HRMP: Runtime API for accessing inbound messages

Also, removing some redundant fully-qualified names.

* HRMP: Add diagnostic logging in acceptance criteria

* HRMP: Additional tests

* Self-review fixes

* save test refactorings for the next time

* Missed a return statement.

* a formatting blip

* Add missing logic for appending HRMP digests

* Remove the channel contents vectors which became empty

* Tighten HRMP channel digests invariants.

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* Remove a note about sorting for channel id

* Add missing rustdocs to the configuration

* Clarify and update the invariant for HrmpChannelDigests

* Make the onboarding invariant less sloppy

Namely, introduce `Paras::is_valid_para` (in fact, it already is present
in the implementation) and hook up the invariant to that.

Note that this says "within a session" because I don't want to make it
super strict on the session boundary. The logic on the session boundary
should be extremely careful.

* Make `CandidateCheckContext` use T::BlockNumber for hrmp_watermark

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

We need to distribute the PoV after we have seconded it. Other nodes
that will receive our `Secondded` statement and want to validate the
candidate another time will request this PoV from us.

* Adds a comment as requested

* Update node/core/bitfield-signing/src/lib.rs

Co-authored-by: Fedor Sakharov <[email protected]>

Co-authored-by: Fedor Sakharov <[email protected]>

If a core is freed, we need to make sure it is inserted as free into
availability cores, because we can not be sure that the core is occupied
directly again.

* Make `CandidateHash` a real type

This pr adds a new type `CandidateHash` that is used instead of the
opaque `Hash` type. This helps to ensure on the type system level that
we are passing the correct types.

This pr also fixes wrong usage of `relay_parent` as `candidate_hash`
when communicating with the av storage.

* Update core-primitives/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* Wrap the lines

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* backing: extract log target

* bitfield-signing: extract log target

* utils: fix a typo

* provisioner: extract log target

* candidate selection: remove unused error variant

* bitfield-distribution: change the return type of run

* pov-distribution: extract log target

* collator-protocol: simplify runtime request

* collation-generation: do not exit early on error

* collation-generation: do not exit on double init

* collator-protocol: do not exit on errors and rename LOG_TARGET

* collator-protocol: a workaround for ununused imports warning

* Update node/network/bitfield-distribution/src/lib.rs

* collation-generation: elevate warn! to error!

* collator-protocol: fix imports

* post merge fix

* fix compilation

Besides that the pr also adds a simple test.

* Do not validate a candidate in candidate selection

The candidate selection subsystem should not validate a candidate, as
this is done by the backing subsystem on a `Second` request. Otherwise
we validate one candidate twice.

* Update candidate-selection.md

* Moare fixes for parachains

- Sending data to a job should always contain a relay parent. Done this
for the provisioner
- Fixed the `select_availability_bitfields` function. It was assuming we
have one core per validator, while we only have one core per parachain.
- Drive by async "rewrite" in proposer

* Make tests compile

* Update primitives/src/v1.rs

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* Update proxy filters

* add session

* new polkadot weights

* kusama weights

* update for westend

* cargo update -p sp-io

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=pallet_session --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=pallet_session --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=pallet_session --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/

Co-authored-by: Parity Benchmarking Bot <[email protected]>

* Parachain improvements

- Set the parachains configuration in Rococo genesis
- Don't stop the overseer when a subsystem job is stopped
- Several small code changes

* Remove unused functionality

* Return error from the runtime instead of printing it

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* Update primitives/src/v1.rs

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* Update primitives/src/v1.rs

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* Fix test

* Revert "Update primitives/src/v1.rs"

This reverts commit 11fce2785acd1de481ca57815b8e18400f09fd52.

* Revert "Update primitives/src/v1.rs"

This reverts commit d6439fed4f954360c89fb1e12b73954902c76a41.

* Revert "Return error from the runtime instead of printing it"

This reverts commit cb4b5c0830ac516a6d54b2c24197e9354f2b98cb.

* Revert "Fix test"

This reverts commit 0c5fa1b5566d4cd3c55a55d485e707165ce7a59e.

* Update runtime/parachains/src/runtime_api_impl/v1.rs

Co-authored-by: Sergei Shulepov <[email protected]>

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>
Co-authored-by: Sergei Shulepov <[email protected]>

* add native logging to check_upward_messages

This doesn't affect the WASM builds, but ensures that native versions
(such as are used for testing) emit diagnostics anytime the
check_upward_messages function exits, which should reduce the pain
of debugging when something goes wrong.

* Apply suggestions from code review

Co-authored-by: Sergei Shulepov <[email protected]>

* verb tense

Co-authored-by: Sergei Shulepov <[email protected]>

* Fix adder collator README

* Add trivial logging to show that collator collates

* Update parachain/test-parachains/adder/collator/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>

* utils: remove unused error

* av-store: do not exit early on errors

* av-store: revert logging change on master

* av-store: add a test

* Commit my changes

* some backing fixes

* indentation

* fix backing tests

* tweak includability rules

* comment

* Update node/core/backing/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/core/backing/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/core/backing/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/core/backing/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* fix: ensure candidate validation gets code based on occupied core assumption

* guide: runtime API for historical validation code

* add historical runtime API

* integrate into runtime API subsystem

* remove blocked TODO

* fix service build: enable notifications protocol only under real overseer

* Update node/subsystem/src/messages.rs

Co-authored-by: Sergei Shulepov <[email protected]>

* fix compilation

Co-authored-by: Robert Habermeier <[email protected]>
Co-authored-by: Sergei Shulepov <[email protected]>

* UMP: Update the impl guide

* UMP: Incorporate XCM related changes into the guide

* UMP: Data structures and configuration

* UMP: Initial plumbing

* UMP: Data layout

* UMP: Acceptance criteria & enactment

* UMP: Fix dispatcher bug and add the test for it

* UMP: Constrain the maximum size of an UMP message

This commit addresses the UMP part of https://github.com/paritytech/polkadot/issues/1869

* Fix failing test due to misconfiguration

* Make the type of RelayDispatchQueueSize be more apparent in the guide

* Revert renaming `max_upward_queue_capacity` to `max_upward_queue_count`

* convert spaces to tabs

Co-authored-by: Bernhard Schuster <[email protected]>

* Update runtime/parachains/src/router/ump.rs

Co-authored-by: Bernhard Schuster <[email protected]>

Co-authored-by: Bernhard Schuster <[email protected]>

* Fixes bug that collator wasn't sending `Declare` message

* Set authority discovery config

* Fixes bug that collator wasn't sending `Declare` message

* Adds real overseer feature and makes the wasm_validation fail with a
proper error

* Adds README

* Remove debug stuff

* Add feature

* Make adder collator use the correct parent when building a new block

* Fixes bug that collator wasn't sending `Declare` message

* Apply suggestions from code review

I'm an idiot

Co-authored-by: Robert Habermeier <[email protected]>

* start working on building the real overseer

Unfortunately, this fails to compile right now due to an upstream
failure to compile which is probably brought on by a recent upgrade
to rustc v1.47.

* fill in AllSubsystems internal constructors

* replace fn make_metrics with Metrics::attempt_to_register

* update to account for #1740

* remove Metrics::register, rename Metrics::attempt_to_register

* add 'static bounds to real_overseer type params

* pass authority_discovery and network_service to real_overseer

It's not straightforwardly obvious that this is the best way to handle
the case when there is no authority discovery service, but it seems
to be the best option available at the moment.

* select a proper database configuration for the availability store db

* use subdirectory for av-store database path

* apply Basti's patch which avoids needing to parameterize everything on Block

* simplify path extraction

* get all tests to compile

* Fix Prometheus double-registry error

for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476:

```rust
Some(registry) => Self::try_register(registry).map_err(|err| {
	eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err);
	err
}),
```

That pointed out where the registration was failing, which led to
this fix. The test still doesn't pass, but it now fails in a new
and different way!

* authorities must have authority discovery, but not necessarily overseer handlers

* fix broken SpawnedSubsystem impls

detailed logging determined that using the `Box::new` style of
future generation, the `self.run` method was never being called,
leading to dropped receivers / closed senders for those subsystems,
causing the overseer to shut down immediately.

This is not the final fix needed to get things working properly,
but it's a good start.

* use prometheus properly

Prometheus lets us register simple counters, which aren't very
interesting. It also allows us to register CounterVecs, which are.
With a CounterVec, you can provide a set of labels, which can
later be used to filter the counts.

We were using them wrong, though. This pattern was repeated in a
variety of places in the code:

```rust
// panics with an cardinality mismatch
let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?;
my_counter.with_label_values(&["succeeded"]).inc()
```

The problem is that the labels provided in the constructor are not
the set of legal values which can be annotated, but a set of individual
label names which can have individual, arbitrary values.

This commit fixes that.

* get av-store subsystem to actually run properly and not die on first signal

* typo fix: incomming -> incoming

* don't disable authority discovery in test nodes

* Fix rococo-v1 missing session keys

* Update node/core/av-store/Cargo.toml

* try dummying out av-store on non-full-nodes

* overseer and subsystems are required only for full nodes

* Reduce the amount of warnings on browser target

* Fix two more warnings

* InclusionInherent should actually have an Inherent module on rococo

* Ancestry: don't return genesis' parent hash

* Update Cargo.lock

* fix broken test

* update test script: specify chainspec as script argument

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/service/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* node/service/src/lib: Return error via ? operator

* post-merge blues

* add is_collator flag

* prevent occasional av-store test panic

* simplify fix; expand application

* run authority_discovery in Role::Discover when collating

* distinguish between proposer closed channel errors

* add IsCollator enum, remove is_collator CLI flag

* improve formatting

* remove nop loop

* Fix some stuff

* Adds test parachain adder collator

* Add sudo to Rococo, change session length to 30 seconds and some renaming

* Update to the latest changes on master

* Some fixes

* Fix compilation

* Update parachain/test-parachains/adder/collator/src/lib.rs

Co-authored-by: Sergei Shulepov <[email protected]>

* Review comments

* Downgrade transaction version

* Fixes

* MOARE

* Register notification protocols

* utils: remove unused error

* av-store: more resilient to some errors

* address review nits

* address more review nits

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>
Co-authored-by: Andronik Ordian <[email protected]>
Co-authored-by: Fedor Sakharov <[email protected]>
Co-authored-by: Robert Habermeier <[email protected]>
Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>
Co-authored-by: Max Inden <[email protected]>
Co-authored-by: Sergey Shulepov <[email protected]>
Co-authored-by: Sergei Shulepov <[email protected]>

* A real overseer feature

* Fix build without feature

* Update node/service/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Robert Habermeier <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* add storage for approvals module

* basics of approval logic

* fix links

* Session info module

* create an approvals_inherent module

* integrate approvals module with inclusion

* Remove approvals runtime

* tweak mentions of on-chain logic

* add note on finality-grandpa voting rule

* elaborate on node-side components

* stub for availability recovery

* add another note on voting rule

* Beginnings of approval subsystems

* flesh out approval voting now

* logic for checking assignment certs

* initial scheduler logic

* scheduler logic

* adjst tranche taking logic

* approval voting import

* approval work (voting side)

* amend some TODOs

* mark some TODOs

* describe `ApprovedAncestor`

* reference protocol-approval.md

* clarity on bitfield

* remove approvals_inherent

* tweak session_info module according to review

* formatting & nits

Co-authored-by: Robert Habermeier <[email protected]>

* adjustments for substrate PR

https://github.com/paritytech/substrate/pull/6912

* Update runtime/kusama/src/lib.rs

Co-authored-by: Guillaume Thiolliere <[email protected]>

* Apply suggestions from code review

Co-authored-by: Guillaume Thiolliere <[email protected]>

* update substrate

Co-authored-by: Guillaume Thiolliere <[email protected]>

* switch to using paritytech/srtool

* update release link

* Make sure validator discovery works with a delayed peer to validator mapping

Currently the implementation checks on connect of a peer if this peer is
a validator by asking the authority discovery. It can now happen that
the authority discovery is not yet aware that a given peer is an
authority. This can for example happen on start up of the node.

This pr changes the behavior, to make it possible to later associate a
peer to a validator id. Instead of just storing the connected
validators, we now store all connected peers with a vector of associated
validator ids. When we get a request to connect to a given given set of
validators, we start by checking the connected peers. If we didn't find
a validator id in the connected peers, we ask the authority discovery
for the peerid of a given authority id. When the returned peerid is part
of our connected peers set, we cache and return the authority id.

* Update node/network/bridge/Cargo.toml

Co-authored-by: Pierre Krieger <[email protected]>

* Update node/network/bridge/src/validator_discovery.rs

Co-authored-by: Pierre Krieger <[email protected]>

* Update `Cargo.lock`

Co-authored-by: Pierre Krieger <[email protected]>