* Companion for substrate#8079

* More changes

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Adapt code to changes in substrate

* WIP

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Adapt service creation to new substrate API

* Fix test

* WIP

* Revert "WIP"

This reverts commit 816a3633e91abc943b12b2bfa77ce98b959e78b2.

* WIP

* Adapt to current code

* Fix tests

* Yet another fix

* CLEANUP

* WIP

* WIP

* WIP

* Adapt code to changes on substrate

* Adapt code

* Introduce kick.

* Fixes

* WIP

* WIP

* WIP

* WIP

* Bump

* Update sp-io

* WIP

Co-authored-by: Gav Wood <[email protected]>
Co-authored-by: Parity Benchmarking Bot <[email protected]>

* Update for new service config

* Naming

* "Update Substrate"

Co-authored-by: parity-processbot <>

Co-authored-by: Pierre Krieger <[email protected]>
Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* remove low information density error doc comments

* another round of error dancing

* fix compilation

* remove stale `None` argument

* adjust test, minor slip in command

* only add AvailabilityError for full node features

* another None where none shuld be

* adding missing param

* updating to latest master

* include new parameter in test `Configuration`

* update calls to `init_logger`

* "Update Substrate"

* cargo update -p sp-io

Co-authored-by: Matt <[email protected]>
Co-authored-by: parity-processbot <>

* *: Update authority discovery and remove WorkerConfig

With https://github.com/paritytech/substrate/pull/7545 the authority
discovery module queries and publishes addresses on an exponentially
increasing interval. Doing so should make custom configurations
obsolete, as operations are retried in a timely fashion in the first
minutes.

* */Cargo.{lock,toml}: Point to mxinden substrate auth-disc-timing

* Revert "*/Cargo.{lock,toml}: Point to mxinden substrate auth-disc-timing"

This reverts commit 0785943a1e377454f088814ef20f4432de09da7a.

* "Update Substrate"

* Revert ""Update Substrate""

This reverts commit 377b221e1853b2c383f0c416d686535b545796cb.

* Cargo.lock: Manual Substrate update

* node/test/service/src/lib: Remove unused import

* parachain/test-parachains/adder: Remove unused import

Co-authored-by: parity-processbot <>

* send_message should not return an error

* Apply suggestions from code review

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* s/send_logging_error/send_and_log_error

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* Rename ExecutionMode to IsolationStrategy

Execution mode is too generic name and can imply a lot of different
aspects of execution. The notion of isolation better describes the
meant aspect.

And while I am at it, I also renamed mode -> strategy cause it seems a
bit more appropriate, although that is way more subjective.

* Fix compilation in wasm_executor tests.

* Add a comment to IsolationStrategy

* Update comments on IsolationStrategy

* Update node/core/candidate-validation/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Accomodate the point on interruption

* Update parachain/src/wasm_executor/mod.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Naming nits

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Andronik Ordian <[email protected]>

* Introduce CollatorFn type alias

* Make test-runtime imports consistent with rococo-runtime

* Update node/primitives/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* fix warnings

Co-authored-by: Bastian Köcher <[email protected]>

* Adds integration test based on adder collator

This adds an integration test for parachains that uses the adder
collator. The test will start two relay chain nodes and one collator and
waits until 4 blocks are build and enacted by the parachain.

* Make sure the integration test is run in CI

* Fix wasm compilation

* Update parachain/test-parachains/adder/collator/src/lib.rs

Co-authored-by: Sergei Shulepov <[email protected]>

* Update cli/src/command.rs

Co-authored-by: Sergei Shulepov <[email protected]>

* Fixes bug that collator wasn't sending `Declare` message

* Set authority discovery config

* Fixes bug that collator wasn't sending `Declare` message

* Adds real overseer feature and makes the wasm_validation fail with a
proper error

* Adds README

* Remove debug stuff

* Add feature

* Make adder collator use the correct parent when building a new block

* start working on building the real overseer

Unfortunately, this fails to compile right now due to an upstream
failure to compile which is probably brought on by a recent upgrade
to rustc v1.47.

* fill in AllSubsystems internal constructors

* replace fn make_metrics with Metrics::attempt_to_register

* update to account for #1740

* remove Metrics::register, rename Metrics::attempt_to_register

* add 'static bounds to real_overseer type params

* pass authority_discovery and network_service to real_overseer

It's not straightforwardly obvious that this is the best way to handle
the case when there is no authority discovery service, but it seems
to be the best option available at the moment.

* select a proper database configuration for the availability store db

* use subdirectory for av-store database path

* apply Basti's patch which avoids needing to parameterize everything on Block

* simplify path extraction

* get all tests to compile

* Fix Prometheus double-registry error

for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476:

```rust
Some(registry) => Self::try_register(registry).map_err(|err| {
	eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err);
	err
}),
```

That pointed out where the registration was failing, which led to
this fix. The test still doesn't pass, but it now fails in a new
and different way!

* authorities must have authority discovery, but not necessarily overseer handlers

* fix broken SpawnedSubsystem impls

detailed logging determined that using the `Box::new` style of
future generation, the `self.run` method was never being called,
leading to dropped receivers / closed senders for those subsystems,
causing the overseer to shut down immediately.

This is not the final fix needed to get things working properly,
but it's a good start.

* use prometheus properly

Prometheus lets us register simple counters, which aren't very
interesting. It also allows us to register CounterVecs, which are.
With a CounterVec, you can provide a set of labels, which can
later be used to filter the counts.

We were using them wrong, though. This pattern was repeated in a
variety of places in the code:

```rust
// panics with an cardinality mismatch
let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?;
my_counter.with_label_values(&["succeeded"]).inc()
```

The problem is that the labels provided in the constructor are not
the set of legal values which can be annotated, but a set of individual
label names which can have individual, arbitrary values.

This commit fixes that.

* get av-store subsystem to actually run properly and not die on first signal

* typo fix: incomming -> incoming

* don't disable authority discovery in test nodes

* Fix rococo-v1 missing session keys

* Update node/core/av-store/Cargo.toml

* try dummying out av-store on non-full-nodes

* overseer and subsystems are required only for full nodes

* Reduce the amount of warnings on browser target

* Fix two more warnings

* InclusionInherent should actually have an Inherent module on rococo

* Ancestry: don't return genesis' parent hash

* Update Cargo.lock

* fix broken test

* update test script: specify chainspec as script argument

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/service/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* node/service/src/lib: Return error via ? operator

* post-merge blues

* add is_collator flag

* prevent occasional av-store test panic

* simplify fix; expand application

* run authority_discovery in Role::Discover when collating

* distinguish between proposer closed channel errors

* add IsCollator enum, remove is_collator CLI flag

* improve formatting

* remove nop loop

* Fix some stuff

Co-authored-by: Andronik Ordian <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Fedor Sakharov <[email protected]>
Co-authored-by: Robert Habermeier <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Max Inden <[email protected]>

* runtime: annonate some types

* cargo update -p sp-io

* add wasm overwrites to Configuration in test-service

* rename overwrite to override

Co-authored-by: Andrew Plaza <[email protected]>

* Improve and unify testing facilities

This improves the testing facilities by making the test client easier to
use. It also removes code that is not required for the test client.
Besides that it also moves the test service and test client under
`node/test`.

* Update Cargo.lock

* Update node/test/client/src/block_builder.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Remove explicit lifetime annotation

* Fix warnings and add extra `BlockBuilderExt`

Co-authored-by: Andronik Ordian <[email protected]>

* WIP

* WIP

* WIP

* WIP

* WIP

* WIP

* Update node/service/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* WIP

* "Update Substrate"

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: parity-processbot <>

* Make register/deregister parachain dispatchables

This makes `register_parachain` and `deregister_parachain` of
`Registrar` dispatchables.

Besides that it brings back the functionality of the test node to
register a parachain.

* Fix tests

* PR review comments

This reverts commit a7b6c91f.

The authority discovery module was initially enabled by default on
validator and sentry nodes with commit 59f5eb4c. This change was later on
reverted in a7b6c91f. With this commit the authority discovery module is
again enabled by default.

They are not node handles anymore ;)

The collator is now enabled in a different way and we don't require this
argument anymore.

* Remove old service, 3rd try

i.e.
Revert "Revert "Remove Old Service, 2nd try (#1732)" (#1758)"

This reverts commit 9a0f08bf

.

Closes #1757.

We now have some evidence that the polkadot validator was producing
blocks after all; the reason the blocks_constructed metric was 0 was
that as a new metric it hadn't yet been incorporated into that
branch's codebase. See
https://github.com/paritytech/polkadot/issues/1757#issuecomment-700977602

As this PR is based on a newer `master` branch than the previous one,
that should hopefully no longer be an issue.

* paras trait now has an Origin type

* initial work running a two node local net

* use the right incantations so the nodes produce blocks together

* improve internal documentation

Co-authored-by: Bastian Köcher <[email protected]>

This reverts commit 96f26155.

* Restore "Remove service, migrate all to service-new (#1630)"

i.e.
Revert "Revert "Remove service, migrate all to service-new (#1630)" (#1731)"

This reverts commit b4457f55

.

This allows us to get the changeset from #1630 into a new branch
which can be merged sometime in the future after appropriate burnin
tests have completed.

* remove ',)' from codebase outside of macros

* restore bdfl-preferred formatting

* attempt to improve destructuring formatting

* rename polkadot-service-new -> polkadot-service

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* remove unused import

* Update runtime/rococo-v1/README.md

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Andronik Ordian <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

This reverts commit d531ba56.

* import rococo into chain-spec

* make a few stabs at moving forward

* wip: rococo readme

* remove /service crate

- Move the chain-spec files to node-service
- update sufficient cargo files that polkadot-service-new builds
- not everything else builds yet

* wip: chase down some build errors in polkadot-cli

There's a lot more to go, but some progress has happened.

* make more progress getting polkadot-cli to build

* don't ignore polkadot.json within the res directory

* don't recreate pathbufs

* Prepare Polkadot to be used by Cumulus

This begins to make Polkadot usable from Cumulus.

* Remove old test

* migrate new_chain_ops fix from /service

* partially remove node/test-service

* Reset some changes

* Revert "partially remove node/test-service"

This reverts commit 7b8f9ba5bfc286a309df89853ae11facf3277ffb.

* WIP: replace v0 ParachainHost impl with v1 for test runtime

This is necessary because one of the current errors when building
the test service boils down to:

the trait bound `polkadot_test_runtime::RuntimeApiImpl<...>`:
  `polkadot_primitives::v1::ParachainHost<...>` is not satisfied

This is WIP because it appears to be causing some std leakage into
the wasm environment, or something; the compiler is currently
complaining about duplicate definitions of `panic_handler` and `oom`.
Presumably I have to identify all std types (Vec etc) and replace
them with sp_std equivalents.

* fix test runtime build

it wasn't std leakage, after all

* bump westend spec version

* use service-new as service within cli

* to revert: demo that forwarding the test runtime to the real impl blows up

* Revert "to revert: demo that forwarding the test runtime to the real impl blows up"

This reverts commit 68d2f385f378721c7433e3e39133434610cd2a51.

* Revert "Revert "to revert: demo that forwarding the test runtime to the real impl blows up""

This reverts commit 04cb1cbf8873b4429cb9c9fdccb7f4bb137dc720.

Might have just forgotten to disable default features

* More reverts

* MOARE

* plug in the runtime as the generic instantiation

This feels closer to a solution, but it still has problems: in particular,
it's assumed that Runtime implements all appropriate Trait traits,
which this one apparently does not.

* implement necessary traits to get the test runtime compiling

This is almost certainly not correct in some way; it really
looks like I need to mess with the construct_runtime! macro
somehow, to inject the inclusion trait's event type as a Event
variant. Still, better lock down this changeset while it all
compiles.

* add inclusion::Event as variant into Event enum

* implement unimplemented bits in kusama

* implement unimplemented bits in polkadot runtime

* implement unimplemented bits in westend runtime

* migrate client upgrades from master

* update test service with new node changes

* package metadata--that wasn't intended to be removed

* add parachains v1 modules to each runtime

It's not clear what precisely this does, but it's probably the right
thing to do.

* enable cli to opt out of full node features

* adjust rococo chainspec per example

https://github.com/paritytech/polkadot/blob/26f1fa47f7836ab4bee5d4aad127ebce748320dd/service/src/chain_spec.rs#L362

* try to fix Cargo.lock

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Andronik Ordian <[email protected]>

Companion PR for ` Add a `build-sync-spec` subcommand and remove the CHT roots from the light sync state.` (#1670)

* Add BuildSyncSpec

* Remove accidental whitespace

* Update substrate

* Update substrate

* Fix wasm compilation

* clean out v0 consensus crates

* remove service dependencies on old consensus code

* fix cli

* kill adder-collator

* bump Cargo.lock

* remove Arc<RpcHandlers>

* "Update Substrate"

Co-authored-by: parity-processbot <>

* remove v0 parachains modules and switch to v1 primitives

* get tests compiling for runtime-common

* remove registrar module

* Add a dummy module

* remove runtime-parachains

* mostly remove old parachains code from polkadot-runtime

* remove slots::Trait implementation

* remove sp_std prelude import

* add a ZeroSizedTypeDifferentiator to dummy

* finish porting over polkadot runtime

* ZeroSizedTypeDifferentiator was actually unnecessary

* westend

* kusama

* test-runtime (no dummy modules)

* fix warning

* fix chain-specs

* fix test-service

* test-client

* remove dead import

* remove unused needed_extrinsics parameter

* runtimes compile

* remove rococo-v0

* remove remaining references to Rococo

* bump versions

* WIP

Forked at: e91642361133dba084a9c9b3efa45fcb6cd3c5ca
Parent branch: origin/rococo-branch

* WIP

Forked at: e91642361133dba084a9c9b3efa45fcb6cd3c5ca
Parent branch: origin/rococo-branch

* WIP

Forked at: e91642361133dba084a9c9b3efa45fcb6cd3c5ca
Parent branch: origin/rococo-branch

* WIP

Forked at: e91642361133dba084a9c9b3efa45fcb6cd3c5ca
Parent branch: origin/rococo-branch

* WIP

Forked at: e91642361133dba084a9c9b3efa45fcb6cd3c5ca
Parent branch: origin/rococo-branch

* WIP

Forked at: e91642361133dba084a9c9b3efa45fcb6cd3c5ca
Parent branch: origin/rococo-branch

* WIP

Forked at: e91642361133dba084a9c9b3efa45fcb6cd3c5ca
Parent branch: origin/rococo-branch

* WIP

Forked at: e91642361133dba084a9c9b3efa45fcb6cd3c5ca
Parent branch: origin/rococo-branch

* Apply suggestions from code review

* WIP

Forked at: e91642361133dba084a9c9b3efa45fcb6cd3c5ca
Parent branch: origin/rococo-branch

* WIP

Forked at: e91642361133dba084a9c9b3efa45fcb6cd3c5ca
Parent branch: origin/rococo-branch

* WIP

Forked at: e91642361133dba084a9c9b3efa45fcb6cd3c5ca
Parent branch: origin/rococo-branch

* CLEANUP

Forked at: e91642361133dba084a9c9b3efa45fcb6cd3c5ca
Parent branch: origin/rococo-branch

* link in doc

* doc

* Rewrite client handling

We are supporting muliple polkadot-like chains and all have different
client types. This pr reworks the client handling by having all of them
in one enum combined. Besides that, there is added a special trait
`ExecuteWithClient` to use the internal client.

* Apply suggestions from code review

Co-authored-by: Robert Habermeier <[email protected]>

* Up the versions

* Fix Cargo.lock

* Fix merge conflict

* ......................

* ....v2

* yep

* I'm dumb...

* Browser lol

Co-authored-by: Robert Habermeier <[email protected]>

* Fix transaction payment runtime api companion

Companion for Substrate pr: https://github.com/paritytech/substrate/pull/6792

* 'Update substrate'

Co-authored-by: parity-processbot <>

* Switch branch

* Update branch

* Change service code

* Change light service stuff to be functions ^_^

* Update substrate branch

* Remove accidental 'f'

* Rework LightBackend/LightClient types

* Update substrate branch

* Remove unused imports in test-service

* Add #[cfg(feature = full-node)]

* *: Enable authority discovery by default

Instead of having to explicitly enable the authority discovery module on
validator and sentry nodes, this commit enables the module by default.

Today there is no way for non validator or sentry nodes to run the
module. That might change in the future.

* service/src/lib: Fix typo in new_full! for test

* create a v1 primitives module

* Improve guide on availability types

* punctuate

* new parachains runtime uses new primitives

* tests of new runtime now use new primitives

* add ErasureChunk to guide

* export erasure chunk from v1 primitives

* subsystem crate uses v1 primitives

* node-primitives uses new v1 primitives

* port overseer to new primitives

* new-proposer uses v1 primitives (no ParachainHost anymore)

* fix no-std compilation for primitives

* service-new uses v1 primitives

* network-bridge uses new primitives

* statement distribution uses v1 primitives

* PoV distribution uses v1 primitives; add PoV::hash fn

* move parachain to v0

* remove inclusion_inherent module and place into v1

* remove everything from primitives crate root

* remove some unused old types from v0 primitives

* point everything else at primitives::v0

* squanch some warns up

* add RuntimeDebug import to no-std as well

* port over statement-table and validation

* fix final errors in validation and node-primitives

* add dummy Ord impl to committed candidate receipt

* guide: update CandidateValidationMessage

* add primitive for validationoutputs

* expand CandidateValidationMessage further

* bikeshed

* add some impls to omitted-validation-data and available-data

* expand CandidateValidationMessage

* make erasure-coding generic over v1/v0

* update usages of erasure-coding

* implement commitments.hash()

* use Arc<Pov> for CandidateValidation

* improve new erasure-coding method names

* fix up candidate backing

* update docs a bit

* fix most tests and add short-circuiting to make_pov_available

* fix remainder of candidate backing tests

* squanching warns

* squanch it up

* some fallout

* overseer fallout

* free from polkadot-test-service hell

Fixes #833