* Remove telemetry external transport member

* Make it compile

* Remove more

Co-authored-by: Bastian Köcher <[email protected]>

* cargo +nightly fmt

* add cargo-fmt check to ci

* update ci

* fmt

* fmt

* skip macro

* ignore bridges

* update rustls to be compat with jsronrpsee 0.2.0

* Stub try-cli when feat not enabled

* update Substrate

Co-authored-by: parity-processbot <>

* polkadot-service: Make native runtime configurable

This pull requests adds support for configuring the native runtimes used
by polkadot-service. While this whole pr doesn't change that much for
polkadot, besides not having the light-node enabled for the default
polkadot binary. However, downstream projects (parachains) will have a
much better compile time. In cumulus for example the `cargo test --all
--release` is about 4m faster to compile.

* Fixes

* Fix

* Enable rococo-native

* Fix light client

* 🤦

* Fixes

Instead of relying on the name of the chain spec file, we now use the
`is_*` methods for this.

* Add --disable-beefy CLI flag

* Update cli/src/command.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Update node/test/service/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update parachain/test-parachains/adder/collator/src/main.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update cli/src/command.rs

Co-authored-by: André Silva <[email protected]>

* Update cli/src/cli.rs

Co-authored-by: André Silva <[email protected]>

* Update cli/src/cli.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update cli/src/cli.rs

Co-authored-by: André Silva <[email protected]>

Co-authored-by: Andronik Ordian <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: André Silva <[email protected]>

* Add `rococo-dev` chain option

* Add `wococo-dev` CLI option

* Sort Chain ID match arms

* runtime: remove mmr and beefy from westend runtime

* runtime: westend: remove pallet_beefy config

* node: only start beefy gadget on rococo

* node: remove beefy keys from westend chain spec

* node: add wococo testnet chain spec

* node: add comments about beefy gadget task

* runtime: update wococo chainspec

* Remove stale comment

Co-authored-by: Tomasz Drwięga <[email protected]>
Co-authored-by: adoerr <[email protected]>

* Fix bench bot

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=pallet_membership --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/

* Fix weights files

* Fix'

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=pallet_membership --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/

Co-authored-by: Parity Benchmarking Bot <[email protected]>

* Implement PVF validation host

* WIP: Diener

* Increase the alloted compilation time

* Add more comments

* Minor clean up

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* Fix pruning artifact removal

* Fix formatting and newlines

* Fix the thread pool

* Update node/core/pvf/src/executor_intf.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Remove redundant test declaration

* Don't convert the path into an intermediate string

* Try to workaround the test failure

* Use the puppet_worker trick again

* Fix a blip

* Move `ensure_wasmtime_version` under the tests mod

* Add a macro for puppet_workers

* fix build for not real-overseer

* Rename the puppet worker for adder collator

* play it safe with the name of adder puppet worker

* Typo: triggered

* Add more comments

* Do not kill exec worker on every error

* Plumb Duration for timeouts

* typo: critical

* Add proofs

* Clean unused imports

* Revert "WIP: Diener"

This reverts commit b9f54e513366c7a6dfdd117ac19fbdc46b900b4d.

* Sync version of wasmtime

* Update cargo.lock

* Update Substrate

* Merge fixes still

* Update wasmtime version in test

* bastifmt

Co-authored-by: Bastian Köcher <[email protected]>

* Squash spaces

* Trailing new line for testing.rs

* Remove controversial code

* comment about biasing

* Fix suggestion

* Add comments

* make it more clear why unwrap_err

* tmpfile retry

* proper proofs for claim_idle

* Remove mutex from ValidationHost

* Add some more logging

* Extract exec timeout into a constant

* Add some clarifying logging

* Use blake2_256

* Clean up the merge

Specifically the leftovers after removing real-overseer

* Update parachain/test-parachains/adder/collator/Cargo.toml

Co-authored-by: Andronik Ordian <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Andronik Ordian <[email protected]>

* proper executor/block type for benchmarks and try-runtime

* Fix

* update benches as well to latest work

* Remvoe dbg

Companion for https://github.com/paritytech/substrate/pull/8143

* Template

* Midway..

* Make all 3 runtimes work

* Update node/service/Cargo.toml

* undo brach updates

* fix with latest version

* Update runtime/kusama/src/lib.rs

Co-authored-by: Shawn Tabrizi <[email protected]>

* Update runtime/rococo/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Final nits

* Update runtime/kusama/src/lib.rs

Co-authored-by: Guillaume Thiolliere <[email protected]>

Co-authored-by: Shawn Tabrizi <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Guillaume Thiolliere <[email protected]>

* add tracing to approval voting

* notify if session info is not working

* add dispute period to chain specs

* propagate genesis session to parachains runtime

* use `on_genesis_session`

* protect against zero cores in computation

* tweak voting rule to be based off of best and add logs

* genesis configuration should use VRF slots only

* swallow more keystore errors

* add some docs

* make validation-worker args non-optional and update clap

* better tracing for bitfield signing and provisioner

* pass amount of bits in bitfields to inclusion instead of recomputing

* debug -> warn for some logs

* better tracing for availability recovery

* a little av-store tracing

* bridge: forward availability recovery messages

* add missing try_from impl

* some more tracing

* improve approval distribution tracing

* guide: hold onto pending approval messages until NewBlocks

* Hold onto pending approval messages until NewBlocks

* guide: adjust comment

* process all actions for one wakeup at a time

* vec

* fix network bridge test

* replace randomness-collective-flip with Babe

* remove PairNotFound

* Wasm caching

* Fix compilation errors

* Rename `cache_path` to `cache_base_path`

* "Update Substrate"

* Fix check-web-wasm build

Co-authored-by: parity-processbot <>

* Telemetry API struct renamed

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Adapt service creation to new substrate API

* Fix test

* WIP

* Revert "WIP"

This reverts commit 816a3633e91abc943b12b2bfa77ce98b959e78b2.

* WIP

* Adapt to current code

* Fix tests

* Yet another fix

* CLEANUP

* WIP

* WIP

* WIP

* Adapt code to changes on substrate

* Adapt code

* Introduce kick.

* Fixes

* WIP

* WIP

* WIP

* WIP

* Bump

* Update sp-io

* WIP

Co-authored-by: Gav Wood <[email protected]>
Co-authored-by: Parity Benchmarking Bot <[email protected]>

* update futures to 0.3.9

* companion changes for generic error types

* improve errors, keep error sources

* command compile fix

* use the correct Error type, add annotation

* avoid async { fut.await.map_err(E::Variant) } with TryFuture

* fix cargo lock, bump futures to 0.3.10

futures 0.3.9 got yanked

* chore cargo update -p sp-io

* minor quirk

* make rustc 1.48 happy by bouning the error more tightly

* be more explicit

* cargo lock futures

* Companion for Substrate#7795

https://github.com/paritytech/substrate/pull/7795

* Fix missing test

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Companion for Substrate#7775

https://github.com/paritytech/substrate/pull/7775

* "Update Substrate"

Co-authored-by: parity-processbot <>

Co-authored-by: Pierre Krieger <[email protected]>
Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* remove low information density error doc comments

* another round of error dancing

* fix compilation

* remove stale `None` argument

* adjust test, minor slip in command

* only add AvailabilityError for full node features

* another None where none shuld be

* include new parameter in test `Configuration`

* update calls to `init_logger`

* "Update Substrate"

* cargo update -p sp-io

Co-authored-by: Matt <[email protected]>
Co-authored-by: parity-processbot <>

* *: Update authority discovery and remove WorkerConfig

With https://github.com/paritytech/substrate/pull/7545 the authority
discovery module queries and publishes addresses on an exponentially
increasing interval. Doing so should make custom configurations
obsolete, as operations are retried in a timely fashion in the first
minutes.

* */Cargo.{lock,toml}: Point to mxinden substrate auth-disc-timing

* Revert "*/Cargo.{lock,toml}: Point to mxinden substrate auth-disc-timing"

This reverts commit 0785943a1e377454f088814ef20f4432de09da7a.

* "Update Substrate"

* Revert ""Update Substrate""

This reverts commit 377b221e1853b2c383f0c416d686535b545796cb.

* Cargo.lock: Manual Substrate update

* node/test/service/src/lib: Remove unused import

* parachain/test-parachains/adder: Remove unused import

Co-authored-by: parity-processbot <>

* Adds integration test based on adder collator

This adds an integration test for parachains that uses the adder
collator. The test will start two relay chain nodes and one collator and
waits until 4 blocks are build and enacted by the parachain.

* Make sure the integration test is run in CI

* Fix wasm compilation

* Update parachain/test-parachains/adder/collator/src/lib.rs

Co-authored-by: Sergei Shulepov <[email protected]>

* Update cli/src/command.rs

Co-authored-by: Sergei Shulepov <[email protected]>

* Fixes bug that collator wasn't sending `Declare` message

* Set authority discovery config

* Fixes bug that collator wasn't sending `Declare` message

* Adds real overseer feature and makes the wasm_validation fail with a
proper error

* Adds README

* Remove debug stuff

* Add feature

* Make adder collator use the correct parent when building a new block

* start working on building the real overseer

Unfortunately, this fails to compile right now due to an upstream
failure to compile which is probably brought on by a recent upgrade
to rustc v1.47.

* fill in AllSubsystems internal constructors

* replace fn make_metrics with Metrics::attempt_to_register

* update to account for #1740

* remove Metrics::register, rename Metrics::attempt_to_register

* add 'static bounds to real_overseer type params

* pass authority_discovery and network_service to real_overseer

It's not straightforwardly obvious that this is the best way to handle
the case when there is no authority discovery service, but it seems
to be the best option available at the moment.

* select a proper database configuration for the availability store db

* use subdirectory for av-store database path

* apply Basti's patch which avoids needing to parameterize everything on Block

* simplify path extraction

* get all tests to compile

* Fix Prometheus double-registry error

for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476:

```rust
Some(registry) => Self::try_register(registry).map_err(|err| {
	eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err);
	err
}),
```

That pointed out where the registration was failing, which led to
this fix. The test still doesn't pass, but it now fails in a new
and different way!

* authorities must have authority discovery, but not necessarily overseer handlers

* fix broken SpawnedSubsystem impls

detailed logging determined that using the `Box::new` style of
future generation, the `self.run` method was never being called,
leading to dropped receivers / closed senders for those subsystems,
causing the overseer to shut down immediately.

This is not the final fix needed to get things working properly,
but it's a good start.

* use prometheus properly

Prometheus lets us register simple counters, which aren't very
interesting. It also allows us to register CounterVecs, which are.
With a CounterVec, you can provide a set of labels, which can
later be used to filter the counts.

We were using them wrong, though. This pattern was repeated in a
variety of places in the code:

```rust
// panics with an cardinality mismatch
let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?;
my_counter.with_label_values(&["succeeded"]).inc()
```

The problem is that the labels provided in the constructor are not
the set of legal values which can be annotated, but a set of individual
label names which can have individual, arbitrary values.

This commit fixes that.

* get av-store subsystem to actually run properly and not die on first signal

* typo fix: incomming -> incoming

* don't disable authority discovery in test nodes

* Fix rococo-v1 missing session keys

* Update node/core/av-store/Cargo.toml

* try dummying out av-store on non-full-nodes

* overseer and subsystems are required only for full nodes

* Reduce the amount of warnings on browser target

* Fix two more warnings

* InclusionInherent should actually have an Inherent module on rococo

* Ancestry: don't return genesis' parent hash

* Update Cargo.lock

* fix broken test

* update test script: specify chainspec as script argument

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/service/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* node/service/src/lib: Return error via ? operator

* post-merge blues

* add is_collator flag

* prevent occasional av-store test panic

* simplify fix; expand application

* run authority_discovery in Role::Discover when collating

* distinguish between proposer closed channel errors

* add IsCollator enum, remove is_collator CLI flag

* improve formatting

* remove nop loop

* Fix some stuff

Co-authored-by: Andronik Ordian <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Fedor Sakharov <[email protected]>
Co-authored-by: Robert Habermeier <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Max Inden <[email protected]>

* Add sync_state_items

* Add SyncStateRpc

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Companion for Substrate#7318

* "Update Substrate"

Co-authored-by: parity-processbot <>

This reverts commit a7b6c91f.

The authority discovery module was initially enabled by default on
validator and sentry nodes with commit 59f5eb4c. This change was later on
reverted in a7b6c91f. With this commit the authority discovery module is
again enabled by default.

The collator is now enabled in a different way and we don't require this
argument anymore.

* Remove old service, 3rd try

i.e.
Revert "Revert "Remove Old Service, 2nd try (#1732)" (#1758)"

This reverts commit 9a0f08bf

.

Closes #1757.

We now have some evidence that the polkadot validator was producing
blocks after all; the reason the blocks_constructed metric was 0 was
that as a new metric it hadn't yet been incorporated into that
branch's codebase. See
https://github.com/paritytech/polkadot/issues/1757#issuecomment-700977602

As this PR is based on a newer `master` branch than the previous one,
that should hopefully no longer be an issue.

* paras trait now has an Origin type

* initial work running a two node local net

* use the right incantations so the nodes produce blocks together

* improve internal documentation

Co-authored-by: Bastian Köcher <[email protected]>

This reverts commit 96f26155.

* Restore "Remove service, migrate all to service-new (#1630)"

i.e.
Revert "Revert "Remove service, migrate all to service-new (#1630)" (#1731)"

This reverts commit b4457f55

.

This allows us to get the changeset from #1630 into a new branch
which can be merged sometime in the future after appropriate burnin
tests have completed.

* remove ',)' from codebase outside of macros

* restore bdfl-preferred formatting

* attempt to improve destructuring formatting

* rename polkadot-service-new -> polkadot-service

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* remove unused import

* Update runtime/rococo-v1/README.md

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Andronik Ordian <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

This reverts commit d531ba56.