* test/malus: craft the first maliciously disputing actor

* initial draft

* Add Dockerfile and instructions how to use it to build malus image locally

* Forgot one flag for the build cmd

* we are not docker specific, we are happy to use any containerruntime

* shuffle things around

* add initial tera based integration test

* chores

* fixins

* simple setup to start

* other samples (WIP)

* add Docker version with cargo-chef

* update substarte and small change of orders in commands in the container file

* metrics one

* fmt

* minor

* fixin

* fix metric names

* -d

* add open gauge

* fmt

* spellcheck

* fix test

* adjust to changed error messages

* refactor, more malus impls

* more malus changes

* foo

* minor cleanup

* suggest garbage candidate

* chore

* fix suggest garabge malus

* malus: back garbage candidate

* cargo lock

* re-introduce metrics

* chore: cargo fmt

* undoe 1.54.0 output, CI uses 1.53.0 rustc

* update location of js types

* Fix trybuild

* add tag to image name also; this will be replaced in the prod version

* Tests fixed

* add some fix me

* add dockerfile for ci

* Add docker file for malus for ci

* use variables in .toml file

* add chnages for malus test

* some fixes

* some more fixes

* Update .gitlab-ci.yml

* add local build for polkadot and malus

* some fixes

* enable disputes feature in CI

* ok, ok

* rename: MsgFilter -> MessageInterceptor

* remove TODO that would not have worked

* intercept

* refactor

* fix README and containers

* fix

* chore: cargo fmt

* avoid some more malus-$VARIANT references

* fix argument order

* chore: add about

* Update sanity check in relay chain selection

* fix order, add dispute-unavailable-block malus

* fixup: avoid underflow issue

* it's all u32

* fix conditional use

* Revert "it's all u32"

This reverts commit 6b3ae25bfd0bbf0b51d90d743642a75a4a815d6e.

* Revert "fixup: avoid underflow issue"

This reverts commit 336cbe2938e9720f870d37d8feeab7ca69200f47.

* Revert "Update sanity check in relay chain selection"

This reverts commit 970647f35e1116136e12fd91cd9f2fb7e18ad28d.

* update the malus bin

* Update node/malus/integrationtests/0003-dispute-unavailable-block.feature

Co-authored-by: Andronik Ordian <[email protected]>

* add some FIXME reminders

* update path to index.js

* Update .gitlab-ci.yml

* Update node/malus/integrationtests/0001-dispute-valid-block.toml

* try 1: make malus test run

* chore: cargo fmt

* temporary fix

* use subcommand syntax from latest gurke

* cargo +nightly fmt

* add collator to a a test

* docs: add env vars to README

* update ci to run dispute-valid-block test

* needs the polkadot image

* Fix path for nodejs container

* post merge fix

* download proper dir  containg configs for malus test

* update the malus ci job

* rm a whitespace

* temp build for malus

* use correct build command for temp malus

* remove subcommands for now

* set max validators per core in the default HostConfig

* tabs

* update beefy

* fixup

* fixup II

* make one variant compile

* make other variants compile

* revert changes to chain_spec

* fmt

* build malus image from polkadot-test-malus again

* revert unrelated changes

* try fixing build-malus job

* Revert "remove subcommands for now"

This reverts commit 5d8292bc49252124937affec4b7c28181a5deb7e.

* try fixing build-malus job II

* MVP working dispute-ancestor

* renames

* fix PVF execution on malus

* fix test

* fix typo

* fmt

* checkmate

* try something

* make it actually work

* some tweaks to 01 feature test

* fmt

* sleep a bit more

* complete wococoization

* some tweaks to 01 feature test

* typo fix

* use correct metric names

* fix

* ffs

* .

* try some rearrangement

* Attempt to wait till initial node bootstrap in test

* Fix test syntax

* Run malus tests with v2 script

* Proper symlink created

* simnet v14

* add zombienet tests

* add zombie net test - draft

* add more tests to dispute suite

* add within to fix tests

* replace test directory and start test migration

* migrate all the tests

* add timeout to tests

* reduce debug

* make easy to test in dev

* set appropriated debug

* use image from ci

* fix config for test

* set images from ci

* fix config

* add COLIMAGE env

* tweek tests

* reduce debug

* typo

* wip, migrate old test to zombie-net

* adjunt test config for zombie-net

* run mauls 0001 test only

* clean and setup smoke-test in zombie-net

* add extra time to assertinons

* clean code to merge and improve README

* add info to access logs

* improved readme

* merge master and resolve conflicts

* Update zombienet_tests/README.md

Co-authored-by: Bernhard Schuster <[email protected]>

* clean and consolidate zombienet name

* change runner in gitlab

* add comment explain why we use wococo

* change tag for runner

* remove unused tests

* remove dup Dockerfile and update description

* fmt

* fix compilation post-merge

* fmt

* cut me Some slack

Co-authored-by: Bernhard Schuster <[email protected]>
Co-authored-by: radupopa2010 <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Anton Gavrilov <[email protected]>
Co-authored-by: Andronik Ordian <[email protected]>
Co-authored-by: Lldenaurois <[email protected]>

* enable beefy explicitly

* update beefy cli flag docs

* Companion PR for ss58 crate

* cargo fmt

* Rename

* into()

* into()

* updating substrate (ideally use bot merge)

* introduce the OverseerConnector, use it

* introduce is_relay_chain to RelayChainSelection

* Update node/service/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

* avoid the deferred setting of `is_relay_chain` in `RelayChainSelection`

* positive assertion is not mandated, only the negative one, to avoid a stall

* cleanup: overseer residue

* spellcheck

* fixin

* groundwork to obsolete Overseer::new and AllSubsystemsGen proc-macro

* Now all malus & tests can be ported to the builder pattern.

Obsoletes `Overseer::new`, `AllSubsystemsGen` derive macro, `AllSubsystems`.

* spellcheck

* adjust tests, minor fixes

* remove derive macro AllSubsystemsGen

* add forgotten file dummy.rs

* remove residue

* good news everyone!

* spellcheck

* address review comments

* fixup imports

* make it conditional

* fixup docs

* reduce import

* chore: fmt

* chore: fmt

* chore: spellcheck / nlprules

* fixup malus variant-a

* fmt

* fix

* fixins

* pfmt

* fixins

* chore: fmt

* remove expanded overseer generation

* tracing version

* Update node/network/statement-distribution/src/lib.rs

Co-authored-by: Robert Habermeier <[email protected]>

* use future::ready instead

* silence warning

* chore: fmt

Co-authored-by: Andronik Ordian <[email protected]>
Co-authored-by: Robert Habermeier <[email protected]>

* make polkadot-runtime optional feature

* sprinkle some cfg statements

* ok

* ok, ok

* add CI check

* set -e

* chmod +x

* fixes

* fmt

* nicer compile errors

* Update outdated comments

Co-authored-by: parity-processbot <>
Co-authored-by: kianenigma <[email protected]>

* compile try-runtime

* cargo fmt lol

Companion PR for 'Make choosing an executor an explicit part of service construction' (#9525) (#3615)

* Companion PR

* Update a few files

* Run cargo fmt

* Do better at renaming things

* More renamings

* More fixes

* oops

* Fix simnet problems

* fix compilation

* Update substrate

Co-authored-by: thiolliere <[email protected]>

* Remove telemetry external transport member

* Make it compile

* Remove more

Co-authored-by: Bastian Köcher <[email protected]>

* cargo +nightly fmt

* add cargo-fmt check to ci

* update ci

* fmt

* fmt

* skip macro

* ignore bridges

* CI: add spellcheck

* revert me

* CI: explicit command for spellchecker

* spellcheck: edit misspells

* CI: run spellcheck on diff

* spellcheck: edits

* spellcheck: edit misspells

* spellcheck: add rules

* spellcheck: mv configs

* spellcheck: more edits

* spellcheck: chore

* spellcheck: one more thing

* spellcheck: and another one

* spellcheck: seems like it doesn't get to an end

* spellcheck: new words after rebase

* spellcheck: new words appearing out of nowhere

* chore

* review edits

* more review edits

* more edits

* wonky behavior

* wonky behavior 2

* wonky behavior 3

* change git behavior

* spellcheck: another bunch of new edits

* spellcheck: new words are koming out of nowhere

* CI: finding the master

* CI: fetching master implicitly

* CI: undebug

* new errors

* a bunch of new edits

* and some more

* Update node/core/approval-voting/src/approval_db/v1/mod.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Update xcm/xcm-executor/src/assets.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Apply suggestions from code review

Co-authored-by: Andronik Ordian <[email protected]>

* Suggestions from the code review

* CI: scan only changed files

Co-authored-by: Andronik Ordian <[email protected]>

* bump a bunch of deps in parity-common

* update Substrate

* fix browser check

Co-authored-by: parity-processbot <>

* update rustls to be compat with jsronrpsee 0.2.0

* Stub try-cli when feat not enabled

* update Substrate

Co-authored-by: parity-processbot <>

* polkadot-service: Make native runtime configurable

This pull requests adds support for configuring the native runtimes used
by polkadot-service. While this whole pr doesn't change that much for
polkadot, besides not having the light-node enabled for the default
polkadot binary. However, downstream projects (parachains) will have a
much better compile time. In cumulus for example the `cargo test --all
--release` is about 4m faster to compile.

* Fixes

* Fix

* Enable rococo-native

* Fix light client

* 🤦

* Fixes

Instead of relying on the name of the chain spec file, we now use the
`is_*` methods for this.

* Add --disable-beefy CLI flag

* Update cli/src/command.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Update node/test/service/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update parachain/test-parachains/adder/collator/src/main.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update cli/src/command.rs

Co-authored-by: André Silva <[email protected]>

* Update cli/src/cli.rs

Co-authored-by: André Silva <[email protected]>

* Update cli/src/cli.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update cli/src/cli.rs

Co-authored-by: André Silva <[email protected]>

Co-authored-by: Andronik Ordian <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: André Silva <[email protected]>

* Add `rococo-dev` chain option

* Add `wococo-dev` CLI option

* Sort Chain ID match arms

* runtime: remove mmr and beefy from westend runtime

* runtime: westend: remove pallet_beefy config

* node: only start beefy gadget on rococo

* node: remove beefy keys from westend chain spec

* node: add wococo testnet chain spec

* node: add comments about beefy gadget task

* runtime: update wococo chainspec

* Remove stale comment

Co-authored-by: Tomasz Drwięga <[email protected]>
Co-authored-by: adoerr <[email protected]>

* Fix bench bot

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=pallet_membership --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/

* Fix weights files

* Fix'

* cargo run --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=pallet_membership --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/

Co-authored-by: Parity Benchmarking Bot <[email protected]>

* Implement PVF validation host

* WIP: Diener

* Increase the alloted compilation time

* Add more comments

* Minor clean up

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* Fix pruning artifact removal

* Fix formatting and newlines

* Fix the thread pool

* Update node/core/pvf/src/executor_intf.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Remove redundant test declaration

* Don't convert the path into an intermediate string

* Try to workaround the test failure

* Use the puppet_worker trick again

* Fix a blip

* Move `ensure_wasmtime_version` under the tests mod

* Add a macro for puppet_workers

* fix build for not real-overseer

* Rename the puppet worker for adder collator

* play it safe with the name of adder puppet worker

* Typo: triggered

* Add more comments

* Do not kill exec worker on every error

* Plumb Duration for timeouts

* typo: critical

* Add proofs

* Clean unused imports

* Revert "WIP: Diener"

This reverts commit b9f54e513366c7a6dfdd117ac19fbdc46b900b4d.

* Sync version of wasmtime

* Update cargo.lock

* Update Substrate

* Merge fixes still

* Update wasmtime version in test

* bastifmt

Co-authored-by: Bastian Köcher <[email protected]>

* Squash spaces

* Trailing new line for testing.rs

* Remove controversial code

* comment about biasing

* Fix suggestion

* Add comments

* make it more clear why unwrap_err

* tmpfile retry

* proper proofs for claim_idle

* Remove mutex from ValidationHost

* Add some more logging

* Extract exec timeout into a constant

* Add some clarifying logging

* Use blake2_256

* Clean up the merge

Specifically the leftovers after removing real-overseer

* Update parachain/test-parachains/adder/collator/Cargo.toml

Co-authored-by: Andronik Ordian <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Andronik Ordian <[email protected]>

* proper executor/block type for benchmarks and try-runtime

* Fix

* update benches as well to latest work

* Remvoe dbg

Companion for https://github.com/paritytech/substrate/pull/8143

* test: add unit test to catch missing distribution to subsystems faster

* add a simple count

* introduce proc macro to generate dispatch type

* refactor

* refactor

* chore: add license

* fixup unit test

* fixup merge

* better errors

* better fmt

* fix error spans

* better docs

* better error messages

* ui test foo

* Update node/subsystem/dispatch-gen/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/network/bridge/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/subsystem/Cargo.toml

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/subsystem/dispatch-gen/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/subsystem/dispatch-gen/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update node/network/bridge/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

* fix compilation

* use find_map

* drop the silly 2, use _inner instead

* Update node/network/bridge/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Update node/subsystem/dispatch-gen/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* nail deps down

* more into()

* flatten

* missing use statement

* fix messages order

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Andronik Ordian <[email protected]>

* Template

* Midway..

* Make all 3 runtimes work

* Update node/service/Cargo.toml

* undo brach updates

* fix with latest version

* Update runtime/kusama/src/lib.rs

Co-authored-by: Shawn Tabrizi <[email protected]>

* Update runtime/rococo/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Final nits

* Update runtime/kusama/src/lib.rs

Co-authored-by: Guillaume Thiolliere <[email protected]>

Co-authored-by: Shawn Tabrizi <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Guillaume Thiolliere <[email protected]>

* add tracing to approval voting

* notify if session info is not working

* add dispute period to chain specs

* propagate genesis session to parachains runtime

* use `on_genesis_session`

* protect against zero cores in computation

* tweak voting rule to be based off of best and add logs

* genesis configuration should use VRF slots only

* swallow more keystore errors

* add some docs

* make validation-worker args non-optional and update clap

* better tracing for bitfield signing and provisioner

* pass amount of bits in bitfields to inclusion instead of recomputing

* debug -> warn for some logs

* better tracing for availability recovery

* a little av-store tracing

* bridge: forward availability recovery messages

* add missing try_from impl

* some more tracing

* improve approval distribution tracing

* guide: hold onto pending approval messages until NewBlocks

* Hold onto pending approval messages until NewBlocks

* guide: adjust comment

* process all actions for one wakeup at a time

* vec

* fix network bridge test

* replace randomness-collective-flip with Babe

* remove PairNotFound

* Wasm caching

* Fix compilation errors

* Rename `cache_path` to `cache_base_path`

* "Update Substrate"

* Fix check-web-wasm build

Co-authored-by: parity-processbot <>

* Telemetry API struct renamed

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Adapt service creation to new substrate API

* Fix test

* WIP

* Revert "WIP"

This reverts commit 816a3633e91abc943b12b2bfa77ce98b959e78b2.

* WIP

* Adapt to current code

* Fix tests

* Yet another fix

* CLEANUP

* WIP

* WIP

* WIP

* Adapt code to changes on substrate

* Adapt code

* Introduce kick.

* Fixes

* WIP

* WIP

* WIP

* WIP

* Bump

* Update sp-io

* WIP

Co-authored-by: Gav Wood <[email protected]>
Co-authored-by: Parity Benchmarking Bot <[email protected]>

* update futures to 0.3.9

* companion changes for generic error types

* improve errors, keep error sources

* command compile fix

* use the correct Error type, add annotation

* avoid async { fut.await.map_err(E::Variant) } with TryFuture

* fix cargo lock, bump futures to 0.3.10

futures 0.3.9 got yanked

* chore cargo update -p sp-io

* minor quirk

* make rustc 1.48 happy by bouning the error more tightly

* be more explicit

* cargo lock futures

* Companion for Substrate#7795

https://github.com/paritytech/substrate/pull/7795

* Fix missing test

* "Update Substrate"

Co-authored-by: parity-processbot <>

* Companion for Substrate#7775

https://github.com/paritytech/substrate/pull/7775

* "Update Substrate"

Co-authored-by: parity-processbot <>

Co-authored-by: Pierre Krieger <[email protected]>
Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* remove low information density error doc comments

* another round of error dancing

* fix compilation

* remove stale `None` argument

* adjust test, minor slip in command

* only add AvailabilityError for full node features

* another None where none shuld be

* include new parameter in test `Configuration`

* update calls to `init_logger`

* "Update Substrate"

* cargo update -p sp-io

Co-authored-by: Matt <[email protected]>
Co-authored-by: parity-processbot <>

* *: Update authority discovery and remove WorkerConfig

With https://github.com/paritytech/substrate/pull/7545 the authority
discovery module queries and publishes addresses on an exponentially
increasing interval. Doing so should make custom configurations
obsolete, as operations are retried in a timely fashion in the first
minutes.

* */Cargo.{lock,toml}: Point to mxinden substrate auth-disc-timing

* Revert "*/Cargo.{lock,toml}: Point to mxinden substrate auth-disc-timing"

This reverts commit 0785943a1e377454f088814ef20f4432de09da7a.

* "Update Substrate"

* Revert ""Update Substrate""

This reverts commit 377b221e1853b2c383f0c416d686535b545796cb.

* Cargo.lock: Manual Substrate update

* node/test/service/src/lib: Remove unused import

* parachain/test-parachains/adder: Remove unused import

Co-authored-by: parity-processbot <>

* Adds integration test based on adder collator

This adds an integration test for parachains that uses the adder
collator. The test will start two relay chain nodes and one collator and
waits until 4 blocks are build and enacted by the parachain.

* Make sure the integration test is run in CI

* Fix wasm compilation

* Update parachain/test-parachains/adder/collator/src/lib.rs

Co-authored-by: Sergei Shulepov <[email protected]>

* Update cli/src/command.rs

Co-authored-by: Sergei Shulepov <[email protected]>

* Fixes bug that collator wasn't sending `Declare` message

* Set authority discovery config

* Fixes bug that collator wasn't sending `Declare` message

* Adds real overseer feature and makes the wasm_validation fail with a
proper error

* Adds README

* Remove debug stuff

* Add feature

* Make adder collator use the correct parent when building a new block