* client/network-gossip: Move sink IO outside of state_machine

`ConsensusGossip` is supposed to be a deterministic state machine.
`GossipEngine` wrapping `ConsensusGossip` should handle IO operations.

This commit moves the `message_sink` IO operations to `GossipEngine`.
More specifically on incoming messages a `GossipEngine` calls
`ConsensusGossip::on_incoming` to validate and register the messages.
`ConsensusGossip` returns the valid messages which are then forwarded by
`GossipEngine` to the upper layer via the `message_sinks`.

* client/network-gossip: Adjust and extend tests

* Update client/network-gossip/src/bridge.rs

Co-authored-by: Benjamin Kampmann <[email protected]>

Co-Authored-By: thiolliere <[email protected]>

Co-Authored-By: Pierre Krieger <[email protected]>

* Improve tracing implementation

* Enable tracing in runtime interfaces

* Switch to `TRACE` level

* Add transaction_version

* Semantic versioning for runtimes

* Move new field to bottom

* Versioning

* Runtime versioning stuff.

* Fix test

* Adds tests and fixes bugs

* Bump runtime

Co-authored-by: Bastian Köcher <[email protected]>

* make new contructor

* add metrics to txpool

* fix review

* fix doc comment

* change to counters

* Update client/transaction-pool/src/metrics.rs

Co-Authored-By: Max Inden <[email protected]>

* Update client/transaction-pool/src/metrics.rs

Co-Authored-By: Max Inden <[email protected]>

* Update client/transaction-pool/src/metrics.rs

Co-Authored-By: Max Inden <[email protected]>

* Update client/transaction-pool/src/lib.rs

Co-Authored-By: Max Inden <[email protected]>

* Update client/transaction-pool/src/lib.rs

Co-Authored-By: Max Inden <[email protected]>

* use dedicated wrapper

Co-authored-by: Max Inden <[email protected]>

* Clean phragmen API and equalise()

* Stabilize new api

* Fix phragmen fuzzers

* More fixes

* Make fuzzers reproducible

* improvements

* Make equalize update assignments as well.

* total function for staked_assignment.

* Fix fuzzer build

* remvoe TODO

* Fix a bunch more.

* clean stray debug stuff

* Update primitives/phragmen/src/lib.rs

Co-Authored-By: thiolliere <[email protected]>

* fix range function

* fix number generator

Co-authored-by: thiolliere <[email protected]>

* Validate transaction timeouts

* Add timing

* Add tracing to transaction validation.

* Fix docs.

Co-authored-by: NikVolf <[email protected]>

* create parallel tasks extension

* make type system happy

* basic externalities

* test for dynamic extensions

* batching test

* remove premature verify_batch

* shnschnorrkel batch

* alter test

* shnschnorrkel test

* executive batching

* some docs

* also multi/any signatgures

* error propagation

* styling

* make verification extension optional

* experimental ed25519 parallelization

* some merge fallout

* utilize task executor

* merge fallout

* utilize task executor more

* another merge fallout

* feature-gate sp-io

* arrange toml

* fix no-std

* sr25519 batching and refactoring

* add docs

* fix name

* add newline

* fix block import test

* long sr25519 test

* blocking instead of parking

* move everything in crypto

* return batch_verify to check :)

* use condvars

* use multi-threaded executor for benches

* don't call via host interface

* try no spawning

* add true

* cleanup

* straighten batching

* remove signature check from this test (?)

* remove now pointless test

* remove another now useless test

* fix warnings

* Revert "remove another now useless test"

This reverts commit bbdec24b.

* rethink the sp-io-part

* Revert "remove now pointless test"

This reverts commit 4d553066

.

* fix wording

* add  wording

* add todo and fix

* return check and fix

* add logging in sp-io

* Update primitives/io/src/batch_verifier.rs

Co-Authored-By: cheme <[email protected]>

* address review and use std condvar

* account for early exit

* address reivew

* address review

* more suggestions

* add docs for batch verification

* remove unused

* more review suggestions

* move to sp-runtime

* add expects

* remove blocks

* use entry

* Update primitives/io/src/batch_verifier.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update primitives/externalities/src/extensions.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* update overlooked note

* remove stupid return

* Update primitives/io/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update primitives/io/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* fix wording

* bump spec_version

Co-authored-by: cheme <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

This release includes a fix for an out of bound access that was reported
by one of the validators.

Co-Authored-By: Kian Paimani <[email protected]>

* Introduce trait

* The trait

* Generic

* Basic impls.

* Remove unneeded bounds

* Minor changes

* Switch over to the new DB trait

* Integrated parity-db and added CLI for db selection

* Default impl.

* Fix logs.

* Started integrating subdb

* Apply suggestions from code review

Co-Authored-By: Cecile Tonglet <[email protected]>

* Apply suggestions from code review

Co-Authored-By: Nikolay Volf <[email protected]>

* Enable subdb

* Bump parity-db

* Fixed CLI macro

* Fixed browser build

* Fixed features

* Sort out features

* Use parity-db from crates.io

* Typo

Co-authored-by: arkpar <[email protected]>
Co-authored-by: Cecile Tonglet <[email protected]>
Co-authored-by: Nikolay Volf <[email protected]>

Co-Authored-By: Bastian Köcher <[email protected]>

* bumping version

* cargo update

* adding changelog

* Bump trie-db

* Bumped version in Cargo.toml

* add fixed128

* fix build

* re-export

* fix test

* saturating_pow

* Update primitives/arithmetic/src/fixed128.rs

Co-Authored-By: Shawn Tabrizi <[email protected]>

Co-authored-by: Shawn Tabrizi <[email protected]>

* Native version text was borked.

* Unfix benchmarking.

* Update to libp2p 0.18

* Update Cargo.lock

* Also bump in browser-utils

* in-mem state for benchmarks

* Use caching state

* Update Cargo.lock

Co-authored-by: Shawn Tabrizi <[email protected]>

* Add new RPC method to get the chain type

This adds a new RPC method to get the chain type of the running chain.
The chain type needs to be specified in the chain spec. This should make
it easier for tools/UI to display extra information without needing to
rely on parsing the chain name.

* Update client/rpc-api/src/system/mod.rs

Co-Authored-By: Tomasz Drwięga <[email protected]>

* Primitive crate

* Feedback

Co-authored-by: Tomasz Drwięga <[email protected]>

* Preserve a single wasmtime instance.

* Sketch of wasm instance reusing.

* Refactor and docs.

* Rename state_snapshot to util module.

* Renaming.

* Comments.

* Error handling

* More fixes.

* Adapt to rust-libp2p#1440.

* Further adapt to libp2p/master.

* Update to libp2p-0.17

* Finishing touches.

* Remove stray TODO.

* Incorporate review feedback.

* Remove unused import.

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Rename IntoConfiguration to CliConfiguration

* Renamed into_configuration to create_configuration

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Move keystore params to its own module

* Use in-memory keystore even for build-spec

* Enforce proper value for node name

* dev_key_seed

* Telemetry endpoints

* rustfmt

* Converted all RunCmd

* rustfmt

* Added export-blocks

* Missed something

* Removed config_path in NetworkConfiguration (not used)

* Fixed warnings

* public_addresses is used but never set, keeping it

* Merge Configuration.node and NetworkConfiguration.node_name

...because they are the same thing

* Added: import-blocks

* Adding a proc_macro to help impl SubstrateCli

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Re-export spec_factory from sc_cli

* Re-added all the commands

* Refactored node_key_params

* Fixed previous refucktoring

* Clean-up and removed full_version()

* Renamed get_is_dev to not confuse with Configuration field

* Fixed sc-cli-derive example

* Fixing tests

* Fixing tests and removing some (will re-add later)

* Fixing more tests

* Removes the need of type parameter

* Converting bin/node and simplifying API

* Converting more

* Converting last command

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Fixing tests and added default for WasmExecutionMethod

* Fixing stuff

* Fixed something I broke oops

* Update Cargo.lock

* Moving things around

* Convert everything to Result

* Added new macros to simplify the impl of CliConfiguration

* Added a macro to generate CliConfiguration automatically for subcommands

* Revert... too many macros (this one is not really useful)

This reverts commit 9c516dd3

.

* Renamed is_dev to get_is_dev

Good enough for now

* Fixed name roles (this is plural, not singular)

* Clean-up

* Re-export NodeKeyConfig and TelemetryEndpoints from sc_service

* Improve styling/formatting

* Added copyrights

* Added doc and fixed warnings

* Added myself to code owners

* Yes it is needed according to the history

* Revert formatting

* Fixing conflict

* Updated build.rs

* Cargo.lock

* Clean-up

* Update client/cli-derive/Cargo.toml

Co-Authored-By: Seun Lanlege <[email protected]>

* Fail if using proc_macro and build.rs is not set properly

* Dropped all get_ in front of methods

* Clean-up

* Fixing proc macro missing env var

* Get the configuration inside the Runtime (needed for polkadot)

* Clean-up

* Get is_dev from argument like the others

* Get chain ID instead of chain spec from shared params

* &self is passed to spec_factory/load_spec

* Wrong text

* Fix example

* Officialize macro and made a cool doc

* Renamed spec_factory to load_spec (substrate_cli_configuration)

* Removed not so useful ChainSpec

* Renamed SubstrateCLI to SubstrateCli

* Added changelog for impl_version being full now

* Renamed Runtime to Runner

* Update changelog to show example

* Removed option on database cache size

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Fix on removal of option

* typo

* Clean-up imports

* Added info in Cargo.toml

* typo

* remarks

* Moved function for build.rs to substrate-build-script-utils

* Fixed example & test of cli-derive

* Moved function for build.rs to substrate-build-script-utils

* Renamed substrate_cli_configuration to substrate_cli oops

It implements SubstrateCli not CliConfiguration!

* Added documentation and wrapper macro

* Removed option on database cache size

* Removed option on database cache size

* Clean-up

* Reduce risk of errors due to typos

* Removed option on database cache size

* Added NOTE as suggested

* Added doc as suggested

* Fixed test

* typo

* renamed runtime to runner

* Fixed weird argument

* More commas

* Moved client/cli-derive to client/cli/derive

* Added 7 tests for the macros

* Improve error message

* Upgrade assert_cmd

* Fixing missing stuff

* Fixed unused import

* Improve SubstrateCli doc

* Applied suggestions

* Fix and clean-up imports

* Started replacing macros WIP

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Started removing substrate_cli

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81


Parent branch: origin/master

* fixed bug introduced while refactoring

* Renamed NetworkConfigurationParams to NetworkParams for consistency sake

* Fixed test

* Update client/cli/src/commands/runcmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/runcmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/check_block_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update bin/node/cli/src/command.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update bin/node/cli/src/command.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Revert "Update client/cli/src/commands/export_blocks_cmd.rs"

This reverts commit 59067769.

* Revert "Update client/cli/src/commands/check_block_cmd.rs"

This reverts commit f705f42b.

* Revert "Update client/cli/src/commands/export_blocks_cmd.rs"

This reverts commit 8d57c055.

* Revert "Update client/cli/src/commands/runcmd.rs"

This reverts commit 93e74cf5.

* Revert "Update client/cli/src/commands/runcmd.rs"

This reverts commit 11d527ba

.

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/import_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Changed ::sc_cli to $crate in the macro

* fixed tests

* fixed conflicts

* Fixing test

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/params/pruning_params.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Remove comment as suggested

* Apply suggestion

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update utils/frame/benchmarking-cli/src/command.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/params/pruning_params.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/params/node_key_params.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/params/network_params.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/config.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Added doc

* Fixed error introduced after applying suggestion

* Revert "Update client/cli/src/params/pruning_params.rs"

This reverts commit 0574d06a

.

* Print error

* Apply suggestions from code review

* Remove useless Results

* Fixed CI failing on polkadot approval

Co-authored-by: Seun Lanlege <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* Add initial report_ofence bench.

* Remove unused imports

* Style nit

* Add nominators

* Remove logs.

* Nits.

* Add nominators param.

* Reorg, comment.

* Remove whitespaces.

* Apply review suggestion: move benchmark to own crate.

* Remove import.

* Remove line.

* Add feature flag.

* Pass can_report.

* Cleaning up.

* More cleaning

* Update to latest staking

* generate tests for benchmarking

* add tests, fix warnings

* starting on democracy

* impl_benchmark_tests

* Way more readable

* add test feature flag (does this work?)

* Fix `successful_origin` impl

* democracry benchmark tests

* Fix example benchmarks, add tests

* identity benchmark tests

* Update im-online benchmark tests

* try to add session benchmarking tests (problem with mock)

* staking and timestamp

* add test for treasury, issue with dynamic contains

* utility

* Vesting

* test instead of check

* hide until we figure out what is wrong

* add docs

* close code

* Create custom mock for session-pallet-benchmarking

* Use refcell pattern

* make un-pub

* test-linux-stable includes `runtime-benchmarks` feature

* Revert "test-linux-stable includes `runtime-benchmarks` feature"

This reverts commit a2dab38a.

* run tests in `--release`

* undo balance change

* build wasm

This PR refactors the metrics measuring and Prometheus exposing entity in sc-service into its own submodule and extends the parameters it exposes by:

- system load average (over one, five and 15min)
- the TCP connection state of the process (lsof), refs #5304
- number of tokio threads
- number of known forks
- counter for items in each unbounded queue (with internal unbounded channels)
- number of file descriptors opened by this process (*nix only at this point)
- number of system threads (*nix only at this point)

refs #4679

Co-authored-by: Max Inden <[email protected]>
Co-authored-by: Ashley <[email protected]>

* Simple Payouts

* explicit test for out of order claiming

* Add `payout_all` benchmark

* Fix merge

* add docs

* change event to controller

* Fix timestamp test warnings

* Revert "change event to controller"

This reverts commit 5d4a9783

.

* Update Reward event doc

* Add "non-production" test

* add unlock chunk to test

* fix merge

* End payout early if no reward points

* payout_validator -> payout_stakers

* bring back payout nominator/validator, but limit their use to before migration era

* Add test for before migration

* New payout works for the era that we migrate

* Fix logic, check that migration era works

* Migrate Era tests (copypasta)

* Move comment

* Add mock back to external functions

* Fixes based on review from gui

* Update Cargo.lock

* Update Cargo.lock

* small docs update

Co-authored-by: joepetrowski <[email protected]>

* custom benchmark runner

* add license preamble

* delete old benchmarks

* update toml

* imporove macro

* tabs

* add size

* size fixes

* add docs for cli

* Initial draft of the logic

* Build and tests

* Make work with new initialize infratructure.

* Update frame/scheduler/src/lib.rs

Co-Authored-By: Marcio Diaz <[email protected]>

* Update frame/scheduler/src/lib.rs

Co-Authored-By: Marcio Diaz <[email protected]>

* Update frame/scheduler/src/lib.rs

Co-Authored-By: Marcio Diaz <[email protected]>

* Update frame/scheduler/src/lib.rs

Co-Authored-By: Marcio Diaz <[email protected]>

* Fix test

* Update frame/scheduler/src/lib.rs

Co-Authored-By: Marcio Diaz <[email protected]>

* Rejig interface to make it more useful for democracy.

* Try to get democraxy module to make use of scheduler.

* Make democracy use scheduler.

* Use actual max weight for enactent

* Remove TODO

* Fix runtime build

* Minor cleanup

* Fix scheduler.

* Fix benchmarks

* Fix

* Fix

* Fix

* More bench fixes

* Fix

* Fix.

* Add more bench constants.

* Fix cancel_queued bench.

* Fix test comment.

* Update frame/scheduler/src/lib.rs

Co-Authored-By: Marcio Diaz <[email protected]>

Co-authored-by: Marcio Diaz <[email protected]>

* Reset by removing pos-genesis changes

* CLI option for DB cache size

* Update Cargo.lock

Co-authored-by: Shawn Tabrizi <[email protected]>