* in-mem state for benchmarks

* Use caching state

* Update Cargo.lock

Co-authored-by: Shawn Tabrizi <[email protected]>

* Report local force-closing connections to Prometheus

* Also report ping timeouts separately

* Address concerns of #5571

* 128 -> 1024

* add trace and set = 512

* fix doc comment

* client/authority-discovery: Add test ensure termination on network termi

* client/authority-discovery: Terminate when network does

When the dht event stream returns Poll::Ready(None) it is likely due to
the network terminating. When the network terminates due to the node
itself shutting down or due to a fatal error, there is no purpose in
continuing to run the authority discovery module.

* client/authority-discovery/src/lib: Apply suggestions

Co-Authored-By: André Silva <[email protected]>

Co-authored-by: André Silva <[email protected]>

* Move contracts wasm test code into separate files

* Move crypto_hashes into a separate file

* Load wasm code at runtime

* Move wasm files

* Fix wasm formatting

* Update crypto_hashes file

* Add new RPC method to get the chain type

This adds a new RPC method to get the chain type of the running chain.
The chain type needs to be specified in the chain spec. This should make
it easier for tools/UI to display extra information without needing to
rely on parsing the chain name.

* Update client/rpc-api/src/system/mod.rs

Co-Authored-By: Tomasz Drwięga <[email protected]>

* Primitive crate

* Feedback

Co-authored-by: Tomasz Drwięga <[email protected]>

* Preserve a single wasmtime instance.

* Sketch of wasm instance reusing.

* Refactor and docs.

* Rename state_snapshot to util module.

* Renaming.

* Comments.

* Error handling

* More fixes.

There was a bug in determining the correct git files that should be
tracked. The ref extracted from `.git/HEAD` pointed to a relative path
and thus could not be found by cargo and resulted in always recompiling
the binary.

* Add a failing test

* Make test not freeze

* Fix the bug

* Fix spaces

* Fix tests

* Apply suggestions from code review

Co-Authored-By: Toralf Wittner <[email protected]>

* Make sure test doesn't succeed if nothing happened

* Fix build

* Do the events change

Co-authored-by: Toralf Wittner <[email protected]>

* Add verify block to benchmarks macro.

* Update all benchmarks.

* Add tests, add params.

* Should panic.

* ups, add closures

* Update tests.rs

* update macro syntax

* Revert benchmark syntax change

* verify only in tests

* Update tests.rs

* Uncomment staking

* Fix tests for benchmark instance

* Add docs

* Update frame/benchmarking/src/lib.rs

* add trace logs to benchmarks

* Verify returns closure

Co-authored-by: Shawn Tabrizi <[email protected]>

* Configurable Unsigned Priority.

* Use the new builder.

* Fix tests.

* Fix benches.

* Remove unused import.

* Rename for_pallet

* Add more Prometheus metrics

* Update client/network/src/service.rs

Co-Authored-By: Max Inden <[email protected]>

* Update client/network/src/service.rs

Co-Authored-By: Max Inden <[email protected]>

Co-authored-by: Gavin Wood <[email protected]>
Co-authored-by: Max Inden <[email protected]>

* Refactor SignedExtension

* Move DispatchInfo Associated type to Dispatchable
* Bound Call: Dispatchable
* Pass PostDispatchInfo to post_dispatch
* Pass DispatchInfo by reference to avoid clones

* Whitespace fix

Co-Authored-By: Tomasz Drwięga <[email protected]>

* Style changes from code review

Co-Authored-By: Kian Paimani <[email protected]>

* Only decalre in test mod to remove warning

* Deduplicate Call definition

* Bound frame_system::trait::Call by Dispatchable

* Introduce DispatchInfoOf type alias

* Whitespace fix from review

Co-Authored-By: Bastian Köcher <[email protected]>

Co-authored-by: Tomasz Drwięga <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* Adapt to rust-libp2p#1440.

* Further adapt to libp2p/master.

* Update to libp2p-0.17

* Finishing touches.

* Remove stray TODO.

* Incorporate review feedback.

* Remove unused import.

* babe: restrict info logging during initial sync

* grandpa: restrict info logging during initial sync

* grandpa: fix test compilation

* grandpa: remove afg_log macro mod

* manual-seal uses Finalizer trait for finalization instead of Backend

* fix tests

* use Transaction type

* refactor import_queue

* ugh

* line-width

* Update client/consensus/manual-seal/src/lib.rs

Co-Authored-By: Joshy Orndorff <[email protected]>

* fix tests

* update docs

* Update client/consensus/manual-seal/src/seal_new_block.rs

Co-Authored-By: André Silva <[email protected]>

* Don't auto-finalize on verification

* Explicity don't finalize on import.

Co-authored-by: Joshy Orndorff <[email protected]>
Co-authored-by: Joshy Orndorff <[email protected]>
Co-authored-by: André Silva <[email protected]>

* Fix compilation of service metrics on Windows

* Fix browser node

* Ahh

* Remove duplicate code...

* Make `application-crypto` `std` feature internal

The macros should not generate code that requires that the calling crate
has a feature with the name `std` defined.

* Use a proper panic!

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Rename IntoConfiguration to CliConfiguration

* Renamed into_configuration to create_configuration

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Move keystore params to its own module

* Use in-memory keystore even for build-spec

* Enforce proper value for node name

* dev_key_seed

* Telemetry endpoints

* rustfmt

* Converted all RunCmd

* rustfmt

* Added export-blocks

* Missed something

* Removed config_path in NetworkConfiguration (not used)

* Fixed warnings

* public_addresses is used but never set, keeping it

* Merge Configuration.node and NetworkConfiguration.node_name

...because they are the same thing

* Added: import-blocks

* Adding a proc_macro to help impl SubstrateCli

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Re-export spec_factory from sc_cli

* Re-added all the commands

* Refactored node_key_params

* Fixed previous refucktoring

* Clean-up and removed full_version()

* Renamed get_is_dev to not confuse with Configuration field

* Fixed sc-cli-derive example

* Fixing tests

* Fixing tests and removing some (will re-add later)

* Fixing more tests

* Removes the need of type parameter

* Converting bin/node and simplifying API

* Converting more

* Converting last command

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Fixing tests and added default for WasmExecutionMethod

* Fixing stuff

* Fixed something I broke oops

* Update Cargo.lock

* Moving things around

* Convert everything to Result

* Added new macros to simplify the impl of CliConfiguration

* Added a macro to generate CliConfiguration automatically for subcommands

* Revert... too many macros (this one is not really useful)

This reverts commit 9c516dd3

.

* Renamed is_dev to get_is_dev

Good enough for now

* Fixed name roles (this is plural, not singular)

* Clean-up

* Re-export NodeKeyConfig and TelemetryEndpoints from sc_service

* Improve styling/formatting

* Added copyrights

* Added doc and fixed warnings

* Added myself to code owners

* Yes it is needed according to the history

* Revert formatting

* Fixing conflict

* Updated build.rs

* Cargo.lock

* Clean-up

* Update client/cli-derive/Cargo.toml

Co-Authored-By: Seun Lanlege <[email protected]>

* Fail if using proc_macro and build.rs is not set properly

* Dropped all get_ in front of methods

* Clean-up

* Fixing proc macro missing env var

* Get the configuration inside the Runtime (needed for polkadot)

* Clean-up

* Get is_dev from argument like the others

* Get chain ID instead of chain spec from shared params

* &self is passed to spec_factory/load_spec

* Wrong text

* Fix example

* Officialize macro and made a cool doc

* Renamed spec_factory to load_spec (substrate_cli_configuration)

* Removed not so useful ChainSpec

* Renamed SubstrateCLI to SubstrateCli

* Added changelog for impl_version being full now

* Renamed Runtime to Runner

* Update changelog to show example

* Removed option on database cache size

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Fix on removal of option

* typo

* Clean-up imports

* Added info in Cargo.toml

* typo

* remarks

* Moved function for build.rs to substrate-build-script-utils

* Fixed example & test of cli-derive

* Moved function for build.rs to substrate-build-script-utils

* Renamed substrate_cli_configuration to substrate_cli oops

It implements SubstrateCli not CliConfiguration!

* Added documentation and wrapper macro

* Removed option on database cache size

* Removed option on database cache size

* Clean-up

* Reduce risk of errors due to typos

* Removed option on database cache size

* Added NOTE as suggested

* Added doc as suggested

* Fixed test

* typo

* renamed runtime to runner

* Fixed weird argument

* More commas

* Moved client/cli-derive to client/cli/derive

* Added 7 tests for the macros

* Improve error message

* Upgrade assert_cmd

* Fixing missing stuff

* Fixed unused import

* Improve SubstrateCli doc

* Applied suggestions

* Fix and clean-up imports

* Started replacing macros WIP

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Started removing substrate_cli

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81


Parent branch: origin/master

* fixed bug introduced while refactoring

* Renamed NetworkConfigurationParams to NetworkParams for consistency sake

* Fixed test

* Update client/cli/src/commands/runcmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/runcmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/check_block_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update bin/node/cli/src/command.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update bin/node/cli/src/command.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Revert "Update client/cli/src/commands/export_blocks_cmd.rs"

This reverts commit 59067769.

* Revert "Update client/cli/src/commands/check_block_cmd.rs"

This reverts commit f705f42b.

* Revert "Update client/cli/src/commands/export_blocks_cmd.rs"

This reverts commit 8d57c055.

* Revert "Update client/cli/src/commands/runcmd.rs"

This reverts commit 93e74cf5.

* Revert "Update client/cli/src/commands/runcmd.rs"

This reverts commit 11d527ba

.

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/import_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Changed ::sc_cli to $crate in the macro

* fixed tests

* fixed conflicts

* Fixing test

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/params/pruning_params.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Remove comment as suggested

* Apply suggestion

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update utils/frame/benchmarking-cli/src/command.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/params/pruning_params.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/params/node_key_params.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/params/network_params.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/config.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Added doc

* Fixed error introduced after applying suggestion

* Revert "Update client/cli/src/params/pruning_params.rs"

This reverts commit 0574d06a

.

* Print error

* Apply suggestions from code review

* Remove useless Results

* Fixed CI failing on polkadot approval

Co-authored-by: Seun Lanlege <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* Add initial report_ofence bench.

* Remove unused imports

* Style nit

* Add nominators

* Remove logs.

* Nits.

* Add nominators param.

* Reorg, comment.

* Remove whitespaces.

* Apply review suggestion: move benchmark to own crate.

* Remove import.

* Remove line.

* Add feature flag.

* Pass can_report.

* Cleaning up.

* More cleaning

* Update to latest staking

* generate tests for benchmarking

* add tests, fix warnings

* starting on democracy

* impl_benchmark_tests

* Way more readable

* add test feature flag (does this work?)

* Fix `successful_origin` impl

* democracry benchmark tests

* Fix example benchmarks, add tests

* identity benchmark tests

* Update im-online benchmark tests

* try to add session benchmarking tests (problem with mock)

* staking and timestamp

* add test for treasury, issue with dynamic contains

* utility

* Vesting

* test instead of check

* hide until we figure out what is wrong

* add docs

* close code

* Create custom mock for session-pallet-benchmarking

* Use refcell pattern

* make un-pub

* test-linux-stable includes `runtime-benchmarks` feature

* Revert "test-linux-stable includes `runtime-benchmarks` feature"

This reverts commit a2dab38a.

* run tests in `--release`

* undo balance change

* build wasm

* change (ci): merge check warnings into test linux job

* change (ci): newline doesn't make sense here

* Add call hash as parameters of multisig related event

* update docs

* Turn notifications_total into notifications_sizes

* Address review

* update externalities docs

* Update primitives/externalities/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update primitives/externalities/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update primitives/externalities/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update primitives/externalities/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>

This reverts commit 25fb92ab.

* Make verbosity level mandatory instead of defaulting to 0 when using --telemetry-url

* Update README docs

* Change TelemetryError struct to enum

* Return TelemetryParsingError instead of a Boxed dyn error

* Replace spaces by tabs

* Add example of expected format for telemetry-url

* Remove UrlParsingError; Call to_string instead of parse for TelemetryEndpoints url

* Sort members in elections-phragmen genesis

* panic if duplicate member

* print duplicate member

* Mandatory dispatch class

* Tweaks

* Docs

* Fix test

* Update frame/support/src/weights.rs

Co-Authored-By: joe petrowski <[email protected]>

* Introduce logic that was stated in PR.

* Use

* Docs.

* Fix test

* Fix merge

* Update frame/support/src/weights.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Fix.

* Fix

Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* some fixes

* Update tips benchmark to be resiliant to genesis

* Update Cargo.lock

* procfs is linux only

* also for the import

* fixup non-linux build

This PR refactors the metrics measuring and Prometheus exposing entity in sc-service into its own submodule and extends the parameters it exposes by:

- system load average (over one, five and 15min)
- the TCP connection state of the process (lsof), refs #5304
- number of tokio threads
- number of known forks
- counter for items in each unbounded queue (with internal unbounded channels)
- number of file descriptors opened by this process (*nix only at this point)
- number of system threads (*nix only at this point)

refs #4679

Co-authored-by: Max Inden <[email protected]>
Co-authored-by: Ashley <[email protected]>

* Simple Payouts

* explicit test for out of order claiming

* Add `payout_all` benchmark

* Fix merge

* add docs

* change event to controller

* Fix timestamp test warnings

* Revert "change event to controller"

This reverts commit 5d4a9783

.

* Update Reward event doc

* Add "non-production" test

* add unlock chunk to test

* fix merge

* End payout early if no reward points

* payout_validator -> payout_stakers

* bring back payout nominator/validator, but limit their use to before migration era

* Add test for before migration

* New payout works for the era that we migrate

* Fix logic, check that migration era works

* Migrate Era tests (copypasta)

* Move comment

* Add mock back to external functions

* Fixes based on review from gui

* Update Cargo.lock

* Update Cargo.lock

* small docs update

Co-authored-by: joepetrowski <[email protected]>

* EnsureOrigin is a frame abstraction - it should be in frame_support

* Fixes

* support: add lateness trait

* babe: implement the lateness trait

* babe: add docs about lateness entry lifetime

* babe: don't use option for lateness storage entry