* in-mem state for benchmarks

* Use caching state

* Update Cargo.lock

Co-authored-by: Shawn Tabrizi <[email protected]>

* Add new RPC method to get the chain type

This adds a new RPC method to get the chain type of the running chain.
The chain type needs to be specified in the chain spec. This should make
it easier for tools/UI to display extra information without needing to
rely on parsing the chain name.

* Update client/rpc-api/src/system/mod.rs

Co-Authored-By: Tomasz Drwięga <[email protected]>

* Primitive crate

* Feedback

Co-authored-by: Tomasz Drwięga <[email protected]>

* Preserve a single wasmtime instance.

* Sketch of wasm instance reusing.

* Refactor and docs.

* Rename state_snapshot to util module.

* Renaming.

* Comments.

* Error handling

* More fixes.

* Adapt to rust-libp2p#1440.

* Further adapt to libp2p/master.

* Update to libp2p-0.17

* Finishing touches.

* Remove stray TODO.

* Incorporate review feedback.

* Remove unused import.

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Rename IntoConfiguration to CliConfiguration

* Renamed into_configuration to create_configuration

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Move keystore params to its own module

* Use in-memory keystore even for build-spec

* Enforce proper value for node name

* dev_key_seed

* Telemetry endpoints

* rustfmt

* Converted all RunCmd

* rustfmt

* Added export-blocks

* Missed something

* Removed config_path in NetworkConfiguration (not used)

* Fixed warnings

* public_addresses is used but never set, keeping it

* Merge Configuration.node and NetworkConfiguration.node_name

...because they are the same thing

* Added: import-blocks

* Adding a proc_macro to help impl SubstrateCli

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Re-export spec_factory from sc_cli

* Re-added all the commands

* Refactored node_key_params

* Fixed previous refucktoring

* Clean-up and removed full_version()

* Renamed get_is_dev to not confuse with Configuration field

* Fixed sc-cli-derive example

* Fixing tests

* Fixing tests and removing some (will re-add later)

* Fixing more tests

* Removes the need of type parameter

* Converting bin/node and simplifying API

* Converting more

* Converting last command

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Fixing tests and added default for WasmExecutionMethod

* Fixing stuff

* Fixed something I broke oops

* Update Cargo.lock

* Moving things around

* Convert everything to Result

* Added new macros to simplify the impl of CliConfiguration

* Added a macro to generate CliConfiguration automatically for subcommands

* Revert... too many macros (this one is not really useful)

This reverts commit 9c516dd3

.

* Renamed is_dev to get_is_dev

Good enough for now

* Fixed name roles (this is plural, not singular)

* Clean-up

* Re-export NodeKeyConfig and TelemetryEndpoints from sc_service

* Improve styling/formatting

* Added copyrights

* Added doc and fixed warnings

* Added myself to code owners

* Yes it is needed according to the history

* Revert formatting

* Fixing conflict

* Updated build.rs

* Cargo.lock

* Clean-up

* Update client/cli-derive/Cargo.toml

Co-Authored-By: Seun Lanlege <[email protected]>

* Fail if using proc_macro and build.rs is not set properly

* Dropped all get_ in front of methods

* Clean-up

* Fixing proc macro missing env var

* Get the configuration inside the Runtime (needed for polkadot)

* Clean-up

* Get is_dev from argument like the others

* Get chain ID instead of chain spec from shared params

* &self is passed to spec_factory/load_spec

* Wrong text

* Fix example

* Officialize macro and made a cool doc

* Renamed spec_factory to load_spec (substrate_cli_configuration)

* Removed not so useful ChainSpec

* Renamed SubstrateCLI to SubstrateCli

* Added changelog for impl_version being full now

* Renamed Runtime to Runner

* Update changelog to show example

* Removed option on database cache size

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Fix on removal of option

* typo

* Clean-up imports

* Added info in Cargo.toml

* typo

* remarks

* Moved function for build.rs to substrate-build-script-utils

* Fixed example & test of cli-derive

* Moved function for build.rs to substrate-build-script-utils

* Renamed substrate_cli_configuration to substrate_cli oops

It implements SubstrateCli not CliConfiguration!

* Added documentation and wrapper macro

* Removed option on database cache size

* Removed option on database cache size

* Clean-up

* Reduce risk of errors due to typos

* Removed option on database cache size

* Added NOTE as suggested

* Added doc as suggested

* Fixed test

* typo

* renamed runtime to runner

* Fixed weird argument

* More commas

* Moved client/cli-derive to client/cli/derive

* Added 7 tests for the macros

* Improve error message

* Upgrade assert_cmd

* Fixing missing stuff

* Fixed unused import

* Improve SubstrateCli doc

* Applied suggestions

* Fix and clean-up imports

* Started replacing macros WIP

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Started removing substrate_cli

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81


Parent branch: origin/master

* fixed bug introduced while refactoring

* Renamed NetworkConfigurationParams to NetworkParams for consistency sake

* Fixed test

* Update client/cli/src/commands/runcmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/runcmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/check_block_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update bin/node/cli/src/command.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update bin/node/cli/src/command.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Revert "Update client/cli/src/commands/export_blocks_cmd.rs"

This reverts commit 59067769.

* Revert "Update client/cli/src/commands/check_block_cmd.rs"

This reverts commit f705f42b.

* Revert "Update client/cli/src/commands/export_blocks_cmd.rs"

This reverts commit 8d57c055.

* Revert "Update client/cli/src/commands/runcmd.rs"

This reverts commit 93e74cf5.

* Revert "Update client/cli/src/commands/runcmd.rs"

This reverts commit 11d527ba

.

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/import_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Changed ::sc_cli to $crate in the macro

* fixed tests

* fixed conflicts

* Fixing test

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/params/pruning_params.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Remove comment as suggested

* Apply suggestion

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update utils/frame/benchmarking-cli/src/command.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/params/pruning_params.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/params/node_key_params.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/params/network_params.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/config.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Added doc

* Fixed error introduced after applying suggestion

* Revert "Update client/cli/src/params/pruning_params.rs"

This reverts commit 0574d06a

.

* Print error

* Apply suggestions from code review

* Remove useless Results

* Fixed CI failing on polkadot approval

Co-authored-by: Seun Lanlege <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* Add initial report_ofence bench.

* Remove unused imports

* Style nit

* Add nominators

* Remove logs.

* Nits.

* Add nominators param.

* Reorg, comment.

* Remove whitespaces.

* Apply review suggestion: move benchmark to own crate.

* Remove import.

* Remove line.

* Add feature flag.

* Pass can_report.

* Cleaning up.

* More cleaning

* Update to latest staking

* generate tests for benchmarking

* add tests, fix warnings

* starting on democracy

* impl_benchmark_tests

* Way more readable

* add test feature flag (does this work?)

* Fix `successful_origin` impl

* democracry benchmark tests

* Fix example benchmarks, add tests

* identity benchmark tests

* Update im-online benchmark tests

* try to add session benchmarking tests (problem with mock)

* staking and timestamp

* add test for treasury, issue with dynamic contains

* utility

* Vesting

* test instead of check

* hide until we figure out what is wrong

* add docs

* close code

* Create custom mock for session-pallet-benchmarking

* Use refcell pattern

* make un-pub

* test-linux-stable includes `runtime-benchmarks` feature

* Revert "test-linux-stable includes `runtime-benchmarks` feature"

This reverts commit a2dab38a.

* run tests in `--release`

* undo balance change

* build wasm

This PR refactors the metrics measuring and Prometheus exposing entity in sc-service into its own submodule and extends the parameters it exposes by:

- system load average (over one, five and 15min)
- the TCP connection state of the process (lsof), refs #5304
- number of tokio threads
- number of known forks
- counter for items in each unbounded queue (with internal unbounded channels)
- number of file descriptors opened by this process (*nix only at this point)
- number of system threads (*nix only at this point)

refs #4679

Co-authored-by: Max Inden <[email protected]>
Co-authored-by: Ashley <[email protected]>

* Simple Payouts

* explicit test for out of order claiming

* Add `payout_all` benchmark

* Fix merge

* add docs

* change event to controller

* Fix timestamp test warnings

* Revert "change event to controller"

This reverts commit 5d4a9783

.

* Update Reward event doc

* Add "non-production" test

* add unlock chunk to test

* fix merge

* End payout early if no reward points

* payout_validator -> payout_stakers

* bring back payout nominator/validator, but limit their use to before migration era

* Add test for before migration

* New payout works for the era that we migrate

* Fix logic, check that migration era works

* Migrate Era tests (copypasta)

* Move comment

* Add mock back to external functions

* Fixes based on review from gui

* Update Cargo.lock

* Update Cargo.lock

* small docs update

Co-authored-by: joepetrowski <[email protected]>

* custom benchmark runner

* add license preamble

* delete old benchmarks

* update toml

* imporove macro

* tabs

* add size

* size fixes

* add docs for cli

* Initial draft of the logic

* Build and tests

* Make work with new initialize infratructure.

* Update frame/scheduler/src/lib.rs

Co-Authored-By: Marcio Diaz <[email protected]>

* Update frame/scheduler/src/lib.rs

Co-Authored-By: Marcio Diaz <[email protected]>

* Update frame/scheduler/src/lib.rs

Co-Authored-By: Marcio Diaz <[email protected]>

* Update frame/scheduler/src/lib.rs

Co-Authored-By: Marcio Diaz <[email protected]>

* Fix test

* Update frame/scheduler/src/lib.rs

Co-Authored-By: Marcio Diaz <[email protected]>

* Rejig interface to make it more useful for democracy.

* Try to get democraxy module to make use of scheduler.

* Make democracy use scheduler.

* Use actual max weight for enactent

* Remove TODO

* Fix runtime build

* Minor cleanup

* Fix scheduler.

* Fix benchmarks

* Fix

* Fix

* Fix

* More bench fixes

* Fix

* Fix.

* Add more bench constants.

* Fix cancel_queued bench.

* Fix test comment.

* Update frame/scheduler/src/lib.rs

Co-Authored-By: Marcio Diaz <[email protected]>

Co-authored-by: Marcio Diaz <[email protected]>

* Reset by removing pos-genesis changes

* CLI option for DB cache size

* Update Cargo.lock

Co-authored-by: Shawn Tabrizi <[email protected]>

* Move tests

* Move around feature cfgs

* Start benchmarks

* as_multi bench

* finish

* fix tests

* Update frame/utility/src/benchmarking.rs

Co-Authored-By: Marcio Diaz <[email protected]>

* Update Cargo.toml

* fix test

Co-authored-by: Marcio Diaz <[email protected]>

* A patch

* Fix

* Better approach.

* Fix grumbles

* imporoved intervallier version

* remove extra event

* use version 0.3.1

* fix warning

* Added option to disable default block announce

* Added on_block_imported on NetworkService

* Revert "Added on_block_imported on NetworkService"

This reverts commit ba360cad.

* Do not announce block if set to not announce block

* Revert fix

* Moving default announce block to NetworkConfig

* WIP

Forked at: 41bb2193
Parent branch: origin/master

* WIP

Forked at: 41bb2193
Parent branch: origin/master

* Removing boolean in favor of explicit call

* Fixing tests

* WIP

Forked at: 41bb2193
Parent branch: origin/master

* WIP

Forked at: 41bb2193
Parent branch: origin/master

* increase spec_version

* increase spec_version

* Fixed test

* Fixing test

* Renamed should_announce_imported_blocks to announce_imported_blocks

* Updated assert_cmd

The signal handler will be reset to the default system one. This means
the program will be exited instead of panicking. This is required to
support piping in the console.

* Initial skeleton for offchain phragmen

* Basic compact encoding decoding for results

* add compact files

* Bring back Self::ensure_storage_upgraded();

* Make staking use compact stuff.

* First seemingly working version of reduce, full of todos

* Everything phragmen related works again.

* Signing made easier, still issues.

* Signing from offchain compile fine 😎

* make compact work with staked asssignment

* Evaluation basics are in place.

* Move reduce into crate. Document stuff

* move reduce into no_std

* Add files

* Remove other std deps. Runtime compiles

* Seemingly it is al stable; cycle implemented but not integrated.

* Add fuzzing code.

* Cleanup reduce a bit more.

* a metric ton of tests for staking; wip 🔨

* Implement a lot more of the tests.

* wip getting the unsigned stuff to work

* A bit gleanup for unsigned debug

* Clean and finalize compact code.

* Document reduce.

* Still problems with signing

* We officaly duct taped the transaction submission stuff. 🤓

* Deadlock with keys again

* Runtime builds

* Unsigned test works 🙌

* Some cleanups

* Make all the tests compile and stuff

* Minor cleanup

* fix more merge stuff

* Most tests work again.

* a very nasty bug in reduce

* Fix all integrations

* Fix more todos

* Revamp everything and everything

* Remove bogus test

* Some review grumbles.

* Some fixes

* Fix doc test

* loop for submission

* Fix cli, keyring etc.

* some cleanup

* Fix staking tests again

* fix per-things; bring patches from benchmarking

* better score prediction

* Add fuzzer, more patches.

* Some fixes

* More docs

* Remove unused generics

* Remove max-nominator footgun

* Better fuzzer

* Disable it ❌



* Bump.

* Another round of self-review

* Refactor a lot

* More major fixes in perThing

* Add new fuzz file

* Update lock

* fix fuzzing code.

* Fix nominator retain test

* Add slashing check

* Update frame/staking/src/tests.rs

Co-Authored-By: Joshy Orndorff <[email protected]>

* Some formatting nits

* Review comments.

* Fix cargo file

* Almost all tests work again

* Update frame/staking/src/tests.rs

Co-Authored-By: thiolliere <[email protected]>

* Fix review comments

* More review stuff

* Some nits

* Fix new staking / session / babe relation

* Update primitives/phragmen/src/lib.rs

Co-Authored-By: thiolliere <[email protected]>

* Update primitives/phragmen/src/lib.rs

Co-Authored-By: thiolliere <[email protected]>

* Update primitives/phragmen/compact/src/lib.rs

Co-Authored-By: thiolliere <[email protected]>

* Some doc updates to slashing

* Fix derive

* Remove imports

* Remove unimplemented tests

* nits

* Remove dbg

* Better fuzzing params

* Remove unused pref map

* Deferred Slashing/Offence for offchain Phragmen  (#5151)

* Some boilerplate

* Add test

* One more test

* Review comments

* Fix build

* review comments

* fix more

* fix build

* Some cleanups and self-reviews

* More minor self reviews

* Final nits

* Some merge fixes.

* opt comment

* Fix build

* Fix build again.

* Update frame/staking/fuzz/fuzz_targets/submit_solution.rs

Co-Authored-By: Gavin Wood <[email protected]>

* Update frame/staking/src/slashing.rs

Co-Authored-By: Gavin Wood <[email protected]>

* Update frame/staking/src/offchain_election.rs

Co-Authored-By: Gavin Wood <[email protected]>

* Fix review comments

* fix test

* === 🔑

 Revamp without staking key.

* final round of changes.

* Fix cargo-deny

* Update frame/staking/src/lib.rs

Co-Authored-By: Gavin Wood <[email protected]>

Co-authored-by: Joshy Orndorff <[email protected]>
Co-authored-by: thiolliere <[email protected]>
Co-authored-by: Gavin Wood <[email protected]>

* Add origin bounds to benchmark macro.

* Add democracy benchmark.

* Fix tests

* Remove collective from frame/benchmarking, partially use EnsureOrigin.

* Remove collective stuff.

* Make previous benches compile again.

* Remove comments.

* Make prev bench to work again.

* Add remove votes.

* Add new proxy calls.

* Add runtime-benchmarks guard to EnsureOrigin and implementations.

* Refactor.

* Add missing import.

* Remove duplicated import

* Fix features.

* Add some missing features.

* Update frame/collective/Cargo.toml

Co-Authored-By: Kian Paimani <[email protected]>

* Update frame/democracy/src/benchmarking.rs

Co-Authored-By: Kian Paimani <[email protected]>

* Update frame/democracy/src/benchmarking.rs

Co-Authored-By: Kian Paimani <[email protected]>

* Add referendums to state.

* populate vecs before call

* Update weight docs

* More fixes and weight docs

* More updates

Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Shawn Tabrizi <[email protected]>

* add new trait, still migration to make

* Apply suggestions from code review

Co-Authored-By: Kian Paimani <[email protected]>

* implement migration

* better naming

* fix test

* no longer require DeprecatedTime

* add test

* fix version

* upgrade only from kusama

* add test

* fix test

* Update frame/timestamp/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* always depend on latest alpha version

* integrate arithmetic fuzzer into root workspace

* Init

* Fix execute.

* Duplicate macro for instances temporarilly

* Add propose.

* Add vote, close.

* Propose from own module

* Add old members to set_members.

* Add previous proposals to propose.

* Compress a bit the macro.

Co-authored-by: Shawn Tabrizi <[email protected]>

* use versioning for deprecated api, remove deprecated api from regular verification

* Update primitives/core/src/sr25519.rs

* add test to transaction pool

Co-authored-by: Bastian Köcher <[email protected]>

* bump pre-version

* Changelog alpha.4->alpha.5

* repo + homepage to sp-consensus-vrf

* Add default rocksdb feature to `frame-benchmarking-cli` (#5367)

* Add default rocksdb feature

* Update Cargo.toml

Co-authored-by: Shawn Tabrizi <[email protected]>

* Get rid of rustc_hex in favor of hex

* Use HexDisplay

* versionned runtime-interface

* use only one additional wasm blob

* alter docs

* formatting, naming and docs

* add comment for test

* version duplicate err

* RuntimeInterfaceItem -> RuntimeInterfaceFunction<

* test naming

* version checking

* remove spaces

* Update primitives/runtime-interface/proc-macro/src/runtime_interface/bare_function_interface.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* remove sanity checks and reduce coverage

* add doc comment

Co-authored-by: Bastian Köcher <[email protected]>

* Add linregress

* Regression, initial stuff.

* More analytics

* whitespace

* Fix

* Median slopes regression algo

* Warnings

* Update to lastest benchmark changes.

* Use both algorithms temp.

* Move analysis to frame.

* Fix tests.

* Only build analysis in std

Co-authored-by: Gav Wood <[email protected]>

* pallet-evm: configurable gasometer config

* Bump runtime impl_version

* Update evm to 0.16.1

Documentation updates

Co-authored-by: Gavin Wood <[email protected]>

* Init vrf crate without type alias

* Generic PreDigest definition for BABE

* Fix BABE vrf interface change

* Missing default-features def in sp-consensus-babe

* Fix sp-consensus-babe compile

* frame-babe: fix type inference

* Unify type definitions of vrf output/proof and randomness

* frame-babe: fix tests

* Bump node impl version

* Update cargo lock

* Derive Copy for RawVRFOutput and RawVRFProof

* Fix duplicated derive

Co-authored-by: Gavin Wood <[email protected]>

* Make sure frame examples compile for wasm

This makes sure that `frame-example` and `frame-example-offchain-worker`
compile for wasm.

This also fixes compilation for these crates. The offchain worker
example doesn't use serde-json anymore as that is too heavy and breaks
`no_std` compilation.

* Apply suggestions from code review

Co-Authored-By: Nikolay Volf <[email protected]>

Co-authored-by: Nikolay Volf <[email protected]>

* Start benchmarks

* try_origin or root

* More benches

* stuck

* Custom trait functions for benchmarks

* finish benchmarks

* Bump impl

* More comments

* Bump spec

* Remove import

* Update frame/elections-phragmen/src/lib.rs

Co-Authored-By: thiolliere <[email protected]>

* Update frame/support/src/traits.rs

Co-Authored-By: thiolliere <[email protected]>

* Fix merge

Co-authored-by: thiolliere <[email protected]>