* Add initial report_ofence bench.

* Remove unused imports

* Style nit

* Add nominators

* Remove logs.

* Nits.

* Add nominators param.

* Reorg, comment.

* Remove whitespaces.

* Apply review suggestion: move benchmark to own crate.

* Remove import.

* Remove line.

* Add feature flag.

* Pass can_report.

* Cleaning up.

* More cleaning

* Update to latest staking

* generate tests for benchmarking

* add tests, fix warnings

* starting on democracy

* impl_benchmark_tests

* Way more readable

* add test feature flag (does this work?)

* Fix `successful_origin` impl

* democracry benchmark tests

* Fix example benchmarks, add tests

* identity benchmark tests

* Update im-online benchmark tests

* try to add session benchmarking tests (problem with mock)

* staking and timestamp

* add test for treasury, issue with dynamic contains

* utility

* Vesting

* test instead of check

* hide until we figure out what is wrong

* add docs

* close code

* Create custom mock for session-pallet-benchmarking

* Use refcell pattern

* make un-pub

* test-linux-stable includes `runtime-benchmarks` feature

* Revert "test-linux-stable includes `runtime-benchmarks` feature"

This reverts commit a2dab38a.

* run tests in `--release`

* undo balance change

* build wasm

This PR refactors the metrics measuring and Prometheus exposing entity in sc-service into its own submodule and extends the parameters it exposes by:

- system load average (over one, five and 15min)
- the TCP connection state of the process (lsof), refs #5304
- number of tokio threads
- number of known forks
- counter for items in each unbounded queue (with internal unbounded channels)
- number of file descriptors opened by this process (*nix only at this point)
- number of system threads (*nix only at this point)

refs #4679

Co-authored-by: Max Inden <[email protected]>
Co-authored-by: Ashley <[email protected]>

* Simple Payouts

* explicit test for out of order claiming

* Add `payout_all` benchmark

* Fix merge

* add docs

* change event to controller

* Fix timestamp test warnings

* Revert "change event to controller"

This reverts commit 5d4a9783

.

* Update Reward event doc

* Add "non-production" test

* add unlock chunk to test

* fix merge

* End payout early if no reward points

* payout_validator -> payout_stakers

* bring back payout nominator/validator, but limit their use to before migration era

* Add test for before migration

* New payout works for the era that we migrate

* Fix logic, check that migration era works

* Migrate Era tests (copypasta)

* Move comment

* Add mock back to external functions

* Fixes based on review from gui

* Update Cargo.lock

* Update Cargo.lock

* small docs update

Co-authored-by: joepetrowski <[email protected]>

* custom benchmark runner

* add license preamble

* delete old benchmarks

* update toml

* imporove macro

* tabs

* add size

* size fixes

* add docs for cli

* Initial draft of the logic

* Build and tests

* Make work with new initialize infratructure.

* Update frame/scheduler/src/lib.rs

Co-Authored-By: Marcio Diaz <[email protected]>

* Update frame/scheduler/src/lib.rs

Co-Authored-By: Marcio Diaz <[email protected]>

* Update frame/scheduler/src/lib.rs

Co-Authored-By: Marcio Diaz <[email protected]>

* Update frame/scheduler/src/lib.rs

Co-Authored-By: Marcio Diaz <[email protected]>

* Fix test

* Update frame/scheduler/src/lib.rs

Co-Authored-By: Marcio Diaz <[email protected]>

* Rejig interface to make it more useful for democracy.

* Try to get democraxy module to make use of scheduler.

* Make democracy use scheduler.

* Use actual max weight for enactent

* Remove TODO

* Fix runtime build

* Minor cleanup

* Fix scheduler.

* Fix benchmarks

* Fix

* Fix

* Fix

* More bench fixes

* Fix

* Fix.

* Add more bench constants.

* Fix cancel_queued bench.

* Fix test comment.

* Update frame/scheduler/src/lib.rs

Co-Authored-By: Marcio Diaz <[email protected]>

Co-authored-by: Marcio Diaz <[email protected]>

* Reset by removing pos-genesis changes

* CLI option for DB cache size

* Update Cargo.lock

Co-authored-by: Shawn Tabrizi <[email protected]>

* Move tests

* Move around feature cfgs

* Start benchmarks

* as_multi bench

* finish

* fix tests

* Update frame/utility/src/benchmarking.rs

Co-Authored-By: Marcio Diaz <[email protected]>

* Update Cargo.toml

* fix test

Co-authored-by: Marcio Diaz <[email protected]>

* A patch

* Fix

* Better approach.

* Fix grumbles

* imporoved intervallier version

* remove extra event

* use version 0.3.1

* fix warning

* Added option to disable default block announce

* Added on_block_imported on NetworkService

* Revert "Added on_block_imported on NetworkService"

This reverts commit ba360cad.

* Do not announce block if set to not announce block

* Revert fix

* Moving default announce block to NetworkConfig

* WIP

Forked at: 41bb2193
Parent branch: origin/master

* WIP

Forked at: 41bb2193
Parent branch: origin/master

* Removing boolean in favor of explicit call

* Fixing tests

* WIP

Forked at: 41bb2193
Parent branch: origin/master

* WIP

Forked at: 41bb2193
Parent branch: origin/master

* increase spec_version

* increase spec_version

* Fixed test

* Fixing test

* Renamed should_announce_imported_blocks to announce_imported_blocks

* Updated assert_cmd

The signal handler will be reset to the default system one. This means
the program will be exited instead of panicking. This is required to
support piping in the console.

* Initial skeleton for offchain phragmen

* Basic compact encoding decoding for results

* add compact files

* Bring back Self::ensure_storage_upgraded();

* Make staking use compact stuff.

* First seemingly working version of reduce, full of todos

* Everything phragmen related works again.

* Signing made easier, still issues.

* Signing from offchain compile fine 😎

* make compact work with staked asssignment

* Evaluation basics are in place.

* Move reduce into crate. Document stuff

* move reduce into no_std

* Add files

* Remove other std deps. Runtime compiles

* Seemingly it is al stable; cycle implemented but not integrated.

* Add fuzzing code.

* Cleanup reduce a bit more.

* a metric ton of tests for staking; wip 🔨

* Implement a lot more of the tests.

* wip getting the unsigned stuff to work

* A bit gleanup for unsigned debug

* Clean and finalize compact code.

* Document reduce.

* Still problems with signing

* We officaly duct taped the transaction submission stuff. 🤓

* Deadlock with keys again

* Runtime builds

* Unsigned test works 🙌

* Some cleanups

* Make all the tests compile and stuff

* Minor cleanup

* fix more merge stuff

* Most tests work again.

* a very nasty bug in reduce

* Fix all integrations

* Fix more todos

* Revamp everything and everything

* Remove bogus test

* Some review grumbles.

* Some fixes

* Fix doc test

* loop for submission

* Fix cli, keyring etc.

* some cleanup

* Fix staking tests again

* fix per-things; bring patches from benchmarking

* better score prediction

* Add fuzzer, more patches.

* Some fixes

* More docs

* Remove unused generics

* Remove max-nominator footgun

* Better fuzzer

* Disable it ❌



* Bump.

* Another round of self-review

* Refactor a lot

* More major fixes in perThing

* Add new fuzz file

* Update lock

* fix fuzzing code.

* Fix nominator retain test

* Add slashing check

* Update frame/staking/src/tests.rs

Co-Authored-By: Joshy Orndorff <[email protected]>

* Some formatting nits

* Review comments.

* Fix cargo file

* Almost all tests work again

* Update frame/staking/src/tests.rs

Co-Authored-By: thiolliere <[email protected]>

* Fix review comments

* More review stuff

* Some nits

* Fix new staking / session / babe relation

* Update primitives/phragmen/src/lib.rs

Co-Authored-By: thiolliere <[email protected]>

* Update primitives/phragmen/src/lib.rs

Co-Authored-By: thiolliere <[email protected]>

* Update primitives/phragmen/compact/src/lib.rs

Co-Authored-By: thiolliere <[email protected]>

* Some doc updates to slashing

* Fix derive

* Remove imports

* Remove unimplemented tests

* nits

* Remove dbg

* Better fuzzing params

* Remove unused pref map

* Deferred Slashing/Offence for offchain Phragmen  (#5151)

* Some boilerplate

* Add test

* One more test

* Review comments

* Fix build

* review comments

* fix more

* fix build

* Some cleanups and self-reviews

* More minor self reviews

* Final nits

* Some merge fixes.

* opt comment

* Fix build

* Fix build again.

* Update frame/staking/fuzz/fuzz_targets/submit_solution.rs

Co-Authored-By: Gavin Wood <[email protected]>

* Update frame/staking/src/slashing.rs

Co-Authored-By: Gavin Wood <[email protected]>

* Update frame/staking/src/offchain_election.rs

Co-Authored-By: Gavin Wood <[email protected]>

* Fix review comments

* fix test

* === 🔑

 Revamp without staking key.

* final round of changes.

* Fix cargo-deny

* Update frame/staking/src/lib.rs

Co-Authored-By: Gavin Wood <[email protected]>

Co-authored-by: Joshy Orndorff <[email protected]>
Co-authored-by: thiolliere <[email protected]>
Co-authored-by: Gavin Wood <[email protected]>

* Add origin bounds to benchmark macro.

* Add democracy benchmark.

* Fix tests

* Remove collective from frame/benchmarking, partially use EnsureOrigin.

* Remove collective stuff.

* Make previous benches compile again.

* Remove comments.

* Make prev bench to work again.

* Add remove votes.

* Add new proxy calls.

* Add runtime-benchmarks guard to EnsureOrigin and implementations.

* Refactor.

* Add missing import.

* Remove duplicated import

* Fix features.

* Add some missing features.

* Update frame/collective/Cargo.toml

Co-Authored-By: Kian Paimani <[email protected]>

* Update frame/democracy/src/benchmarking.rs

Co-Authored-By: Kian Paimani <[email protected]>

* Update frame/democracy/src/benchmarking.rs

Co-Authored-By: Kian Paimani <[email protected]>

* Add referendums to state.

* populate vecs before call

* Update weight docs

* More fixes and weight docs

* More updates

Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Shawn Tabrizi <[email protected]>

* add new trait, still migration to make

* Apply suggestions from code review

Co-Authored-By: Kian Paimani <[email protected]>

* implement migration

* better naming

* fix test

* no longer require DeprecatedTime

* add test

* fix version

* upgrade only from kusama

* add test

* fix test

* Update frame/timestamp/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* always depend on latest alpha version

* integrate arithmetic fuzzer into root workspace

* Init

* Fix execute.

* Duplicate macro for instances temporarilly

* Add propose.

* Add vote, close.

* Propose from own module

* Add old members to set_members.

* Add previous proposals to propose.

* Compress a bit the macro.

Co-authored-by: Shawn Tabrizi <[email protected]>

* use versioning for deprecated api, remove deprecated api from regular verification

* Update primitives/core/src/sr25519.rs

* add test to transaction pool

Co-authored-by: Bastian Köcher <[email protected]>

* bump pre-version

* Changelog alpha.4->alpha.5

* repo + homepage to sp-consensus-vrf

* Add default rocksdb feature to `frame-benchmarking-cli` (#5367)

* Add default rocksdb feature

* Update Cargo.toml

Co-authored-by: Shawn Tabrizi <[email protected]>

* Get rid of rustc_hex in favor of hex

* Use HexDisplay

* versionned runtime-interface

* use only one additional wasm blob

* alter docs

* formatting, naming and docs

* add comment for test

* version duplicate err

* RuntimeInterfaceItem -> RuntimeInterfaceFunction<

* test naming

* version checking

* remove spaces

* Update primitives/runtime-interface/proc-macro/src/runtime_interface/bare_function_interface.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* remove sanity checks and reduce coverage

* add doc comment

Co-authored-by: Bastian Köcher <[email protected]>

* Add linregress

* Regression, initial stuff.

* More analytics

* whitespace

* Fix

* Median slopes regression algo

* Warnings

* Update to lastest benchmark changes.

* Use both algorithms temp.

* Move analysis to frame.

* Fix tests.

* Only build analysis in std

Co-authored-by: Gav Wood <[email protected]>

* pallet-evm: configurable gasometer config

* Bump runtime impl_version

* Update evm to 0.16.1

Documentation updates

Co-authored-by: Gavin Wood <[email protected]>

* Init vrf crate without type alias

* Generic PreDigest definition for BABE

* Fix BABE vrf interface change

* Missing default-features def in sp-consensus-babe

* Fix sp-consensus-babe compile

* frame-babe: fix type inference

* Unify type definitions of vrf output/proof and randomness

* frame-babe: fix tests

* Bump node impl version

* Update cargo lock

* Derive Copy for RawVRFOutput and RawVRFProof

* Fix duplicated derive

Co-authored-by: Gavin Wood <[email protected]>

* Make sure frame examples compile for wasm

This makes sure that `frame-example` and `frame-example-offchain-worker`
compile for wasm.

This also fixes compilation for these crates. The offchain worker
example doesn't use serde-json anymore as that is too heavy and breaks
`no_std` compilation.

* Apply suggestions from code review

Co-Authored-By: Nikolay Volf <[email protected]>

Co-authored-by: Nikolay Volf <[email protected]>

* Start benchmarks

* try_origin or root

* More benches

* stuck

* Custom trait functions for benchmarks

* finish benchmarks

* Bump impl

* More comments

* Bump spec

* Remove import

* Update frame/elections-phragmen/src/lib.rs

Co-Authored-By: thiolliere <[email protected]>

* Update frame/support/src/traits.rs

Co-Authored-By: thiolliere <[email protected]>

* Fix merge

Co-authored-by: thiolliere <[email protected]>

* Initial benchmarking setup

* Add keystore

* validate unsigned

* Update frame/im-online/src/benchmarking.rs

Co-Authored-By: Marcio Diaz <[email protected]>

* Fix verify_unsigned benchmark

* add variable for teting the external addresss length

* Update frame/im-online/src/benchmarking.rs

Co-Authored-By: Kian Paimani <[email protected]>

* Update utils/frame/benchmarking-cli/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

Co-authored-by: Marcio Diaz <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Benjamin Kampmann <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* switching to released wasmtime

* update depdencies in general

* Update fdlimit

* Update parity common deps

* Also update schnorrkel

* update kvdb-rocksdb

* update further dependents

* also update primitive types

* update cargo.lock

* update merlin

* Bumping evm version

* client/network-gossip/src/bridge: Finish when network even stream closes

Previously within `<GossipEngine as Future>::poll` one would poll the
`network_event_stream` ignoring all messages other than
`Poll::Ready(Some())`. Ignoring `Poll::Ready(None)` leads to a panic on
the next poll of the stream, gien that it is not fused.

By design `network_event_stream` does not close unless an unbounded send
into it fails, or the `NetworkWorker` gets shut down.

> The stream never ends (unless the `NetworkWorker` gets shut down).
> (client/network/src/service.rs)

An `unbounded_send` to fail on an unbounded channel is unlikely. The
`NetworkWorker` shutting down is not unlikely. In such case the
`GossipEngine` should shut down as well.

With this patch a `<GossipEngine as Future>` finishes on
`Poll::Ready(None)` returned from `network_event_stream`.

* client/finality-grandpa/communication: Error on gossip engine finished

Have `<NetworkBridge as Future>::poll` return `Poll::Ready(Err)` instead
of `Poll::Ready(Ok)` to be consistent with the handling of the neighbor
packet worker stream and the gossip validator report stream. Both `Err`
as well as `Ok` shut down the `NetworkBridge` as well as the
`VoterWorker`.

* client/network-gossip/src/bridge: Add regression test

* client/network-gossip: Move substrate test client to dev dependencies

* client/network-gossip: Remove TODO

Addressed in a follow up pull request.

* client/network-gossip/bridge: Put match on newline after loop

* client/finality-grandpa/src/observer: Fix regression test

Make sure the event stream sender side is not dropped till the end.

* Adding missing metadata for benchmark-pallet

* Missed to bump a few

* release on roling tag

* Wasm executors were missed

* Version bump

* Split generate_changelog.sh into separate script

Can be run in the format `generate_changelog.sh $previous_version $version`.

* remove early exit from publish_draft_release.sh

* adding changelog

* ci: change last_github_release to also find pre-releases

Co-authored-by: Martin Pugh <[email protected]>