* A few missing emojies

* P2p message emojis

* Add a bit of colour

* format

* Introduce a couple of spaces

* Some spaces

* A patch

* Fix

* Better approach.

* Fix grumbles

The signal handler will be reset to the default system one. This means
the program will be exited instead of panicking. This is required to
support piping in the console.

* Initial skeleton for offchain phragmen

* Basic compact encoding decoding for results

* add compact files

* Bring back Self::ensure_storage_upgraded();

* Make staking use compact stuff.

* First seemingly working version of reduce, full of todos

* Everything phragmen related works again.

* Signing made easier, still issues.

* Signing from offchain compile fine 

* make compact work with staked asssignment

* Evaluation basics are in place.

* Move reduce into crate. Document stuff

* move reduce into no_std

* Add files

* Remove other std deps. Runtime compiles

* Seemingly it is al stable; cycle implemented but not integrated.

* Add fuzzing code.

* Cleanup reduce a bit more.

* a metric ton of tests for staking; wip 

* Implement a lot more of the tests.

* wip getting the unsigned stuff to work

* A bit gleanup for unsigned debug

* Clean and finalize compact code.

* Document reduce.

* Still problems with signing

* We officaly duct taped the transaction submission stuff. 

* Deadlock with keys again

* Runtime builds

* Unsigned test works 

* Some cleanups

* Make all the tests compile and stuff

* Minor cleanup

* fix more merge stuff

* Most tests work again.

* a very nasty bug in reduce

* Fix all integrations

* Fix more todos

* Revamp everything and everything

* Remove bogus test

* Some review grumbles.

* Some fixes

* Fix doc test

* loop for submission

* Fix cli, keyring etc.

* some cleanup

* Fix staking tests again

* fix per-things; bring patches from benchmarking

* better score prediction

* Add fuzzer, more patches.

* Some fixes

* More docs

* Remove unused generics

* Remove max-nominator footgun

* Better fuzzer

* Disable it 



* Bump.

* Another round of self-review

* Refactor a lot

* More major fixes in perThing

* Add new fuzz file

* Update lock

* fix fuzzing code.

* Fix nominator retain test

* Add slashing check

* Update frame/staking/src/tests.rs

Co-Authored-By: Joshy Orndorff <JoshOrndorff@users.noreply.github.com>

* Some formatting nits

* Review comments.

* Fix cargo file

* Almost all tests work again

* Update frame/staking/src/tests.rs

Co-Authored-By: thiolliere <gui.thiolliere@gmail.com>

* Fix review comments

* More review stuff

* Some nits

* Fix new staking / session / babe relation

* Update primitives/phragmen/src/lib.rs

Co-Authored-By: thiolliere <gui.thiolliere@gmail.com>

* Update primitives/phragmen/src/lib.rs

Co-Authored-By: thiolliere <gui.thiolliere@gmail.com>

* Update primitives/phragmen/compact/src/lib.rs

Co-Authored-By: thiolliere <gui.thiolliere@gmail.com>

* Some doc updates to slashing

* Fix derive

* Remove imports

* Remove unimplemented tests

* nits

* Remove dbg

* Better fuzzing params

* Remove unused pref map

* Deferred Slashing/Offence for offchain Phragmen  (#5151)

* Some boilerplate

* Add test

* One more test

* Review comments

* Fix build

* review comments

* fix more

* fix build

* Some cleanups and self-reviews

* More minor self reviews

* Final nits

* Some merge fixes.

* opt comment

* Fix build

* Fix build again.

* Update frame/staking/fuzz/fuzz_targets/submit_solution.rs

Co-Authored-By: Gavin Wood <gavin@parity.io>

* Update frame/staking/src/slashing.rs

Co-Authored-By: Gavin Wood <gavin@parity.io>

* Update frame/staking/src/offchain_election.rs

Co-Authored-By: Gavin Wood <gavin@parity.io>

* Fix review comments

* fix test

* === 

 Revamp without staking key.

* final round of changes.

* Fix cargo-deny

* Update frame/staking/src/lib.rs

Co-Authored-By: Gavin Wood <gavin@parity.io>

Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com>
Co-authored-by: thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Gavin Wood <gavin@parity.io>

* Check for url validity when creating TelemetryEndpoints

* Update code that used TelemetryEndpoints::new()

* Update commennts that referred to TelemetryEndpoints::new()

* Add tests for telemetry

* Fix typo and fix code in docs

* Return error on failing to override telemetry

* Use expect instead of suppressing errors on must-be-valid telemetry endpoints

* Update telemetry unit tests to use expect instead of unwrap

* Implement custom deserializer for TelemetryEndpoints

* Fix typo

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* make sure return ready iterator once state is updated

* update sc_basic_authorship tests

* update node tests

* fix manual seal

* actually fix service test

* add tests

* Update client/basic-authorship/src/basic_authorship.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* helper function

* review suggestions

* warning and continue

* add debug log

* use futures::chennel::oneshot

* use declaration bound

* no option for updated_at

* no allocation

* ready_at / ready

* Update client/transaction-pool/src/lib.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/transaction-pool/src/lib.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/transaction-pool/src/lib.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/transaction-pool/src/lib.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/transaction-pool/src/lib.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/transaction-pool/src/lib.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* fork-tree: prune returns all pruned node data

* epoch-changes: split EpochHeader vs epoch data

* EpochChanges::viable_epoch and add missing comments

* Incoperate the new epoch_changes interface for BABE

* Fix BABE tests

* Fix fork-tree pruning issue

* Fix tests

* Fix pruning algorithm

* fork-tree: implement map function for mapping one value type to another

* Add migration script for new epoch changes scheme

* Update utils/fork-tree/src/lib.rs

Co-Authored-By: André Silva <andre.beat@gmail.com>

* Update client/consensus/slots/src/lib.rs

Co-Authored-By: André Silva <andre.beat@gmail.com>

* Remove authorities_len.is_none check, which is duplicate of unwrap_or(false)

* Update client/consensus/epochs/src/lib.rs

Co-Authored-By: André Silva <andre.beat@gmail.com>

* Update client/consensus/epochs/src/lib.rs

Co-Authored-By: André Silva <andre.beat@gmail.com>

* No trailing ; for return statement

* Use VERSION_KEY for migration

* Fix issues that removed nodes are not properly added into removed list

* Add comments indicating end_slot is non-inclusive

* fork-tree: use &mut F for map type declaration

* Add tests for v0 epoch_changes migration

* Fix babe RPC tests

Co-authored-by: André Silva <andre.beat@gmail.com>

* removes use of sc_client::Client from node-transaction-factory

* move sc-block-builder to [dev-dependencies] in substrate-test-runtime

* ChainSpec trait

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Added docs

* Fixed build

* Fixed build

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* client/authority-discovery: Instrument code with Prometheus

Introduce Prometheus metrics into the authority discovery module
enabling one to observe:

- authority_discovery_times_published_total

- authority_discovery_amount_external_addresses_last_published

- authority_discovery_times_requested_total

- authority_discovery_dht_event_received

* client/authority-discovery/src/lib.rs: Reword metric help texts

Co-Authored-By: Ashley <ashley.ruglys@gmail.com>

* client/authority-discovery/src/lib.rs: Reword metric help text

Co-Authored-By: Ashley <ashley.ruglys@gmail.com>

* client/authority-discovery/tests: Fix struct instantiation + basic test

Co-authored-by: Ashley <ashley.ruglys@gmail.com>

* Update to libp2p 0.16.2

* Use libp2ps implementation of a wasm websocket transport

* Remove explicit Configuration type in node-cli

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* removes use of sc_client::Client from sc-rpc

* remove Client impl from sc-finality-benches

* remove client impl from sc-finality-grandpa

* read_proof accepts iterator

* remove generic Executor param from ExecutorProvider

* fix long ass line

* code style changes

* merge with master

Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>

* TODOs

* Remove superfluous:

* partial implementation

* full implementation

* fix preferences

* update comments

* upgrade test WIP

* fix more tests

* fix cutoff

* fix saturation

* comment

* upgrade mock

* upgrade test

* WIP migration

* WIP migration

* remove slot stake stuff

* fix merge

* migration of ledger

* remove equalize from test

* add test

* fix

* update doc

* fix compilation

* improve test readibility

* improve doc

* fix most todo

* fix migration and test

* remove println

* WIP

* add test and spec

* weight

* update doc

* safer end_era

* fix exposure of conversion

* Revert "safer end_era"

This reverts commit 72ff737d27be67d87308514b13e2574bc5f09fce.

* fix useless put

* exposure clipped

* doc

* fix payout with clipped

* fix node runtime

* add doc

* pluggable and generalized staking module

* remove print

* update doc

* refactor

* improve documentation and implementation

* fix test

* Fix test

* fix test

* fix test

* fix remove lowest stake from exposure, not biggest.

* nomination index arguments in nominator_payout

* add test

* try to fix offence

* apply slashed and bond eras until active era

* doc

* update spec version

* add test upgrade from previous test environment

* Apply suggestions from code review

Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com>

* nominators upgrade has been cleaned

* dynamic history depth implementation

* make current_era - history_depth included

* Change equality check to start era to less than or equal

* Use era specific validator prefs

* Add print statement and comment about start era if <

* fix next_reward overflow

* make more check for bad era claim for zero cost

* small refactor

* code refactor + fix use of deprecated storage

* fix wasm build

* add comment

* Fix tests

* remove outdated comment

* Apply suggestions from code review

Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com>

* gather active era information into one storage

Co-authored-by: thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Add a few things

* Add finality_grandpa_round

* fix fg tests

* Nitpicks

* Nitpicks

* Fix name of prometheus crate

* removes use of sc-client from sc-basic-authorship

* refactor use of ProposerFactory

* correct dep path

* removes use of sc_client::Client from sc_finality_grandpa

* code formatting

* code formatting

* removes use of sc_client::Client from sc_finality_grandpa

* Squash and rebase from gav-lazy-reaping

* Bump version

* Bump runtime again

* Docs.

* Remove old functions

* Update frame/balances/src/lib.rs

Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com>

* Update frame/contracts/src/lib.rs

Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com>

* Warnings

* Bump runtime version

* Update frame/democracy/src/lib.rs

Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com>

* Update frame/system/src/lib.rs

* Clean up OnReapAccount

* Use frame_support debug

* Bump spec

* Renames and fix

* Fix

* Fix rename

* Fix

* Increase time for test

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com>

* remove networkspecialization

* Fix most of the fallout

* get all tests compiling

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

Related to #4776 
Related to https://github.com/paritytech/polkadot/pull/832

To summarize the changes:
1. I did not manage to validate with types the service's Configuration. But I did reduce the possibility of errors by moving all the "fill" functions to their respective structopts
2. I split params.rs to multiple modules: one module params for just CLI parameters and one module commands for CLI subcommands (and RunCmd). Every command and params are in their own file so things are grouped better together and easier to remove
3. I removed the run and run_subcommand helpers as they are not helping much anymore. Running a command is always a set of 3 commands: 1. init 2. update config 3. run. This still allow the user to change the config before arguments get parsed or right after.
4. I added tests for all subcommands.
5. [deleted]

Overall the aim is to improve the situation with the Configuration and the optional parameters, add tests, make the API more consistent and simpler.

* removes use of sc_client::Client from sc_consensus_babe

* Update client/consensus/babe/src/lib.rs

Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com>

* node-template: disable grandpa observer

* node: disable grandpa observer

* node: add doc about grandpa-voter infallible task

* grandpa: remove grandpa observer from public API

* grandpa: ignore observer_enabled field in config

* Remove deprecated API

* Remove (some) allow(dead_code)

* Bump spec_version

* Fix import, remove allow dead code.

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

This prs cleans up some of the frame benchmarking stuff:
- Move CLI into `frame-benchmarking-cli`. No frame related CLI should
exists in the default Substrate CLI.
- Move all traits and types related to frame benchmarking into the
`frame-benchmarking` trait. Frame types should be isolated in Frame.

* Initial inspect.

* WiP

* Add parsing tests.

* Finalize CLI.

* Update to latest substrate.

* Remove unused imports.

* Support ImportParams as well, to get the right pruning setting.

* Mention in docs that hash is no 0x.

* Move bytes above extrinsics.

* Switch to fill helper from sc_cli.

* Remove overwrite.

* Fix error.

* Fix error message.

* Remove extra allow.

* init_config

* babe_epochAuthorship
remove test-helpers from sp-keyring, bump spec_version, impl_version

* bump Cargo.lock

* add BabeRPC to node-rpc

* rename to BabeApi, remove err_derive

* pass &ServiceBuilder to with_rpc_extensions callback

* sc-consensus-babe-rpc

* Update client/consensus/babe/src/lib.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Better docs, code style chanegs

Co-Authored-By: André Silva <andre.beat@gmail.com>

* new line at the end of Cargo.toml

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
Co-authored-by: André Silva <andre.beat@gmail.com>

* Revalidation queeue.

* add docs and license

* move test

* refactor worker to async/await

* address review

* fix warnings

* update Cargo.lock

* move background task to service

* use tomusdrw loop

* naming

* return From::from

* add doc comment

* add more doc comments

* fix merge bug

* add doc comment for test function

* Update client/transaction-pool/src/testing/pool.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* more review fixes

* refactor to allow service keep background tasks from isntantiated subsystems

* use const delay

* fix fallout

* remove fallout

* remove already moved test

* fix doc test

* add valid_at helper

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Refactor BlockImportParams to be non_exhaustive

* Fix cargo check compile

* Basic account composition.

* Add try_mutate_exists

* De-duplicate

* Refactor away the UpdateBalanceOutcome

* Expunge final UpdateBalanceOutcome refs

* Refactor transfer

* Refactor reservable currency stuff.

* Test with the alternative setup.

* Fixes

* Test with both setups.

* Fixes

* Fix

* Fix macros

* Make indices opt-in

* Remove CreationFee, and make indices opt-in.

* Fix construct_runtime

* Fix last few bits

* Fix tests

* Update trait impls

* Don't hardcode the system event

* Make tests build and fix some stuff.

* Pointlessly bump runtime version

* Fix benchmark

* Another fix

* Whitespace

* Make indices module economically safe

* Migrations for indices.

* Fix

* Whilespace

* Trim defunct migrations

* Remove unused storage item

* More contains_key fixes

* Docs.

* Bump runtime

* Remove unneeded code

* Fix test

* Fix test

* Update frame/balances/src/lib.rs

Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com>

* Fix ED logic

* Repatriate reserved logic

* Typo

* Fix typo

* Update frame/system/src/lib.rs

Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com>

* Update frame/system/src/lib.rs

Co-Authored-By: Shawn Tabrizi <shawntabrizi@gmail.com>

* Last few fixes

* Another fix

* Build fix

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Jaco Greeff <jacogr@gmail.com>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Fix chain-spec and make sure it does not breaks again

* REview feedback

`GossipEngine` spawns two tasks, one for a periodic tick, one to forward
messages from the network to subscribers. These tasks hold an `Arc` to a
`GossipEngineInner`.

To reduce the amount of shared ownership (locking) this patch integrates
the two tasks into a `Future` implementation on the `GossipEngine`
struct. This `Future` implementation can now be called from a single
owner, e.g. the `finality-grandpa` `NetworkBridge`.

As a side effect this removes the requirement on the `network-gossip`
crate to spawn tasks and thereby removes the requirement on the
`finality-grandpa` crate to spawn any tasks.

This is part of a greater effort to reduce the number of owners of
components within `finality-grandpa`, `network` and `network-gossip` as
well as to reduce the amount of unbounded channels. For details see
d9837d7d, 5f80929d and 597c0a6c.

* Remove modes.

* Refactor.

* Make WASM browser thing compile

* Fix

* updated exit-future (github repo)

* Switch to broadcast crate

* Migrate client/cli

* Switch exit-future to modernize branch

* Small changes

* Switch to cargo version and fix fg tests

* fix basic-authorship

* Fix crash on grafana macro

* Fix grafana macro

* Switch node python version

* Disable record_metrics_slice in grafana macro on wasm

* Update client/grafana-data-source/src/lib.rs

* Revert "Update client/grafana-data-source/src/lib.rs"

This reverts commit 888009a8e0b7051bd4bfbbfdb0448bcf2e2aae93.

* Add wasm support for state machine

* Switch to my own libp2p version

* Revert "Switch to my own libp2p version"

This reverts commit ce613871b59264b3165b45c37943e6560240daa7.

* Revert "Add wasm support for state machine"

This reverts commit de7eaa0694d9534fc3b164621737968e9a6a7c5f.

* Add sc-browser

* Squash

* remove sc-browser

* Fix keystore on wasm

* stubs for removed functions to make env compatible with old runtimes

* Add test (that doesn't work)

* Fix build scripts

* Revert basic-authorship due to no panics

* Revert cli/informant

* Revert consensus

* revert offchain

* Update utils/browser/Cargo.toml

Co-Authored-By: Benjamin Kampmann <ben@gnunicorn.org>

* export console functions

* Add new chainspec

* Fix ws in chain spec

* revert chainspec

* Fix chainspec

* Use an Option<PathBuf> in keystore instead of cfg flags

* Remove crud

* Only use wasm-timer for instant and systemtime

* Remove telemetry changes

* Assuming this is ok

* Add a KeystoreConfig

* Add stubs back in

* Update libp2p

* Revert "Add stubs back in"

This reverts commit 4690cf1882aa0f99f7f00a58c4080c8aa9b77c36.

* Remove commented js again

* Bump kvdb-web version

* Fix cli

* Switch branch on futures-timer

* Fix tests

* Remove sc-client test build in check-web-wasm because there isn't a good way to build futures-timer with wasm-bindgen support in the build

* Remove more things ^^

* Switch branch on futures-timer back

* Put DB io stats behind a cfg flag

* Fix things

* Don't timeout transports on wasm

* Update branch of futures-timer and fix bad merge

* Spawn informant

* Fix network test

* Fix delay resets

* Changes

* Fix tests

* use wasm_timer for transaction pool

* Fixes

* Switch futures-timer to crates

* Only diagnose futures on native

* Fix sc-network-test tests

* Select log level in js

* Fix syncing ;^)

* Allow disabling colours in the informant

* Use OutputFormat enum for informant

* MallocSizeOf impl on transaction pool broke stuff because wasm_timer::Instant doesnt impl it so just revert the transaction pool to master

* Update futures-diagnose

* Revert "MallocSizeOf impl on transaction pool broke stuff because wasm_timer::Instant doesnt impl it so just revert the transaction pool to master"

This reverts commit baa4ffc94fd968b6660a2c17ba8113e06af15548.

* Pass whole chain spec in start_client

* Get Instant::now to work in transaction pool again

* Informant dep reordering

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Svyatoslav Nikolsky <svyatonik@gmail.com>
Co-authored-by: Benjamin Kampmann <ben.kampmann@googlemail.com>
Co-authored-by: Demi Obenour <48690212+DemiMarie-parity@users.noreply.github.com>

* Fix CLI setup again

We need to set `config_dir` and `database_path` for almost every
command.
This fixes `purge-chain` and also adds a test to make sure we don't
break it again.

* Adds missing test files

* Split methods

* Enable trace timings logs for transaction factory.

* babe: pass epoch data via intermediates

* Switch to use Box<dyn Any> for intermediates

* Set intermediate.epoch to be Option

* Fix proposer should put out an empty intermediate

* Remove unnecessary encode/decode

* Add EpochData to block_import_params in slot worker

* Fix aura compile

* Fix integration test

These are a few changes I missed during the refactoring.

1. Initialization issue and boilerplate

    Most importantly: part of the `Configuration` initialization was done in `sc_cli::init`. This means the user can not benefit from this initialization boilerplate if they have multiple `Configuration` since `sc_cli::init` can only be called once.

2. Boilerplate for `VersionInfo` and `Configuration`

    I'm also answering to the critic of @bkchr on the initialization using version: https://github.com/paritytech/substrate/pull/4692/files/bea809d4c14a2ede953227ac885e3b3f9771c548#r372047238 This will allow initializing a `Configuration` and provide the version by default.

3. Loading the `chain_spec` explicitly

    In the past it was done automatically but in some cases we want to delay this. I moved the code to `Configuration.load_spec()` so it can be called later on. `chain_spec` can also be written directly to the `Configuration` without using this `load_spec` helper.

4. [deleted]

5. Fixing issue that prevents the user to override the port

    In the refactoring I introduced a bug by mistake that could potentially prevent the CLI user to override the ports if defaults where provided for these ports (only on cumulus).

6. Change task_executor from Box to Arc

    This is useful for cumulus where we have 2 nodes with 2 separate Configuration that need to spawn tasks to the same runtime.

7. Renamed TasksExecutorRequired to TaskExecutor

    For consistency.

This is related to https://github.com/paritytech/cumulus/issues/24

This is the continuation (and hopefully the end of) #4692