* service: add spawner for essential tasks

* node: spawn block authoring and grandpa voter as blocking tasks

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Before we only updated the chain info of sync when we have imported
something using the import queue. However, if you import your own
blocks, this is not done using the import queue and so sync is not
updated. If we don't do this, it can lead to sync switching to "major
sync" mode because sync is not informed about new blocks. This
especially happens on Cumulus, where a collator is selected multiple
times to include its block into the relay chain and thus, sync switches
to major sync mode while the node is still building blocks.

* Seperate out the complexity in ServiceBuilder::build_common into seperate functions

* Fix line widths

* Move some functions to their respective crates

This is useful for both security and performance reasons. IPC is faster
than TCP, and it is subject to OS access controls.

Add a feature to create automatically a random temporary directory for base path & remove `Clone` (#6221)

* Initial commit

Forked at: 4adac40c
Parent branch: origin/master

* Add a feature to create automatically a temporary directory for base path

* doc fix and todos

* use parking_lot instead

* use refcell instead since we stay in the main thread

* remove Clone derives

* add test

* solving dependency issue

* clarifying doc

* conflict argument with base-path

* WIP

Forked at: 4adac40c


Parent branch: origin/master

* revert dep deletion

* fixing test and making base_path optional

* hold basepath while the service is running

* fixes

* Update client/cli/src/params/shared_params.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/Cargo.toml

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/mod.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/config.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* WIP

Forked at: 4adac40c


Parent branch: origin/master

* improve doc

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Initial commit

Forked at: 1d93746b
Parent branch: origin/master

* Add a Service Configuration's field + adapt informant + provide means to CLI

* CLEANUP

Forked at: 1d93746b
Parent branch: origin/master

* fix tests

* fixed bad path to object

* Change OutputFormat enum to struct

* Add informant_prefix to builder and service

* Revert "Change OutputFormat enum to struct"

This reverts commit cd86c583c92668426c35cc174401155bf2880c1f.

* Revert "fix tests"

This reverts commit a3c306ebe94720f350c5bc74b9c5fcde2565d340.

* Revert "Add a Service Configuration's field + adapt informant + provide means to CLI"

This reverts commit 9c2e7267423305705916c30d605893524113c8e3.

* Implementation using the ServiceBuilder

* reduce line length

* fix line width again

* WIP

Forked at: 1d93746b
Parent branch: origin/master

* WIP

Forked at: 1d93746b
Parent branch: origin/master

* WIP

Forked at: 1d93746b
Parent branch: origin/master

* use struct instead of enum

* WIP

Forked at: 1d93746b


Parent branch: origin/master

* Update client/service/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* improve doc

* Update client/service/src/builder.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/builder.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* change code

* Update client/informant/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* enable_color

* reorg log

* remove macro

* Removed builder for informant prefix

* fix doc

* Update client/informant/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/informant/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/informant/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/informant/src/lib.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/builder.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/builder.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/service/src/builder.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* service: pass DenyUnsafe to rpc extensions

* node: add DenyUnsafe to rpc full node dependencies

* client: fix whitespace in rpc policy file

* babe: treat epochAuthorship rpc method as unsafe

* babe: add test for unsafe rpc method

* babe: rename babe rpc handler

* service: traitify rpc extension builder

* service: make the rpc extensions builder api non-breaking

* service: revert changes from light node rpc extensions builder

* node: remove unnecessary type in service creation

* service: cleanup with_rpc_extensions implementation

* service: add missing docs to RpcExtensionBuilder

* remove on_block_imported

* Update client/network/src/service.rs

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* change propagation

* add bound

* Initial commit

Forked at: 9acf88f5
Parent branch: origin/master

* Improve error message when essential task failed

* client: Replace `unsafe_rpc_expose` with an `RpcMethods` enum

which can be either Default, Safe or Unsafe. The idea is to have the
following:
|                       | --rpc-external=false  | --rpc-external=true   |
|---------------------  |-------------------    |-----------------      |
| --rpc-methods=Default |                       | unsafe calls denied   |
| --rpc-methods=Safe    | unsafe calls denied   | unsafe calls denied   |
| --rpc-methods=Unsafe  |                       |                       |
Since the previous `unsafe-rpc-expose` option was confusing.

* client: Only warn against exposing externally unsafe RPC method set

* Apply suggestions from code review

Co-Authored-By: Cecile Tonglet <cecile.tonglet@cecton.com>

* cli: Rephrase doc comment for rpc_methods config

* Improve debuggability of build_spec_works

...by printing to stderr the stderr of the command. This is normally
suppressed for s...

Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>

* Drop client from sc-network and sc-client-db, move LongestChain to sc-client-api

* move leaves, cht, in_mem to sc-client-api, drop client from sc-finality-grandpa

* drop sc-service from sc-rpc

* drop sc-service from sc-consensus-aura

* drop sc-client from manual-seal and babe

* drop sc-client from utils/frame/rpc/system and utils/frame/benchmarking-cli

* drop sc-client from bin/node and bin/node-template

* drop sc-client

* fix tests

* remove check -p sc-client from gitlab.yml

* fix warnings

* fixes ui test

* fix light client tests

* adds associated Client type to AbstractService

* adds UsageProvider to Client

* fixed ui test, again

* tried and failed to get node-cli to compile for wasm

* thanks to tomaka for helping me get node-cli to compile for wasmm

* ui test pls pas 



* all tests passing 🪄

* no_run documentation code

* rm -f documentation code

* ClientProvider

* fix mega trait

* move LongestChain to sc-consensus, use adds minimal bounds to AbstractService::Client

* adds license to sc-consensus

Co-authored-by: Benjamin Kampmann <ben@parity.io>

* Remove TaskManagerBuilder

* Clean up use declaration fo SpawnTaskHandle

Co-Authored-By: Pierre Krieger <pierre.krieger1708@gmail.com>

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* sc-cli: Use type-safe constructors for RPC/Prometheus interfaces

* service: Simplify rpc handler creation

Could probably be further simplifies once [this][commit] lands.

[commit]: https://github.com/paritytech/jsonrpc/commit/20485387ed06a48f1a70bf4d609a7cde6cf0accf

* service: Streamline some HTTP & WS server start logic

* client: Introduce a simple RPC policy mechanism

* rpc/system: Check unsafe RPCs

* rpc/offchain: Check unsafe RPCs

* rpc/author: Check unsafe RPCs

* make new contructor

* add metrics to txpool

* fix review

* fix doc comment

* change to counters

* Update client/transaction-pool/src/metrics.rs

Co-Authored-By: Max Inden <mail@max-inden.de>

* Update client/transaction-pool/src/metrics.rs

Co-Authored-By: Max Inden <mail@max-inden.de>

* Update client/transaction-pool/src/metrics.rs

Co-Authored-By: Max Inden <mail@max-inden.de>

* Update client/transaction-pool/src/lib.rs

Co-Authored-By: Max Inden <mail@max-inden.de>

* Update client/transaction-pool/src/lib.rs

Co-Authored-By: Max Inden <mail@max-inden.de>

* use dedicated wrapper

Co-authored-by: Max Inden <mail@max-inden.de>

* Add alternatives to system_networkState

* Fix tests

* Apply suggestions from code review

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Add new RPC method to get the chain type

This adds a new RPC method to get the chain type of the running chain.
The chain type needs to be specified in the chain spec. This should make
it easier for tools/UI to display extra information without needing to
rely on parsing the chain name.

* Update client/rpc-api/src/system/mod.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Primitive crate

* Feedback

Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* WIP

Forked at: d6aa8e95
Parent branch: origin/master

* Rename IntoConfiguration to CliConfiguration

* Renamed into_configuration to create_configuration

* WIP

Forked at: d6aa8e95
Parent branch: origin/master

* WIP

Forked at: d6aa8e95
Parent branch: origin/master

* Move keystore params to its own module

* Use in-memory keystore even for build-spec

* Enforce proper value for node name

* dev_key_seed

* Telemetry endpoints

* rustfmt

* Converted all RunCmd

* rustfmt

* Added export-blocks

* Missed something

* Removed config_path in NetworkConfiguration (not used)

* Fixed warnings

* public_addresses is used but never set, keeping it

* Merge Configuration.node and NetworkConfiguration.node_name

...because they are the same thing

* Added: import-blocks

* Adding a proc_macro to help impl SubstrateCli

* WIP

Forked at: d6aa8e95
Parent branch: origin/master

* WIP

Forked at: d6aa8e95
Parent branch: origin/master

* WIP

Forked at: d6aa8e95
Parent branch: origin/master

* Re-export spec_factory from sc_cli

* Re-added all the commands

* Refactored node_key_params

* Fixed previous refucktoring

* Clean-up and removed full_version()

* Renamed get_is_dev to not confuse with Configuration field

* Fixed sc-cli-derive example

* Fixing tests

* Fixing tests and removing some (will re-add later)

* Fixing more tests

* Removes the need of type parameter

* Converting bin/node and simplifying API

* Converting more

* Converting last command

* WIP

Forked at: d6aa8e95


Parent branch: origin/master

* Fixing tests and added default for WasmExecutionMethod

* Fixing stuff

* Fixed something I broke oops

* Update Cargo.lock

* Moving things around

* Convert everything to Result

* Added new macros to simplify the impl of CliConfiguration

* Added a macro to generate CliConfiguration automatically for subcommands

* Revert... too many macros (this one is not really useful)

This reverts commit 9c516dd38b40fbc420b02c1f8e61d5b2b1a4e434.

* Renamed is_dev to get_is_dev

Good enough for now

* Fixed name roles (this is plural, not singular)

* Clean-up

* Re-export NodeKeyConfig and TelemetryEndpoints from sc_service

* Improve styling/formatting

* Added copyrights

* Added doc and fixed warnings

* Added myself to code owners

* Yes it is needed according to the history

* Revert formatting

* Fixing conflict

* Updated build.rs

* Cargo.lock

* Clean-up

* Update client/cli-derive/Cargo.toml

Co-Authored-By: Seun Lanlege <seunlanlege@gmail.com>

* Fail if using proc_macro and build.rs is not set properly

* Dropped all get_ in front of methods

* Clean-up

* Fixing proc macro missing env var

* Get the configuration inside the Runtime (needed for polkadot)

* Clean-up

* Get is_dev from argument like the others

* Get chain ID instead of chain spec from shared params

* &self is passed to spec_factory/load_spec

* Wrong text

* Fix example

* Officialize macro and made a cool doc

* Renamed spec_factory to load_spec (substrate_cli_configuration)

* Removed not so useful ChainSpec

* Renamed SubstrateCLI to SubstrateCli

* Added changelog for impl_version being full now

* Renamed Runtime to Runner

* Update changelog to show example

* Removed option on database cache size

* WIP

Forked at: d6aa8e95
Parent branch: origin/master

* Fix on removal of option

* typo

* Clean-up imports

* Added info in Cargo.toml

* typo

* remarks

* Moved function for build.rs to substrate-build-script-utils

* Fixed example & test of cli-derive

* Moved function for build.rs to substrate-build-script-utils

* Renamed substrate_cli_configuration to substrate_cli oops

It implements SubstrateCli not CliConfiguration!

* Added documentation and wrapper macro

* Removed option on database cache size

* Removed option on database cache size

* Clean-up

* Reduce risk of errors due to typos

* Removed option on database cache size

* Added NOTE as suggested

* Added doc as suggested

* Fixed test

* typo

* renamed runtime to runner

* Fixed weird argument

* More commas

* Moved client/cli-derive to client/cli/derive

* Added 7 tests for the macros

* Improve error message

* Upgrade assert_cmd

* Fixing missing stuff

* Fixed unused import

* Improve SubstrateCli doc

* Applied suggestions

* Fix and clean-up imports

* Started replacing macros WIP

* WIP

Forked at: d6aa8e95
Parent branch: origin/master

* WIP

Forked at: d6aa8e95
Parent branch: origin/master

* WIP

Forked at: d6aa8e95
Parent branch: origin/master

* Started removing substrate_cli

* WIP

Forked at: d6aa8e95
Parent branch: origin/master

* WIP

Forked at: d6aa8e95
Parent branch: origin/master

* WIP

Forked at: d6aa8e95


Parent branch: origin/master

* fixed bug introduced while refactoring

* Renamed NetworkConfigurationParams to NetworkParams for consistency sake

* Fixed test

* Update client/cli/src/commands/runcmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/runcmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/check_block_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update bin/node/cli/src/command.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update bin/node/cli/src/command.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Revert "Update client/cli/src/commands/export_blocks_cmd.rs"

This reverts commit 5906776953392c02beac6bc0bf50f8cbe1a12a01.

* Revert "Update client/cli/src/commands/check_block_cmd.rs"

This reverts commit f705f42b7f3d732be001141afee210fe46a1ef47.

* Revert "Update client/cli/src/commands/export_blocks_cmd.rs"

This reverts commit 8d57c0550164449e6eb2d3bacb04c750c714fcea.

* Revert "Update client/cli/src/commands/runcmd.rs"

This reverts commit 93e74cf5d2e1c0dc49cdff8608d59fc40fc59338.

* Revert "Update client/cli/src/commands/runcmd.rs"

This reverts commit 11d527ba345c0d79f0d3b5b071933d95474d0614.

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/import_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Changed ::sc_cli to $crate in the macro

* fixed tests

* fixed conflicts

* Fixing test

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/params/pruning_params.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Remove comment as suggested

* Apply suggestion

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update utils/frame/benchmarking-cli/src/command.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/params/pruning_params.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/params/node_key_params.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/params/network_params.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/lib.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Update client/cli/src/config.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Added doc

* Fixed error introduced after applying suggestion

* Revert "Update client/cli/src/params/pruning_params.rs"

This reverts commit 0574d06a4f1efd86e94c1214420a12e7a4be0099.

* Print error

* Apply suggestions from code review

* Remove useless Results

* Fixed CI failing on polkadot approval

Co-authored-by: Seun Lanlege <seunlanlege@gmail.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

This PR refactors the metrics measuring and Prometheus exposing entity in sc-service into its own submodule and extends the parameters it exposes by:

- system load average (over one, five and 15min)
- the TCP connection state of the process (lsof), refs #5304
- number of tokio threads
- number of known forks
- counter for items in each unbounded queue (with internal unbounded channels)
- number of file descriptors opened by this process (*nix only at this point)
- number of system threads (*nix only at this point)

refs #4679

Co-authored-by: Max Inden <mail@max-inden.de>
Co-authored-by: Ashley <ashley.ruglys@gmail.com>

* Split the Roles bitfield in three

* Forgot to include some changes

* Fix cli test

* More test fixes

* Oh God, merging master broke other tests

* Didn't run the doctests

* Address review

* I'm trying to fix the build blindly because it's taking a good hour to compile on my machine

* Address some review

* Also update the peerset's API to make sense

* Fix peerset tests

* Fix browser node

* client: distinguish between local and network authority

Co-authored-by: André Silva <andre.beat@gmail.com>

* More emojies and spaces

* Sync emojis

* More emoji

* Reset by removing pos-genesis changes

* CLI option for DB cache size

* Update Cargo.lock

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

* Added option to disable default block announce

* Added on_block_imported on NetworkService

* Revert "Added on_block_imported on NetworkService"

This reverts commit ba360cad96e0cb041d7047af30df2a35eb112449.

* Do not announce block if set to not announce block

* Revert fix

* Moving default announce block to NetworkConfig

* WIP

Forked at: ad90ab7e
Parent branch: origin/master

* WIP

Forked at: ad90ab7e
Parent branch: origin/master

* Removing boolean in favor of explicit call

* Fixing tests

* WIP

Forked at: ad90ab7e
Parent branch: origin/master

* WIP

Forked at: ad90ab7e
Parent branch: origin/master

* increase spec_version

* increase spec_version

* Fixed test

* Fixing test

* Renamed should_announce_imported_blocks to announce_imported_blocks

* Updated assert_cmd

* WiP

* Support source in the runtime API.

* Finish implementation in txpool.

* Fix warning.

* Fix tests.

* Apply suggestions from code review

Co-Authored-By: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-Authored-By: Nikolay Volf <nikvolf@gmail.com>

* Extra changes.

* Fix test and benches.

* fix test

* Fix test & benches again.

* Fix tests.

* Update bumpalo

* Fix doc test.

* Fix doctest.

* Fix doctest.

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Nikolay Volf <nikvolf@gmail.com>

* fix propagation in network

* Update client/service/src/lib.rs

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* fix suggestion

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* ChainSpec trait

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <bkchr@users.noreply.github.com>

* Added docs

* Fixed build

* Fixed build

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

* Revert "Build block without checking signatures (#4916)"

This reverts commit e50f6109.

* Some further clean ups

* Reorganize tasks into task manager

* move to separate file and improve api

* address api issues

* fix spawning inside closures

* decouple executor

* tasks_setup -> tasks_builder

* remove drops

* add deprecatiion comment

* add pub(super)

* fix identation

* Add a few things

* Add finality_grandpa_round

* fix fg tests

* Nitpicks

* Nitpicks

* Fix name of prometheus crate

* Start removing last few instances of futures01

* Use to_poll on wasm

* Revert "Use to_poll on wasm"

This reverts commit 1c61728f10d520df5f9b28c415a0db68e478b9c7.

* Fix fg test

* Upgrade network test futures

* Update offchain hyper version

* Update service test

* bump tokio to 0.2.10

* Removed some unneeded tokios

* fixes

* fix run_until_all_full

* Make service test debuggable

* Update client/offchain/src/api/http.rs

Co-Authored-By: Demi Obenour <48690212+DemiMarie-parity@users.noreply.github.com>

* Add service_test to test-int output

* nitpicking

* Finally fix test

* Give up and revert client/serviec/test

* Revert gitlab ci too

Co-authored-by: Demi Obenour <demi@parity.io>

* remove networkspecialization

* Fix most of the fallout

* get all tests compiling

Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com>

* Cleanup the exports of sc-network

* Fix RPC tests

* Revert moving the NetworkStateInfo trait

* Fix GrandPa tests