This PR refactors the metrics measuring and Prometheus exposing entity in sc-service into its own submodule and extends the parameters it exposes by:

- system load average (over one, five and 15min)
- the TCP connection state of the process (lsof), refs #5304
- number of tokio threads
- number of known forks
- counter for items in each unbounded queue (with internal unbounded channels)
- number of file descriptors opened by this process (*nix only at this point)
- number of system threads (*nix only at this point)

refs #4679

Co-authored-by: Max Inden <[email protected]>
Co-authored-by: Ashley <[email protected]>

* Split the Roles bitfield in three

* Forgot to include some changes

* Fix cli test

* More test fixes

* Oh God, merging master broke other tests

* Didn't run the doctests

* Address review

* I'm trying to fix the build blindly because it's taking a good hour to compile on my machine

* Address some review

* Also update the peerset's API to make sense

* Fix peerset tests

* Fix browser node

* client: distinguish between local and network authority

Co-authored-by: André Silva <[email protected]>

* Better logging for notifications and buffer size increase

* Address review

* Improve warning about notifications queue and remove spurious triggers

* Add a sub_libp2p_notifications_queues_size Prometheus metric

* Fix network tests

* Address review

* More emojies and spaces

* Sync emojis

* More emoji

This pr ensures that we don't panic because of a deadline that is
already reached in basic authorship. This is mainly useful for debug
builds, as release builds normally don't have debug assertions enabled.

* Registering state from overlay.

* fix

* fix2

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* Switch to the new protocol

* Oops, forgot to remove light_dispatch.rs

* Fix tests

* Address review

This was removed in https://github.com/paritytech/substrate/pull/5291
but is still too early to do it as it requires nodes still on Polkadot
v0.7.26 to resync from scratch.

* A few missing emojies

* P2p message emojis

* Add a bit of colour

* format

* Introduce a couple of spaces

* Some spaces

* Reset by removing pos-genesis changes

* CLI option for DB cache size

* Update Cargo.lock

Co-authored-by: Shawn Tabrizi <[email protected]>

* client/finality-grandpa: Instrument until-imported queue

The `UntilImported` queue takes as input finality grandpa messages that
depend on blocks that are not yet imported and holds them back until
those blocks are imported.

This commit adds a basic metric, the amount of messages waiting in the
queue, to the module. For now this metric is only available for the
global `UntilImported` queue awaiting blocks for commit and catch-up
messages.

* client/finality-grandpa/src/until_imported: Update metric help text

Co-Authored-By: Ashley <[email protected]>

Co-authored-by: Ashley <[email protected]>

* Increase limit on light client response size

* Address review

* Implements mocking of runtime apis

This pr adds support for easily mock runtime api implementations for
tests by using the `mock_impl_runtime_apis!` macro. The syntax is
similar to `impl_runtime_apis!`. The mocked implementation automatically
implements `ApiExt`, `ApiErrorExt` and `Core` as these are required by
the runtime api traits, but not required in tests or only a subset of them.

* Fix warnings

* Update primitives/api/proc-macro/src/utils.rs

Co-Authored-By: Nikolay Volf <[email protected]>

* Review feedback

Co-authored-by: Nikolay Volf <[email protected]>

* imporoved intervallier version

* remove extra event

* use version 0.3.1

* fix warning

* Added option to disable default block announce

* Added on_block_imported on NetworkService

* Revert "Added on_block_imported on NetworkService"

This reverts commit ba360cad.

* Do not announce block if set to not announce block

* Revert fix

* Moving default announce block to NetworkConfig

* WIP

Forked at: 41bb2193
Parent branch: origin/master

* WIP

Forked at: 41bb2193
Parent branch: origin/master

* Removing boolean in favor of explicit call

* Fixing tests

* WIP

Forked at: 41bb2193
Parent branch: origin/master

* WIP

Forked at: 41bb2193
Parent branch: origin/master

* increase spec_version

* increase spec_version

* Fixed test

* Fixing test

* Renamed should_announce_imported_blocks to announce_imported_blocks

* Updated assert_cmd

The library `sysinfo` exposes process memory as kibibytes and not bytes,
thus the value needs to be multiplied by 1024 to comply with the metric
name and the Prometheus base units [1].

[1] https://prometheus.io/docs/practices/naming/#base-units

* Fix tried to send handshake twice

* Fix wrong boolean

* Change to debug

* Add KEY_KIND_ID to the public trait

This change is being introduced for the purpose of identifying a public
key with it's identifier and algorithm "kind".

* Use `sign_with` as implemented in BareCryptoStore

* Implement `sign_with` in sc_keystore

* Fix inconsistencies, use *_KIND_ID in sp_core testing

* Rename KeyKindId to CryptoTypeId

* Remove pair-returning functions from BareCryptoStore trait

* Define CryptoTypeId in app-crypto macros

* Add functions to get keys supported by keystore

* Fix sign_with signature to include CryptoTypePublicPair

* Add `sign_with_any` and `sign_with_all`

* Use keystore.sign_with in auth_discovery

* Rename get_supported_keys -> supported_keys

* Added headers to function docstrings

* Use chain instead of extending a temp vector

* Fixed some code formatting

* Restrict size of CryptoTypeId

This is to be able to use Encode/Decode derives and the overcome having
the size being unknown at compile-time.

* Implement sign_with in the trait itself

* Remove whitespace

* Use key_type also as a CryptoTypeId in app_crypto macros

* Rename `get_keys` to `keys` in BareCryptoStore

* Remove usage of key_pair funcs in tests

* Adjust docstring for *_CYPTO_ID constants

* Fix failures

* Simplify mapping on keys

* Remove one let

* Fixed typo

* PR feedback

* remove whitespace

* Zip keys and signatures

* Use into_iter & remove cloned

* Pass index to MissingSignature

* Use typed errors instead of strings for BareCryptoStore

* Implement Debug for trait error

* Use hashsets for better performance for supported_keys

* Make sure keys are inserted into the keystore

* Make sign_with_all return type consistent with `sign_with`

* Rename Error to BareCryptoStoreError

* Rename CRYPT_TYPE_ID -> CRYPTO_ID

* Remove unnecessary CRYPTO_ID declaration in Public trait

* Convert pub key to CryptoTypePublicPair

* Fix use

* Fix code style

* Implement From on CryptoTypePublicPair in app_crypto macros

* Change CryptoTypePublicPair to a struct

* Implement Display on CryptoTypePublicPair

* Pass CryptoTypePublicPair to MissingSignature error

* Adjust docs according to function signature

* Unify keys implementation

* Fix RPC author tests

* Fix stackoverflow

* Tabify spaces

* Pass KeyTypeId to error for easier debugging

* Fix asserts

* Use ToHex to format public key

* Use constants from sp_core

* Rename testing KeyTypeId constants

* Please compiler

* Restore KeyTypeId names

apparently, they're not only used in tests

* Use BareCryptoStoreError instead of String

* Document return value

* Fix borrow check

* Convert to hashset internally

* WIP - iter_keys

* Return raw_public_keys

* Address PR feedback

* Address PR Feedback

* Fix hexdisplay import error

* Update primitives/core/src/traits.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Make transactions and block announces use notifications

* Add documentation

* Add docs.rs metadata to all cargo.toml files

* Remove docs.rs metada in substrate's cargo.toml

The signal handler will be reset to the default system one. This means
the program will be exited instead of panicking. This is required to
support piping in the console.

* adds state_queryStorageAt

* make at param for query_storage_at optional

* Update client/rpc/src/state/state_full.rs

Co-Authored-By: Tomasz Drwięga <[email protected]>

* adds query_storage_at to StateBackend

Co-authored-by: Tomasz Drwięga <[email protected]>

* client/finality-grandpa/src/until_imported: Refactor schedule_wait

Previously `BlockUntilImported::schedule_wait` took two closures, one to
report ready items and one to report items to await. None of the implementors of
`BlockUntilImported` call both closures. From a symantic perspective it
would as well not make sense to both await something and state that it
is ready.

Instead with this commit `BlockUntilImported::schedule_wait` simply
returns whether the given item needs waiting or is ready to be passed
on.

This reduces complexity by:

- Removing side effects through the two closures.

- Reducing borrowing given that `UntilImported` `ready` and `pending`
don't need to be borrowed from within the two closures.

- Removes the need for trait bounds for the two closures.

* client/finality-grandpa/src/until_imported: Fix comments

Co-Authored-By: André Silva <[email protected]>

Co-authored-by: André Silva <[email protected]>

* Improve sc-network's documentation for network protocols

* Add note about protocol id

* Apply suggestions from code review

Co-Authored-By: Max Inden <[email protected]>

Co-authored-by: Max Inden <[email protected]>

* Allow syncing to peers with finalized common block

* Added test

* Check for url validity when creating TelemetryEndpoints

* Update code that used TelemetryEndpoints::new()

* Update commennts that referred to TelemetryEndpoints::new()

* Add tests for telemetry

* Fix typo and fix code in docs

* Return error on failing to override telemetry

* Use expect instead of suppressing errors on must-be-valid telemetry endpoints

* Update telemetry unit tests to use expect instead of unwrap

* Implement custom deserializer for TelemetryEndpoints

* Fix typo

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* client/finality-grandpa/src/until_imported: Refactor BlockGlobalMessage

`BlockGlobalMessage` owns an `inner` which contains (1) a count for the
amount of outstanding blocks to be waited on and (2) the message itself.

Given that both is already wrapped in an `Arc` there is no need to keep
track of the outstanding blocks, given that it simply corresponds to the
amount of strong reference counts on the `Arc` itself.

This commit removes the atomic counter within `inner` and piggy backs on
the `Arc` reference counter instead.

* client/finality-grandpa/src/until_imported: Remove useless match

* client/finality-grandpa/src/until_imported.rs: Remove unused var

Co-Authored-By: André Silva <[email protected]>

* client/finality-grandpa/src/until_imported: Address comment suggestion

Co-authored-by: André Silva <[email protected]>

* delias outer node template

* dealias pallet template

* Fix mock

* attempt to fix warning

* Revert "attempt to fix warning"

This reverts commit 3d39142b.

* Add call to test interface.

* always depend on latest alpha version

* integrate arithmetic fuzzer into root workspace

* WiP

* Support source in the runtime API.

* Finish implementation in txpool.

* Fix warning.

* Fix tests.

* Apply suggestions from code review

Co-Authored-By: Kian Paimani <[email protected]>
Co-Authored-By: Nikolay Volf <[email protected]>

* Extra changes.

* Fix test and benches.

* fix test

* Fix test & benches again.

* Fix tests.

* Update bumpalo

* Fix doc test.

* Fix doctest.

* Fix doctest.

Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Nikolay Volf <[email protected]>

* fix propagation in network

* Update client/service/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* fix suggestion

Co-authored-by: Bastian Köcher <[email protected]>