* client/finality-grandpa: Instrument until-imported queue

The `UntilImported` queue takes as input finality grandpa messages that
depend on blocks that are not yet imported and holds them back until
those blocks are imported.

This commit adds a basic metric, the amount of messages waiting in the
queue, to the module. For now this metric is only available for the
global `UntilImported` queue awaiting blocks for commit and catch-up
messages.

* client/finality-grandpa/src/until_imported: Update metric help text

Co-Authored-By: Ashley <[email protected]>

Co-authored-by: Ashley <[email protected]>

* Increase limit on light client response size

* Address review

* Implements mocking of runtime apis

This pr adds support for easily mock runtime api implementations for
tests by using the `mock_impl_runtime_apis!` macro. The syntax is
similar to `impl_runtime_apis!`. The mocked implementation automatically
implements `ApiExt`, `ApiErrorExt` and `Core` as these are required by
the runtime api traits, but not required in tests or only a subset of them.

* Fix warnings

* Update primitives/api/proc-macro/src/utils.rs

Co-Authored-By: Nikolay Volf <[email protected]>

* Review feedback

Co-authored-by: Nikolay Volf <[email protected]>

* imporoved intervallier version

* remove extra event

* use version 0.3.1

* fix warning

* Added option to disable default block announce

* Added on_block_imported on NetworkService

* Revert "Added on_block_imported on NetworkService"

This reverts commit ba360cad.

* Do not announce block if set to not announce block

* Revert fix

* Moving default announce block to NetworkConfig

* WIP

Forked at: 41bb2193
Parent branch: origin/master

* WIP

Forked at: 41bb2193
Parent branch: origin/master

* Removing boolean in favor of explicit call

* Fixing tests

* WIP

Forked at: 41bb2193
Parent branch: origin/master

* WIP

Forked at: 41bb2193
Parent branch: origin/master

* increase spec_version

* increase spec_version

* Fixed test

* Fixing test

* Renamed should_announce_imported_blocks to announce_imported_blocks

* Updated assert_cmd

The library `sysinfo` exposes process memory as kibibytes and not bytes,
thus the value needs to be multiplied by 1024 to comply with the metric
name and the Prometheus base units [1].

[1] https://prometheus.io/docs/practices/naming/#base-units

* Fix tried to send handshake twice

* Fix wrong boolean

* Change to debug

* Add KEY_KIND_ID to the public trait

This change is being introduced for the purpose of identifying a public
key with it's identifier and algorithm "kind".

* Use `sign_with` as implemented in BareCryptoStore

* Implement `sign_with` in sc_keystore

* Fix inconsistencies, use *_KIND_ID in sp_core testing

* Rename KeyKindId to CryptoTypeId

* Remove pair-returning functions from BareCryptoStore trait

* Define CryptoTypeId in app-crypto macros

* Add functions to get keys supported by keystore

* Fix sign_with signature to include CryptoTypePublicPair

* Add `sign_with_any` and `sign_with_all`

* Use keystore.sign_with in auth_discovery

* Rename get_supported_keys -> supported_keys

* Added headers to function docstrings

* Use chain instead of extending a temp vector

* Fixed some code formatting

* Restrict size of CryptoTypeId

This is to be able to use Encode/Decode derives and the overcome having
the size being unknown at compile-time.

* Implement sign_with in the trait itself

* Remove whitespace

* Use key_type also as a CryptoTypeId in app_crypto macros

* Rename `get_keys` to `keys` in BareCryptoStore

* Remove usage of key_pair funcs in tests

* Adjust docstring for *_CYPTO_ID constants

* Fix failures

* Simplify mapping on keys

* Remove one let

* Fixed typo

* PR feedback

* remove whitespace

* Zip keys and signatures

* Use into_iter & remove cloned

* Pass index to MissingSignature

* Use typed errors instead of strings for BareCryptoStore

* Implement Debug for trait error

* Use hashsets for better performance for supported_keys

* Make sure keys are inserted into the keystore

* Make sign_with_all return type consistent with `sign_with`

* Rename Error to BareCryptoStoreError

* Rename CRYPT_TYPE_ID -> CRYPTO_ID

* Remove unnecessary CRYPTO_ID declaration in Public trait

* Convert pub key to CryptoTypePublicPair

* Fix use

* Fix code style

* Implement From on CryptoTypePublicPair in app_crypto macros

* Change CryptoTypePublicPair to a struct

* Implement Display on CryptoTypePublicPair

* Pass CryptoTypePublicPair to MissingSignature error

* Adjust docs according to function signature

* Unify keys implementation

* Fix RPC author tests

* Fix stackoverflow

* Tabify spaces

* Pass KeyTypeId to error for easier debugging

* Fix asserts

* Use ToHex to format public key

* Use constants from sp_core

* Rename testing KeyTypeId constants

* Please compiler

* Restore KeyTypeId names

apparently, they're not only used in tests

* Use BareCryptoStoreError instead of String

* Document return value

* Fix borrow check

* Convert to hashset internally

* WIP - iter_keys

* Return raw_public_keys

* Address PR feedback

* Address PR Feedback

* Fix hexdisplay import error

* Update primitives/core/src/traits.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Make transactions and block announces use notifications

* Add documentation

* Add docs.rs metadata to all cargo.toml files

* Remove docs.rs metada in substrate's cargo.toml

The signal handler will be reset to the default system one. This means
the program will be exited instead of panicking. This is required to
support piping in the console.

* adds state_queryStorageAt

* make at param for query_storage_at optional

* Update client/rpc/src/state/state_full.rs

Co-Authored-By: Tomasz Drwięga <[email protected]>

* adds query_storage_at to StateBackend

Co-authored-by: Tomasz Drwięga <[email protected]>

* client/finality-grandpa/src/until_imported: Refactor schedule_wait

Previously `BlockUntilImported::schedule_wait` took two closures, one to
report ready items and one to report items to await. None of the implementors of
`BlockUntilImported` call both closures. From a symantic perspective it
would as well not make sense to both await something and state that it
is ready.

Instead with this commit `BlockUntilImported::schedule_wait` simply
returns whether the given item needs waiting or is ready to be passed
on.

This reduces complexity by:

- Removing side effects through the two closures.

- Reducing borrowing given that `UntilImported` `ready` and `pending`
don't need to be borrowed from within the two closures.

- Removes the need for trait bounds for the two closures.

* client/finality-grandpa/src/until_imported: Fix comments

Co-Authored-By: André Silva <[email protected]>

Co-authored-by: André Silva <[email protected]>

* Improve sc-network's documentation for network protocols

* Add note about protocol id

* Apply suggestions from code review

Co-Authored-By: Max Inden <[email protected]>

Co-authored-by: Max Inden <[email protected]>

* Allow syncing to peers with finalized common block

* Added test

* Check for url validity when creating TelemetryEndpoints

* Update code that used TelemetryEndpoints::new()

* Update commennts that referred to TelemetryEndpoints::new()

* Add tests for telemetry

* Fix typo and fix code in docs

* Return error on failing to override telemetry

* Use expect instead of suppressing errors on must-be-valid telemetry endpoints

* Update telemetry unit tests to use expect instead of unwrap

* Implement custom deserializer for TelemetryEndpoints

* Fix typo

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* client/finality-grandpa/src/until_imported: Refactor BlockGlobalMessage

`BlockGlobalMessage` owns an `inner` which contains (1) a count for the
amount of outstanding blocks to be waited on and (2) the message itself.

Given that both is already wrapped in an `Arc` there is no need to keep
track of the outstanding blocks, given that it simply corresponds to the
amount of strong reference counts on the `Arc` itself.

This commit removes the atomic counter within `inner` and piggy backs on
the `Arc` reference counter instead.

* client/finality-grandpa/src/until_imported: Remove useless match

* client/finality-grandpa/src/until_imported.rs: Remove unused var

Co-Authored-By: André Silva <[email protected]>

* client/finality-grandpa/src/until_imported: Address comment suggestion

Co-authored-by: André Silva <[email protected]>

* delias outer node template

* dealias pallet template

* Fix mock

* attempt to fix warning

* Revert "attempt to fix warning"

This reverts commit 3d39142b.

* Add call to test interface.

* always depend on latest alpha version

* integrate arithmetic fuzzer into root workspace

* WiP

* Support source in the runtime API.

* Finish implementation in txpool.

* Fix warning.

* Fix tests.

* Apply suggestions from code review

Co-Authored-By: Kian Paimani <[email protected]>
Co-Authored-By: Nikolay Volf <[email protected]>

* Extra changes.

* Fix test and benches.

* fix test

* Fix test & benches again.

* Fix tests.

* Update bumpalo

* Fix doc test.

* Fix doctest.

* Fix doctest.

Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Nikolay Volf <[email protected]>

* fix propagation in network

* Update client/service/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* fix suggestion

Co-authored-by: Bastian Köcher <[email protected]>

* Split PrimaryPreDigest and SecondaryPreDigest

* Update client/consensus/babe/src/lib.rs

Co-Authored-By: André Silva <[email protected]>

* Update primitives/consensus/babe/src/digests.rs

Co-Authored-By: André Silva <[email protected]>

* Update primitives/consensus/babe/src/digests.rs

Co-Authored-By: André Silva <[email protected]>

* Update primitives/consensus/babe/src/digests.rs

Co-Authored-By: André Silva <[email protected]>

Co-authored-by: André Silva <[email protected]>

* use versioning for deprecated api, remove deprecated api from regular verification

* Update primitives/core/src/sr25519.rs

* add test to transaction pool

Co-authored-by: Bastian Köcher <[email protected]>

* Print an error when a bootnode is on a different chain

* Fix tests

* bump pre-version

* Changelog alpha.4->alpha.5

* repo + homepage to sp-consensus-vrf

* Add default rocksdb feature to `frame-benchmarking-cli` (#5367)

* Add default rocksdb feature

* Update Cargo.toml

Co-authored-by: Shawn Tabrizi <[email protected]>

* Get rid of rustc_hex in favor of hex

* Use HexDisplay

Currently when we connect to a bootnode and it returns a different peer
id, than the one we provided, we disconnect silently. This pr changes
the behavior to printing an error when a bootnode returns a different
peer id than the provided one.

* Init vrf crate without type alias

* Generic PreDigest definition for BABE

* Fix BABE vrf interface change

* Missing default-features def in sp-consensus-babe

* Fix sp-consensus-babe compile

* frame-babe: fix type inference

* Unify type definitions of vrf output/proof and randomness

* frame-babe: fix tests

* Bump node impl version

* Update cargo lock

* Derive Copy for RawVRFOutput and RawVRFProof

* Fix duplicated derive

Co-authored-by: Gavin Wood <[email protected]>

A gossip validator returns `ValidationResult::Discard` for messages that
should not be processed nor kept due to not being useful for the node.
Examples are out-of-scope messages or messages with bad signatures. The
corresponding log message did not reflect this, thus the update through
this patch.

One caveat is a finality grandpa neighbor packet. Given that the concept
of neighbor packets is handled in the finality grandpa gossip validator
only, valid neighbor packets are always discarded on the
`client/network-gossip` layer.

* minor fixes to CLI arg docs

* Update client/cli/src/params/shared_params.rs

Co-authored-by: Bastian Köcher <[email protected]>

* grandpa: allow noting same set in gossip with different authorities

* grandpa: add test for note_set behavior