* consensus: remove caching functionality from block import pipeline

* client: update docs on Verifier::verify

* node: fix block production benchmark

* Move import queue out of `sc-network`

Add supplementary asynchronous API for the import queue which means
it can be run as an independent task and communicated with through
the `ImportQueueService`.

This commit removes removes block and justification imports from
`sc-network` and provides `ChainSync` with a handle to import queue so
it can import blocks and justifications. Polling of the import queue is
moved complete out of `sc-network` and `sc_consensus::Link` is
implemented for `ChainSyncInterfaceHandled` so the import queue
can still influence the syncing process.

* Move stuff to SyncingEngine

* Move `ChainSync` instanation to `SyncingEngine`

Some of the tests have to be rewritten

* Move peer hashmap to `SyncingEngine`

* Let `SyncingEngine` to implement `ChainSyncInterface`

* Introduce `SyncStatusProvider`

* Move `sync_peer_(connected|disconnected)` to `SyncingEngine`

* Implement `SyncEventStream`

Remove `SyncConnected`/`SyncDisconnected` events from
`NetworkEvenStream` and provide those events through
`ChainSyncInterface` instead.

Modify BEEFY/GRANDPA/transactions protocol and `NetworkGossip` to take
`SyncEventStream` object which they listen to for incoming sync peer
events.

* Introduce `ChainSyncInterface`

This interface provides a set of miscellaneous functions that other
subsystems can use to query, for example, the syncing status.

* Move event stream polling to `SyncingEngine`

Subscribe to `NetworkStreamEvent` and poll the incoming notifications
and substream events from `SyncingEngine`.

The code needs refactoring.

* Make `SyncingEngine` into an asynchronous runner

This commits removes the last hard dependency of syncing from
`sc-network` meaning the protocol now lives completely outside of
`sc-network`, ignoring the hardcoded peerset entry which will be
addressed in the future.

Code needs a lot of refactoring.

* Fix warnings

* Code refactoring

* Use `SyncingService` for BEEFY

* Use `SyncingService` for GRANDPA

* Remove call delegation from `NetworkService`

* Remove `ChainSyncService`

* Remove `ChainSync` service tests

They were written for the sole purpose of verifying that `NetworWorker`
continues to function while the calls are being dispatched to
`ChainSync`.

* Refactor code

* Refactor code

* Update client/finality-grandpa/src/communication/tests.rs

Co-authored-by: Anton <[email protected]>

* Fix warnings

* Apply review comments

* Fix docs

* Fix test

* cargo-fmt

* Update client/network/sync/src/engine.rs

Co-authored-by: Anton <[email protected]>

* Update client/network/sync/src/engine.rs

Co-authored-by: Anton <[email protected]>

* Add missing docs

* Refactor code

---------

Co-authored-by: Anton <[email protected]>

Signed-off-by: acatangiu <[email protected]>

* Move beefy primitives to consensus dir
* Move beefy gadget to client consensus folder
* Rename beefy crates

* Move grandpa under consensus dir
* Rename grandpa folder
* Finish grandpa renaming
* Minor tweaks
* Cargo fmt
* Adjust path to chain spec

* Change copyright year to 2023 from 2022

* Fix incorrect update of copyright year

* Remove years from copy right header

* Fix remaining files

* Fix typo in a header and remove update-copyright.sh

* `BlockId` removal: `BlockBuilderProvider::new_block_at`

It changes the arguments of `BlockBuilderProvider::new_block_at` from:
`BlockId<Block>` to: `Block::Hash`

* fmt

* fix

* more fixes

* BlockId removal: refactor of runtime API

It changes the arguments of:
- `ApiExt` methods:  `has_api`, `has_api_with`, `api_version`
- `CallApiAt` method: `runtime_version_at`
from: `BlockId<Block>` to: `Block::Hash`

It also changes the first argument of all generated runtime API calls from: `BlockId<Block>` to: `Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* BlockId removal: refactor of runtime API - tests

- tests adjusted to new runtime API,
- some tests migrated from block number to block hash

* benchmarking-cli: BlockId(0) migrated to info().genesis_hash

`runtime_api.call()` now requires the block hash instead of BlockId::Number.
To access the genesis hash widely used in benchmarking engine the Client
was constrained to satisfy `sp_blockchain::HeaderBackend<Block>` trait
which provides `info().genesis_hash`.

* trivial: api.call(BlockId) -> api.call(Hash)

- Migrated all `runtime_api.calls` to use Hash
- Noteworthy (?):
-- `validate_transaction_blocking` in transaction pool,

* CallApiAtParams::at changed to Block::Hash

* missed doc updated

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* ".git/.scripts/commands/fmt/fmt.sh"

* BlockId removal: Benchmark::consumed_weight

Little refactor around `Benchmark::consumed_weight`: `BlockId` removed.

* at_hash renamed

* wrong merge fixed

* beefy worker: merged with master

* beefy: tests: missing block problem fixed

* Apply review suggestion

* fix

---------

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: command-bot <>

* client/beefy: simplify self_vote logic

* client/beefy: migrate to new state version

* client/beefy: detect equivocated votes

* fix typos

* sp-beefy: add equivocation primitives

* client/beefy: refactor vote processing

* fix version migration for new rounds struct

* client/beefy: track equivocations and create proofs

* client/beefy: adjust tests for new voting logic

* sp-beefy: fix commitment ordering and equality

* client/beefy: simplify handle_vote() a bit

* client/beefy: add simple equivocation test

* client/beefy: submit equivocation proof - WIP

* frame/beefy: add equivocation report runtime api - part 1

* frame/beefy: report equivocation logic - part 2

* frame/beefy: add pluggable Equivocation handler - part 3

* frame/beefy: impl ValidateUnsigned for equivocations reporting

* client/beefy: submit report equivocation unsigned extrinsic

* primitives/beefy: fix tests

* frame/beefy: add default weights

* frame/beefy: fix tests

* client/beefy: fix tests

* frame/beefy-mmr: fix tests

* frame/beefy: cross-check session index with equivocation report

* sp-beefy: make test Keyring useable in pallet

* frame/beefy: add basic equivocation test

* frame/beefy: test verify equivocation results in slashing

* frame/beefy: test report_equivocation_old_set

* frame/beefy: add more equivocation tests

* sp-beefy: fix docs

* beefy: simplify equivocations and fix tests

* client/beefy: address review comments

* frame/beefy: add ValidateUnsigned to test/mock runtime

* client/beefy: fixes after merge master

* fix missed merge damage

* client/beefy: add test for reporting equivocations

Also validated there's no unexpected equivocations reported in the
other tests.

Signed-off-by: acatangiu <[email protected]>

* sp-beefy: move test utils to their own file

* client/beefy: add negative test for equivocation reports

* sp-beefy: move back MmrRootProvider - used in polkadot-service

* impl review suggestions

* client/beefy: add equivocation metrics

---------

Signed-off-by: acatangiu <[email protected]>
Co-authored-by: parity-processbot <>

* Removed assumptions about ancestry from fork tree prune method

* Tests improvement

* Fork tree prune refactory

* Code refactory

* Correctly handle borderline, but legit, case

* Apply suggestions from code review

Co-authored-by: André Silva <[email protected]>

* Removed duplicated test

---------

Co-authored-by: André Silva <[email protected]>

* Finalization target should be chosed as some ancestor of SelectChain::best_chain

* More test assertions

* Improve docs

* Removed stale docs

* Rename 'target' to 'base' in lookup method

* Fix typo

* Apply suggestions from code review

Co-authored-by: André Silva <[email protected]>

* Rename 'target_hash' to 'base_hash' in 'SelectChain::finality_target()'

* Apply suggestions from code review

Co-authored-by: Anton <[email protected]>

* Docs improvement

* Doc fix

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* Apply more code suggestions

---------

Co-authored-by: André Silva <[email protected]>
Co-authored-by: Anton <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* Add handler and upgrade tests

* Add tests for `behaviour.rs`

* Apply review comments

* Update dependencies

* Apply suggestions from code review

Co-authored-by: Dmitry Markin <[email protected]>

* Apply review comments

* Fix clippy

* Update mockall

* Apply review comment

---------

Co-authored-by: Dmitry Markin <[email protected]>

* scale_info: Derive TypeInfo for types present in runtime API

Signed-off-by: Alexandru Vasile <[email protected]>

* cargo: Update Cargo.lock

Signed-off-by: Alexandru Vasile <[email protected]>

---------

Signed-off-by: Alexandru Vasile <[email protected]>

The code was added without any clear usage. The inherent for example is not benchmarked and not used.

* use non binary pronouns in comments

* cargo fmt

* fix the use of "it" with "they" when dealing about an opperations identity

* Aura: Fix warp syncing

We need to set the fork choice rule! When using Cumulus this is done by the `ParachainsBlockImport`,
but for standalone chains we still need this!

Closes: https://github.com/paritytech/substrate/issues/13220

* Improve fork choice

* Detect and correct epoch-index for skipped epochs

* Code refactory

* Epoch index should be also be fixed for secondary claims with VRF

* Fix typo

* Make clippy happy

* Fix typo

* Trigger pipeline

The `authoring_blocks` test of BABE was calculating the slot based on the timestamp it sometimes
failed in CI. The problem is that we combine all the notifications and authoring futures in one big
future. This one big future may first polls one authoring future to build a block. Then it polls all
notification futures again to import the block. Then some other authoring future is polled and
builds on the imported block using the same slot and making the import fail. The solution is that we
just artificially increase the slot to make the test work.

* Make DispatchError impl MEL

* Upgrade SCALE codec to support `codec(skip)` for MEL

Co-authored-by: Bastian Köcher <[email protected]>

Refactory of `next_slot` method

* Prevents slot worker exit if inherent data provider creation fails
* Failure is not possible anymore
* Fix potential failure after warp-sync where block headers of not already downloaded blocks are used by the inherent data provider

* Stop keeping track of epoch changes data within the sync gap

* Fix docs

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* Fix typo

Co-authored-by: Bastian Köcher <[email protected]>

* upgrade libp2p to 0.50.0

* on_swarm_event and on_connection_handler_event

* replace `Swarm::new` with `Swarm::with_threadpool_executor`

* on_swarm_event and on_connection_handler_event part 2

* on_swarm_event and on_connection_handler_event part 3

* on_swarm_event and on_connection_handler_event part 4

* update libp2p

* libp2p 0.50.0

* rename OutboundQueryCompleted to OutboundQueryProgressed

refs https://github.com/libp2p/rust-libp2p/pull/2712

* remove unused var

* accumulate outbound_query_records until query is finished

* format code

* use p_handler instead of new_handler

https://github.com/paritytech/substrate/pull/12734#discussion_r1027640610

* pass ListenFailure to kademlia

https://github.com/paritytech/substrate/pull/12734#discussion_r1034716664

* use tokio executor in tests

https://github.com/paritytech/substrate/pull/12734#discussion_r1039291776

* use chrono Local::now

instead of deprecated Local::today

* remove unused vars from request_responses tests

* attempt to fix pallet UI tests

* restart CI

* restart CI

* restart CI

* restart CI

* restart CI

* restart CI

* restart CI

* restart CI

* Aura: Do not verify on state import

When we import the state, we can not fetch authorities to verify the seal etc. So, we can directly
skip any verification.

* Skip checks as well for gap sync

* Update client/consensus/aura/src/import_queue.rs

Co-authored-by: André Silva <[email protected]>

* Review comment

Co-authored-by: André Silva <[email protected]>

* babe: allow skipping epochs in pallet

* babe: detect and skip epochs on client

* babe: cleaner epoch util functions

* babe: add test for runtime handling of skipped epochs

* babe: simpler implementation of client handling of skipped epochs

* babe: test client-side handling of skipped epochs

* babe: add comments on client-side skipped epochs

* babe: remove emptyline

* babe: make it resilient to forks

* babe: typo

* babe: overflow-safe math

* babe: add test for skipping epochs across different forks

* Fix tests

* FMT

Co-authored-by: Bastian Köcher <[email protected]>

It changes the arguments of `HeaderBackend::status` method from: `BlockId<Block>` to: `Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

Co-authored-by: parity-processbot <>

* BlockId removal: refactor: HeaderBackend::header

It changes the arguments of:
- `HeaderBackend::header`,
- `Client::header`,
- `PeersClient::header`
- `ChainApi::block_header`

methods from: `BlockId<Block>` to: `Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* non-trivial usages of haeder(block_id) refactored

This may required introduction of dedicated function:
header_for_block_num

* fmt

* fix

* doc fixed

* ".git/.scripts/fmt.sh"

* BlockId removal: refactor: HeaderBackend::expect_header

It changes the arguments of `HeaderBackend::expect_header` method from: `BlockId<Block>` to: `Block::Hash`

* ".git/.scripts/fmt.sh"

* readme updated

* ".git/.scripts/fmt.sh"

* fix

Co-authored-by: parity-processbot <>

* *: update rand to v0.8.5

* *: remove useless deps from Cargo.toml

* fix pallet-session-benchmarking

* fix pallet-election-provider-support test

* remove useless rand from dev-dependencies

* Use shared LOG_TARGET in consensus related crates
* Rename target from "afg" to "grandpa" 

* jsonrpsee v0.16

add backwards compatibility

run old http server on http only

* cargo fmt

* update jsonrpsee 0.16.1

* less verbose cors log

* fix nit in log: WS -> HTTP

* revert needless changes in Cargo.lock

* remove unused features in tower

* fix nits; add client-core feature

* jsonrpsee v0.16.2

* Move import queue out of `sc-network`

Add supplementary asynchronous API for the import queue which means
it can be run as an independent task and communicated with through
the `ImportQueueService`.

This commit removes removes block and justification imports from
`sc-network` and provides `ChainSync` with a handle to import queue so
it can import blocks and justifications. Polling of the import queue is
moved complete out of `sc-network` and `sc_consensus::Link` is
implemented for `ChainSyncInterfaceHandled` so the import queue
can still influence the syncing process.

* Fix tests

* Apply review comments

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/network/sync/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>

* Replace deprecated libp2p feature specs with correct ones

* Bump tokio to 1.21.2

* Replace async-std libp2p primitives with tokio ones

* minor: rustfmt

* Fix TestNet to run initialization in the tokio context

* Convert telemetry test from async-std to tokio

* Convert notifications tests from async-std to tokio

* Convert chain sync tests from async-std to tokio

* Ditch async-std completely

* Make executor mandatory

* Bump tokio to 1.22.0

* minor: rustfmt

* Explicitly use tokio runtime in tests

* Move more tests to explicit tokio runtime

* Explicitly set multithreaded runtime in tokio test

* minor: rustfmt

* minor: fix comment

* Replace async-std with tokio in MMR tests

* Move create_inherent_data call to use side

* Make provide_inherent_data async

* Fix tests

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* Log errors

* Fix test

* Fix test

* fix

* Deduplicate test code

* fix

* flag

* Update client/consensus/slots/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Revert "Deduplicate test code"

This reverts commit ba46adbe089329c78cd69ccdb08e27ed67bd77cf.

* Fix test

* remove commented out code

* minor to start CI run

* start CI

* Update client/consensus/slots/src/lib.rs

Co-authored-by: Marcin S. <[email protected]>

* Apply PR suggestions

* Apply PR suggestions

* Update client/consensus/slots/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* minor

* kickoff CI

* PR suggestions

* Compute remaining duration instead of using slot_info.duration

* Don't rely on sub implementation for Instant

* Apply PR suggestions

* Use saturating_duration_since

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Marcin S. <[email protected]>
Co-authored-by: parity-processbot <>

* Move MMR utils methods from pallet to primitives

Signed-off-by: Serban Iorga <[email protected]>

* Add method to MmrApi

* Move forks expanding logic from babe to primitives

* Implement MMR gadget

* Remove prunning logic from the MMR pallet

* Code review changes: 1st iteration

* Replace MaybeCanonEngine with CanonEngineBuilder

* fix mmr_leaves_count() for kitchen sink demo

* Update client/merkle-mountain-range/src/canon_engine.rs

Co-authored-by: Adrian Catangiu <[email protected]>

* Code review changes: 2nd iteration

* fix INDEXING_PREFIX

* impl review comments

* add documentation and minor rename

Signed-off-by: Serban Iorga <[email protected]>
Co-authored-by: Adrian Catangiu <[email protected]>

* Prevent epochs pruning while on epoch 0

* Bump sp-keyring

* Bump pallet-contracts-primitives

* Cargo.lock