* client/beefy: simplify self_vote logic

* client/beefy: migrate to new state version

* client/beefy: detect equivocated votes

* fix typos

* sp-beefy: add equivocation primitives

* client/beefy: refactor vote processing

* fix version migration for new rounds struct

* client/beefy: track equivocations and create proofs

* client/beefy: adjust tests for new voting logic

* sp-beefy: fix commitment ordering and equality

* client/beefy: simplify handle_vote() a bit

* client/beefy: add simple equivocation test

* client/beefy: submit equivocation proof - WIP

* frame/beefy: add equivocation report runtime api - part 1

* frame/beefy: report equivocation logic - part 2

* frame/beefy: add pluggable Equivocation handler - part 3

* frame/beefy: impl ValidateUnsigned for equivocations reporting

* client/beefy: submit report equivocation unsigned extrinsic

* primitives/beefy: fix tests

* frame/beefy: add default weights

* frame/beefy: fix tests

* client/beefy: fix tests

* frame/beefy-mmr: fix tests

* frame/beefy: cross-check session index with equivocation report

* sp-beefy: make test Keyring useable in pallet

* frame/beefy: add basic equivocation test

* frame/beefy: test verify equivocation results in slashing

* frame/beefy: test report_equivocation_old_set

* frame/beefy: add more equivocation tests

* sp-beefy: fix docs

* beefy: simplify equivocations and fix tests

* client/beefy: address review comments

* frame/beefy: add ValidateUnsigned to test/mock runtime

* client/beefy: fixes after merge master

* fix missed merge damage

* client/beefy: add test for reporting equivocations

Also validated there's no unexpected equivocations reported in the
other tests.

Signed-off-by: acatangiu <[email protected]>

* sp-beefy: move test utils to their own file

* client/beefy: add negative test for equivocation reports

* sp-beefy: move back MmrRootProvider - used in polkadot-service

* impl review suggestions

* client/beefy: add equivocation metrics

---------

Signed-off-by: acatangiu <[email protected]>
Co-authored-by: parity-processbot <>

* few beefy metrics

* more beefy metrics

* some beefy metrics

* some beefy metrics

* more metrics

* other metrics

* fix tests

* merge changes

* Apply suggestions from code review

* client/beefy: fix metrics

* client/beefy: separate metrics per component, avoid double registering

* client/beefy: deduplicate metrics registration code

* remove unused metric

* impl review suggestions

---------

Co-authored-by: Adrian Catangiu <[email protected]>

For on-demand justifications, peer selection is based on witnessed
gossip votes. This commit changes the condition for selecting a peer
to request justification for `block` from
"last voted on >= `block`" to "peer last voted on strict > `block`".

When allowing `>=` we see nodes continuously spamming unsuccessful
on-demand requests to nodes which are still voting on a block without
having a justification available.

One way to fix the spam would be to add some rate-limiting or backoff
period when requesting justifications.

The other solution (present in this commit) is to simply request
justifications from peers that are voting on future blocks so we know
they're _guaranteed_ to have the wanted mandatory justification
available to send back.

Signed-off-by: acatangiu <[email protected]>

* client/beefy: detect equivocated votes

* client/beefy: make sure to persist state after voting

* client/beefy: drop never-used aux-schema v2 migration

* impl review suggestion

---------

Signed-off-by: Adrian Catangiu <[email protected]>

* beefy: add support to configure BEEFY genesis

* client/beefy: more flexible test runtime api

* client/beefy: add tests for custom BEEFY genesis

* client/beefy: ignore old state that didn't account for pallet genesis

* client/beefy: fix clippy

* frame/beefy: default BEEFY-genesis is block One::one()

* frame/beefy: add extra doc comments

---------

Co-authored-by: parity-processbot <>

* use non binary pronouns in comments

* cargo fmt

* fix the use of "it" with "they" when dealing about an opperations identity

* LOG_TARGET const
* fmt

Still allows custom message hasher, but ties together the crypto
types used for private+public keys and the signature.

Signed-off-by: acatangiu <[email protected]>

Minor change to logs but required to minimize spamming during initial BEEFY deployment.

* BlockId removal: refactor: HeaderBackend::header

It changes the arguments of:
- `HeaderBackend::header`,
- `Client::header`,
- `PeersClient::header`
- `ChainApi::block_header`

methods from: `BlockId<Block>` to: `Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* non-trivial usages of haeder(block_id) refactored

This may required introduction of dedicated function:
header_for_block_num

* fmt

* fix

* doc fixed

* ".git/.scripts/fmt.sh"

* BlockId removal: refactor: HeaderBackend::expect_header

It changes the arguments of `HeaderBackend::expect_header` method from: `BlockId<Block>` to: `Block::Hash`

* ".git/.scripts/fmt.sh"

* readme updated

* ".git/.scripts/fmt.sh"

* fix

Co-authored-by: parity-processbot <>

* sc-network-test::Peer: block push methods return hashes vec

This commit reworks the block generation/push methods in
sc-network-test::Peer.

Now methods are providing the vector of hashes that were built.

This allows to get rid of redundant `block_hash_from_id` call, as all
hashes are known just after being built.

Similar approach was taken in BeefyTestNet::generate_blocks_and_sync
method.

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* fix

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>

Introduce bounds on the justifications and votes queues, so they do not grow forever if voter cannot make progress and consume from them. When bounds are hit, new votes or justifications get dropped.

* use a BTreeMap and check for bounds

* cargo fmt

* use usize

Co-authored-by: Adrian Catangiu <[email protected]>

* Replace deprecated libp2p feature specs with correct ones

* Bump tokio to 1.21.2

* Replace async-std libp2p primitives with tokio ones

* minor: rustfmt

* Fix TestNet to run initialization in the tokio context

* Convert telemetry test from async-std to tokio

* Convert notifications tests from async-std to tokio

* Convert chain sync tests from async-std to tokio

* Ditch async-std completely

* Make executor mandatory

* Bump tokio to 1.22.0

* minor: rustfmt

* Explicitly use tokio runtime in tests

* Move more tests to explicit tokio runtime

* Explicitly set multithreaded runtime in tokio test

* minor: rustfmt

* minor: fix comment

* Replace async-std with tokio in MMR tests

* client/beefy: fix on-demand justif sync for old blocks

When receiving BEEFY justifications for old blocks the state might
be pruned for them, in which case justification verification fails
because BEEFY validator set cannot be retrieved from runtime state.

Fix this by having the voter give the validator set to the
`OnDemandJustificationsEngine` as request information. On receiving
a BEEFY justification for requested block, the provided validator
set will be used to validate the justification.

Signed-off-by: acatangiu <[email protected]>

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* impl review suggestions

* client/beefy: fail initialization if state unavailable

* beefy: remove spammy log

Signed-off-by: acatangiu <[email protected]>
Co-authored-by: parity-processbot <>
Co-authored-by: Bastian Köcher <[email protected]>

* client/beefy: remove high-freq network events from main loop

Network events are many and very frequent, remove the net-event-stream
from the main voter loop and drastically reduce BEEFY voter task
'wakeups'.

Instead have the `GossipValidator` track known peers as it already
has callbacks for that coming from `GossipEngine`.

Signed-off-by: acatangiu <[email protected]>

* client/beefy: prepare worker for persisting state

* client/beefy: persist voter state

* client/beefy: initialize persistent state

* client/beefy: try to vote from the very beginning

Now that voter is initialized from persistent state, it makes
sense that it can attempt voting right away. This also helps
the genesis case when we consider block `One` as mandatory.

* client/beefy: add tests for voter state db
* client/beefy: persist voter state as soon as initialized
* client/beefy: make sure min-block-delta is at least 1
* client/beefy: persist state after voting

Persist state after handling self vote to avoid double voting in case
of voter restarts.

* client/beefy: persist state after handling mandatory block vote

For mandatory blocks we want to make sure we're not losing votes
in case of crashes or restarts, since voter will not make further
progress without finalizing them.

* frame/beefy: use GENESIS_AUTHORITY_SET_ID on pallet genesis

* client/beefy: initialize voter at either genesis or last finalized
To guarantee unbroken chain of mandatory blocks justifications, voter
will always resume from either last BEEFY-justified block or
`pallet-beefy` genesis, whichever is more recent.

Initialization walks back the chain from latest GRANDPA finalized
block looking for one of the above. Along the way, it also records
and enqueues for processing any BEEFY mandatory blocks that have
been already GRANDPA finalized but not BEEFY finalized.

* client/beefy: decouple voter init from aux db state load
* client/beefy: fix voter init tests
* remove debug prints
* gadget future must be type ()
* fix init from last justification

Signed-off-by: Adrian Catangiu <[email protected]>

* histor. batch proof: make best block arg optional

* correct testing range

* make generate_batch_proof stub for historical

* merge generate_{historical_}batch_proof functions

* merge generate_{batch_}proof functions

* merge verify_{batch_}proof functions

* merge verify_{batch_}proof_stateless functions

* remove {Leaf}Proof

Not utilized by API anymore, so superfluous.
Removal consistent with prior changes to just use "batch" proof API.

* rename BatchProof->Proof

no need to qualify if only one universal proof type.

* cleanup

* expose verify_proof rpc api

* document verify_proof

* expose verify_proof_stateless rpc api

* add optional BlockHash to mmr_root rpc api

* fixup! expose verify_proof rpc api

* fix documentation phrasing

Co-authored-by: Adrian Catangiu <[email protected]>

* documentation grammar

Co-authored-by: Adrian Catangiu <[email protected]>

* define mmr error msgs together with error enum

Co-authored-by: Serban Iorga <[email protected]>

* fixup! define mmr error msgs together with error enum

* map decoding errors to CallError::InvalidParams

Co-authored-by: Serban Iorga <[email protected]>

* fixup! map decoding errors to CallError::InvalidParams

Co-authored-by: Adrian Catangiu <[email protected]>
Co-authored-by: parity-processbot <>
Co-authored-by: Serban Iorga <[email protected]>

It changes &Block::Hash argument to Block::Hash.

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* BlockId removal: refactor: Backend::justifications

It changes the arguments of `Backend::justifications` method from: `BlockId<Block>` to: `&Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* trigger CI job

* trigger CI job

* bug fix

* match -> if

Co-authored-by: Adrian Catangiu <[email protected]>

Co-authored-by: Adrian Catangiu <[email protected]>

Signed-off-by: Adrian Catangiu <[email protected]>

* BlockId removal: refactor: Backend::append_justification

It changes the arguments of `Backend::append_justification`
from: block: `BlockId<Block>` to: hash: `&Block::Hash`

This PR is part of `BlockId::Number` refactoring analysis (paritytech/substrate#11292)

* Error message improved

Co-authored-by: Adrian Catangiu <[email protected]>

* single error message in beefy::finalize

* println removed

Co-authored-by: Adrian Catangiu <[email protected]>

* Normalize keystore type and usage across tests

* Extract peer index from array index

* BlockId removal: refactor: BlockImportOperation+Bknd::finalize_block

It changes the arguments of methods of `BlockImportOperation` trait
from: block: `BlockId<Block>` to: hash: `&Block::Hash`
`Backend::finalize_block` was also changed.

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* Review suggestion applied

thx to @davxy

* trigger CI job

* BlockId removal: refactor: Finalizer

It changes the arguments of methods of `Finalizer` trait from:
block: `BlockId<Block>` to: hash: `&Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* minor corrections

* failing test corrected

* minor rework

The runtime api implementation contained invalid unsafe trait bounds. `Sync` was never correct there
and `Send` should have not been "force implemented".

* pallet-mmr: RPC API works with block_numbers

* fixes

* update rpc

* fmt

* final touches in the rpc

* temporary fix

* fix

* fmt

* docs

* Update lib.rs

* use NumberFor

* validate input

* update runtime

* convert block_number to u64

* small edit

* update runtime api

* test fix

* runtime fix

* update test function

* fmt

* fix nits

* remove block_num_to_leaf_index from runtime api

* Update frame/merkle-mountain-range/src/lib.rs

Co-authored-by: Robert Hambrock <[email protected]>

* fix tests

* get the code to compile after merge

* get the tests to compile

* fix in tests?

* fix test

* Update frame/merkle-mountain-range/src/tests.rs

Co-authored-by: Adrian Catangiu <[email protected]>

* Update frame/merkle-mountain-range/src/lib.rs

Co-authored-by: Adrian Catangiu <[email protected]>

* Update primitives/merkle-mountain-range/src/lib.rs

Co-authored-by: Adrian Catangiu <[email protected]>

* fix errors & nits

* change block_num_to_leaf_index

* don't make any assumptions

* Update frame/merkle-mountain-range/src/tests.rs

Co-authored-by: Adrian Catangiu <[email protected]>

* Update frame/merkle-mountain-range/src/tests.rs

Co-authored-by: Adrian Catangiu <[email protected]>

* Update frame/merkle-mountain-range/src/tests.rs

Co-authored-by: Adrian Catangiu <[email protected]>

* fix

* small fix

* use best_known_block_number

* best_known_block_number instead of leaves_count

* more readable?

* remove warning

* Update frame/merkle-mountain-range/src/lib.rs

Co-authored-by: Robert Hambrock <[email protected]>

* simplify

* update docs

* nits

* fmt & fix

* merge fixes

* fix

* small fix

* docs & nit fixes

* Nit fixes

* remove leaf_indices_to_block_numbers()

* fmt

Co-authored-by: Robert Hambrock <[email protected]>
Co-authored-by: Adrian Catangiu <[email protected]>

* Define CustomVerify trait

Signed-off-by: Serban Iorga <[email protected]>

* Use ECDSA CustomVerify for MultiSignature

Signed-off-by: Serban Iorga <[email protected]>

* beefy: small simplifications

Signed-off-by: Serban Iorga <[email protected]>

* Revert "Use ECDSA CustomVerify for MultiSignature"

This reverts commit 136cff82505662dd92c864491814629d2bc349f0.

* Revert "Define CustomVerify trait"

This reverts commit adf91e9e6d1bdea6f00831f6067b74c3d945f9a2.

* Define BeefyAuthorityId and BeefyVerify traits

* Improve BeefyVerify unit tests

Co-authored-by: Robert Hambrock <[email protected]>

* fmt & import sp_core::blake2_256

* Renamings

* remove SignerToAccountId

* fix

Signed-off-by: Serban Iorga <[email protected]>
Co-authored-by: Robert Hambrock <[email protected]>

* primitives/beefy: move Payload to its own file

* primitives/beefy: add Payload tests

* primitives/beefy: add MmrRootProvider as custom BEEFY payload provider

* client/beefy: use generic BEEFY 'PayloadProvider'

* primitives/beefy: rename Payload::new to Payload::from_single_entry for clarity

* fix visibility

* fix cargo doc

* BEEFY client: avoid unnecessary clone

* MMR: impl TypeInfo for some structures

* client/beefy: remove bounds on type definitions

* client/beefy: remove gossip protocol legacy name

* client/beefy: simplify justification request response engine

Signed-off-by: Adrian Catangiu <[email protected]>

* client/beefy: create communication module and move gossip there

* client/beefy: move beefy_protocol_name module to communication

* client/beefy: move notification module under communication

* client/beefy: add incoming request_response protocol handler

* client/beefy: keep track of connected peers and their progress

* client/beefy: add logic for generating Justif requests

* client/beefy: cancel outdated on-demand justification requests

* try Andre's suggestion for JustificationEngine

* justif engine add justifs validation

* client/beefy: impl OnDemandJustificationsEngine async next()

* move beefy proto name test

* client/beefy: initialize OnDemandJustificationsEngine

* client/tests: allow for custom req-resp protocols

* client/beefy: on-demand-justif: implement simple peer selection strategy

* client/beefy: fix voter initialization

Fix corner case where voter gets a single burst of finality
notifications just when it starts.

The notification stream was consumed by "wait_for_pallet" logic,
then main loop would subscribe to finality notifications, but by that
time some notifications might've been lost.

Fix this by subscribing the main loop to notifications before waiting
for pallet to become available. Share the same stream with the main loop
so that notifications for blocks before pallet available are ignored,
while _all_ notifications after pallet available are processed.

Add regression test for this.

Signed-off-by: acatangiu <[email protected]>

* client/beefy: make sure justif requests are always out for mandatory blocks

* client/beefy: add test for on-demand justifications sync

* client/beefy: tweak main loop event processing order

* client/beefy: run on-demand-justif-handler under same async task as voter

* client/beefy: add test for known-peers

* client/beefy: reorg request-response module

* client/beefy: add issue references for future work todos

* client/beefy: consolidate on-demand-justifications engine state machine

Signed-off-by: acatangiu <[email protected]>

* client/beefy: fix for polkadot companion

* client/beefy: implement review suggestions

* cargo fmt and clippy

* fix merge damage

* fix rust-doc

* fix merge damage

* fix merge damage

* client/beefy: add test for justif proto name

Signed-off-by: acatangiu <[email protected]>

* BEEFY: generate historical proofs

Signed-off-by: Serban Iorga <[email protected]>

* Update frame/merkle-mountain-range/rpc/src/lib.rs

Co-authored-by: Adrian Catangiu <[email protected]>

* Update primitives/merkle-mountain-range/src/lib.rs

Co-authored-by: Adrian Catangiu <[email protected]>

* Update frame/merkle-mountain-range/src/lib.rs

Co-authored-by: Adrian Catangiu <[email protected]>

* cargo fmt

* fix off-by-one in leaves powerset generation

* test all possible mmr sizes for historical proofs

* remove now redundant simple_historical_proof

* cargo fmt

Signed-off-by: Serban Iorga <[email protected]>
Co-authored-by: Adrian Catangiu <[email protected]>
Co-authored-by: Robert Hambrock <[email protected]>

* Move transaction protocol to its own crate

* Update Cargo.lock

* Fix binaries

* Update client/network/transactions/src/lib.rs

Co-authored-by: Dmitry Markin <[email protected]>

* Update client/service/src/builder.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Apply review comments

* Revert one change and apply cargo-fmt

* Remove Transaction from Message

* Add array-bytes

* trigger CI

* Add comment about codec index

Co-authored-by: Dmitry Markin <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* Use `array-bytes` for All Array/Bytes/Hex Operations

Signed-off-by: Xavier Lau <[email protected]>

* Reorder

* Self Review

* Format

* Fix Tests

* Bump `array-bytes`

* Optimize large test res

Signed-off-by: Xavier Lau <[email protected]>
Co-authored-by: parity-processbot <>

Fix corner case where voter gets a single burst of finality
notifications just when it starts.

The notification stream was consumed by "wait_for_pallet" logic,
then main loop would subscribe to finality notifications, but by that
time some notifications might've been lost.

Fix this by subscribing the main loop to notifications before waiting
for pallet to become available. Share the same stream with the main loop
so that notifications for blocks before pallet available are ignored,
while _all_ notifications after pallet available are processed.

Add regression test for this.

Signed-off-by: acatangiu <[email protected]>

* client/beefy: use backend instead of client where possible

* client/beefy: initialize voter from genesis

Now that we have justifications import, we can drop the "lean beefy"
behaviour and start building justifications chain from Genesis with
containing all past sessions' mandatory blocks justifications.

* client/beefy: walk finality tree_route to catch session changes

* client/beefy: fix block import

During initial block import blocks are not finalized, so trying to
validate and append justifications within block import fails (for
initial network sync imported blocks).

Changes:

- Move justification validation to _after_ `inner.block_import()`,
  so block is imported in backend and runtime api can be called to
  get the BEEFY authorities for said block.
- Move append-to-backend for imported BEEFY justification to voter,
  because it already has the required logic to BEEFY-finalize blocks
  only after GRANDPA finalized them.
- Mark voting rounds as concluded when finalizing through
  imported justifications as well as when finalizing through voting.

* client/beefy: valid justifications are one per block number

The only way we'd get _different_ _validated_ justifications for same
block number is if authorities are double voting, which will be handled
later.

* client/beefy: process incoming justifs during major sync

* client/beefy: correct voter initialization

BEEFY voter should resume voting from either:
  - last BEEFY finalized block,
  - session start,
whichever is closest to head.

* client/beefy: test voter initialization

* client/beefy: impl review suggestions

Signed-off-by: acatangiu <[email protected]>

* Add ProtocolName custom type

* Use new ProtocolName in sc_network_common

* Use new ProtocolName in sc_network

* Use new ProtocolName for BEEFY and GRANDPA

* Use new ProtocolName for notifications

* Use new ProtocolName in sc_network (part 2)

* Use new ProtocolName in sc_network_gossip

* Use new ProtocolName in sc_offchain

* Remove unused imports

* Some more fixes

* Add tests

* Fix minor import issues

* Re-export ProtocolName in sc_network

* Revert "Re-export ProtocolName in sc_network"

This reverts commit 8d8ff71927e7750757f29c9bbd88dc0ba181d214.

* Re-export ProtocolName in sc_network

* Remove dependency on sc-network-common from beefy-gadget

* declone and close the door

* cargo fmt

* remove brackets