Skip to content
  1. Feb 16, 2023
    • dharjeezy's avatar
      client/beefy: add more metrics for production visibility (#12910) · 292e5ee4
      dharjeezy authored 
      

* few beefy metrics

* more beefy metrics

* some beefy metrics

* some beefy metrics

* more metrics

* other metrics

* fix tests

* merge changes

* Apply suggestions from code review

* client/beefy: fix metrics

* client/beefy: separate metrics per component, avoid double registering

* client/beefy: deduplicate metrics registration code

* remove unused metric

* impl review suggestions

---------

Co-authored-by: default avatarAdrian Catangiu <[email protected]>
      292e5ee4
  3. Feb 13, 2023
    • Adrian Catangiu's avatar
      client/beefy: request justifs from peers further in consensus (#13343) · d01e675f
      Adrian Catangiu authored 
      

For on-demand justifications, peer selection is based on witnessed
gossip votes. This commit changes the condition for selecting a peer
to request justification for `block` from
"last voted on >= `block`" to "peer last voted on strict > `block`".

When allowing `>=` we see nodes continuously spamming unsuccessful
on-demand requests to nodes which are still voting on a block without
having a justification available.

One way to fix the spam would be to add some rate-limiting or backoff
period when requesting justifications.

The other solution (present in this commit) is to simply request
justifications from peers that are voting on future blocks so we know
they're _guaranteed_ to have the wanted mandatory justification
available to send back.

Signed-off-by: default avataracatangiu <[email protected]>
      d01e675f
  5. Feb 06, 2023
  7. Feb 03, 2023
    • Adrian Catangiu's avatar
      BEEFY: define on-chain beefy-genesis and use it to coordinate voter initialization (#13215) · 981ffb29
      Adrian Catangiu authored 
      * beefy: add support to configure BEEFY genesis

* client/beefy: more flexible test runtime api

* client/beefy: add tests for custom BEEFY genesis

* client/beefy: ignore old state that didn't account for pallet genesis

* client/beefy: fix clippy

* frame/beefy: default BEEFY-genesis is block One::one()

* frame/beefy: add extra doc comments

---------

Co-authored-by: parity-processbot <>
      981ffb29
  9. Jan 27, 2023
  11. Jan 24, 2023
  13. Jan 13, 2023
  15. Jan 12, 2023
  17. Dec 23, 2022
  19. Dec 20, 2022
    • Michal Kucharczyk's avatar
      `BlockId` removal: refactor: `HeaderBackend::header` (#12874) · 548955a7
      Michal Kucharczyk authored 
      * BlockId removal: refactor: HeaderBackend::header

It changes the arguments of:
- `HeaderBackend::header`,
- `Client::header`,
- `PeersClient::header`
- `ChainApi::block_header`

methods from: `BlockId<Block>` to: `Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* non-trivial usages of haeder(block_id) refactored

This may required introduction of dedicated function:
header_for_block_num

* fmt

* fix

* doc fixed

* ".git/.scripts/fmt.sh"

* BlockId removal: refactor: HeaderBackend::expect_header

It changes the arguments of `HeaderBackend::expect_header` method from: `BlockId<Block>` to: `Block::Hash`

* ".git/.scripts/fmt.sh"

* readme updated

* ".git/.scripts/fmt.sh"

* fix

Co-authored-by: parity-processbot <>
      548955a7
  21. Dec 19, 2022
  23. Dec 16, 2022
  25. Dec 05, 2022
    • dharjeezy's avatar
      client/beefy: add some bounds on enqueued votes (#12562) · 91e6de05
      dharjeezy authored 
      

Introduce bounds on the justifications and votes queues, so they do not grow forever if voter cannot make progress and consume from them. When bounds are hit, new votes or justifications get dropped.

* use a BTreeMap and check for bounds

* cargo fmt

* use usize

Co-authored-by: default avatarAdrian Catangiu <[email protected]>
      91e6de05
    • Dmitry Markin's avatar
      Upgrade tokio to 1.22.0 and replace async-std with tokio (#12646) · 5eb84f9c
      Dmitry Markin authored 
      * Replace deprecated libp2p feature specs with correct ones

* Bump tokio to 1.21.2

* Replace async-std libp2p primitives with tokio ones

* minor: rustfmt

* Fix TestNet to run initialization in the tokio context

* Convert telemetry test from async-std to tokio

* Convert notifications tests from async-std to tokio

* Convert chain sync tests from async-std to tokio

* Ditch async-std completely

* Make executor mandatory

* Bump tokio to 1.22.0

* minor: rustfmt

* Explicitly use tokio runtime in tests

* Move more tests to explicit tokio runtime

* Explicitly set multithreaded runtime in tokio test

* minor: rustfmt

* minor: fix comment

* Replace async-std with tokio in MMR tests
      5eb84f9c
  27. Nov 28, 2022
  29. Nov 23, 2022
  31. Nov 21, 2022
    • Adrian Catangiu's avatar
      client/beefy: persist voter state (#12712) · 3e7a2778
      Adrian Catangiu authored 
      

* client/beefy: prepare worker for persisting state

* client/beefy: persist voter state

* client/beefy: initialize persistent state

* client/beefy: try to vote from the very beginning

Now that voter is initialized from persistent state, it makes
sense that it can attempt voting right away. This also helps
the genesis case when we consider block `One` as mandatory.

* client/beefy: add tests for voter state db
* client/beefy: persist voter state as soon as initialized
* client/beefy: make sure min-block-delta is at least 1
* client/beefy: persist state after voting

Persist state after handling self vote to avoid double voting in case
of voter restarts.

* client/beefy: persist state after handling mandatory block vote

For mandatory blocks we want to make sure we're not losing votes
in case of crashes or restarts, since voter will not make further
progress without finalizing them.

* frame/beefy: use GENESIS_AUTHORITY_SET_ID on pallet genesis

* client/beefy: initialize voter at either genesis or last finalized
To guarantee unbroken chain of mandatory blocks justifications, voter
will always resume from either last BEEFY-justified block or
`pallet-beefy` genesis, whichever is more recent.

Initialization walks back the chain from latest GRANDPA finalized
block looking for one of the above. Along the way, it also records
and enqueues for processing any BEEFY mandatory blocks that have
been already GRANDPA finalized but not BEEFY finalized.

* client/beefy: decouple voter init from aux db state load
* client/beefy: fix voter init tests
* remove debug prints
* gadget future must be type ()
* fix init from last justification

Signed-off-by: default avatarAdrian Catangiu <[email protected]>
      3e7a2778
  33. Nov 09, 2022
    • Robert Hambrock's avatar
      Consolidate and deduplicate MMR API methods (#12530) · 53f97350
      Robert Hambrock authored 
      

* histor. batch proof: make best block arg optional

* correct testing range

* make generate_batch_proof stub for historical

* merge generate_{historical_}batch_proof functions

* merge generate_{batch_}proof functions

* merge verify_{batch_}proof functions

* merge verify_{batch_}proof_stateless functions

* remove {Leaf}Proof

Not utilized by API anymore, so superfluous.
Removal consistent with prior changes to just use "batch" proof API.

* rename BatchProof->Proof

no need to qualify if only one universal proof type.

* cleanup

* expose verify_proof rpc api

* document verify_proof

* expose verify_proof_stateless rpc api

* add optional BlockHash to mmr_root rpc api

* fixup! expose verify_proof rpc api

* fix documentation phrasing

Co-authored-by: default avatarAdrian Catangiu <[email protected]>

* documentation grammar

Co-authored-by: default avatarAdrian Catangiu <[email protected]>

* define mmr error msgs together with error enum

Co-authored-by: default avatarSerban Iorga <[email protected]>

* fixup! define mmr error msgs together with error enum

* map decoding errors to CallError::InvalidParams

Co-authored-by: default avatarSerban Iorga <[email protected]>

* fixup! map decoding errors to CallError::InvalidParams

Co-authored-by: default avatarAdrian Catangiu <[email protected]>
Co-authored-by: parity-processbot <>
Co-authored-by: default avatarSerban Iorga <[email protected]>
      53f97350
  35. Nov 07, 2022
  37. Nov 02, 2022
  39. Oct 31, 2022
  41. Oct 30, 2022
  43. Oct 24, 2022
  45. Oct 20, 2022
    • Michal Kucharczyk's avatar
      BlockId removal: refactor: BlockImportOperation+Bknd::finalize_block (#12535) · 42215038
      Michal Kucharczyk authored 
      * BlockId removal: refactor: BlockImportOperation+Bknd::finalize_block

It changes the arguments of methods of `BlockImportOperation` trait
from: block: `BlockId<Block>` to: hash: `&Block::Hash`
`Backend::finalize_block` was also changed.

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* Review suggestion applied

thx to @davxy

* trigger CI job
      42215038
    • Michal Kucharczyk's avatar
      BlockId removal: refactor: Finalizer (#12528) · 749bcd19
      Michal Kucharczyk authored 
      * BlockId removal: refactor: Finalizer

It changes the arguments of methods of `Finalizer` trait from:
block: `BlockId<Block>` to: hash: `&Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* minor corrections

* failing test corrected

* minor rework
      749bcd19
  47. Oct 17, 2022
  49. Oct 13, 2022
  51. Oct 07, 2022
  53. Oct 06, 2022
    • Adrian Catangiu's avatar
      Add pluggable BEEFY payload constructors (#12428) · b91d2dfd
      Adrian Catangiu authored 
      * primitives/beefy: move Payload to its own file

* primitives/beefy: add Payload tests

* primitives/beefy: add MmrRootProvider as custom BEEFY payload provider

* client/beefy: use generic BEEFY 'PayloadProvider'

* primitives/beefy: rename Payload::new to Payload::from_single_entry for clarity

* fix visibility

* fix cargo doc
      b91d2dfd
  55. Oct 05, 2022
  57. Oct 04, 2022
  59. Oct 03, 2022
    • Adrian Catangiu's avatar
      Beefy on-demand justifications as a custom RequestResponse protocol (#12124) · 2a27545a
      Adrian Catangiu authored 
      

* client/beefy: create communication module and move gossip there

* client/beefy: move beefy_protocol_name module to communication

* client/beefy: move notification module under communication

* client/beefy: add incoming request_response protocol handler

* client/beefy: keep track of connected peers and their progress

* client/beefy: add logic for generating Justif requests

* client/beefy: cancel outdated on-demand justification requests

* try Andre's suggestion for JustificationEngine

* justif engine add justifs validation

* client/beefy: impl OnDemandJustificationsEngine async next()

* move beefy proto name test

* client/beefy: initialize OnDemandJustificationsEngine

* client/tests: allow for custom req-resp protocols

* client/beefy: on-demand-justif: implement simple peer selection strategy

* client/beefy: fix voter initialization

Fix corner case where voter gets a single burst of finality
notifications just when it starts.

The notification stream was consumed by "wait_for_pallet" logic,
then main loop would subscribe to finality notifications, but by that
time some notifications might've been lost.

Fix this by subscribing the main loop to notifications before waiting
for pallet to become available. Share the same stream with the main loop
so that notifications for blocks before pallet available are ignored,
while _all_ notifications after pallet available are processed.

Add regression test for this.

Signed-off-by: default avataracatangiu <[email protected]>

* client/beefy: make sure justif requests are always out for mandatory blocks

* client/beefy: add test for on-demand justifications sync

* client/beefy: tweak main loop event processing order

* client/beefy: run on-demand-justif-handler under same async task as voter

* client/beefy: add test for known-peers

* client/beefy: reorg request-response module

* client/beefy: add issue references for future work todos

* client/beefy: consolidate on-demand-justifications engine state machine

Signed-off-by: default avataracatangiu <[email protected]>

* client/beefy: fix for polkadot companion

* client/beefy: implement review suggestions

* cargo fmt and clippy

* fix merge damage

* fix rust-doc

* fix merge damage

* fix merge damage

* client/beefy: add test for justif proto name

Signed-off-by: default avataracatangiu <[email protected]>
      2a27545a
  61. Sep 30, 2022
  63. Sep 26, 2022
  65. Sep 21, 2022
  67. Sep 16, 2022
    • Adrian Catangiu's avatar
      client/beefy: fix voter initialization (#12274) · d9a4d835
      Adrian Catangiu authored 
      

Fix corner case where voter gets a single burst of finality
notifications just when it starts.

The notification stream was consumed by "wait_for_pallet" logic,
then main loop would subscribe to finality notifications, but by that
time some notifications might've been lost.

Fix this by subscribing the main loop to notifications before waiting
for pallet to become available. Share the same stream with the main loop
so that notifications for blocks before pallet available are ignored,
while _all_ notifications after pallet available are processed.

Add regression test for this.

Signed-off-by: default avataracatangiu <[email protected]>
      d9a4d835
  69. Sep 05, 2022
    • Adrian Catangiu's avatar
      beefy: initialize voter from genesis and fix initial sync (#11959) · 84acfd5f
      Adrian Catangiu authored 
      

* client/beefy: use backend instead of client where possible

* client/beefy: initialize voter from genesis

Now that we have justifications import, we can drop the "lean beefy"
behaviour and start building justifications chain from Genesis with
containing all past sessions' mandatory blocks justifications.

* client/beefy: walk finality tree_route to catch session changes

* client/beefy: fix block import

During initial block import blocks are not finalized, so trying to
validate and append justifications within block import fails (for
initial network sync imported blocks).

Changes:

- Move justification validation to _after_ `inner.block_import()`,
  so block is imported in backend and runtime api can be called to
  get the BEEFY authorities for said block.
- Move append-to-backend for imported BEEFY justification to voter,
  because it already has the required logic to BEEFY-finalize blocks
  only after GRANDPA finalized them.
- Mark voting rounds as concluded when finalizing through
  imported justifications as well as when finalizing through voting.

* client/beefy: valid justifications are one per block number

The only way we'd get _different_ _validated_ justifications for same
block number is if authorities are double voting, which will be handled
later.

* client/beefy: process incoming justifs during major sync

* client/beefy: correct voter initialization

BEEFY voter should resume voting from either:
  - last BEEFY finalized block,
  - session start,
whichever is closest to head.

* client/beefy: test voter initialization

* client/beefy: impl review suggestions

Signed-off-by: default avataracatangiu <[email protected]>
      84acfd5f
  71. Sep 03, 2022
    • Dmitry Markin's avatar
      Use custom type for ProtocolName (#12172) · 24d09fe8
      Dmitry Markin authored 
      * Add ProtocolName custom type

* Use new ProtocolName in sc_network_common

* Use new ProtocolName in sc_network

* Use new ProtocolName for BEEFY and GRANDPA

* Use new ProtocolName for notifications

* Use new ProtocolName in sc_network (part 2)

* Use new ProtocolName in sc_network_gossip

* Use new ProtocolName in sc_offchain

* Remove unused imports

* Some more fixes

* Add tests

* Fix minor import issues

* Re-export ProtocolName in sc_network

* Revert "Re-export ProtocolName in sc_network"

This reverts commit 8d8ff71927e7750757f29c9bbd88dc0ba181d214.

* Re-export ProtocolName in sc_network

* Remove dependency on sc-network-common from beefy-gadget
      24d09fe8
  73. Aug 15, 2022
  75. Aug 04, 2022