Skip to content
  1. Feb 16, 2023
    • dharjeezy's avatar
      client/beefy: add more metrics for production visibility (#12910) · 292e5ee4
      dharjeezy authored 
      

* few beefy metrics

* more beefy metrics

* some beefy metrics

* some beefy metrics

* more metrics

* other metrics

* fix tests

* merge changes

* Apply suggestions from code review

* client/beefy: fix metrics

* client/beefy: separate metrics per component, avoid double registering

* client/beefy: deduplicate metrics registration code

* remove unused metric

* impl review suggestions

---------

Co-authored-by: default avatarAdrian Catangiu <[email protected]>
      292e5ee4
  3. Feb 13, 2023
    • Adrian Catangiu's avatar
      client/beefy: request justifs from peers further in consensus (#13343) · d01e675f
      Adrian Catangiu authored 
      

For on-demand justifications, peer selection is based on witnessed
gossip votes. This commit changes the condition for selecting a peer
to request justification for `block` from
"last voted on >= `block`" to "peer last voted on strict > `block`".

When allowing `>=` we see nodes continuously spamming unsuccessful
on-demand requests to nodes which are still voting on a block without
having a justification available.

One way to fix the spam would be to add some rate-limiting or backoff
period when requesting justifications.

The other solution (present in this commit) is to simply request
justifications from peers that are voting on future blocks so we know
they're _guaranteed_ to have the wanted mandatory justification
available to send back.

Signed-off-by: default avataracatangiu <[email protected]>
      d01e675f
  5. Feb 06, 2023
  7. Feb 03, 2023
    • Adrian Catangiu's avatar
      BEEFY: define on-chain beefy-genesis and use it to coordinate voter initialization (#13215) · 981ffb29
      Adrian Catangiu authored 
      * beefy: add support to configure BEEFY genesis

* client/beefy: more flexible test runtime api

* client/beefy: add tests for custom BEEFY genesis

* client/beefy: ignore old state that didn't account for pallet genesis

* client/beefy: fix clippy

* frame/beefy: default BEEFY-genesis is block One::one()

* frame/beefy: add extra doc comments

---------

Co-authored-by: parity-processbot <>
      981ffb29
  9. Jan 27, 2023
  11. Jan 24, 2023
  13. Jan 19, 2023
  15. Jan 13, 2023
  17. Jan 12, 2023
  19. Dec 23, 2022
  21. Dec 20, 2022
    • Michal Kucharczyk's avatar
      `BlockId` removal: refactor: `HeaderBackend::header` (#12874) · 548955a7
      Michal Kucharczyk authored 
      * BlockId removal: refactor: HeaderBackend::header

It changes the arguments of:
- `HeaderBackend::header`,
- `Client::header`,
- `PeersClient::header`
- `ChainApi::block_header`

methods from: `BlockId<Block>` to: `Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* non-trivial usages of haeder(block_id) refactored

This may required introduction of dedicated function:
header_for_block_num

* fmt

* fix

* doc fixed

* ".git/.scripts/fmt.sh"

* BlockId removal: refactor: HeaderBackend::expect_header

It changes the arguments of `HeaderBackend::expect_header` method from: `BlockId<Block>` to: `Block::Hash`

* ".git/.scripts/fmt.sh"

* readme updated

* ".git/.scripts/fmt.sh"

* fix

Co-authored-by: parity-processbot <>
      548955a7
  23. Dec 19, 2022
    • Michal Kucharczyk's avatar
      sc-network-test::Peer: block push methods return hashes vec (#12944) · 1be51ccb
      Michal Kucharczyk authored 
      

* sc-network-test::Peer: block push methods return hashes vec

This commit reworks the block generation/push methods in
sc-network-test::Peer.

Now methods are providing the vector of hashes that were built.

This allows to get rid of redundant `block_hash_from_id` call, as all
hashes are known just after being built.

Similar approach was taken in BeefyTestNet::generate_blocks_and_sync
method.

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* fix

* Apply suggestions from code review

Co-authored-by: default avatarBastian Köcher <[email protected]>

Co-authored-by: default avatarBastian Köcher <[email protected]>
      1be51ccb
    • Qinxuan Chen's avatar
      *: update `rand` to v0.8.5 (#12962) · 40c04cf1
      Qinxuan Chen authored 
      * *: update rand to v0.8.5

* *: remove useless deps from Cargo.toml

* fix pallet-session-benchmarking

* fix pallet-election-provider-support test

* remove useless rand from dev-dependencies
      40c04cf1
  25. Dec 16, 2022
  27. Dec 15, 2022
  29. Dec 12, 2022
    • Niklas Adolfsson's avatar
      rpc server with HTTP/WS on the same socket (#12663) · 84303ca7
      Niklas Adolfsson authored 
      * jsonrpsee v0.16

add backwards compatibility

run old http server on http only

* cargo fmt

* update jsonrpsee 0.16.1

* less verbose cors log

* fix nit in log: WS -> HTTP

* revert needless changes in Cargo.lock

* remove unused features in tower

* fix nits; add client-core feature

* jsonrpsee v0.16.2
      84303ca7
  31. Dec 06, 2022
  33. Dec 05, 2022
    • dharjeezy's avatar
      client/beefy: add some bounds on enqueued votes (#12562) · 91e6de05
      dharjeezy authored 
      

Introduce bounds on the justifications and votes queues, so they do not grow forever if voter cannot make progress and consume from them. When bounds are hit, new votes or justifications get dropped.

* use a BTreeMap and check for bounds

* cargo fmt

* use usize

Co-authored-by: default avatarAdrian Catangiu <[email protected]>
      91e6de05
    • Dmitry Markin's avatar
      Upgrade tokio to 1.22.0 and replace async-std with tokio (#12646) · 5eb84f9c
      Dmitry Markin authored 
      * Replace deprecated libp2p feature specs with correct ones

* Bump tokio to 1.21.2

* Replace async-std libp2p primitives with tokio ones

* minor: rustfmt

* Fix TestNet to run initialization in the tokio context

* Convert telemetry test from async-std to tokio

* Convert notifications tests from async-std to tokio

* Convert chain sync tests from async-std to tokio

* Ditch async-std completely

* Make executor mandatory

* Bump tokio to 1.22.0

* minor: rustfmt

* Explicitly use tokio runtime in tests

* Move more tests to explicit tokio runtime

* Explicitly set multithreaded runtime in tokio test

* minor: rustfmt

* minor: fix comment

* Replace async-std with tokio in MMR tests
      5eb84f9c
  35. Nov 28, 2022
  37. Nov 23, 2022
  39. Nov 21, 2022
    • Adrian Catangiu's avatar
      client/beefy: persist voter state (#12712) · 3e7a2778
      Adrian Catangiu authored 
      

* client/beefy: prepare worker for persisting state

* client/beefy: persist voter state

* client/beefy: initialize persistent state

* client/beefy: try to vote from the very beginning

Now that voter is initialized from persistent state, it makes
sense that it can attempt voting right away. This also helps
the genesis case when we consider block `One` as mandatory.

* client/beefy: add tests for voter state db
* client/beefy: persist voter state as soon as initialized
* client/beefy: make sure min-block-delta is at least 1
* client/beefy: persist state after voting

Persist state after handling self vote to avoid double voting in case
of voter restarts.

* client/beefy: persist state after handling mandatory block vote

For mandatory blocks we want to make sure we're not losing votes
in case of crashes or restarts, since voter will not make further
progress without finalizing them.

* frame/beefy: use GENESIS_AUTHORITY_SET_ID on pallet genesis

* client/beefy: initialize voter at either genesis or last finalized
To guarantee unbroken chain of mandatory blocks justifications, voter
will always resume from either last BEEFY-justified block or
`pallet-beefy` genesis, whichever is more recent.

Initialization walks back the chain from latest GRANDPA finalized
block looking for one of the above. Along the way, it also records
and enqueues for processing any BEEFY mandatory blocks that have
been already GRANDPA finalized but not BEEFY finalized.

* client/beefy: decouple voter init from aux db state load
* client/beefy: fix voter init tests
* remove debug prints
* gadget future must be type ()
* fix init from last justification

Signed-off-by: default avatarAdrian Catangiu <[email protected]>
      3e7a2778
  41. Nov 16, 2022
  43. Nov 15, 2022
  45. Nov 09, 2022
    • Robert Hambrock's avatar
      Consolidate and deduplicate MMR API methods (#12530) · 53f97350
      Robert Hambrock authored 
      

* histor. batch proof: make best block arg optional

* correct testing range

* make generate_batch_proof stub for historical

* merge generate_{historical_}batch_proof functions

* merge generate_{batch_}proof functions

* merge verify_{batch_}proof functions

* merge verify_{batch_}proof_stateless functions

* remove {Leaf}Proof

Not utilized by API anymore, so superfluous.
Removal consistent with prior changes to just use "batch" proof API.

* rename BatchProof->Proof

no need to qualify if only one universal proof type.

* cleanup

* expose verify_proof rpc api

* document verify_proof

* expose verify_proof_stateless rpc api

* add optional BlockHash to mmr_root rpc api

* fixup! expose verify_proof rpc api

* fix documentation phrasing

Co-authored-by: default avatarAdrian Catangiu <[email protected]>

* documentation grammar

Co-authored-by: default avatarAdrian Catangiu <[email protected]>

* define mmr error msgs together with error enum

Co-authored-by: default avatarSerban Iorga <[email protected]>

* fixup! define mmr error msgs together with error enum

* map decoding errors to CallError::InvalidParams

Co-authored-by: default avatarSerban Iorga <[email protected]>

* fixup! map decoding errors to CallError::InvalidParams

Co-authored-by: default avatarAdrian Catangiu <[email protected]>
Co-authored-by: parity-processbot <>
Co-authored-by: default avatarSerban Iorga <[email protected]>
      53f97350
  47. Nov 07, 2022
  49. Nov 02, 2022
  51. Oct 31, 2022
  53. Oct 30, 2022
  55. Oct 24, 2022
  57. Oct 20, 2022
    • Michal Kucharczyk's avatar
      BlockId removal: refactor: BlockImportOperation+Bknd::finalize_block (#12535) · 42215038
      Michal Kucharczyk authored 
      * BlockId removal: refactor: BlockImportOperation+Bknd::finalize_block

It changes the arguments of methods of `BlockImportOperation` trait
from: block: `BlockId<Block>` to: hash: `&Block::Hash`
`Backend::finalize_block` was also changed.

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* Review suggestion applied

thx to @davxy

* trigger CI job
      42215038
    • Michal Kucharczyk's avatar
      BlockId removal: refactor: Finalizer (#12528) · 749bcd19
      Michal Kucharczyk authored 
      * BlockId removal: refactor: Finalizer

It changes the arguments of methods of `Finalizer` trait from:
block: `BlockId<Block>` to: hash: `&Block::Hash`

This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)

* minor corrections

* failing test corrected

* minor rework
      749bcd19
  59. Oct 17, 2022
  61. Oct 13, 2022
  63. Oct 07, 2022
  65. Oct 06, 2022
    • Adrian Catangiu's avatar
      Add pluggable BEEFY payload constructors (#12428) · b91d2dfd
      Adrian Catangiu authored 
      * primitives/beefy: move Payload to its own file

* primitives/beefy: add Payload tests

* primitives/beefy: add MmrRootProvider as custom BEEFY payload provider

* client/beefy: use generic BEEFY 'PayloadProvider'

* primitives/beefy: rename Payload::new to Payload::from_single_entry for clarity

* fix visibility

* fix cargo doc
      b91d2dfd
  67. Oct 05, 2022
  69. Oct 04, 2022
  71. Oct 03, 2022
    • Adrian Catangiu's avatar
      Beefy on-demand justifications as a custom RequestResponse protocol (#12124) · 2a27545a
      Adrian Catangiu authored 
      

* client/beefy: create communication module and move gossip there

* client/beefy: move beefy_protocol_name module to communication

* client/beefy: move notification module under communication

* client/beefy: add incoming request_response protocol handler

* client/beefy: keep track of connected peers and their progress

* client/beefy: add logic for generating Justif requests

* client/beefy: cancel outdated on-demand justification requests

* try Andre's suggestion for JustificationEngine

* justif engine add justifs validation

* client/beefy: impl OnDemandJustificationsEngine async next()

* move beefy proto name test

* client/beefy: initialize OnDemandJustificationsEngine

* client/tests: allow for custom req-resp protocols

* client/beefy: on-demand-justif: implement simple peer selection strategy

* client/beefy: fix voter initialization

Fix corner case where voter gets a single burst of finality
notifications just when it starts.

The notification stream was consumed by "wait_for_pallet" logic,
then main loop would subscribe to finality notifications, but by that
time some notifications might've been lost.

Fix this by subscribing the main loop to notifications before waiting
for pallet to become available. Share the same stream with the main loop
so that notifications for blocks before pallet available are ignored,
while _all_ notifications after pallet available are processed.

Add regression test for this.

Signed-off-by: default avataracatangiu <[email protected]>

* client/beefy: make sure justif requests are always out for mandatory blocks

* client/beefy: add test for on-demand justifications sync

* client/beefy: tweak main loop event processing order

* client/beefy: run on-demand-justif-handler under same async task as voter

* client/beefy: add test for known-peers

* client/beefy: reorg request-response module

* client/beefy: add issue references for future work todos

* client/beefy: consolidate on-demand-justifications engine state machine

Signed-off-by: default avataracatangiu <[email protected]>

* client/beefy: fix for polkadot companion

* client/beefy: implement review suggestions

* cargo fmt and clippy

* fix merge damage

* fix rust-doc

* fix merge damage

* fix merge damage

* client/beefy: add test for justif proto name

Signed-off-by: default avataracatangiu <[email protected]>
      2a27545a
  73. Sep 30, 2022