Skip to content
  1. Oct 06, 2022
    • Adrian Catangiu's avatar
      Add pluggable BEEFY payload constructors (#12428) · b91d2dfd
      Adrian Catangiu authored 
      * primitives/beefy: move Payload to its own file

* primitives/beefy: add Payload tests

* primitives/beefy: add MmrRootProvider as custom BEEFY payload provider

* client/beefy: use generic BEEFY 'PayloadProvider'

* primitives/beefy: rename Payload::new to Payload::from_single_entry for clarity

* fix visibility

* fix cargo doc
      b91d2dfd
  3. Oct 05, 2022
  5. Oct 04, 2022
  7. Oct 03, 2022
    • Adrian Catangiu's avatar
      Beefy on-demand justifications as a custom RequestResponse protocol (#12124) · 2a27545a
      Adrian Catangiu authored 
      

* client/beefy: create communication module and move gossip there

* client/beefy: move beefy_protocol_name module to communication

* client/beefy: move notification module under communication

* client/beefy: add incoming request_response protocol handler

* client/beefy: keep track of connected peers and their progress

* client/beefy: add logic for generating Justif requests

* client/beefy: cancel outdated on-demand justification requests

* try Andre's suggestion for JustificationEngine

* justif engine add justifs validation

* client/beefy: impl OnDemandJustificationsEngine async next()

* move beefy proto name test

* client/beefy: initialize OnDemandJustificationsEngine

* client/tests: allow for custom req-resp protocols

* client/beefy: on-demand-justif: implement simple peer selection strategy

* client/beefy: fix voter initialization

Fix corner case where voter gets a single burst of finality
notifications just when it starts.

The notification stream was consumed by "wait_for_pallet" logic,
then main loop would subscribe to finality notifications, but by that
time some notifications might've been lost.

Fix this by subscribing the main loop to notifications before waiting
for pallet to become available. Share the same stream with the main loop
so that notifications for blocks before pallet available are ignored,
while _all_ notifications after pallet available are processed.

Add regression test for this.

Signed-off-by: default avataracatangiu <[email protected]>

* client/beefy: make sure justif requests are always out for mandatory blocks

* client/beefy: add test for on-demand justifications sync

* client/beefy: tweak main loop event processing order

* client/beefy: run on-demand-justif-handler under same async task as voter

* client/beefy: add test for known-peers

* client/beefy: reorg request-response module

* client/beefy: add issue references for future work todos

* client/beefy: consolidate on-demand-justifications engine state machine

Signed-off-by: default avataracatangiu <[email protected]>

* client/beefy: fix for polkadot companion

* client/beefy: implement review suggestions

* cargo fmt and clippy

* fix merge damage

* fix rust-doc

* fix merge damage

* fix merge damage

* client/beefy: add test for justif proto name

Signed-off-by: default avataracatangiu <[email protected]>
      2a27545a
  9. Sep 30, 2022
  11. Sep 26, 2022
  13. Sep 21, 2022
  15. Sep 16, 2022
    • Adrian Catangiu's avatar
      client/beefy: fix voter initialization (#12274) · d9a4d835
      Adrian Catangiu authored 
      

Fix corner case where voter gets a single burst of finality
notifications just when it starts.

The notification stream was consumed by "wait_for_pallet" logic,
then main loop would subscribe to finality notifications, but by that
time some notifications might've been lost.

Fix this by subscribing the main loop to notifications before waiting
for pallet to become available. Share the same stream with the main loop
so that notifications for blocks before pallet available are ignored,
while _all_ notifications after pallet available are processed.

Add regression test for this.

Signed-off-by: default avataracatangiu <[email protected]>
      d9a4d835
  17. Sep 05, 2022
    • Adrian Catangiu's avatar
      beefy: initialize voter from genesis and fix initial sync (#11959) · 84acfd5f
      Adrian Catangiu authored 
      

* client/beefy: use backend instead of client where possible

* client/beefy: initialize voter from genesis

Now that we have justifications import, we can drop the "lean beefy"
behaviour and start building justifications chain from Genesis with
containing all past sessions' mandatory blocks justifications.

* client/beefy: walk finality tree_route to catch session changes

* client/beefy: fix block import

During initial block import blocks are not finalized, so trying to
validate and append justifications within block import fails (for
initial network sync imported blocks).

Changes:

- Move justification validation to _after_ `inner.block_import()`,
  so block is imported in backend and runtime api can be called to
  get the BEEFY authorities for said block.
- Move append-to-backend for imported BEEFY justification to voter,
  because it already has the required logic to BEEFY-finalize blocks
  only after GRANDPA finalized them.
- Mark voting rounds as concluded when finalizing through
  imported justifications as well as when finalizing through voting.

* client/beefy: valid justifications are one per block number

The only way we'd get _different_ _validated_ justifications for same
block number is if authorities are double voting, which will be handled
later.

* client/beefy: process incoming justifs during major sync

* client/beefy: correct voter initialization

BEEFY voter should resume voting from either:
  - last BEEFY finalized block,
  - session start,
whichever is closest to head.

* client/beefy: test voter initialization

* client/beefy: impl review suggestions

Signed-off-by: default avataracatangiu <[email protected]>
      84acfd5f
  19. Sep 03, 2022
    • Dmitry Markin's avatar
      Use custom type for ProtocolName (#12172) · 24d09fe8
      Dmitry Markin authored 
      * Add ProtocolName custom type

* Use new ProtocolName in sc_network_common

* Use new ProtocolName in sc_network

* Use new ProtocolName for BEEFY and GRANDPA

* Use new ProtocolName for notifications

* Use new ProtocolName in sc_network (part 2)

* Use new ProtocolName in sc_network_gossip

* Use new ProtocolName in sc_offchain

* Remove unused imports

* Some more fixes

* Add tests

* Fix minor import issues

* Re-export ProtocolName in sc_network

* Revert "Re-export ProtocolName in sc_network"

This reverts commit 8d8ff71927e7750757f29c9bbd88dc0ba181d214.

* Re-export ProtocolName in sc_network

* Remove dependency on sc-network-common from beefy-gadget
      24d09fe8
  21. Aug 15, 2022
  23. Aug 04, 2022
  25. Jul 29, 2022
    • Adrian Catangiu's avatar
      Lean BEEFY to Full BEEFY - don't skip (older) mandatory blocks and import justifications (#11821) · 1c6867c6
      Adrian Catangiu authored 
      

* client/beefy: don't accept vote for older rounds

* client/beefy: clean up and reorg the worker struct

* client/beefy: first step towards Full BEEFY

The first step from Lean->Full BEEFY is to have the worker
enforce uninterrupted line of BEEFY finalized mandatory blocks.

There is one mandatory block per session (the first block in the
session). As such, votes processing and votes generation now
enforces that all mandatory blocks are finalized in strict
monotonically increasing sequence and no block 'N' will be worked
on if there is any GRANDPA finalized but BEEFY non-final mandatory
block 'M', where 'M < N'.

Implementation details:

- Introduced 'VoterOracle' to separate the voting decisions logic,
  and track new/pending sessions.

- New sessions get queued up with the worker operating either:
  1. up-to-date - all mandatory blocks leading up to current GRANDPA
     finalized: queue has ONE element, the 'current session' where
     `mandatory_done == true`,
  2. lagging behind GRANDPA: queue has [1, N] elements, where all
     `mandatory_done == false`.
     In this state, everytime a session gets its mandatory block
     BEEFY finalized, the session is popped off the queue,
     eventually getting to operating mode `1. up-to-date`.

- Votes get triaged and those that fall withing the `VoterOracle`
  allowed window get processed, the others get dropped if stale,
  or buffered for later processing (when they reach the window).

- Worker general code was also updated to fall in one of two roles:
  1. react to external events and change internal 'state',
  2. generate events/votes based on internal 'state'.

Signed-off-by: default avataracatangiu <[email protected]>

* client/beefy: sketch idea for block import and sync

Signed-off-by: default avataracatangiu <[email protected]>

* client/beefy: add BEEFY block import

* client/beefy: process justifications from block import

* client/beefy: add TODOs for sync protocol

* client/beefy: add more docs and comments

* client/beefy-rpc: fix RPC error

* client/beefy: verify justification validity on block import

* client/beefy: more tests

* client/beefy: small fixes

- first handle and note the self vote before gossiping it,
- don't shortcircuit on err when processing pending votes.

* client/beefy: remove invalid justifications at block import

* todo: beefy block import tests

* RFC: ideas for multiple justifications per block

* Revert "RFC: ideas for multiple justifications per block"

This reverts commit 8256fb07d3124db69daf252720b3c0208202624d.

* client/beefy: append justif to backend on block import

* client/beefy: groundwork for block import test

* client/beefy: groundwork2 for block import test

* client/beefy: groundwork3 for block import test

* client/beefy: add block import test

* client/beefy: add required trait bounds to block import builder

* remove client from beefy block import, backend gets the job done

Signed-off-by: default avataracatangiu <[email protected]>
      1c6867c6
  27. Jul 26, 2022
    • Sebastian Kunert's avatar
      Prepare for rust 1.62.1 (#11903) · 9f409dc0
      Sebastian Kunert authored 
      

* Update UI test output for rust 1.62.1

* switch ci to staging image to check that everything works

* fix artifacts node-bench-regression-guard

* Imeplement `scale_info::TypeInfo` manually to silence aggressive rust warning

* Fix more clippy lints

* Make clippy happy by relying on auto-deref were possible

* Add tracking issue to the comments

* pin ci image

Co-authored-by: default avataralvicsam <[email protected]>
      9f409dc0
  29. Jul 12, 2022
    • Nazar Mokrynskyi's avatar
      Network sync refactoring (part 4) (#11412) · 5896072b
      Nazar Mokrynskyi authored 
      

* Remove direct dependency of `sc-network` on `sc-network-light`

* Move `WarpSyncProvider` trait and surrounding data structures into `sc-network-common`

* Move `WarpSyncProvider` trait and surrounding data structures into `sc-network-common`

* Create `sync` module in `sc-network-common`, create `ChainSync` trait there (not used yet), move a bunch of associated data structures from `sc-network-sync`

* Switch from concrete implementation to `ChainSync` trait from `sc-network-common`

* Introduce `OpaqueStateRequest`/`OpaqueStateResponse` to remove generics from `StateSync` trait

* Introduce `OpaqueBlockRequest`/`OpaqueBlockResponse`, make `scheme` module of `sc-network-sync` private

* Surface `sc-network-sync` into `sc-service` and make `sc-network` not depend on it anymore

* Remove now unnecessary dependency from `sc-network`

* Replace crate links with just text since dependencies are gone now

* Remove `warp_sync` re-export from `sc-network-common`

* Update copyright in network-related files

* Address review comments about documentation

* Apply review suggestion

* Rename `extra_requests` module to `metrics`

Co-authored-by: default avatarBastian Köcher <[email protected]>
      5896072b
  31. Jun 20, 2022
  33. Jun 14, 2022
  35. May 04, 2022
    • Web3 Smith's avatar
      Update MMR Runtime API with functionality to generate MMR proof for a series... · fd45676d
      Web3 Smith authored 
      
Update MMR Runtime API with functionality to generate MMR proof for a series of leaf indices  (#10635)

* updated mmr rpc api with functions for batch generation of proof

* update code comments

* fix build errors

* added tests to mmr-rpc

* add tests to pallet-mmr

* update comments

* minor comment fix

* remove unused variables

* fix rust doc errors

* refactor mmr runtime api

* fix tests

* minor fix

* minor fix

* fix node-runtime

* revert to initial api

* impl from proof fot batchproof

* minor fix

* minor fix

* use explicit functions to convert btw batch proof and single proof

* minor fix

* add new variant to mmr error

* fmt

* update conversion to single leaf proof

* fix style nit

Co-authored-by: default avatarAdrian Catangiu <[email protected]>
      fd45676d
    • Adrian Catangiu's avatar
      BEEFY voter bugfixes (#11335) · 1e2424ec
      Adrian Catangiu authored 
      

* beefy: gadget should always use current validator set

The gadget/client-voter was using previous' session validator set
to sign the 1st block in the new session (to have chained validator
set handoffs).

This is not necessary because:
1. BEEFY piggy-backs on GRANDPA and only works on canonical chain,
   so it need not concern itself with the validity of the block header
   (which contains digest with the new session's validator set). It
   can safely assume header is valid and simply use new validator set.
2. The BEEFY payload itself already contains a merkle root for the
   next validator set keys. So at the BEEFY-payload layer we already
   have a validated/trusted hand-off of authority.

Signed-off-by: default avataracatangiu <[email protected]>

* beefy: buffer votes for not yet finalized blocks

Signed-off-by: default avataracatangiu <[email protected]>

* beefy: add buffered votes regression test
      1e2424ec
  37. Apr 30, 2022
  39. Apr 28, 2022
  41. Apr 21, 2022
  43. Apr 13, 2022
    • Adrian Catangiu's avatar
      Expose MMR root through runtime API - use it in BEEFY client (#11183) · 80ce5c57
      Adrian Catangiu authored 
      * beefy-gadget: allow custom runtime api provider

* beefy-gadget: use mock runtime api in tests

* pallet-mmr: expose mmr root from state through runtime API

* beefy-gadget: get mmr root from runtime state

* pallet-beefy-mmr: remove MmrRoot from header digests

* frame/mmr: move mmr primitives out of frame

* frame/mmr: completely move primitives out of frame

* address review comments

* beefy-mmr: bring back mmr root from header digest

* clippy fixes for rustc 1.60

* address review comments
      80ce5c57
  45. Apr 03, 2022
  47. Mar 25, 2022
    • Adrian Catangiu's avatar
      Implement Lean BEEFY (#10882) · 498e8c18
      Adrian Catangiu authored 
      

Simplified BEEFY worker logic based on the invariant that GRANDPA
will always finalize 1st block of each new session, meaning BEEFY
worker is guaranteed to receive finality notification for the
BEEFY mandatory blocks.

Under these conditions the current design is as follows:
- session changes are detected based on BEEFY Digest present in
  BEEFY mandatory blocks,
- on each new session new `Rounds` of voting is created, with old
  rounds being dropped (for gossip rounds, last 3 are still alive
  so votes are still being gossiped),
- after processing finality for a block, the worker votes if
  a new voting target has become available as a result of said
  block finality processing,
- incoming votes as well as self-created votes are processed
  and signed commitments are created for completed BEEFY voting
  rounds,
- the worker votes if a new voting target becomes available
  once a round successfully completes.

On worker startup, the current validator set is retrieved from
the BEEFY pallet. If it is the genesis validator set, worker
starts voting right away considering Block #1 as session start.

Otherwise (not genesis), the worker will vote starting with
mandatory block of the next session.

Later on when we add the BEEFY initial-sync (catch-up) logic,
the worker will sync all past mandatory blocks Signed Commitments
and will be able to start voting right away.

BEEFY mandatory block is the block with header containing the BEEFY
`AuthoritiesChange` Digest, this block is guaranteed to be finalized
by GRANDPA.

This session-boundary block is signed by the ending-session's
validator set. Next blocks will be signed by the new session's
validator set. This behavior is consistent with what GRANDPA does
as well.

Also drop the limit N on active gossip rounds. In an adversarial
network, a bad actor could create and gossip N invalid votes with
round numbers larger than the current correct round number. This
would lead to votes for correct rounds to no longer be gossiped.

Add unit-tests for all components, including full voter consensus
tests.

Signed-off-by: default avatarAdrian Catangiu <[email protected]>
Co-authored-by: default avatarTomasz Drwięga <[email protected]>
Co-authored-by: David Salami <Wizdave97>
      498e8c18
  49. Mar 04, 2022
  51. Feb 09, 2022
    • Koute's avatar
      Add a new host function for reporting fatal errors; make WASM backtraces... · 9a31b2c3
      Koute authored 
      Add a new host function for reporting fatal errors; make WASM backtraces readable when printing out errors (#10741)

* Add a new host function for reporting fatal errors

* Fix one of the wasmtime executor tests

* Have `#[runtime_interface(wasm_only)]` actually mean WASM-only, and not no_std-only

* Print out errors through `Display` instead of `Debug`

* Switch one more trait to require `Error` for its error instead of only `Debug`

* Align to review comments
      9a31b2c3
  53. Jan 06, 2022
    • Adrian Catangiu's avatar
      Add BEEFY `latestFinalized` RPC and deduplicate code between BEEFY and GRANDPA (#10568) · fe8d2bc7
      Adrian Catangiu authored 
      

* beefy: add dummy latest_finalized() RPC

* beefy: rpc latest_best_beefy() using shared mem

* beefy: rpc populate latest_best_beefy()

* beefy: rpc handle readiness

* beefy: best block over channel - wip

Not working because channel can't be simply opened and receiver passed
to `rpc_extensions_builder` because `rpc_extensions_builder` has to be
`Fn` and not `FnOnce`... and and Receiver side of mpsc can't be cloned

yay!..

* beefy: make notification channels payload-agnostic

* beefy: use notification mechanism instead of custom channel

* beefy: add tracing key to notif channels

* sc-utils: add notification channel - wip

* beefy: use sc-utils generic notification channel

* grandpa: use sc-utils generic notification channel

* fix grumbles

* beefy-rpc: get best block header instead of number

* beefy-rpc: rename to `beefy_getFinalizedHead`

* fix nitpicks

* client-rpc-notifications: move generic Error from struct to fn

* beefy: use header from notification instead of getting from database

* beefy-rpc: get best block hash instead of header

* beefy-rpc: fix and improve latestHead test

* beefy-rpc: bubble up errors from rpc-handler instantiation

* update lockfile

* Apply suggestions from code review

Co-authored-by: default avatarAndré Silva <[email protected]>

* fix errors and warnings

* fix nit

Co-authored-by: default avatarAndré Silva <[email protected]>
      fe8d2bc7
  55. Jan 05, 2022
  57. Jan 03, 2022
  59. Dec 21, 2021
  61. Dec 20, 2021
  63. Dec 09, 2021
  65. Dec 01, 2021
  67. Nov 25, 2021
  69. Nov 17, 2021
  71. Sep 23, 2021