Skip to content
  1. Mar 30, 2023
    • Adrian Catangiu's avatar
      BEEFY: gossip finality proofs (#13727) · 92c1229e
      Adrian Catangiu authored 
      * sc-consensus-beefy: add justifications to gossip protocol

* sc-consensus-beefy: voter gossips finality proofs

* sc-consensus-beefy: add finality proof gossip test

* sc-consensus-beefy: always gossip finality proof

Gossip finality proof in _both_ cases of reaching finality threshold
through votes:
1. threshold reached through self vote,
2. threshold reached through incoming vote.

* address comments
      92c1229e
    • Aaro Altonen's avatar
      Attempt to relieve pressure on `mpsc_network_worker` (#13725) · 4240490d
      Aaro Altonen authored 
      * Attempt to relieve pressure on `mpsc_network_worker`

`SyncingEngine` interacting with `NetworkWorker` can put a lot of strain
on the channel if the number of inbound connections is high. This is
because `SyncingEngine` is notified of each inbound substream which it
then can either accept or reject and this causes a lot of message
exchange on the already busy channel.

Use a direct channel pair between `Protocol` and `SyncingEngine`
to exchange notification events. It is a temporary change to alleviate
the problems caused by syncing being an independent protocol and the
fix will be removed once `NotificationService` is implemented.

* Apply review comments

* fixes

* trigger ci

* Fix tests

Verify that both peers have a connection now that the validation goes
through `SyncingEngine`. Depending on how the tasks are scheduled,
one of them might not have the peer registered in `SyncingEngine` at which
point the test won't make any progress because block announcement received
from an unknown peer is discarded.

Move polling of `ChainSync` at the end of the function so that if a block
announcement causes a block request to be sent, that can be sent in the
same call to `SyncingEngine::poll()`.

---------

Co-authored-by: parity-processbot <>
      4240490d
  3. Mar 29, 2023
  5. Mar 28, 2023
  7. Mar 27, 2023
  9. Mar 26, 2023
  11. Mar 24, 2023
    • Aaro Altonen's avatar
      Get the correct number of connected peers in `SyncState` (#13700) · cef4b363
      Aaro Altonen authored 
      `Protocol` is not a reliable source for the information of connected
peers because it doesn't have real-time information of the actual
connectivity state because it's not resposible for accepting/rejecting
connections and gets that information with delay from `SyncinEngine`.
      cef4b363
    • Bastian Köcher's avatar
      try-runtime: Use configured wasm execution method (#13694) · 685b43c3
      Bastian Köcher authored 
      * try-runtime: Use configured wasm execution method

* Fix compilation
      685b43c3
    • Davide Galassi's avatar
      Keystore overhaul (final) (#13683) · 40e1704e
      Davide Galassi authored 
      

* Introduce keystore specialized sign methods

* Get rid of 'AppKey::UntypedGeneric' associated type.

Untyped generics are accessible using associated types 'Generic' associated type.
I.e. <T as AppKey>::Public::Generic

* Get rid of 'CryptoTypePublicPair'

* Trivial fix

* Small refactory of local keystore implementations

* Remove 'crypto_id' method from 'Public'

* Trivial rename of 'AppKey' to 'AppCrypto'

* Remove unused import

* Improve docs

* Better signature related errors for authority-discovery

* Apply review suggestion

* Apply review suggestions

Co-authored-by: default avatarKoute <[email protected]>

* Authority discoverty signing error revisited

* Signing error revisited for babe and aura as well

* Further cleanup

---------

Co-authored-by: default avatarKoute <[email protected]>
      40e1704e
  13. Mar 23, 2023
  15. Mar 22, 2023
  17. Mar 21, 2023
  19. Mar 20, 2023
    • Davide Galassi's avatar
      Keystore overhaul (iter 2) (#13634) · 480396fe
      Davide Galassi authored 
      * Remove bloat about remote keystore

* Update docs and remove unused 'KeystoreRef' trait

* Use wherever possible, MemoryKeystore for testing

* Remove unrequired fully qualified method syntax for Keystore
      480396fe
    • Mira Ressel's avatar
      address clippy & rustdoc warnings · faaa0c28
      Mira Ressel authored 
      The comment was out of date anyway, State::Live no longer takes a
snapshot_path argument.
      faaa0c28
  21. Mar 17, 2023
  23. Mar 16, 2023
    • Bastian Köcher's avatar
      Try to fix flaky `temp-base-path-work` test (#13505) · 3e73b755
      Bastian Köcher authored 
      

* Try to fix flaky `temp-base-path-work` test

The test is most of the time failing when checking if the database path was deleted. The assumption
is that it takes a little bit more time by the OS to actually clean up the temp path under high
load. The pr tries to fix this by checking multiple times if the path was deleted. Besides that it
also ensures that the tests that require the benchmark feature don't fail when compiled without the feature.

* ".git/.scripts/commands/fmt/fmt.sh"

* Capture signals earlier

* Rewrite tests to let them having one big timeout

* Remove unneeded dep

* Update bin/node/cli/tests/common.rs

Co-authored-by: default avatarKoute <[email protected]>

* Review feedback

* Update bin/node/cli/tests/common.rs

Co-authored-by: default avatarAnton <[email protected]>

---------

Co-authored-by: command-bot <>
Co-authored-by: default avatarKoute <[email protected]>
Co-authored-by: default avatarAnton <[email protected]>
      3e73b755
    • Adrian Catangiu's avatar
      sc-consensus-beefy: improve beefy gossip validator (#13606) · 3708b156
      Adrian Catangiu authored 
      

* sc-consensus-beefy: improve beefy gossip validator

Old gossip validator was pretty dumb, being very permissive with
incoming votes - only condition it had was to be newer than best
finalized.

New filter conditions:
 - voter rounds are initialized (discarding votes until voter is
   actually active),
 - only votes for current active set id are accepted,
 - only votes for rounds in the current voting session are accepted,
 - only votes for GRANDPA finalized blocks are accepted,
 - when BEEFY voter reaches mandatory round, only votes for said
   mandatory round are accepted.

New validator uses the VoterOracle to easily implement above conditions
and only allow through votes that are immediately useful to the voter.

After every GRANDPA or BEEFY finality, the gossip validator filter is
updated.

* sc-consensus-beefy: remove votes enqueueing

Since gossip validator will simply disallow votes for future rounds,
and only allow votes that the voter can immediately process, there
is no need for the voter to enqueue votes.

It will see these "future" votes later in rebroadcasts, when voter
will also be able to process them. Only at that point does gossip
accept and consume them.

* sc-consensus-beefy: refactor persistent state

Move best-beefy and best-grandpa into VoterOracle instead
of passing them around as params.
VoterOracle ultimately needs to know best-beefy and/or best-grandpa
for most of its functions.

* sc-consensus-beefy: further restrict gossip validator

Assuming mandatory done in current session:
Instead of allowing votes for any round in the current session, only
accept votes for rounds equal or better than best BEEFY finalized.

* sc-consensus-beefy: add a couple of comments

* sc-consensus-beefy: fix tests involving multiple tasks

Finalize blocks one a time in tests where we want gossip to happen
in a certain round. Otherwise, some tasks may be left behind in
terms of gossip round numbers because once "scheduled" a task will
greedily process as much as possible.

This change should be in line with the real-world scenario where
voters run "in parallel" across nodes, the only points of
synchronization being the finality notifications.

* sc-consensus-beefy: address review comments

---------

Signed-off-by: default avataracatangiu <[email protected]>
      3708b156
  25. Mar 15, 2023
  27. Mar 14, 2023
  29. Mar 13, 2023
  31. Mar 09, 2023
  33. Mar 07, 2023
  35. Mar 06, 2023
    • Liu-Cheng Xu's avatar
      Expose `new_full_parts_with_genesis_builder` from sc-service (#13539) · 58d1d9e1
      Liu-Cheng Xu authored 
      I forgot this in https://github.com/paritytech/substrate/pull/12291
      58d1d9e1
    • Aaro Altonen's avatar
      Extract syncing protocol from `sc-network` (#12828) · 1a7f5be0
      Aaro Altonen authored 
      

* Move import queue out of `sc-network`

Add supplementary asynchronous API for the import queue which means
it can be run as an independent task and communicated with through
the `ImportQueueService`.

This commit removes removes block and justification imports from
`sc-network` and provides `ChainSync` with a handle to import queue so
it can import blocks and justifications. Polling of the import queue is
moved complete out of `sc-network` and `sc_consensus::Link` is
implemented for `ChainSyncInterfaceHandled` so the import queue
can still influence the syncing process.

* Move stuff to SyncingEngine

* Move `ChainSync` instanation to `SyncingEngine`

Some of the tests have to be rewritten

* Move peer hashmap to `SyncingEngine`

* Let `SyncingEngine` to implement `ChainSyncInterface`

* Introduce `SyncStatusProvider`

* Move `sync_peer_(connected|disconnected)` to `SyncingEngine`

* Implement `SyncEventStream`

Remove `SyncConnected`/`SyncDisconnected` events from
`NetworkEvenStream` and provide those events through
`ChainSyncInterface` instead.

Modify BEEFY/GRANDPA/transactions protocol and `NetworkGossip` to take
`SyncEventStream` object which they listen to for incoming sync peer
events.

* Introduce `ChainSyncInterface`

This interface provides a set of miscellaneous functions that other
subsystems can use to query, for example, the syncing status.

* Move event stream polling to `SyncingEngine`

Subscribe to `NetworkStreamEvent` and poll the incoming notifications
and substream events from `SyncingEngine`.

The code needs refactoring.

* Make `SyncingEngine` into an asynchronous runner

This commits removes the last hard dependency of syncing from
`sc-network` meaning the protocol now lives completely outside of
`sc-network`, ignoring the hardcoded peerset entry which will be
addressed in the future.

Code needs a lot of refactoring.

* Fix warnings

* Code refactoring

* Use `SyncingService` for BEEFY

* Use `SyncingService` for GRANDPA

* Remove call delegation from `NetworkService`

* Remove `ChainSyncService`

* Remove `ChainSync` service tests

They were written for the sole purpose of verifying that `NetworWorker`
continues to function while the calls are being dispatched to
`ChainSync`.

* Refactor code

* Refactor code

* Update client/finality-grandpa/src/communication/tests.rs

Co-authored-by: default avatarAnton <[email protected]>

* Fix warnings

* Apply review comments

* Fix docs

* Fix test

* cargo-fmt

* Update client/network/sync/src/engine.rs

Co-authored-by: default avatarAnton <[email protected]>

* Update client/network/sync/src/engine.rs

Co-authored-by: default avatarAnton <[email protected]>

* Add missing docs

* Refactor code

---------

Co-authored-by: default avatarAnton <[email protected]>
      1a7f5be0
  37. Mar 03, 2023
  39. Mar 02, 2023
  41. Mar 01, 2023
  43. Feb 28, 2023