Skip to content
  1. Aug 02, 2022
  3. Jul 29, 2022
    • Adrian Catangiu's avatar
      Lean BEEFY to Full BEEFY - don't skip (older) mandatory blocks and import justifications (#11821) · 1c6867c6
      Adrian Catangiu authored 
      

* client/beefy: don't accept vote for older rounds

* client/beefy: clean up and reorg the worker struct

* client/beefy: first step towards Full BEEFY

The first step from Lean->Full BEEFY is to have the worker
enforce uninterrupted line of BEEFY finalized mandatory blocks.

There is one mandatory block per session (the first block in the
session). As such, votes processing and votes generation now
enforces that all mandatory blocks are finalized in strict
monotonically increasing sequence and no block 'N' will be worked
on if there is any GRANDPA finalized but BEEFY non-final mandatory
block 'M', where 'M < N'.

Implementation details:

- Introduced 'VoterOracle' to separate the voting decisions logic,
  and track new/pending sessions.

- New sessions get queued up with the worker operating either:
  1. up-to-date - all mandatory blocks leading up to current GRANDPA
     finalized: queue has ONE element, the 'current session' where
     `mandatory_done == true`,
  2. lagging behind GRANDPA: queue has [1, N] elements, where all
     `mandatory_done == false`.
     In this state, everytime a session gets its mandatory block
     BEEFY finalized, the session is popped off the queue,
     eventually getting to operating mode `1. up-to-date`.

- Votes get triaged and those that fall withing the `VoterOracle`
  allowed window get processed, the others get dropped if stale,
  or buffered for later processing (when they reach the window).

- Worker general code was also updated to fall in one of two roles:
  1. react to external events and change internal 'state',
  2. generate events/votes based on internal 'state'.

Signed-off-by: default avataracatangiu <[email protected]>

* client/beefy: sketch idea for block import and sync

Signed-off-by: default avataracatangiu <[email protected]>

* client/beefy: add BEEFY block import

* client/beefy: process justifications from block import

* client/beefy: add TODOs for sync protocol

* client/beefy: add more docs and comments

* client/beefy-rpc: fix RPC error

* client/beefy: verify justification validity on block import

* client/beefy: more tests

* client/beefy: small fixes

- first handle and note the self vote before gossiping it,
- don't shortcircuit on err when processing pending votes.

* client/beefy: remove invalid justifications at block import

* todo: beefy block import tests

* RFC: ideas for multiple justifications per block

* Revert "RFC: ideas for multiple justifications per block"

This reverts commit 8256fb07d3124db69daf252720b3c0208202624d.

* client/beefy: append justif to backend on block import

* client/beefy: groundwork for block import test

* client/beefy: groundwork2 for block import test

* client/beefy: groundwork3 for block import test

* client/beefy: add block import test

* client/beefy: add required trait bounds to block import builder

* remove client from beefy block import, backend gets the job done

Signed-off-by: default avataracatangiu <[email protected]>
      1c6867c6
    • Koute's avatar
      Prevent double allocation of the payload when calling `sp_io::storage::get` (#11523) · c4b607d4
      Koute authored 
      

* Expose allocation stats in `FreeingBumpHeapAllocator`

* Return allocation stats when calling into the runtime

* Bump `parity-scale-codec` to 3.1.3 (fork)

* Prevent double allocation of the payload when calling `sp_io::storage::get`

* Fix tests

* Remove unnecessary `mut`

* Enable the `bytes` feature for `parity-scale-codec` in `sp-runtime-interface`

* Update client/allocator/src/freeing_bump.rs

Co-authored-by: default avatarBastian Köcher <[email protected]>

* Bump `parity-scale-codec` to 3.1.3

* Fix some of the UI tests

Co-authored-by: default avatarBastian Köcher <[email protected]>
      c4b607d4
  5. Jul 26, 2022
    • Sebastian Kunert's avatar
      Prepare for rust 1.62.1 (#11903) · 9f409dc0
      Sebastian Kunert authored 
      

* Update UI test output for rust 1.62.1

* switch ci to staging image to check that everything works

* fix artifacts node-bench-regression-guard

* Imeplement `scale_info::TypeInfo` manually to silence aggressive rust warning

* Fix more clippy lints

* Make clippy happy by relying on auto-deref were possible

* Add tracking issue to the comments

* pin ci image

Co-authored-by: default avataralvicsam <[email protected]>
      9f409dc0
  7. Jul 25, 2022
  9. Jul 18, 2022
  11. Jul 16, 2022
  13. Jul 07, 2022
    • Adrian Catangiu's avatar
      pallet-mmr: handle forks without collisions in offchain storage (#11594) · 27b53822
      Adrian Catangiu authored 
      

* pallet-mmr: fix some typos

* pallet-mmr: make the MMR resilient to chain forks

* pallet-mmr: get hash for block that added node

* beefy-mmr: add debug logging

* add explanatory comment

* account for block offset of pallet activation

* add support for finding all nodes added by leaf

* minor improvements

* add helper to return all nodes added to mmr with a leaf append

* simplify leaf_node_index_to_leaf_index

summing the (shifted) differences in peak positions adds up to the (shifted) final position, so
don't need to fold over positions.

* dead fish: this also doesn't work

The idea was to keep a rolling window of `(parent_hash, pos)` leaf
entries in the offchain db, with the window matching the one
that provides `block_num -> block_hash` mappings in `frame_system`.

Once a leaf exits the window it would be "canonicalized" by switching
its offchain db key from `(parent_hash, pos)` to simple `pos`.

This doesn't work however because there's no way to get leaf contents
from offchain db while in runtime context.. so no way to get+clear+set
leaf to change its key in offchain db.

Ideas:
1. move the "canonicalization" logic to offchain worker
2. enhance IndexingApi with "offchain::move(old_key, new_key)"
   This is weird, but correct, deterministic and safe AFAICT, so
   it could be exposed to runtime.

* simplify rightmost_leaf_node_index_from_pos

* minor fix

* move leaf canonicalization to offchain worker

* move storage related code to storage.rs

* on offchain reads use canonic key for old leaves

* fix offchain worker write using canon key

* fix pallet-mmr tests

* add documentation and fix logging

* add offchain mmr canonicalization test

* test canon + generate + verify

* fix pallet-beefy-mmr tests

* implement review suggestions

* improve test

* pallet-mmr: add offchain pruning of forks

* pallet-mmr: improve offchain pruning

Instead of keeping pruning map as single blob in offchain db,
keep individual parent-hash lists with block-num identifier as part
of the offchain key.

Signed-off-by: default avataracatangiu <[email protected]>

* pallet-mmr: improve MMRStore<OffchainStorage>::get()

Do the math and retrieve node using correct (canon or non-canon)
offchain db key, instead of blindly looking in both canon and non-canon
offchain db locations for each node.

Still fallback on looking at both if for any reason it's not where
expected.

Signed-off-by: default avataracatangiu <[email protected]>

* pallet-mmr: storage: improve logs

* fix tests: correctly persist overlay

runtime indexing API works on overlay, whereas offchain context
bypasses overlay, so for loops > canon-window, canon would fail.

* pallet-mmr: fix numeric typo in test

* add comment around LeafData requirements

Signed-off-by: default avataracatangiu <[email protected]>

Co-authored-by: default avatarRobert Hambrock <[email protected]>
      27b53822
  15. Jun 24, 2022
  17. Jun 22, 2022
    • Adrian Catangiu's avatar
      pallet-beefy-mmr: add API for BEEFY Authority Sets (#11406) · 3018051c
      Adrian Catangiu authored 
      

* pallet-beefy: add Config::OnNewValidatorSet type

Add a hook to pallet-beefy for doing specific work when
BEEFY validator set changes.

For example, this can be used by pallet-beefy-mmr to cache
a lightweight MMR root over validators and make it available
to light clients.

* pallet-beefy-mmr: implement OnNewValidatorSet

Implement pallet-beefy::OnNewValidatorSet to be notified of BEEFY
validator set changes. Use the notifications to compute and cache
a light weight 'BEEFY authority set' which is an MMR root over
BEEFY validator set plus some extra info.

Previously, pallet-beefy-mmr was interogating pallet-beefy about
validator set id on every block to find out when it needs to recompute
the authority set.
By using the event-driven approach in this commit, we also save one
extra state interogation per block.

* pallet-beefy-mmr: add new authority_set() API

Expose current and next BEEFY authority sets through runtime API.
These can be directly used by light clients to avoid having them
compute them themselves based on BEEFY validator sets.

Signed-off-by: default avataracatangiu <[email protected]>

* rename BeefyMmr exposed runtime api
      3018051c
  19. Jun 21, 2022
  21. Jun 20, 2022
  23. Jun 19, 2022
  25. Jun 17, 2022
  27. Jun 16, 2022
  29. Jun 15, 2022
    • Georges's avatar
      combine iteratons and tolerance in sp-npos-elections API (#11498) · b71e1804
      Georges authored 
      

* Initial implementation of mms

* Some more attempts at `mms`

* Functioning `MMS` algorithm implementation.
Adding some tests too

* More tests and typos fixed.

* Adding fuzzer for `mms`
(but could not test it on Mac M1)

* Missing imports

* Fixing rustdoc

* More accurate implementation of `mms`

* Removing the fuzzer `mms` implementation

* Implementing `NposSolver` for `MMS`
had to add the `Clone` trait, maybe I could see if I can get rid of it.

* Fixing rust docs by adding () to resolve ambiguity

* Amending `unwrap` to `expect`
removing unneeded `Clone` trait

* Removing redundant `mms3.rs`

* Implementing `BalancingConfig` and rustdoc changes

* Implementing `weight` for `MMS`

* Implementing `weight` for `MMS`

* Fixing post merge

* Initial implementation of mms

* Some more attempts at `mms`

* Functioning `MMS` algorithm implementation.
Adding some tests too

* More tests and typos fixed.

* Adding fuzzer for `mms`
(but could not test it on Mac M1)

* Missing imports

* Fixing rustdoc

* More accurate implementation of `mms`

* Removing the fuzzer `mms` implementation

* Implementing `NposSolver` for `MMS`
had to add the `Clone` trait, maybe I could see if I can get rid of it.

* Amending `unwrap` to `expect`
removing unneeded `Clone` trait

* Fixing rust docs by adding () to resolve ambiguity

* Removing redundant `mms3.rs`

* Implementing `BalancingConfig` and rustdoc changes

* Implementing `weight` for `MMS`

* Implementing `weight` for `MMS`

* Fixing post merge

* Removing left over from rebase

* Fixing tests

* Removing unneeded import

* Removing unneeded functions

* Removing useless imports

Co-authored-by: default avatarkianenigma <[email protected]>
      b71e1804
  31. Jun 14, 2022
  33. Jun 13, 2022
  35. Jun 11, 2022
  37. Jun 01, 2022
  39. May 31, 2022
    • Bastian Köcher's avatar
      Fix clippy on master (#11559) · 7f8b64c4
      Bastian Köcher authored
      7f8b64c4
    • Gavin Wood's avatar
      Introduce `EnsureOrigin::try_successul_origin` (#11558) · 9107ae41
      Gavin Wood authored 
      * Introduce `EnsureOrigin::try_successul_origin`

* Formatting

* Fixes

* Add Morph

* Fixes

* Formatting
      9107ae41
    • Gavin Wood's avatar
      Several tweaks needed for Governance 2.0 (#11124) · 7808b0c3
      Gavin Wood authored 
      

* Add stepped curve for referenda

* Treasury SpendOrigin

* Add tests

* Better Origin Or-gating

* Reciprocal curve

* Tests for reciprical and rounding in PerThings

* Tweaks and new quad curve

* Const derivation of reciprocal curve parameters

* Remove some unneeded code

* Actually useful linear curve

* Fixes

* Provisional curves

* Rejig 'turnout' as 'support'

* Use TypedGet

* Fixes

* Enable curve's ceil to be configured

* Formatting

* Fixes

* Fixes

* Fixes

* Remove EnsureOneOf

* Fixes

* Fixes

* Fixes

* Formatting

* Fixes

* Update frame/support/src/traits/dispatch.rs

Co-authored-by: default avatarKian Paimani <[email protected]>

* Grumbles

* Formatting

* Fixes

* APIs of VoteTally should include class

* Fixes

* Fix overlay prefix removal result

* Second part of the overlay prefix removal fix.

* Formatting

* Fixes

* Add some tests and make clear rounding algo

* Fixes

* Formatting

* Revert questionable fix

* Introduce test for kill_prefix

* Fixes

* Formatting

* Fixes

* Fix possible overflow

* Docs

* Add benchmark test

* Formatting

* Update frame/referenda/src/types.rs

Co-authored-by: default avatarKeith Yeung <[email protected]>

* Docs

* Fixes

* Use latest API in tests

* Formatting

* Whitespace

* Use latest API in tests

Co-authored-by: default avatarKian Paimani <[email protected]>
Co-authored-by: default avatarKeith Yeung <[email protected]>
      7808b0c3
    • yjh's avatar
      sp-core: impl serde for some offchain types (#11512) · c808340d
      yjh authored 
      

* sp-core: impl serde for some offchain types

* Update primitives/core/src/offchain/mod.rs

Co-authored-by: default avatarBastian Köcher <[email protected]>

* remove serde impls from OpaqueNetworkState/OpaqueMultiaddr

* derive default

Co-authored-by: default avatarBastian Köcher <[email protected]>
      c808340d
  41. May 30, 2022
  43. May 29, 2022
    • Gavin Wood's avatar
      Safe and sane multi-item storage removal (#11490) · ecbd65fb
      Gavin Wood authored 
      * Fix overlay prefix removal result

* Second part of the overlay prefix removal fix.

* Report only items deleted from storage in clear_prefix

* Fix kill_prefix

* Formatting

* Remove unused code

* Fixes

* Fixes

* Introduce clear_prefix host function v3

* Formatting

* Use v2 for now

* Fixes

* Formatting

* Docs

* Child prefix removal should also hide v3 for now

* Fixes

* Fixes

* Formatting

* Fixes

* apply_to_keys_whle takes start_at

* apply_to_keys_whle takes start_at

* apply_to_keys_whle takes start_at

* Cursor API; force limits

* Use unsafe deprecated functions

* Formatting

* Fixes

* Grumbles

* Fixes

* Docs

* Some nitpicks 🙈



* Update primitives/externalities/src/lib.rs

Co-authored-by: default avatarBastian Köcher <[email protected]>

* Formatting

* Fixes

* cargo fmt

* Fixes

* Update primitives/io/src/lib.rs

Co-authored-by: default avatarKeith Yeung <[email protected]>

* Formatting

* Fixes

Co-authored-by: default avatarBastian Köcher <[email protected]>
Co-authored-by: default avatarBastian Köcher <[email protected]>
Co-authored-by: default avatarKeith Yeung <[email protected]>
      ecbd65fb
  45. May 24, 2022
  47. May 21, 2022
  49. May 18, 2022
    • Bastian Köcher's avatar
      trie: Optimize `keys` function (#11457) · 516a8fa4
      Bastian Köcher authored 
      * trie: Optimize `keys` function

Instead of iterating the entire state and collecting all keys that match the given prefix, we can
directly use the optimized prefix iterator.

* Add a test
      516a8fa4
  51. May 17, 2022
  53. May 16, 2022