Skip to content
  1. Aug 15, 2022
  3. Aug 13, 2022
    • Tsvetomir Dimitrov's avatar
      Runtime API versioning (#11779) · 2bff2f84
      Tsvetomir Dimitrov authored 
      

* Runtime API versioning

Related to issue #11577

Add support for multiple versions of a Runtime API. The purpose is to
have one main version of the API, which is considered stable and
multiple unstable (aka staging) ones.

How it works
===========
Some methods of the API trait can be tagged with `#[api_version(N)]`
attribute where N is version number bigger than the main one. Let's call
them **staging methods** for brevity.

The implementor of the API decides which version to implement.

Example (from https://github.com/paritytech/substrate/issues/11577#issuecomment-1145347025):

```
decl_runtime_apis! {
    #{api_version(10)]
    trait Test {
         fn something() -> Vec<u8>;
         #[api_version(11)]
         fn new_cool_function() -> u32;
    }
}
```

```
impl_runtime_apis! {
    #[api_version(11)]
    impl Test for Runtime {
         fn something() -> Vec<u8> { vec![1, 2, 3] }

         fn new_cool_function() -> u32 {
             10
         }
    }
}
```

Version safety checks (currently not implemented)
=================================================
By default in the API trait all staging methods has got default
implementation calling `unimplemented!()`. This is a problem because if
the developer wants to implement version 11 in the example above and
forgets to add `fn new_cool_function()` in `impl_runtime_apis!` the
runtime will crash when the function is executed.

Ideally a compilation error should be generated in such cases.

TODOs
=====

Things not working well at the moment:
[ ] Version safety check
[ ] Integration tests of `primitives/api` are messed up a bit. More
specifically `primitives/api/test/tests/decl_and_impl.rs`
[ ] Integration test covering the new functionality.
[ ] Some duplicated code

* Update primitives/api/proc-macro/src/impl_runtime_apis.rs

Code review feedback and formatting

Co-authored-by: default avatarasynchronous rob <[email protected]>

* Code review feedback

Applying suggestions from @bkchr



* fmt

* Apply suggestions from code review

Co-authored-by: default avatarBastian Köcher <[email protected]>

* Code review feedback

* dummy trait -> versioned trait

* Implement only versioned traits (not compiling)

* Remove native API calls (still not compiling)

* fmt

* Fix compilation

* Comments

* Remove unused code

* Remove native runtime tests

* Remove unused code

* Fix UI tests

* Code review feedback

* Code review feedback

* attribute_names -> common

* Rework `append_api_version`

* Code review feedback

* Apply suggestions from code review

Co-authored-by: default avatarBastian Köcher <[email protected]>

* Code review feedback

* Code review feedback

* Code review feedback

* Use type alias for the default trait - doesn't compile

* Fixes

* Better error for `method_api_ver < trait_api_version`

* fmt

* Rework how we call runtime functions

* Update UI tests

* Fix warnings

* Fix doctests

* Apply suggestions from code review

Co-authored-by: default avatarBastian Köcher <[email protected]>

* Apply suggestions from code review

Co-authored-by: default avatarBastian Köcher <[email protected]>

* Fix formatting and small compilation errors

* Update primitives/api/proc-macro/src/impl_runtime_apis.rs

Co-authored-by: default avatarBastian Köcher <[email protected]>

Co-authored-by: default avatarasynchronous rob <[email protected]>
Co-authored-by: default avatarBastian Köcher <[email protected]>
Co-authored-by: default avatarBastian Köcher <[email protected]>
      2bff2f84
  5. Aug 12, 2022
  7. Aug 11, 2022
  9. Aug 10, 2022
  11. Aug 09, 2022
    • Nazar Mokrynskyi's avatar
      Network sync refactoring (part 6) (#11940) · a685582b
      Nazar Mokrynskyi authored 
      * Extract `NetworkKVProvider` trait in `sc-authority-discovery` and remove unnecessary dependency

* Extract `NetworkSyncForkRequest` trait in `sc-finality-grandpa`

* Relax requirements on `SyncOracle` trait, remove extra native methods from `NetworkService` that are already provided by trait impls

* Move `NetworkSigner` trait from `sc-authority-discovery` into `sc-network-common` and de-duplicate methods on `NetworkService`

* Move `NetworkKVProvider` trait from `sc-authority-discovery` into `sc-network-common` and de-duplicate methods on `NetworkService`

* Minimize `sc-authority-discovery` dependency on `sc-network`

* Move `NetworkSyncForkRequest` trait from `sc-finality-grandpa` to `sc-network-common` and de-duplicate methods in `NetworkService`

* Extract `NetworkStatusProvider` trait and de-duplicate methods on `NetworkService`

* Extract `NetworkPeers` trait and de-duplicate methods on `NetworkService`

* Extract `NetworkEventStream` trait and de-duplicate methods on `NetworkService`

* Move more methods from `NetworkService` into `NetworkPeers` trait

* Move `NetworkStateInfo` trait into `sc-network-common`

* Extract `NetworkNotification` trait and de-duplicate methods on `NetworkService`

* Extract `NetworkRequest` trait and de-duplicate methods on `NetworkService`

* Remove `NetworkService::local_peer_id()`, it is already provided by `NetworkStateInfo` impl

* Extract `NetworkTransaction` trait and de-duplicate methods on `NetworkService`

* Extract `NetworkBlock` trait and de-duplicate methods on `NetworkService`

* Remove dependencies on `NetworkService` from most of the methods of `sc-service`

* Address simple review comments
      a685582b
  13. Aug 03, 2022
  15. Aug 02, 2022
  17. Jul 29, 2022
    • Adrian Catangiu's avatar
      Lean BEEFY to Full BEEFY - don't skip (older) mandatory blocks and import justifications (#11821) · 1c6867c6
      Adrian Catangiu authored 
      

* client/beefy: don't accept vote for older rounds

* client/beefy: clean up and reorg the worker struct

* client/beefy: first step towards Full BEEFY

The first step from Lean->Full BEEFY is to have the worker
enforce uninterrupted line of BEEFY finalized mandatory blocks.

There is one mandatory block per session (the first block in the
session). As such, votes processing and votes generation now
enforces that all mandatory blocks are finalized in strict
monotonically increasing sequence and no block 'N' will be worked
on if there is any GRANDPA finalized but BEEFY non-final mandatory
block 'M', where 'M < N'.

Implementation details:

- Introduced 'VoterOracle' to separate the voting decisions logic,
  and track new/pending sessions.

- New sessions get queued up with the worker operating either:
  1. up-to-date - all mandatory blocks leading up to current GRANDPA
     finalized: queue has ONE element, the 'current session' where
     `mandatory_done == true`,
  2. lagging behind GRANDPA: queue has [1, N] elements, where all
     `mandatory_done == false`.
     In this state, everytime a session gets its mandatory block
     BEEFY finalized, the session is popped off the queue,
     eventually getting to operating mode `1. up-to-date`.

- Votes get triaged and those that fall withing the `VoterOracle`
  allowed window get processed, the others get dropped if stale,
  or buffered for later processing (when they reach the window).

- Worker general code was also updated to fall in one of two roles:
  1. react to external events and change internal 'state',
  2. generate events/votes based on internal 'state'.

Signed-off-by: default avataracatangiu <[email protected]>

* client/beefy: sketch idea for block import and sync

Signed-off-by: default avataracatangiu <[email protected]>

* client/beefy: add BEEFY block import

* client/beefy: process justifications from block import

* client/beefy: add TODOs for sync protocol

* client/beefy: add more docs and comments

* client/beefy-rpc: fix RPC error

* client/beefy: verify justification validity on block import

* client/beefy: more tests

* client/beefy: small fixes

- first handle and note the self vote before gossiping it,
- don't shortcircuit on err when processing pending votes.

* client/beefy: remove invalid justifications at block import

* todo: beefy block import tests

* RFC: ideas for multiple justifications per block

* Revert "RFC: ideas for multiple justifications per block"

This reverts commit 8256fb07d3124db69daf252720b3c0208202624d.

* client/beefy: append justif to backend on block import

* client/beefy: groundwork for block import test

* client/beefy: groundwork2 for block import test

* client/beefy: groundwork3 for block import test

* client/beefy: add block import test

* client/beefy: add required trait bounds to block import builder

* remove client from beefy block import, backend gets the job done

Signed-off-by: default avataracatangiu <[email protected]>
      1c6867c6
    • Koute's avatar
      Prevent double allocation of the payload when calling `sp_io::storage::get` (#11523) · c4b607d4
      Koute authored 
      

* Expose allocation stats in `FreeingBumpHeapAllocator`

* Return allocation stats when calling into the runtime

* Bump `parity-scale-codec` to 3.1.3 (fork)

* Prevent double allocation of the payload when calling `sp_io::storage::get`

* Fix tests

* Remove unnecessary `mut`

* Enable the `bytes` feature for `parity-scale-codec` in `sp-runtime-interface`

* Update client/allocator/src/freeing_bump.rs

Co-authored-by: default avatarBastian Köcher <[email protected]>

* Bump `parity-scale-codec` to 3.1.3

* Fix some of the UI tests

Co-authored-by: default avatarBastian Köcher <[email protected]>
      c4b607d4
  19. Jul 26, 2022
    • Sebastian Kunert's avatar
      Prepare for rust 1.62.1 (#11903) · 9f409dc0
      Sebastian Kunert authored 
      

* Update UI test output for rust 1.62.1

* switch ci to staging image to check that everything works

* fix artifacts node-bench-regression-guard

* Imeplement `scale_info::TypeInfo` manually to silence aggressive rust warning

* Fix more clippy lints

* Make clippy happy by relying on auto-deref were possible

* Add tracking issue to the comments

* pin ci image

Co-authored-by: default avataralvicsam <[email protected]>
      9f409dc0
  21. Jul 25, 2022
  23. Jul 18, 2022
  25. Jul 16, 2022
  27. Jul 07, 2022
    • Adrian Catangiu's avatar
      pallet-mmr: handle forks without collisions in offchain storage (#11594) · 27b53822
      Adrian Catangiu authored 
      

* pallet-mmr: fix some typos

* pallet-mmr: make the MMR resilient to chain forks

* pallet-mmr: get hash for block that added node

* beefy-mmr: add debug logging

* add explanatory comment

* account for block offset of pallet activation

* add support for finding all nodes added by leaf

* minor improvements

* add helper to return all nodes added to mmr with a leaf append

* simplify leaf_node_index_to_leaf_index

summing the (shifted) differences in peak positions adds up to the (shifted) final position, so
don't need to fold over positions.

* dead fish: this also doesn't work

The idea was to keep a rolling window of `(parent_hash, pos)` leaf
entries in the offchain db, with the window matching the one
that provides `block_num -> block_hash` mappings in `frame_system`.

Once a leaf exits the window it would be "canonicalized" by switching
its offchain db key from `(parent_hash, pos)` to simple `pos`.

This doesn't work however because there's no way to get leaf contents
from offchain db while in runtime context.. so no way to get+clear+set
leaf to change its key in offchain db.

Ideas:
1. move the "canonicalization" logic to offchain worker
2. enhance IndexingApi with "offchain::move(old_key, new_key)"
   This is weird, but correct, deterministic and safe AFAICT, so
   it could be exposed to runtime.

* simplify rightmost_leaf_node_index_from_pos

* minor fix

* move leaf canonicalization to offchain worker

* move storage related code to storage.rs

* on offchain reads use canonic key for old leaves

* fix offchain worker write using canon key

* fix pallet-mmr tests

* add documentation and fix logging

* add offchain mmr canonicalization test

* test canon + generate + verify

* fix pallet-beefy-mmr tests

* implement review suggestions

* improve test

* pallet-mmr: add offchain pruning of forks

* pallet-mmr: improve offchain pruning

Instead of keeping pruning map as single blob in offchain db,
keep individual parent-hash lists with block-num identifier as part
of the offchain key.

Signed-off-by: default avataracatangiu <[email protected]>

* pallet-mmr: improve MMRStore<OffchainStorage>::get()

Do the math and retrieve node using correct (canon or non-canon)
offchain db key, instead of blindly looking in both canon and non-canon
offchain db locations for each node.

Still fallback on looking at both if for any reason it's not where
expected.

Signed-off-by: default avataracatangiu <[email protected]>

* pallet-mmr: storage: improve logs

* fix tests: correctly persist overlay

runtime indexing API works on overlay, whereas offchain context
bypasses overlay, so for loops > canon-window, canon would fail.

* pallet-mmr: fix numeric typo in test

* add comment around LeafData requirements

Signed-off-by: default avataracatangiu <[email protected]>

Co-authored-by: default avatarRobert Hambrock <[email protected]>
      27b53822
  29. Jun 24, 2022
  31. Jun 22, 2022
    • Adrian Catangiu's avatar
      pallet-beefy-mmr: add API for BEEFY Authority Sets (#11406) · 3018051c
      Adrian Catangiu authored 
      

* pallet-beefy: add Config::OnNewValidatorSet type

Add a hook to pallet-beefy for doing specific work when
BEEFY validator set changes.

For example, this can be used by pallet-beefy-mmr to cache
a lightweight MMR root over validators and make it available
to light clients.

* pallet-beefy-mmr: implement OnNewValidatorSet

Implement pallet-beefy::OnNewValidatorSet to be notified of BEEFY
validator set changes. Use the notifications to compute and cache
a light weight 'BEEFY authority set' which is an MMR root over
BEEFY validator set plus some extra info.

Previously, pallet-beefy-mmr was interogating pallet-beefy about
validator set id on every block to find out when it needs to recompute
the authority set.
By using the event-driven approach in this commit, we also save one
extra state interogation per block.

* pallet-beefy-mmr: add new authority_set() API

Expose current and next BEEFY authority sets through runtime API.
These can be directly used by light clients to avoid having them
compute them themselves based on BEEFY validator sets.

Signed-off-by: default avataracatangiu <[email protected]>

* rename BeefyMmr exposed runtime api
      3018051c
  33. Jun 21, 2022
  35. Jun 20, 2022
  37. Jun 19, 2022
  39. Jun 17, 2022
  41. Jun 16, 2022
  43. Jun 15, 2022
    • Georges's avatar
      combine iteratons and tolerance in sp-npos-elections API (#11498) · b71e1804
      Georges authored 
      

* Initial implementation of mms

* Some more attempts at `mms`

* Functioning `MMS` algorithm implementation.
Adding some tests too

* More tests and typos fixed.

* Adding fuzzer for `mms`
(but could not test it on Mac M1)

* Missing imports

* Fixing rustdoc

* More accurate implementation of `mms`

* Removing the fuzzer `mms` implementation

* Implementing `NposSolver` for `MMS`
had to add the `Clone` trait, maybe I could see if I can get rid of it.

* Fixing rust docs by adding () to resolve ambiguity

* Amending `unwrap` to `expect`
removing unneeded `Clone` trait

* Removing redundant `mms3.rs`

* Implementing `BalancingConfig` and rustdoc changes

* Implementing `weight` for `MMS`

* Implementing `weight` for `MMS`

* Fixing post merge

* Initial implementation of mms

* Some more attempts at `mms`

* Functioning `MMS` algorithm implementation.
Adding some tests too

* More tests and typos fixed.

* Adding fuzzer for `mms`
(but could not test it on Mac M1)

* Missing imports

* Fixing rustdoc

* More accurate implementation of `mms`

* Removing the fuzzer `mms` implementation

* Implementing `NposSolver` for `MMS`
had to add the `Clone` trait, maybe I could see if I can get rid of it.

* Amending `unwrap` to `expect`
removing unneeded `Clone` trait

* Fixing rust docs by adding () to resolve ambiguity

* Removing redundant `mms3.rs`

* Implementing `BalancingConfig` and rustdoc changes

* Implementing `weight` for `MMS`

* Implementing `weight` for `MMS`

* Fixing post merge

* Removing left over from rebase

* Fixing tests

* Removing unneeded import

* Removing unneeded functions

* Removing useless imports

Co-authored-by: default avatarkianenigma <[email protected]>
      b71e1804
  45. Jun 14, 2022
  47. Jun 13, 2022
  49. Jun 11, 2022
  51. Jun 01, 2022
  53. May 31, 2022
    • Bastian Köcher's avatar
      Fix clippy on master (#11559) · 7f8b64c4
      Bastian Köcher authored
      7f8b64c4
    • Gavin Wood's avatar
      Introduce `EnsureOrigin::try_successul_origin` (#11558) · 9107ae41
      Gavin Wood authored 
      * Introduce `EnsureOrigin::try_successul_origin`

* Formatting

* Fixes

* Add Morph

* Fixes

* Formatting
      9107ae41
    • Gavin Wood's avatar
      Several tweaks needed for Governance 2.0 (#11124) · 7808b0c3
      Gavin Wood authored 
      

* Add stepped curve for referenda

* Treasury SpendOrigin

* Add tests

* Better Origin Or-gating

* Reciprocal curve

* Tests for reciprical and rounding in PerThings

* Tweaks and new quad curve

* Const derivation of reciprocal curve parameters

* Remove some unneeded code

* Actually useful linear curve

* Fixes

* Provisional curves

* Rejig 'turnout' as 'support'

* Use TypedGet

* Fixes

* Enable curve's ceil to be configured

* Formatting

* Fixes

* Fixes

* Fixes

* Remove EnsureOneOf

* Fixes

* Fixes

* Fixes

* Formatting

* Fixes

* Update frame/support/src/traits/dispatch.rs

Co-authored-by: default avatarKian Paimani <[email protected]>

* Grumbles

* Formatting

* Fixes

* APIs of VoteTally should include class

* Fixes

* Fix overlay prefix removal result

* Second part of the overlay prefix removal fix.

* Formatting

* Fixes

* Add some tests and make clear rounding algo

* Fixes

* Formatting

* Revert questionable fix

* Introduce test for kill_prefix

* Fixes

* Formatting

* Fixes

* Fix possible overflow

* Docs

* Add benchmark test

* Formatting

* Update frame/referenda/src/types.rs

Co-authored-by: default avatarKeith Yeung <[email protected]>

* Docs

* Fixes

* Use latest API in tests

* Formatting

* Whitespace

* Use latest API in tests

Co-authored-by: default avatarKian Paimani <[email protected]>
Co-authored-by: default avatarKeith Yeung <[email protected]>
      7808b0c3