Skip to content
  1. Nov 06, 2023
    • Andrei Sandu's avatar
      approval-voting improvement: include all tranche0 assignments in one certificate (#1178) · 0570b6fa
      Andrei Sandu authored 
      **_PR migrated from https://github.com/paritytech/polkadot/pull/6782_** 

This PR will upgrade the network protocol to version 3 -> VStaging which
will later be renamed to V3. This version introduces a new kind of
assignment certificate that will be used for tranche0 assignments.
Instead of issuing/importing one tranche0 assignment per candidate,
there will be just one certificate per relay chain block per validator.
However, we will not be sending out the new assignment certificates,
yet. So everything should work exactly as before. Once the majority of
the validators have been upgraded to the new protocol version we will
enable the new certificates (starting at a specific relay chain block)
with a new client update.

There are still a few things that need to be done:

- [x] Use bitfield instead of Vec<CandidateIndex>:
https://github.com/paritytech/polkadot/pull/6802


  - [x] Fix existing approval-distribution and approval-voting tests
  - [x] Fix bitfield-distribution and statement-distribution tests
  - [x] Fix network bridge tests
  - [x] Implement todos in the code
  - [x] Add tests to cover new code
  - [x] Update metrics
  - [x] Remove the approval distribution aggression levels: TBD PR
  - [x] Parachains DB migration 
  - [x] Test network protocol upgrade on Versi
  - [x] Versi Load test
  - [x] Add Zombienet test
  - [x] Documentation updates
- [x] Fix for sending DistributeAssignment for each candidate claimed by
a v2 assignment (warning: Importing locally an already known assignment)
 - [x]  Fix AcceptedDuplicate
 - [x] Fix DB migration so that we can still keep old data.
 - [x] Final Versi burn in

---------

Signed-off-by: default avatarAndrei Sandu <[email protected]>
Signed-off-by: default avatarAlexandru Gheorghe <[email protected]>
Co-authored-by: default avatarAlexandru Gheorghe <[email protected]>
      0570b6fa
  3. Sep 07, 2023
    • ordian's avatar
      polkadot: pin one block per session (#1220) · 15503883
      ordian authored 
      * polkadot: propagate UnpinHandle to ActiveLeafUpdate

Also extract the leaf creation for tests
into a common function.

* dispute-coordinator: try pinned blocks for slashin

* apparently 1.72 is smarter than 1.70

* address nits

* rename fresh_leaf to new_leaf
      15503883
  5. Sep 05, 2023
  7. Sep 04, 2023
  9. Sep 01, 2023
  11. Aug 29, 2023
  13. Aug 28, 2023
  15. Aug 25, 2023
  17. Aug 16, 2023
  19. Aug 15, 2023
    • Bastian Köcher's avatar
      Remove superflous parameter `overseer_enable_anyways` and make parachain node... · cf3271fe
      Bastian Köcher authored 
      Remove superflous parameter `overseer_enable_anyways` and make parachain node type more explicit (#7617)

* Remove superflous parameter `overseer_enable_anyways`

We don't need this flag, as we don't need the overseer enabled when the
node isn't a collator or validator.

* Rename `IsCollator` to `IsParachainNode`

`IsParachainNode` is more expressive and also encapsulates the state of
the parachain node being a full node. Some functionality like the
overseer needs to run always when the node runs alongside a parachain
node. The parachain node needs the overseer to e.g. recover PoVs. Other
things like candidate validation or pvf checking are only required for
when the node is running as validator.

* FMT

* Fix CI
      cf3271fe
  21. Aug 14, 2023
  23. Aug 10, 2023
  25. Aug 04, 2023
  27. Jul 25, 2023
    • Anton's avatar
      [Substrate companion] update libp2p to 0.52.0 (#7472) · 7a1d96e1
      Anton authored 
      * update tinyvec to 1.6.0

* update once_cell to 1.18.0

* update data-encoding to 2.4.0

* update libc

* update js-sys

* update wasm-bindgen-futures

* update pin-project

* update tokio

* update syn

* p2p protocol now contains `PeerId`

not multihash

* update arrayvec to 0.7.4

* update sha2

* update smallvec

* updates to Cargo.lock after merge

* redo dep updates

* update lru

in attempt to compile polkadot

* update lockfile for {"substrate"}

---------

Co-authored-by: parity-processbot <>
      7a1d96e1
  29. Jul 04, 2023
  31. Jun 21, 2023
  33. May 30, 2023
    • Tsvetomir Dimitrov's avatar
      `RollingSessionWindow` cleanup (#7204) · eb1ed63b
      Tsvetomir Dimitrov authored 
      

* Replace `RollingSessionWindow` with `RuntimeInfo` - initial commit

* Fix tests in import

* Fix the rest of the tests

* Remove dead code

* Fix todos

* Simplify session caching

* Comments for `SessionInfoProvider`

* Separate `SessionInfoProvider` from `State`

* `cache_session_info_for_head` becomes freestanding function

* Remove unneeded `mut` usage

* fn session_info -> fn get_session_info() to avoid name clashes. The function also tries to initialize `SessionInfoProvider`

* Fix SessionInfo retrieval

* Code cleanup

* Don't wrap `SessionInfoProvider` in an `Option`

* Remove `earliest_session()`

* Remove pre-caching -> wip

* Fix some tests and code cleanup

* Fix all tests

* Fixes in tests

* Fix comments, variable names and small style changes

* Fix a warning

* impl From<SessionWindowSize> for NonZeroUsize

* Fix logging for `get_session_info` - remove redundant logs and decrease log level to DEBUG

* Code review feedback

* Storage migration removing `COL_SESSION_WINDOW_DATA` from parachains db

* Remove `col_session_data` usages

* Storage migration clearing columns w/o removing them

* Remove session data column usages from `approval-voting` and `dispute-coordinator` tests

* Add some test cases from `RollingSessionWindow` to `dispute-coordinator` tests

* Fix formatting in initialized.rs

* Fix a corner case in `SessionInfo` caching for `dispute-coordinator`

* Remove `RollingSessionWindow` ;(

* Revert "Fix formatting in initialized.rs"

This reverts commit 0f94664ec9f3a7e3737a30291195990e1e7065fc.

* v2 to v3 migration drops `COL_DISPUTE_COORDINATOR_DATA` instead of clearing it

* Fix `NUM_COLUMNS` in `approval-voting`

* Use `columns::v3::NUM_COLUMNS` when opening db

* Update node/service/src/parachains_db/upgrade.rs

Co-authored-by: default avatarAndrei Sandu <[email protected]>

* Don't write in `COL_DISPUTE_COORDINATOR_DATA` for `test_rocksdb_migrate_2_to_3`

* Fix `NUM+COLUMNS` in approval_voting

* Fix formatting

* Fix columns usage

* Clarification comments about the different db versions

---------

Co-authored-by: default avatarAndrei Sandu <[email protected]>
      eb1ed63b
  35. May 12, 2023
    • Tsvetomir Dimitrov's avatar
      Replace `RollingSessionWindow` in approval-voting with `RuntimeInfo` (#7123) · 12dd9277
      Tsvetomir Dimitrov authored 
      * Replace `RollingSessionWindow` with `RuntimeInfo` - initial commit

* Fix tests in import

* Fix the rest of the tests

* Remove dead code

* Fix todos

* Simplify session caching

* Comments for `SessionInfoProvider`

* Separate `SessionInfoProvider` from `State`

* `cache_session_info_for_head` becomes freestanding function

* Remove unneeded `mut` usage

* fn session_info -> fn get_session_info() to avoid name clashes. The function also tries to initialize `SessionInfoProvider`

* Fix SessionInfo retrieval

* Code cleanup

* Don't wrap `SessionInfoProvider` in an `Option`

* Remove `earliest_session()`

* Remove pre-caching -> wip

* Fix some tests and code cleanup

* Fix all tests

* Fixes in tests

* Fix comments, variable names and small style changes

* Fix a warning

* impl From<SessionWindowSize> for NonZeroUsize

* Fix logging for `get_session_info` - remove redundant logs and decrease log level to DEBUG

* Code review feedback
      12dd9277
  37. May 04, 2023
  39. Apr 19, 2023
    • Davide Galassi's avatar
      Companion for Substrate #13889 (#7063) · 83fb51a2
      Davide Galassi authored 
      * Companion for substrate #13889

* Remove leftover

* Remove removed dependency

* Remove sp-consensus-vrf from lock

* Revert "Remove sp-consensus-vrf from lock"

This reverts commit 2269ca1e32df89272e8fd4544292204db387f436.

* Fix after substrate modifications

* Fix tests to use new VRF Signature type

* Don't rely of Deref trait

* Fix test

* Further code simplification

* Reuse garbage_vrf_signature

* update lockfile for {"substrate"}

---------

Co-authored-by: parity-processbot <>
      83fb51a2
  41. Apr 11, 2023
    • Mira Ressel's avatar
      apply clippy 1.68 suggestions · 0e9b0a64
      Mira Ressel authored 
      * Simplify &(ref foo, _) tuple pattern matches
* Drop unneccessary .clone() calls
* Replace .position(cond).is_some() by .any(cond)
* Drop unneccessary lifetime specs
      0e9b0a64
  43. Apr 08, 2023
  45. Mar 31, 2023
    • Mattia L.V. Bradascio's avatar
      Consolidate subsystem spans so they are all children of the leaf-activated root span (#6458) · 713f6625
      Mattia L.V. Bradascio authored 
      * Pass the PerLeafSpan as mutable reference to handle_new_head function

* cargo +nightly fmt --all

* Add mock span for test

* cargo +nightly fmt --all

* add new-blocks-hashes to span

* ref span in match statement, set span to disabled if not passed

* remove second match clause, make handle_new_head_span mutable

* cargo +nightly fmt --all

* improve tag on error and warning

* add imported blocks and info span

* cargo +nightly fmt --all

* Improve error for imported_blocks_and_info trace

* format tags on get_header_span

* add lost-to-finality tag

* add missing bracket

* - Add bitfield child span
- Add block db insertion span

* - fix update-bitfield span tag

* - Fix type conversion to u64
- Add missing argument

* - Cargo fmt

* - Test add_follows_from

* - Revert as  relationship between spans not working correctly

* - use drop to test if parent-child relationship can be re-established

* - remove bitfield span, check if parent-child relationship can be reestablished

* - Remove dangling bitfield span which is not used, to see if parent-child relationship can be re-established

* Another dangling bitfield span

* cargo fmt

* - add imported blocks and info span
- add candidate span per candidate

* add tags before moving block_header to push scope

* - Add db-insertion span

* cargo fmt

* fix types

* * Pass mutable reference to span in handle_new_head
* Change get-header-span tags in handle_new_head
* Create cache-session-info span in handle_new_head
* Create optional argument in determine_new_blocks
* Pass mutable reference to handle_new_head_span in determine_new_blocks in handle_new_head function
* Add candidate-hash, candidate-number, lost-to-finality tags to candidate_span in handle_new_head function
* Manually drop db_insertion_span and remove superfluous tags  to it, only keeping approved-bitfields tag
* Add ApprovalVoting stage in jaeger

* * Pass mutable reference to jaeger::Span in stead of PerLeafSpan
* Add block-import span

* *Pass optional_span (optional argument) to determine_new_blocks util function

* * Add num-candidates int tag to block_import_span

* * Add head tag to cache_session_span

* * Create PerLeafSpan in handle_from_overseer (this is required to establish parent-child relationship between approval-voting span, and leaf-activated root span)

* * Add candidate-import-span as child of block-import-span
* Add candidate-hash and num-approval tags to candidate-import-span

* * Fix num-candidate tag to bitvec-len tag in candidate-import-span

* *Fix imported_blocKs_and_info span to create new-block-span as not dealing with candidates

* Consider the future::select! block

* Use HashMap<Hash, jaeger::PerLeafSpan>

* Remove Stage 9

* Add missing spans

* cargo +nightly fmt --all

* Remove optional span argument for determine_new_blocks

* * Remove no-longer needed default PerLeafSpan implementation
* Remove no-longer necessary mock span given re-factoring of handle_new_head() no longer neeing mutable span
* Split validation-result and request-data (availability and validation code) spans into two by dropping request_validation_data_spans
* Remove drop statements for cache_session_info_span
*

* Remove unnecessary span

* Remove another excessively spammy span

* Add missing spans from State in import tests

* Use functional approach to get spans

* - Add functional approach for the approval-voting span
- Add doc on block_numbers given labelling ambiguity
- Add span pruning logic
- Use .add_para_id on validation_result_span

* Replace for hash_set in hash_set_iter with map closure

* cargo +nightly fmt --all

* Change from unconsumed `map` to `.for_each`

* cargo +nightly fmt --all

* Refactor add_para_id to validation_result_span

* cargo +nightly fmt --all

* Remove duplicate tag

* Add missing tag to handle-approved-ancestor span

* Refactor span pruning to only invoke retain once

* Typo in span name

* - Replace unwrap_or with unwrap_or_else due to lazy evaluation of trace-identifier in polkadot_node_jaeger
- Remove some redundant spans

* Add approval-distribution spans

* - Add unwrap_or_else on note-approved-in-chain-selection
- Use child_with_trace_id to add traceID string tag on span (note this does not change the traceID, but just adds a tag)

* cargo +nightly fmt --all

* - Add traceID tags were necessary in approval-voting and availability-distribution
- Always use block-hash tag in stead of relay-parent tag in approval-distribution

* Remove schedule-wakeup span as it will duplicate spans on existing wakeups (which should be a no-op)

* Remove a couple of warnings related to mutability

* Fix failing tests in availability distribution

* Add traceID tag to launch-approval and validation-result

* Reshuffle the validation and validation result spans to where more appropriate and add block-hash tag

* - Add tranche and should-trigger tag to process-wakeup span
- Add candidate-hash and traceID to check-and-import-approval span

* cargo fmt

* - Adjustments after PR comments

* Move span pruning after other pruning logic

* Remove DerefMut - no longer needed

* Relabel request-chunk spans

* - Fix typo in span label
- Add docs for drops

* Add new approval-voting span pruning logic

* Undo removal of !

* cargo fmt
      713f6625
  47. Mar 23, 2023
  49. Mar 17, 2023
  51. Mar 08, 2023
  53. Feb 03, 2023
  55. Jan 22, 2023
  57. Jan 11, 2023
  59. Dec 15, 2022
  61. Dec 06, 2022
    • Marcin S.'s avatar
      Companion for paritytech/substrate#12795 (#6374) · 3f951cce
      Marcin S. authored 
      

* Begin removing `parity-util-mem`; remove `collect_memory_stats`

* Update some dependencies that were using `parity-util-mem`

* Remove `trie-memory-tracker` feature

* Update Cargo.lock

* Update `kvdb-shared-tests`

* Add back jemalloc

* Add missing license header

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>
Co-authored-by: default avatarAndronik <[email protected]>
      3f951cce
  63. Dec 05, 2022
  65. Nov 30, 2022
    • alexgparity's avatar
      Clippyfy (#6341) · 9ea14e66
      alexgparity authored 
      

* Add clippy config and remove .cargo from gitignore

* first fixes

* Clippyfied

* Add clippy CI job

* comment out rusty-cachier

* minor

* fix ci

* remove DAG from check-dependent-project

* add DAG to clippy

Co-authored-by: default avataralvicsam <[email protected]>
      9ea14e66
    • Mara Broda's avatar
      sync versions with current release (0.9.33) (#6363) · b76086c6
      Mara Broda authored 
      * westend: update transaction version

* polkadot: update transaction version

* kusama: update transaction version

* Bump spec_version to 9330

* bump versions to 0.9.33
      b76086c6
  67. Nov 15, 2022
    • Marcin S.'s avatar
      Fixes "for loop over an `Option`" warnings (#6291) · d53513ff
      Marcin S. authored 
      Was seeing these warnings when running `cargo check --all`:

```
warning: for loop over an `Option`. This is more readably written as an `if let` statement
    --> node/core/approval-voting/src/lib.rs:1147:21
     |
1147 |             for activated in update.activated {
     |                              ^^^^^^^^^^^^^^^^
     |
     = note: `#[warn(for_loops_over_fallibles)]` on by default
help: to check pattern in a loop use `while let`
     |
1147 |             while let Some(activated) = update.activated {
     |             ~~~~~~~~~~~~~~~         ~~~
help: consider using `if let` to clear intent
     |
1147 |             if let Some(activated) = update.activated {
     |             ~~~~~~~~~~~~         ~~~
```

My guess is that `activated` used to be a SmallVec or similar, as is
`deactivated`. It was changed to an `Option`, the `for` still compiled (it's
technically correct, just weird), and the compiler didn't catch it until now.
      d53513ff
  69. Nov 12, 2022
  71. Nov 08, 2022
  73. Oct 31, 2022
  75. Oct 22, 2022