Skip to content
  1. Nov 08, 2022
  3. Nov 07, 2022
  5. Nov 01, 2022
    • Marcin S.'s avatar
      PVF timeouts follow-up (#6151) · 1f821976
      Marcin S. authored 
      * Rename timeout consts and timeout parameter; bump leniency

* Update implementor's guide with info about PVFs

* Make glossary a bit easier to read

* Add a note to LENIENT_PREPARATION_TIMEOUT

* Remove PVF-specific section from glossary

* Fix some typos
      1f821976
  7. Oct 31, 2022
  9. Oct 26, 2022
    • Marcin S.'s avatar
      Make some fixes to logging in PVF subsystem (#6180) · d4e3501e
      Marcin S. authored 
      * Log exit status code for workers

* Make log for execute job conclusion match prepare job conclusion

Trace log for conclusion of prepare job:

```rs
gum::debug!(
	target: LOG_TARGET,
	validation_code_hash = ?artifact_id.code_hash,
	?worker,
	?rip,
	"prepare worker concluded",
);
```

Co-authored-by: parity-processbot <>
      d4e3501e
  11. Oct 22, 2022
    • Boluwatife Bakre's avatar
      Use a more typesafe approach for managing indexed data (#6150) · 8eb1f461
      Boluwatife Bakre authored 
      

* Fix for issue #2403

* Nightly fmt

* Quick documentation fixes

* Default Implementation

* iter() function integrated

* Implemented iter functionalities

* Fmt

* small change

* updates node-network

* updates in dispute-coordinator

* Updates

* benchmarking fix

* minor fix

* test fixes in runtime api

* Update primitives/src/v2/mod.rs

Co-authored-by: default avatarAndronik <[email protected]>

* Update primitives/src/v2/mod.rs

Co-authored-by: default avatarAndronik <[email protected]>

* Update primitives/src/v2/mod.rs

Co-authored-by: default avatarAndronik <[email protected]>

* Update primitives/src/v2/mod.rs

Co-authored-by: default avatarAndronik <[email protected]>

* Update primitives/src/v2/mod.rs

Co-authored-by: default avatarAndronik <[email protected]>

* Removal of [index], shorting of FromIterator, Renaming of GroupValidators to ValidatorGroups

* Removal of ops import

* documentation fixes for spell check

* implementation of generic type

* Refactoring

* Test and documentation fixes

* minor test fix

* minor test fix

* minor test fix

* Update node/network/statement-distribution/src/lib.rs

Co-authored-by: default avatarAndronik <[email protected]>

* Update primitives/src/v2/mod.rs

Co-authored-by: default avatarAndronik <[email protected]>

* Update primitives/src/v2/mod.rs

Co-authored-by: default avatarAndronik <[email protected]>

* removed IterMut

* Update node/core/dispute-coordinator/src/import.rs

Co-authored-by: default avatarAndronik <[email protected]>

* Update node/core/dispute-coordinator/src/initialized.rs

Co-authored-by: default avatarAndronik <[email protected]>

* Update primitives/src/v2/mod.rs

Co-authored-by: default avatarAndronik <[email protected]>

* fmt

* IterMut

* documentation update

Co-authored-by: default avatarAndronik <[email protected]>

* minor adjustments and new TypeIndex trait

* spelling fix

* TypeIndex fix

Co-authored-by: default avatarAndronik <[email protected]>
      8eb1f461
    • Mara Broda's avatar
      sync versions with current release (0.9.31) (#6176) · f8cc39a7
      Mara Broda authored 
      * Bump spec_version to 9310

* bump transaction_version (0.9.31) (#6171)

* Bump transaction_version for polkadot

* Bump transaction_version for kusama

* Bump transaction_version for rococo

* Bump transaction_version for westend

* Bump transaction_version for polkadot

* Bump transaction_version for kusama

* Bump transaction_version for rococo

* Bump transaction_version for westend

* Bump crate versions (0.9.31)
      f8cc39a7
  13. Oct 13, 2022
  15. Oct 05, 2022
    • ordian's avatar
      update kvdb & co (#6111) · af6a5cd9
      ordian authored 
      

* toml changes

* REVERTME: patch

* adapt parachains db interface

* fix Cargo.toml patch after master rebase

* fix av-store

* fix chain-selection

* fix parachains-db?

* Revert "fix Cargo.toml patch after master rebase"

This reverts commit 3afcbf033c86027b3f2b909d83ec703591bdd287.

* Revert "REVERTME: patch"

This reverts commit 464b717cf4142d3d09c3d77b83700b632d8c5f54.

* Use `Ok` imported from prelude

Co-authored-by: default avatarBastian Köcher <[email protected]>

* update lockfile for {"substrate"}

* Revert "update lockfile for {"substrate"}"

This reverts commit fdc623de226f7645741b86c4b1a7d030fed2172d.

* cargo update -p sp-io

Co-authored-by: default avatarBastian Köcher <[email protected]>
Co-authored-by: parity-processbot <>
      af6a5cd9
  17. Oct 04, 2022
  19. Sep 30, 2022
  21. Sep 29, 2022
    • Robert Klotzner's avatar
      Demote warning (#6080) · 260f70e3
      Robert Klotzner authored 
      This can happen under heavy load - no value in warning here.

Scenario this happens:

- New slot
- We get asked about what fork to build on
- We start building (create inherent gets called)
- We learn about a better fork (but Babe does not care as it already has a fork)
- 6 seconds passed - New slot
- Babe asks about what for to build on - we answer with the best block we learned about 6 seconds ago (slightly after we got asked the last time)
- We start building (on that old block)
- Milliseconds later we learn about a new block (the one from the slot we are actually in now)
- We kick the old leaf -> create inherent fails

Full discussion: https://github.com/paritytech/substrate/issues/12375
      260f70e3
  23. Sep 27, 2022
  25. Sep 19, 2022
    • Tsvetomir Dimitrov's avatar
      Improved dispute votes import in provisioner (#5567) · 6ae9720c
      Tsvetomir Dimitrov authored 
      * Add `DisputeState` to `DisputeCoordinatorMessage::RecentDisputes`

The new signature of the message is:
```
RecentDisputes(oneshot::Sender<Vec<(SessionIndex, CandidateHash, DisputeStatus)>>),
```

As part of the change also add `DispiteStatus` to
`polkadot_node_primitives`.

* Move dummy_signature() in primitives/test-helpers

* Enable staging runtime api on Rococo

* Implementation

* Move disputes to separate module
* Vote prioritisation
* Duplicates handling
* Double vote handling
* Unit tests
* Logs and metrics
* Code review feedback
* Fix ACTIVE/INACTIVE separation and update partition names
* Add `fn dispute_is_inactive` to node primitives and refactor `fn get_active_with_status()` logic
* Keep the 'old' logic if the staging api is not enabled
* Fix some comments in tests
* Add warning message if there are any inactive_unknown_onchain disputes
* Add file headers and remove `use super::*;` usage outside tests
* Adding doc comments
* Fix test methods names

* Fix staging api usage

* Fix `get_disputes` runtime function implementation

* Fix compilation error

* Fix arithmetic operations in tests

* Use smaller test data

* Rename `RuntimeApiRequest::StagingDisputes` to `RuntimeApiRequest::Disputes`

* Remove `staging-client` feature flag

* fmt

* Remove `vstaging` feature flag

* Some comments regarding the staging api

* Rename dispute selection modules in provisioner
with_staging_api -> prioritized_selection
without_staging_api -> random_selection

* Comments for staging api

* Comments

* Additional logging

* Code review feedback

process_selected_disputes -> into_multi_dispute_statement_set
typo
In trait VoteType: vote_value -> is_valid

* Code review feedback

* Fix metrics

* get_disputes -> disputes

* Get time only once during partitioning

* Fix partitioning

* Comments

* Reduce the number of hardcoded api versions

* Code review feedback

* Unused import

* Comments

* More precise log messages

* Code review feedback

* Code review feedback

* Code review feedback - remove `trait VoteType`

* Code review feedback

* Trace log for DisputeCoordinatorMessage::QueryCandidateVotes counter in vote_selection
      6ae9720c
  27. Sep 15, 2022
  29. Sep 14, 2022
  31. Sep 09, 2022
  33. Sep 06, 2022
  35. Sep 05, 2022
  37. Sep 02, 2022
  39. Sep 01, 2022
    • Robert Klotzner's avatar
      Don't store available data on disputes (#5950) · 254582df
      Robert Klotzner authored 
      

* Don't store available data on disputes

If there are lots of disputes, this leads to blowing up disk space on
validators. Rob luckily remembered that we do store the full
availability in participation.

The argument in the code does not make too much sense with the current
implementation, as no validator will ever request anything else from us,
than the one piece we are meant to posess.

* Fix warnings.

* Fix compile warnings

* Remove redundant field.

Co-authored-by: default avatarVsevolod Stakhov <[email protected]>
      254582df
  41. Aug 30, 2022
  43. Aug 29, 2022
  45. Aug 24, 2022
  47. Aug 16, 2022
    • Robert Klotzner's avatar
      Break cycle collator-procol - backing (#5883) · 16a35652
      Robert Klotzner authored
      16a35652
    • Robert Klotzner's avatar
      Don't request votes if there is nothing to request. (#5888) · 05a79665
      Robert Klotzner authored 
      * Don't request votes if there is nothing to request.

* Add trace log
      05a79665
    • Robert Klotzner's avatar
      Reduce dispute coordinator load (#5785) · ca6297c8
      Robert Klotzner authored 
      * Don't import backing statements directly

into the dispute coordinator. This also gets rid of a redundant
signature check. Both should have some impact on backing performance.
In general this PR should make us scale better in the number of parachains.

Reasoning (aka why this is fine):

For the signature check: As mentioned, it is a redundant check. The
signature has already been checked at this point. This is even made
obvious by the used types. The smart constructor is not perfect as
discussed [here](https://github.com/paritytech/polkadot/issues/3455

),
but is still a reasonable security.

For not importing to the dispute-coordinator: This should be good as the
dispute coordinator does scrape backing votes from chain. This suffices
in practice as a super majority of validators must have seen a backing
fork in order for a candidate to get included and only included
candidates pose a threat to our system. The import from chain is
preferable over direct import of backing votes for two reasons:

1. The import is batched, greatly improving import performance. All
   backing votes for a candidate are imported with a single import.
   And indeed we were able to see in metrics that importing votes
   from chain is fast.
2. We do less work in general as not every candidate for which
   statements are gossiped might actually make it on a chain. The
   dispute coordinator as with the current implementation would still
   import and keep those votes around for six sessions.

While redundancy is good for reliability in the event of bugs, this also
comes at a non negligible cost. The dispute-coordinator right now is the
subsystem with the highest load, despite the fact that it should not be
doing much during mormal operation and it is only getting worse
with more parachains as the load is a direct function of the number of statements.

We'll see on Versi how much of a performance improvement this PR

* Get rid of dead code.

* Dont send approval vote

* Make it pass CI

* Bring back tests for fixing them later.

* Explicit signature check.

* Resurrect approval-voting tests (not fixed yet)

* Send out approval votes in dispute-distribution.

Use BTreeMap for ordered dispute votes.

* Bring back an important warning.

* Fix approval voting tests.

* Don't send out dispute message on import + test

+ Some cleanup.

* Guide changes.

Note that the introduced complexity is actually redundant.

* WIP: guide changes.

* Finish guide changes about dispute-coordinator

conceputally. Requires more proof read still.

Also removed obsolete implementation details, where the code is better
suited as the source of truth.

* Finish guide changes for now.

* Remove own approval vote import logic.

* Implement logic for retrieving approval-votes

into approval-voting and approval-distribution subsystems.

* Update roadmap/implementers-guide/src/node/disputes/dispute-coordinator.md

Co-authored-by: default avatarasynchronous rob <[email protected]>

* Review feedback.

In particular: Add note about disputes of non included candidates.

* Incorporate Review Remarks

* Get rid of superfluous space.

* Tidy up import logic a bit.

Logical vote import is now separated, making the code more readable and
maintainable.

Also: Accept import if there is at least one invalid signer that has not
exceeded its spam slots, instead of requiring all of them to not exceed
their limits. This is more correct and a preparation for vote batching.

* We don't need/have empty imports.

* Fix tests and bugs.

* Remove error prone redundancy.

* Import approval votes on dispute initiated/concluded.

* Add test for approval vote import.

* Make guide checker happy (hopefully)

* Another sanity check + better logs.

* Reasoning about boundedness.

* Use `CandidateIndex` as opposed to `CoreIndex`.

* Remove redundant import.

* Review remarks.

* Add metric for calls to request signatures

* More review remarks.

* Add metric on imported approval votes.

* Include candidate hash in logs.

* More trace log

* Break cycle.

* Add some tracing.

* Cleanup allowed messages.

* fmt

* Tracing + timeout for get inherent data.

* Better error.

* Break cycle in all places.

* Clarified comment some more.

* Typo.

* Break cycle approval-distribution - approval-voting.

Co-authored-by: default avatarasynchronous rob <[email protected]>
      ca6297c8
  49. Aug 09, 2022
  51. Jul 28, 2022
  53. Jul 20, 2022
    • Sebastian Kunert's avatar
      Introduce async runtime calling trait for runtime-api subsystem (#5782) · 72bde288
      Sebastian Kunert authored 
      

* Implement OverseerRuntimeClient

* blockchainevents

* Update patches

* Finish merging rntime-api subsystem

* First version that is able to produce blocks

* Make OverseerRuntimeClient async

* Move overseer notification stream forwarding to cumulus

* Remove unused imports

* Add more logging to collator-protocol

* Lockfile

* Use hashes in OverseerRuntimeClient

* Move OverseerRuntimeClient into extra module

* Fix old session info call and make HeadSupportsParachain async

* Improve naming of trait

* Cleanup

* Remove unused From trait implementation

* Remove unwanted debug print

* Move trait to polkadot-node-subsystem-types

* Add sections to runtime client

Co-authored-by: default avatarDavide Galassi <[email protected]>

* Reorder methods

* Fix spelling

* Fix spacing in Cargo.toml

Co-authored-by: default avatarBastian Köcher <[email protected]>

* Remove unused babe methods

Co-authored-by: default avatarDavide Galassi <[email protected]>
Co-authored-by: default avatarBastian Köcher <[email protected]>
      72bde288
  55. Jul 09, 2022
    • Vsevolod Stakhov's avatar
      Implement prune only stagnant check mode (#5761) · c11c1f38
      Vsevolod Stakhov authored 
      

* Limit number of elements loaded from the stagnant key

This will likely be required if we enable stagnant prunning as currently database has way
too many entries to be prunned in a single iteration

* Fmt run

* Slightly improve logging

* Some more debug nits

* Fmt pass

* Add stagnant prunning delay

* Enable stagnant check worker

* Implement stagnant pruning without stagnant checks

* Update node/core/chain-selection/src/tree.rs

Co-authored-by: default avatarAndronik <[email protected]>

* Apply suggestions from code review

Co-authored-by: default avatarAndronik <[email protected]>

Co-authored-by: default avatarAndronik <[email protected]>
      c11c1f38
  57. Jul 06, 2022
  59. Jul 04, 2022
    • Vsevolod Stakhov's avatar
      Limit stagnant checks to a certain amount of entries (#5742) · 9dc99f02
      Vsevolod Stakhov authored 
      * Limit number of elements loaded from the stagnant key

This will likely be required if we enable stagnant prunning as currently database has way
too many entries to be prunned in a single iteration

* Fmt run

* Slightly improve logging

* Some more debug nits

* Fmt pass
      9dc99f02
  61. Jun 27, 2022
  63. Jun 24, 2022