Skip to content
  1. May 04, 2023
  3. Apr 28, 2023
  5. Apr 26, 2023
  7. Apr 25, 2023
  9. Apr 24, 2023
    • Andrei Sandu's avatar
      Fix rolling session window (#7126) · 634b2f6a
      Andrei Sandu authored 
      * Use stored sessions 🤦



Signed-off-by: default avatarAndrei Sandu <[email protected]>

* Add test

Signed-off-by: default avatarAndrei Sandu <[email protected]>

* remove redundant

Signed-off-by: default avatarAndrei Sandu <[email protected]>

---------

Signed-off-by: default avatarAndrei Sandu <[email protected]>
      634b2f6a
    • Tsvetomir Dimitrov's avatar
      Rework `dispute-coordinator` to use `RuntimeInfo` for obtaining session... · 3f4ce632
      Tsvetomir Dimitrov authored 
      
Rework `dispute-coordinator` to use `RuntimeInfo` for obtaining session information instead of `RollingSessionWindow` (#6968)

* Pass `SessionInfo` directly to `CandidateEnvironment::new` otherwise it should be an async function

* Replace calls to `RollingSessionWindow` with `RuntimeInfo`

Adjust `dispute-coordinator` initialization to use `RuntimeInfo`

* Modify `dispute-coordinator` initialization

* Pass `Hash` to `process_on_chain_votes` so that `RuntimeInfo` calls can be made

Remove some fixmes

* Pass `Hash` to `handle_import_statements` to perform `RuntimeInfo` calls

* remove todo comments

* Remove `error` from `Initialized`

Rework new session handling code

* Remove db code which is no longer used

* Update stale comment and remove unneeded type specification

* Cache SessionInfo on startup

* Use `DISPUTE_WINDOW` from primitives

* Fix caching in `process_active_leaves_update`

* handle_import_statements: leaf_hash -> block_hash

* Restore `ensure_available_session_info`

* Don't interrupt `process_on_chain_votes` if SessionInfo can't be fetched

* Small style improvements in logging

* process_on_chain_votes: leaf_hash -> block_hash

* Restore `note_earliest_session` - it is required to prune disputes and votes

* Cache new sessions only when there is an actual session change

* Fix tests

* `CandidateEnvironment::new` gets `session_idx` and fetches SessionInfo by itself to avoid the invariant where the input SessionIndex and SessionInfo parameters don't match

* Fix handling of missing session info

* Move sessions caching in `handle_startup` and fix tests

* Load `relay_parent` from db in `handle_import_statements` instead of passing it as a parameter via two functions

* Don't do two db reads

* Fix the twisted logic in `handle_import_statements`

* fixup

* Small style fix

* Decrease log levels for caching errors to debug and fix a typo

* Update outdated comment

* Remove `ensure_available_session_info`

* Load relay parent from db in `process_on_chain_votes`

* Revert "Load relay parent from db in `process_on_chain_votes`"

This reverts commit 978ad4f223d517faa7a7fbad96e3f8de4fa17501.

* Keep track of highest seen session and last session cached without gaps.

* Apply suggestions from code review

Co-authored-by: default avatarordian <[email protected]>

* Handle session caching failure on startup correctly

* Update node/core/dispute-coordinator/src/initialized.rs

Co-authored-by: default avatarordian <[email protected]>

* Simplify session caching retries

* Update stale comment

* Fix lower bound calculation for session caching

---------

Co-authored-by: default avatarordian <[email protected]>
      3f4ce632
  11. Apr 21, 2023
    • ordian's avatar
      update rocksdb to 0.20.1 (#7113) · 439ab875
      ordian authored 
      * update rocksdb to 0.20.1

* update lockfile for {"substrate"}

---------

Co-authored-by: parity-processbot <>
      439ab875
    • Marcin S.'s avatar
      PVF: Move PVF workers into separate crate (#7101) · e277f95b
      Marcin S. authored 
      * Move PVF workers into separate crate

* Fix indentation

* Fix compilation errors

* Fix more compilation errors

* Rename `worker.rs` files, make host interface to worker more clear

* Fix more compilation errors

* Fix more compilation errors

* Add link to issue

* Address review comments

* Update comment
      e277f95b
  13. Apr 20, 2023
    • Marcin S.'s avatar
      PVF: Don't dispute on missing artifact (#7011) · 0940cdd1
      Marcin S. authored 
      * PVF: Don't dispute on missing artifact

A dispute should never be raised if the local cache doesn't provide a certain
artifact. You can not dispute based on this reason, as it is a local hardware
issue and not related to the candidate to check.

Design:

Currently we assume that if we prepared an artifact, it remains there on-disk
until we prune it, i.e. we never check again if it's still there.

We can change it so that instead of artifact-not-found triggering a dispute, we
retry once (like we do for AmbiguousWorkerDeath, except we don't dispute if it
still doesn't work). And when enqueuing an execute job, we check for the
artifact on-disk, and start preparation if not found.

Changes:

- [x] Integration test (should fail without the following changes)
- [x] Check if artifact exists when executing, prepare if not
- [x] Return an internal error when file is missing
- [x] Retry once on internal errors
- [x] Document design (update impl guide)

* Add some context to wasm error message (it is quite long)

* Fix impl guide

* Add check for missing/inaccessible file

* Add comment referencing Substrate issue

* Add test for retrying internal errors

---------

Co-authored-by: parity-processbot <>
      0940cdd1
  15. Apr 19, 2023
  17. Apr 18, 2023
  19. Apr 17, 2023
  21. Apr 13, 2023
  23. Apr 12, 2023
    • Sam Johnson's avatar
      Companion for #13846 (#7022) · b3070be0
      Sam Johnson authored 
      * globally upgrade syn to 1.0.109

* globally upgrade quote to 1.0.26

* globally upgrade proc-macro2 to 1.0.56

* globally bump syn to v2.0.13

* update expander to v1.0.0

* temporary commit to prove new version of expander works

(new version hasn't been released yet so using git)

* use expander 2.0.0

* upgrade to syn 2.0.14

* update lock file
      b3070be0
  25. Apr 11, 2023
    • Mira Ressel's avatar
      apply clippy 1.68 suggestions · 0e9b0a64
      Mira Ressel authored 
      * Simplify &(ref foo, _) tuple pattern matches
* Drop unneccessary .clone() calls
* Replace .position(cond).is_some() by .any(cond)
* Drop unneccessary lifetime specs
      0e9b0a64
  27. Apr 08, 2023
  29. Apr 07, 2023
    • Tsvetomir Dimitrov's avatar
      Onchain scraper in `dispute-coordinator` will scrape... · 63a805ba
      Tsvetomir Dimitrov authored 
      Onchain scraper in `dispute-coordinator` will scrape `SCRAPED_FINALIZED_BLOCKS_COUNT` blocks before finality (#7013)

* Onchain scraper in `dispute-coordinator` will scrape `SCRAPED_FINALIZED_BLOCKS_COUNT` blocks before finality

The purpose is to make the availability of a `CandidateReceipt` for finalized candidates more likely.

For details see:  https://github.com/paritytech/polkadot/issues/7009

* Fix off by one error

* Replace `SCRAPED_FINALIZED_BLOCKS_COUNT` with `DISPUTE_CANDIDATE_LIFETIME_AFTER_FINALIZATION`
      63a805ba
    • Marcin S.'s avatar
      PVF: Minor refactor in workers code (#7012) · 5889119f
      Marcin S. authored 
      * Move version check to `worker_event_loop`

* More minor refactors

- More consistent use of `format_invalid` and `format_internal`.
- Fix a doc error.
- Fix some poorly-named local variables.
      5889119f
  31. Apr 05, 2023
  33. Apr 04, 2023
  35. Apr 03, 2023
  37. Mar 31, 2023
    • Mattia L.V. Bradascio's avatar
      Consolidate subsystem spans so they are all children of the leaf-activated root span (#6458) · 713f6625
      Mattia L.V. Bradascio authored 
      * Pass the PerLeafSpan as mutable reference to handle_new_head function

* cargo +nightly fmt --all

* Add mock span for test

* cargo +nightly fmt --all

* add new-blocks-hashes to span

* ref span in match statement, set span to disabled if not passed

* remove second match clause, make handle_new_head_span mutable

* cargo +nightly fmt --all

* improve tag on error and warning

* add imported blocks and info span

* cargo +nightly fmt --all

* Improve error for imported_blocks_and_info trace

* format tags on get_header_span

* add lost-to-finality tag

* add missing bracket

* - Add bitfield child span
- Add block db insertion span

* - fix update-bitfield span tag

* - Fix type conversion to u64
- Add missing argument

* - Cargo fmt

* - Test add_follows_from

* - Revert as  relationship between spans not working correctly

* - use drop to test if parent-child relationship can be re-established

* - remove bitfield span, check if parent-child relationship can be reestablished

* - Remove dangling bitfield span which is not used, to see if parent-child relationship can be re-established

* Another dangling bitfield span

* cargo fmt

* - add imported blocks and info span
- add candidate span per candidate

* add tags before moving block_header to push scope

* - Add db-insertion span

* cargo fmt

* fix types

* * Pass mutable reference to span in handle_new_head
* Change get-header-span tags in handle_new_head
* Create cache-session-info span in handle_new_head
* Create optional argument in determine_new_blocks
* Pass mutable reference to handle_new_head_span in determine_new_blocks in handle_new_head function
* Add candidate-hash, candidate-number, lost-to-finality tags to candidate_span in handle_new_head function
* Manually drop db_insertion_span and remove superfluous tags  to it, only keeping approved-bitfields tag
* Add ApprovalVoting stage in jaeger

* * Pass mutable reference to jaeger::Span in stead of PerLeafSpan
* Add block-import span

* *Pass optional_span (optional argument) to determine_new_blocks util function

* * Add num-candidates int tag to block_import_span

* * Add head tag to cache_session_span

* * Create PerLeafSpan in handle_from_overseer (this is required to establish parent-child relationship between approval-voting span, and leaf-activated root span)

* * Add candidate-import-span as child of block-import-span
* Add candidate-hash and num-approval tags to candidate-import-span

* * Fix num-candidate tag to bitvec-len tag in candidate-import-span

* *Fix imported_blocKs_and_info span to create new-block-span as not dealing with candidates

* Consider the future::select! block

* Use HashMap<Hash, jaeger::PerLeafSpan>

* Remove Stage 9

* Add missing spans

* cargo +nightly fmt --all

* Remove optional span argument for determine_new_blocks

* * Remove no-longer needed default PerLeafSpan implementation
* Remove no-longer necessary mock span given re-factoring of handle_new_head() no longer neeing mutable span
* Split validation-result and request-data (availability and validation code) spans into two by dropping request_validation_data_spans
* Remove drop statements for cache_session_info_span
*

* Remove unnecessary span

* Remove another excessively spammy span

* Add missing spans from State in import tests

* Use functional approach to get spans

* - Add functional approach for the approval-voting span
- Add doc on block_numbers given labelling ambiguity
- Add span pruning logic
- Use .add_para_id on validation_result_span

* Replace for hash_set in hash_set_iter with map closure

* cargo +nightly fmt --all

* Change from unconsumed `map` to `.for_each`

* cargo +nightly fmt --all

* Refactor add_para_id to validation_result_span

* cargo +nightly fmt --all

* Remove duplicate tag

* Add missing tag to handle-approved-ancestor span

* Refactor span pruning to only invoke retain once

* Typo in span name

* - Replace unwrap_or with unwrap_or_else due to lazy evaluation of trace-identifier in polkadot_node_jaeger
- Remove some redundant spans

* Add approval-distribution spans

* - Add unwrap_or_else on note-approved-in-chain-selection
- Use child_with_trace_id to add traceID string tag on span (note this does not change the traceID, but just adds a tag)

* cargo +nightly fmt --all

* - Add traceID tags were necessary in approval-voting and availability-distribution
- Always use block-hash tag in stead of relay-parent tag in approval-distribution

* Remove schedule-wakeup span as it will duplicate spans on existing wakeups (which should be a no-op)

* Remove a couple of warnings related to mutability

* Fix failing tests in availability distribution

* Add traceID tag to launch-approval and validation-result

* Reshuffle the validation and validation result spans to where more appropriate and add block-hash tag

* - Add tranche and should-trigger tag to process-wakeup span
- Add candidate-hash and traceID to check-and-import-approval span

* cargo fmt

* - Adjustments after PR comments

* Move span pruning after other pruning logic

* Remove DerefMut - no longer needed

* Relabel request-chunk spans

* - Fix typo in span label
- Add docs for drops

* Add new approval-voting span pruning logic

* Undo removal of !

* cargo fmt
      713f6625
  39. Mar 30, 2023
  41. Mar 29, 2023
    • s0me0ne-unkn0wn's avatar
      Check spawned worker version vs node version before PVF preparation (#6861) · 55b4aceb
      s0me0ne-unkn0wn authored 
      

* Check spawned worker version vs node version before PVF preparation

* Address discussions

* Propagate errors and shutdown preparation and execution pipelines properly

* Add logs; Fix execution worker checks

* Revert "Propagate errors and shutdown preparation and execution pipelines properly"

This reverts commit b96cc3160ff58db5ff001d8ca0bfea9bd4bdd0f2.

* Don't try to shut down; report the condition and exit worker

* Get rid of `VersionMismatch` preparation error

* Merge master

* Add docs; Fix tests

* Update Cargo.lock

* Kill again, but only the main node process

* Move unsafe code to a common safe function

* Fix libc dependency error on MacOS

* pvf spawning: Add some logging, add a small integration test

* Minor fixes

* Restart CI

---------

Co-authored-by: default avatarMarcin S <[email protected]>
      55b4aceb
    • Oliver Tale-Yazdi's avatar
      Fix feature (#6966) · faa2d5b0
      Oliver Tale-Yazdi authored 
      

Signed-off-by: default avatarOliver Tale-Yazdi <[email protected]>
      faa2d5b0
  43. Mar 28, 2023
    • Tsvetomir Dimitrov's avatar
      Histogram support in runtime metrics (#6935) · 22f404f0
      Tsvetomir Dimitrov authored 
      

* Histogram support in runtime metrics

Add support for histograms to the runtime metrics. Additionally add
`polkadot_parachain_verify_dispute_signature` histogram which tracks the
time needed from the runtime to verify a single validator signature of a
dispute statement.

* Add noops

* u64 instead of f64

* Update buckets

* Wrap `get_current_time()` in runtime metrics

* Change the dimension of the Histogram from usec to sec

* Fix a compilation error

* Update buckets

* Fix `on_signature_check_complete` calculation

* Update buckets

* Update buckets

* formatting

* Another weights update

* Adjust buckets again

* Final buckets adjustment

* Revert "Fix a compilation error"

This reverts commit 06290b40a39eeb78de2602d8916a39edf7a8b714.

* Update primitives/src/v4/metrics.rs

Co-authored-by: default avatarAndrei Sandu <[email protected]>

* Use `saturating_sub` for time difference calculation

* Pass nanoseconds to client instead of seconds (using f64 in runtime is dangerous)

---------

Co-authored-by: default avatarAndrei Sandu <[email protected]>
      22f404f0
  45. Mar 24, 2023