Skip to content
  1. May 07, 2023
  3. May 05, 2023
  5. May 04, 2023
  7. Apr 28, 2023
  9. Apr 26, 2023
  11. Apr 25, 2023
  13. Apr 24, 2023
    • Andrei Sandu's avatar
      Fix rolling session window (#7126) · 634b2f6a
      Andrei Sandu authored 
      * Use stored sessions 🤦



Signed-off-by: default avatarAndrei Sandu <[email protected]>

* Add test

Signed-off-by: default avatarAndrei Sandu <[email protected]>

* remove redundant

Signed-off-by: default avatarAndrei Sandu <[email protected]>

---------

Signed-off-by: default avatarAndrei Sandu <[email protected]>
      634b2f6a
    • Tsvetomir Dimitrov's avatar
      Rework `dispute-coordinator` to use `RuntimeInfo` for obtaining session... · 3f4ce632
      Tsvetomir Dimitrov authored 
      
Rework `dispute-coordinator` to use `RuntimeInfo` for obtaining session information instead of `RollingSessionWindow` (#6968)

* Pass `SessionInfo` directly to `CandidateEnvironment::new` otherwise it should be an async function

* Replace calls to `RollingSessionWindow` with `RuntimeInfo`

Adjust `dispute-coordinator` initialization to use `RuntimeInfo`

* Modify `dispute-coordinator` initialization

* Pass `Hash` to `process_on_chain_votes` so that `RuntimeInfo` calls can be made

Remove some fixmes

* Pass `Hash` to `handle_import_statements` to perform `RuntimeInfo` calls

* remove todo comments

* Remove `error` from `Initialized`

Rework new session handling code

* Remove db code which is no longer used

* Update stale comment and remove unneeded type specification

* Cache SessionInfo on startup

* Use `DISPUTE_WINDOW` from primitives

* Fix caching in `process_active_leaves_update`

* handle_import_statements: leaf_hash -> block_hash

* Restore `ensure_available_session_info`

* Don't interrupt `process_on_chain_votes` if SessionInfo can't be fetched

* Small style improvements in logging

* process_on_chain_votes: leaf_hash -> block_hash

* Restore `note_earliest_session` - it is required to prune disputes and votes

* Cache new sessions only when there is an actual session change

* Fix tests

* `CandidateEnvironment::new` gets `session_idx` and fetches SessionInfo by itself to avoid the invariant where the input SessionIndex and SessionInfo parameters don't match

* Fix handling of missing session info

* Move sessions caching in `handle_startup` and fix tests

* Load `relay_parent` from db in `handle_import_statements` instead of passing it as a parameter via two functions

* Don't do two db reads

* Fix the twisted logic in `handle_import_statements`

* fixup

* Small style fix

* Decrease log levels for caching errors to debug and fix a typo

* Update outdated comment

* Remove `ensure_available_session_info`

* Load relay parent from db in `process_on_chain_votes`

* Revert "Load relay parent from db in `process_on_chain_votes`"

This reverts commit 978ad4f223d517faa7a7fbad96e3f8de4fa17501.

* Keep track of highest seen session and last session cached without gaps.

* Apply suggestions from code review

Co-authored-by: default avatarordian <[email protected]>

* Handle session caching failure on startup correctly

* Update node/core/dispute-coordinator/src/initialized.rs

Co-authored-by: default avatarordian <[email protected]>

* Simplify session caching retries

* Update stale comment

* Fix lower bound calculation for session caching

---------

Co-authored-by: default avatarordian <[email protected]>
      3f4ce632
  15. Apr 21, 2023
    • ordian's avatar
      update rocksdb to 0.20.1 (#7113) · 439ab875
      ordian authored 
      * update rocksdb to 0.20.1

* update lockfile for {"substrate"}

---------

Co-authored-by: parity-processbot <>
      439ab875
    • Marcin S.'s avatar
      PVF: Move PVF workers into separate crate (#7101) · e277f95b
      Marcin S. authored 
      * Move PVF workers into separate crate

* Fix indentation

* Fix compilation errors

* Fix more compilation errors

* Rename `worker.rs` files, make host interface to worker more clear

* Fix more compilation errors

* Fix more compilation errors

* Add link to issue

* Address review comments

* Update comment
      e277f95b
  17. Apr 20, 2023
    • Marcin S.'s avatar
      PVF: Don't dispute on missing artifact (#7011) · 0940cdd1
      Marcin S. authored 
      * PVF: Don't dispute on missing artifact

A dispute should never be raised if the local cache doesn't provide a certain
artifact. You can not dispute based on this reason, as it is a local hardware
issue and not related to the candidate to check.

Design:

Currently we assume that if we prepared an artifact, it remains there on-disk
until we prune it, i.e. we never check again if it's still there.

We can change it so that instead of artifact-not-found triggering a dispute, we
retry once (like we do for AmbiguousWorkerDeath, except we don't dispute if it
still doesn't work). And when enqueuing an execute job, we check for the
artifact on-disk, and start preparation if not found.

Changes:

- [x] Integration test (should fail without the following changes)
- [x] Check if artifact exists when executing, prepare if not
- [x] Return an internal error when file is missing
- [x] Retry once on internal errors
- [x] Document design (update impl guide)

* Add some context to wasm error message (it is quite long)

* Fix impl guide

* Add check for missing/inaccessible file

* Add comment referencing Substrate issue

* Add test for retrying internal errors

---------

Co-authored-by: parity-processbot <>
      0940cdd1
  19. Apr 19, 2023
  21. Apr 18, 2023
  23. Apr 17, 2023
  25. Apr 13, 2023
  27. Apr 12, 2023
    • Sam Johnson's avatar
      Companion for #13846 (#7022) · b3070be0
      Sam Johnson authored 
      * globally upgrade syn to 1.0.109

* globally upgrade quote to 1.0.26

* globally upgrade proc-macro2 to 1.0.56

* globally bump syn to v2.0.13

* update expander to v1.0.0

* temporary commit to prove new version of expander works

(new version hasn't been released yet so using git)

* use expander 2.0.0

* upgrade to syn 2.0.14

* update lock file
      b3070be0
  29. Apr 11, 2023
    • Mira Ressel's avatar
      apply clippy 1.68 suggestions · 0e9b0a64
      Mira Ressel authored 
      * Simplify &(ref foo, _) tuple pattern matches
* Drop unneccessary .clone() calls
* Replace .position(cond).is_some() by .any(cond)
* Drop unneccessary lifetime specs
      0e9b0a64
  31. Apr 08, 2023
  33. Apr 07, 2023
    • Tsvetomir Dimitrov's avatar
      Onchain scraper in `dispute-coordinator` will scrape... · 63a805ba
      Tsvetomir Dimitrov authored 
      Onchain scraper in `dispute-coordinator` will scrape `SCRAPED_FINALIZED_BLOCKS_COUNT` blocks before finality (#7013)

* Onchain scraper in `dispute-coordinator` will scrape `SCRAPED_FINALIZED_BLOCKS_COUNT` blocks before finality

The purpose is to make the availability of a `CandidateReceipt` for finalized candidates more likely.

For details see:  https://github.com/paritytech/polkadot/issues/7009

* Fix off by one error

* Replace `SCRAPED_FINALIZED_BLOCKS_COUNT` with `DISPUTE_CANDIDATE_LIFETIME_AFTER_FINALIZATION`
      63a805ba
    • Marcin S.'s avatar
      PVF: Minor refactor in workers code (#7012) · 5889119f
      Marcin S. authored 
      * Move version check to `worker_event_loop`

* More minor refactors

- More consistent use of `format_invalid` and `format_internal`.
- Fix a doc error.
- Fix some poorly-named local variables.
      5889119f
  35. Apr 05, 2023
  37. Apr 04, 2023
  39. Apr 03, 2023
  41. Mar 31, 2023
    • Mattia L.V. Bradascio's avatar
      Consolidate subsystem spans so they are all children of the leaf-activated root span (#6458) · 713f6625
      Mattia L.V. Bradascio authored 
      * Pass the PerLeafSpan as mutable reference to handle_new_head function

* cargo +nightly fmt --all

* Add mock span for test

* cargo +nightly fmt --all

* add new-blocks-hashes to span

* ref span in match statement, set span to disabled if not passed

* remove second match clause, make handle_new_head_span mutable

* cargo +nightly fmt --all

* improve tag on error and warning

* add imported blocks and info span

* cargo +nightly fmt --all

* Improve error for imported_blocks_and_info trace

* format tags on get_header_span

* add lost-to-finality tag

* add missing bracket

* - Add bitfield child span
- Add block db insertion span

* - fix update-bitfield span tag

* - Fix type conversion to u64
- Add missing argument

* - Cargo fmt

* - Test add_follows_from

* - Revert as  relationship between spans not working correctly

* - use drop to test if parent-child relationship can be re-established

* - remove bitfield span, check if parent-child relationship can be reestablished

* - Remove dangling bitfield span which is not used, to see if parent-child relationship can be re-established

* Another dangling bitfield span

* cargo fmt

* - add imported blocks and info span
- add candidate span per candidate

* add tags before moving block_header to push scope

* - Add db-insertion span

* cargo fmt

* fix types

* * Pass mutable reference to span in handle_new_head
* Change get-header-span tags in handle_new_head
* Create cache-session-info span in handle_new_head
* Create optional argument in determine_new_blocks
* Pass mutable reference to handle_new_head_span in determine_new_blocks in handle_new_head function
* Add candidate-hash, candidate-number, lost-to-finality tags to candidate_span in handle_new_head function
* Manually drop db_insertion_span and remove superfluous tags  to it, only keeping approved-bitfields tag
* Add ApprovalVoting stage in jaeger

* * Pass mutable reference to jaeger::Span in stead of PerLeafSpan
* Add block-import span

* *Pass optional_span (optional argument) to determine_new_blocks util function

* * Add num-candidates int tag to block_import_span

* * Add head tag to cache_session_span

* * Create PerLeafSpan in handle_from_overseer (this is required to establish parent-child relationship between approval-voting span, and leaf-activated root span)

* * Add candidate-import-span as child of block-import-span
* Add candidate-hash and num-approval tags to candidate-import-span

* * Fix num-candidate tag to bitvec-len tag in candidate-import-span

* *Fix imported_blocKs_and_info span to create new-block-span as not dealing with candidates

* Consider the future::select! block

* Use HashMap<Hash, jaeger::PerLeafSpan>

* Remove Stage 9

* Add missing spans

* cargo +nightly fmt --all

* Remove optional span argument for determine_new_blocks

* * Remove no-longer needed default PerLeafSpan implementation
* Remove no-longer necessary mock span given re-factoring of handle_new_head() no longer neeing mutable span
* Split validation-result and request-data (availability and validation code) spans into two by dropping request_validation_data_spans
* Remove drop statements for cache_session_info_span
*

* Remove unnecessary span

* Remove another excessively spammy span

* Add missing spans from State in import tests

* Use functional approach to get spans

* - Add functional approach for the approval-voting span
- Add doc on block_numbers given labelling ambiguity
- Add span pruning logic
- Use .add_para_id on validation_result_span

* Replace for hash_set in hash_set_iter with map closure

* cargo +nightly fmt --all

* Change from unconsumed `map` to `.for_each`

* cargo +nightly fmt --all

* Refactor add_para_id to validation_result_span

* cargo +nightly fmt --all

* Remove duplicate tag

* Add missing tag to handle-approved-ancestor span

* Refactor span pruning to only invoke retain once

* Typo in span name

* - Replace unwrap_or with unwrap_or_else due to lazy evaluation of trace-identifier in polkadot_node_jaeger
- Remove some redundant spans

* Add approval-distribution spans

* - Add unwrap_or_else on note-approved-in-chain-selection
- Use child_with_trace_id to add traceID string tag on span (note this does not change the traceID, but just adds a tag)

* cargo +nightly fmt --all

* - Add traceID tags were necessary in approval-voting and availability-distribution
- Always use block-hash tag in stead of relay-parent tag in approval-distribution

* Remove schedule-wakeup span as it will duplicate spans on existing wakeups (which should be a no-op)

* Remove a couple of warnings related to mutability

* Fix failing tests in availability distribution

* Add traceID tag to launch-approval and validation-result

* Reshuffle the validation and validation result spans to where more appropriate and add block-hash tag

* - Add tranche and should-trigger tag to process-wakeup span
- Add candidate-hash and traceID to check-and-import-approval span

* cargo fmt

* - Adjustments after PR comments

* Move span pruning after other pruning logic

* Remove DerefMut - no longer needed

* Relabel request-chunk spans

* - Fix typo in span label
- Add docs for drops

* Add new approval-voting span pruning logic

* Undo removal of !

* cargo fmt
      713f6625
  43. Mar 30, 2023
  45. Mar 29, 2023
    • s0me0ne-unkn0wn's avatar
      Check spawned worker version vs node version before PVF preparation (#6861) · 55b4aceb
      s0me0ne-unkn0wn authored 
      

* Check spawned worker version vs node version before PVF preparation

* Address discussions

* Propagate errors and shutdown preparation and execution pipelines properly

* Add logs; Fix execution worker checks

* Revert "Propagate errors and shutdown preparation and execution pipelines properly"

This reverts commit b96cc3160ff58db5ff001d8ca0bfea9bd4bdd0f2.

* Don't try to shut down; report the condition and exit worker

* Get rid of `VersionMismatch` preparation error

* Merge master

* Add docs; Fix tests

* Update Cargo.lock

* Kill again, but only the main node process

* Move unsafe code to a common safe function

* Fix libc dependency error on MacOS

* pvf spawning: Add some logging, add a small integration test

* Minor fixes

* Restart CI

---------

Co-authored-by: default avatarMarcin S <[email protected]>
      55b4aceb
    • Oliver Tale-Yazdi's avatar
      Fix feature (#6966) · faa2d5b0
      Oliver Tale-Yazdi authored 
      

Signed-off-by: default avatarOliver Tale-Yazdi <[email protected]>
      faa2d5b0