1. Jan 11, 2023
  2. Jan 10, 2023
    • Marcin S.'s avatar
      Minor fixes (#6533) · 3cf18f11
      Marcin S. authored
      * Minor fixes
      
      * Fix compile errors
      3cf18f11
    • eskimor's avatar
      Fix cycle dispute-coordinator <-> dispute-distribution (#6489) · cc650fe5
      eskimor authored
      
      
      * First iteration of message sender.
      
      * dyn Fn variant (no cloning)
      
      * Full implementation + Clone, without allocs on `Send`
      
      * Further clarifications/cleanup.
      
      * MessageSender -> NestingSender
      
      * Doc update/clarification.
      
      * dispute-coordinator: Send disputes on startup.
      
      + Some fixes, cleanup.
      
      * Fix whitespace.
      
      * Dispute distribution fixes, cleanup.
      
      * Cargo.lock
      
      * Fix spaces.
      
      * More format fixes.
      
      What is cargo fmt doing actually?
      
      * More fmt fixes.
      
      * Fix nesting sender.
      
      * Fixes.
      
      * Whitespace
      
      * Enable logging.
      
      * Guide update.
      
      * Fmt fixes, typos.
      
      * Remove unused function.
      
      * Simplifications, doc fixes.
      
      * Update roadmap/implementers-guide/src/node/disputes/dispute-coordinator.md
      
      Co-authored-by: default avatarMarcin S. <[email protected]>
      
      * Fmt + doc example fix.
      
      Co-authored-by: default avatareskimor <[email protected]>
      Co-authored-by: default avatarMarcin S. <[email protected]>
      cc650fe5
    • Marcin S.'s avatar
      Replace async-std with tokio in PVF subsystem (#6419) · 44fd9566
      Marcin S. authored
      
      
      * Replace async-std with tokio in PVF subsystem
      
      * Rework workers to use `select!` instead of a mutex
      
      The improvement in code readability is more important than the thread overhead.
      
      * Remove unnecessary `fuse`
      
      * Add explanation for `expect()`
      
      * Update node/core/pvf/src/worker_common.rs
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      
      * Update node/core/pvf/src/worker_common.rs
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      
      * Address some review comments
      
      * Shutdown tokio runtime
      
      * Run cargo fmt
      
      * Add a small note about retries
      
      * Fix up merge
      
      * Rework `cpu_time_monitor_loop` to return when other thread finishes
      
      * Add error string to PrepareError::IoErr variant
      
      * Log when artifacts fail to prepare
      
      * Fix `cpu_time_monitor_loop`; fix test
      
      * Fix text
      
      * Fix a couple of potential minor data races.
      
      First data race was due to logging in the CPU monitor thread even if the
      job (other thread) finished. It can technically finish before or after the log.
      
      Maybe best would be to move this log to the `select!`s, where we are guaranteed
      to have chosen the timed-out branch, although there would be a bit of
      duplication.
      
      Also, it was possible for this thread to complete before we executed
      `finished_tx.send` in the other thread, which would trigger an error as the
      receiver has already been dropped. And right now, such a spurious error from
      `send` would be returned even if the job otherwise succeeded.
      
      * Update Cargo.lock
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      44fd9566
  3. Jan 09, 2023
  4. Jan 06, 2023
  5. Jan 05, 2023
  6. Jan 04, 2023
  7. Dec 30, 2022
    • Bradley Olson's avatar
      Update dispute participation on active leaves update (#6303) · 539a9a91
      Bradley Olson authored
      
      
      * Passed candidate events from scraper to participation
      
      * First draft PR 5875
      
      * Added support for timestamp in changes
      
      * Some necessary refactoring
      
      * Removed SessionIndex from unconfirmed_disputes key
      
      * Removed duplicate logic in import statements
      
      * Replaced queue_participation call with re-prio
      
      * Simplifying refactor. Backed were already handled
      
      * Removed unneeded spam slots logic
      
      * Implementers guide edits
      
      * Undid the spam slots refactor
      
      * Added comments and implementers guide edit
      
      * Added test for participation upon backing
      
      * Round of fixes + ran fmt
      
      * Round of changes + fmt
      
      * Error handling draft
      
      * Changed errors to bubble up from reprioritization
      
      * Starting to construct new test
      
      * Clarifying participation function rename
      
      * Reprio test draft
      
      * Very rough bump to priority queue test draft
      
      * Improving logging
      
      * Most concise reproduction of error on third import
      
      * Add `handle_approval_vote_request`
      
      * Removing reprioritization on included event test
      
      * Removing unneeded test config
      
      * cargo fmt
      
      * Test works
      
      * Fixing final nits
      
      * Tweaks to test Tsveto figured out
      
      Co-authored-by: default avatareskimor <[email protected]>
      Co-authored-by: default avatarTsvetomir Dimitrov <[email protected]>
      539a9a91
  8. Dec 26, 2022
  9. Dec 21, 2022
  10. Dec 20, 2022
    • Marcin S.'s avatar
      PVF preparation: do not conflate errors (#6384) · e0a0475a
      Marcin S. authored
      * PVF preparation: do not conflate errors
      
      + Adds some more granularity to the prepare errors.
      + Better distinguish whether errors occur on the host side or the worker.
      + Do not kill the worker if the error happened on the host side.
      + Do not retry preparation if the error was `Panic`.
      + Removes unnecessary indirection with `Selected` type.
      
      * Add missing docs, resolve TODOs
      
      * Address review comments and remove TODOs
      
      * Fix error in CI
      
      * Undo unnecessary change
      
      * Update couple of comments
      
      * Don't return error for stream shutdown
      
      * Update node/core/pvf/src/worker_common.rs
      e0a0475a
    • Michal Kucharczyk's avatar
      BlockId removal: refactor: HeaderBackend::header (#6418) · fcc26d42
      Michal Kucharczyk authored
      * BlockId removal: refactor: HeaderBackend::header
      
      It changes the arguments of:
      - `HeaderBackend::header`,
      - `Client::header`
      
      methods from: `BlockId<Block>` to: `Block::Hash`
      
      This PR is part of BlockId::Number refactoring analysis (paritytech/substrate#11292)
      
      * missed fixes
      
      * BlockId removal: refactor: HeaderBackend::expect_header
      
      It changes the arguments of `HeaderBackend::expect_header` method from: `BlockId<Block>` to: `Block::Hash`
      
      * update lockfile for {"substrate"}
      
      * misspell fixed
      
      Co-authored-by: parity-processbot <>
      fcc26d42
  11. Dec 15, 2022
  12. Dec 07, 2022
  13. Dec 06, 2022
    • Marcin S.'s avatar
      Let the PVF host kill the worker on timeout (#6381) · ab090ab7
      Marcin S. authored
      * Let the PVF host kill the worker on timeout
      
      * Fix comment
      
      * Fix inaccurate comments; add missing return statement
      
      * Fix a comment
      
      * Fix comment
      ab090ab7
    • Marcin S.'s avatar
      Companion for paritytech/substrate#12795 (#6374) · 3f951cce
      Marcin S. authored
      
      
      * Begin removing `parity-util-mem`; remove `collect_memory_stats`
      
      * Update some dependencies that were using `parity-util-mem`
      
      * Remove `trie-memory-tracker` feature
      
      * Update Cargo.lock
      
      * Update `kvdb-shared-tests`
      
      * Add back jemalloc
      
      * Add missing license header
      
      * update lockfile for {"substrate"}
      
      Co-authored-by: parity-processbot <>
      Co-authored-by: default avatarAndronik <[email protected]>
      3f951cce
  14. Dec 05, 2022
  15. Dec 02, 2022
    • alexgparity's avatar
      Reduce provisioner work (#6328) · a7eee7dd
      alexgparity authored
      
      
      * Store values needed to create inherent data when needed instead of creating them early on
      
      * Point deps to substrate branch
      
      * Arc the client
      
      * Cargo update
      
      * Fix main cargo files
      
      * Undo cargo file changes
      
      * Add overseer dep to inherents
      
      * Update deps
      
      * Simplify code
      
      * Update benchmark
      
      * Update node/client/src/benchmarking.rs
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      
      * Update node/core/parachains-inherent/src/lib.rs
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      
      * Update node/core/parachains-inherent/src/lib.rs
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      
      * Revert "Update node/core/parachains-inherent/src/lib.rs"
      
      This reverts commit 8b9555dc2451acfabab173d259e00da2728b7aa2.
      
      * Revert "Update node/core/parachains-inherent/src/lib.rs"
      
      This reverts commit 816c92d0e001e71f677d0acbcf22bdc3f511bc56.
      
      * cargo update -p sp-io
      
      * fmt
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      a7eee7dd
  16. Nov 30, 2022
    • Marcin S.'s avatar
      Use CPU clock timeout for PVF jobs (#6282) · 28a4e909
      Marcin S. authored
      * Put in skeleton logic for CPU-time-preparation
      
      Still needed:
      - Flesh out logic
      - Refactor some spots
      - Tests
      
      * Continue filling in logic for prepare worker CPU time changes
      
      * Fix compiler errors
      
      * Update lenience factor
      
      * Fix some clippy lints for PVF module
      
      * Fix compilation errors
      
      * Address some review comments
      
      * Add logging
      
      * Add another log
      
      * Address some review comments; change Mutex to AtomicBool
      
      * Refactor handling response bytes
      
      * Add CPU clock timeout logic for execute jobs
      
      * Properly handle AtomicBool flag
      
      * Use `Ordering::Relaxed`
      
      * Refactor thread coordination logic
      
      * Fix bug
      
      * Add some timing information to execute tests
      
      * Add section about the mitigation to the IG
      
      * minor: Change more `Ordering`s to `Relaxed`
      
      * candidate-validation: Fix build errors
      28a4e909
    • alexgparity's avatar
      Clippyfy (#6341) · 9ea14e66
      alexgparity authored
      
      
      * Add clippy config and remove .cargo from gitignore
      
      * first fixes
      
      * Clippyfied
      
      * Add clippy CI job
      
      * comment out rusty-cachier
      
      * minor
      
      * fix ci
      
      * remove DAG from check-dependent-project
      
      * add DAG to clippy
      
      Co-authored-by: default avataralvicsam <[email protected]>
      9ea14e66
    • Mara Broda's avatar
      sync versions with current release (0.9.33) (#6363) · b76086c6
      Mara Broda authored
      * westend: update transaction version
      
      * polkadot: update transaction version
      
      * kusama: update transaction version
      
      * Bump spec_version to 9330
      
      * bump versions to 0.9.33
      b76086c6
  17. Nov 29, 2022
  18. Nov 28, 2022
  19. Nov 23, 2022
    • Marcin S.'s avatar
      Add PVF module documentation (#6293) · 1dec2433
      Marcin S. authored
      
      
      * Add PVF module documentation
      
      TODO (once the PRs land):
      
      - [ ] Document executor parametrization.
      
      - [ ] Document CPU time measurement of timeouts.
      
      * Update node/core/pvf/src/lib.rs
      
      Co-authored-by: default avatarAndrei Sandu <[email protected]>
      
      * Clarify meaning of PVF acronym
      
      * Move PVF doc to implementer's guide
      
      * Clean up implementer's guide a bit
      
      * Add page for PVF types
      
      * pvf: Better separation between crate docs and implementer's guide
      
      * ci: Add "prevalidating" to the dictionary
      
      * ig: Remove types/chain.md
      
      The types contained therein did not exist and the file was not referenced
      anywhere.
      
      Co-authored-by: default avatarAndrei Sandu <[email protected]>
      1dec2433
  20. Nov 17, 2022
    • Tsvetomir Dimitrov's avatar
      Change best effort queue behaviour in `dispute-coordinator` (#6275) · ccad411e
      Tsvetomir Dimitrov authored
      * Change best effort queue behaviour in `dispute-coordinator`
      
      Use the same type of queue (`BTreeMap<CandidateComparator,
      ParticipationRequest>`) for best effort and priority in
      `dispute-coordinator`.
      
      Rework `CandidateComparator` to handle unavailable parent
      block numbers.
      
      Best effort queue will order disputes the same way as priority does - by
      parent's block height. Disputes on candidates for which the parent's
      block number can't be obtained will be treated with the lowest priority.
      
      * Fix tests: Handle `ChainApiMessage::BlockNumber` in `handle_sync_queries`
      
      * Some tests are deadlocking on sending messages via overseer so change `SingleItemSink`to `mpsc::Sender` with a buffer of 1
      
      * Fix a race in test after adding a buffered queue for overseer messages
      
      * Fix the rest of the tests
      
      * Guide update - best-effort queue
      
      * Guide update: clarification about spam votes
      
      * Fix tests in `availability-distribution`
      
      * Update comments
      
      * Add `make_buffered_subsystem_context` in `subsystem-test-helpers`
      
      * Code review feedback
      
      * Code review feedback
      
      * Code review feedback
      
      * Don't add best effort candidate if it is already in priority queue
      
      * Remove an old comment
      
      * Fix insert in best_effort
      ccad411e
  21. Nov 16, 2022
    • Tsvetomir Dimitrov's avatar
      Provisioner should ignore unconfirmed disputes (#6294) · a0f4287d
      Tsvetomir Dimitrov authored
      * Fix typos
      
      * Filter unconfirmed disputes in provisioner -  random_selection
      
      * Rework dispute coordinator to return `DisputeStatus` with
      `ActiveDisputes` message.
      * Rework the random_selection implementation of `select_disptues` in
        `provisioner` to return only confirmed disputes.
      
      * Filter unconfirmed disputes in provisioner - prioritized_selection
      
      * Add test for unconfirmed disputes handling
      
      * Fix `dispute-distribution` tests
      a0f4287d
  22. Nov 15, 2022
    • Marcin S.'s avatar
      Fixes "for loop over an `Option`" warnings (#6291) · d53513ff
      Marcin S. authored
      Was seeing these warnings when running `cargo check --all`:
      
      ```
      warning: for loop over an `Option`. This is more readably written as an `if let` statement
          --> node/core/approval-voting/src/lib.rs:1147:21
           |
      1147 |             for activated in update.activated {
           |                              ^^^^^^^^^^^^^^^^
           |
           = note: `#[warn(for_loops_over_fallibles)]` on by default
      help: to check pattern in a loop use `while let`
           |
      1147 |             while let Some(activated) = update.activated {
           |             ~~~~~~~~~~~~~~~         ~~~
      help: consider using `if let` to clear intent
           |
      1147 |             if let Some(activated) = update.activated {
           |             ~~~~~~~~~~~~         ~~~
      ```
      
      My guess is that `activated` used to be a SmallVec or similar, as is
      `deactivated`. It was changed to an `Option`, the `for` still compiled (it's
      technically correct, just weird), and the compiler didn't catch it until now.
      d53513ff
  23. Nov 14, 2022
  24. Nov 12, 2022
  25. Nov 11, 2022
    • Tsvetomir Dimitrov's avatar
      Update disputes prioritisation in `dispute-coordinator` (#6130) · ff09b18d
      Tsvetomir Dimitrov authored
      
      
      * Scraper processes CandidateBacked events
      
      * Change definition of best-effort
      
      * Fix `dispute-coordinator` tests
      
      * Unit test for dispute filtering
      
      * Clarification comment
      
      * Add tests
      
      * Fix logic
      
      If a dispute is not backed, not included and not confirmed we
      don't participate but we do import votes.
      
      * Add metrics for refrained participations
      
      * Revert "Add tests"
      
      This reverts commit 7b8391a087922ced942cde9cd2b50ff3f633efc0.
      
      * Revert "Unit test for dispute filtering"
      
      This reverts commit 92ba5fe678214ab360306313a33c781338e600a0.
      
      * fix dispute-coordinator tests
      
      * Fix scraping
      
      * new tests
      
      * Small fixes in guide
      
      * Apply suggestions from code review
      
      Co-authored-by: default avatarAndrei Sandu <[email protected]>
      
      * Fix some comments and remove a pointless test
      
      * Code review feedback
      
      * Clarification comment in tests
      
      * Some tests
      
      * Reference counted `CandidateHash` in scraper
      
      * Proper handling for Backed and Included candidates in scraper
      
      Backed candidates which are not included should be kept for a
      predetermined window of finalized blocks. E.g. if a candidate is backed
      but not included in block 2, and the window size is 2, the same
      candidate should be cleaned after block 4 is finalized.
      
      Add reference counting for candidates in scraper. A candidate can be
      added on multiple block heights so we have to make sure we don't clean
      it prematurely from the scraper.
      
      Add tests.
      
      * Update comments in tests
      
      * Guide update
      
      * Fix cleanup logic for `backed_candidates_by_block_number`
      
      * Simplify cleanup
      
      * Make spellcheck happy
      
      * Update tests
      
      * Extract candidate backing logic in separate struct
      
      * Code review feedback
      
      * Treat  backed and included candidates in the same fashion
      
      * Update some comments
      
      * Small improvements in test
      
      * spell check
      
      * Fix some more comments
      
      * clean -> prune
      
      * Code review feedback
      
      * Reword comment
      
      * spelling
      
      Co-authored-by: default avatarAndrei Sandu <[email protected]>
      ff09b18d
  26. Nov 08, 2022
  27. Nov 07, 2022