- Apr 08, 2023
-
-
s0me0ne-unkn0wn authored
* Happy New Year! * Remove year entierly Co-authored-by: Oliver Tale-Yazdi <[email protected]> * Remove years from copyright notice in the entire repo --------- Co-authored-by: Oliver Tale-Yazdi <[email protected]>
-
- Apr 07, 2023
-
-
Tsvetomir Dimitrov authored
Onchain scraper in `dispute-coordinator` will scrape `SCRAPED_FINALIZED_BLOCKS_COUNT` blocks before finality (#7013) * Onchain scraper in `dispute-coordinator` will scrape `SCRAPED_FINALIZED_BLOCKS_COUNT` blocks before finality The purpose is to make the availability of a `CandidateReceipt` for finalized candidates more likely. For details see: https://github.com/paritytech/polkadot/issues/7009 * Fix off by one error * Replace `SCRAPED_FINALIZED_BLOCKS_COUNT` with `DISPUTE_CANDIDATE_LIFETIME_AFTER_FINALIZATION`
-
Marcin S. authored
* Move version check to `worker_event_loop` * More minor refactors - More consistent use of `format_invalid` and `format_internal`. - Fix a doc error. - Fix some poorly-named local variables.
-
- Apr 05, 2023
-
-
s0me0ne-unkn0wn authored
-
- Apr 04, 2023
-
-
s0me0ne-unkn0wn authored
* Companion for paritytech/substrate#13811 * Add comment * update lockfile for {"substrate"} * Update Substrate Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Fix pallet weight warnings Signed-off-by: Oliver Tale-Yazdi <[email protected]> --------- Signed-off-by: Oliver Tale-Yazdi <[email protected]> Co-authored-by: parity-processbot <> Co-authored-by: Oliver Tale-Yazdi <[email protected]>
-
Bastian Köcher authored
* Substrate companion: Remove deprecated batch verification * update lockfile for {"substrate"} --------- Co-authored-by: parity-processbot <>
-
- Mar 31, 2023
-
-
Mattia L.V. Bradascio authored
* Pass the PerLeafSpan as mutable reference to handle_new_head function * cargo +nightly fmt --all * Add mock span for test * cargo +nightly fmt --all * add new-blocks-hashes to span * ref span in match statement, set span to disabled if not passed * remove second match clause, make handle_new_head_span mutable * cargo +nightly fmt --all * improve tag on error and warning * add imported blocks and info span * cargo +nightly fmt --all * Improve error for imported_blocks_and_info trace * format tags on get_header_span * add lost-to-finality tag * add missing bracket * - Add bitfield child span - Add block db insertion span * - fix update-bitfield span tag * - Fix type conversion to u64 - Add missing argument * - Cargo fmt * - Test add_follows_from * - Revert as relationship between spans not working correctly * - use drop to test if parent-child relationship can be re-established * - remove bitfield span, check if parent-child relationship can be reestablished * - Remove dangling bitfield span which is not used, to see if parent-child relationship can be re-established * Another dangling bitfield span * cargo fmt * - add imported blocks and info span - add candidate span per candidate * add tags before moving block_header to push scope * - Add db-insertion span * cargo fmt * fix types * * Pass mutable reference to span in handle_new_head * Change get-header-span tags in handle_new_head * Create cache-session-info span in handle_new_head * Create optional argument in determine_new_blocks * Pass mutable reference to handle_new_head_span in determine_new_blocks in handle_new_head function * Add candidate-hash, candidate-number, lost-to-finality tags to candidate_span in handle_new_head function * Manually drop db_insertion_span and remove superfluous tags to it, only keeping approved-bitfields tag * Add ApprovalVoting stage in jaeger * * Pass mutable reference to jaeger::Span in stead of PerLeafSpan * Add block-import span * *Pass optional_span (optional argument) to determine_new_blocks util function * * Add num-candidates int tag to block_import_span * * Add head tag to cache_session_span * * Create PerLeafSpan in handle_from_overseer (this is required to establish parent-child relationship between approval-voting span, and leaf-activated root span) * * Add candidate-import-span as child of block-import-span * Add candidate-hash and num-approval tags to candidate-import-span * * Fix num-candidate tag to bitvec-len tag in candidate-import-span * *Fix imported_blocKs_and_info span to create new-block-span as not dealing with candidates * Consider the future::select! block * Use HashMap<Hash, jaeger::PerLeafSpan> * Remove Stage 9 * Add missing spans * cargo +nightly fmt --all * Remove optional span argument for determine_new_blocks * * Remove no-longer needed default PerLeafSpan implementation * Remove no-longer necessary mock span given re-factoring of handle_new_head() no longer neeing mutable span * Split validation-result and request-data (availability and validation code) spans into two by dropping request_validation_data_spans * Remove drop statements for cache_session_info_span * * Remove unnecessary span * Remove another excessively spammy span * Add missing spans from State in import tests * Use functional approach to get spans * - Add functional approach for the approval-voting span - Add doc on block_numbers given labelling ambiguity - Add span pruning logic - Use .add_para_id on validation_result_span * Replace for hash_set in hash_set_iter with map closure * cargo +nightly fmt --all * Change from unconsumed `map` to `.for_each` * cargo +nightly fmt --all * Refactor add_para_id to validation_result_span * cargo +nightly fmt --all * Remove duplicate tag * Add missing tag to handle-approved-ancestor span * Refactor span pruning to only invoke retain once * Typo in span name * - Replace unwrap_or with unwrap_or_else due to lazy evaluation of trace-identifier in polkadot_node_jaeger - Remove some redundant spans * Add approval-distribution spans * - Add unwrap_or_else on note-approved-in-chain-selection - Use child_with_trace_id to add traceID string tag on span (note this does not change the traceID, but just adds a tag) * cargo +nightly fmt --all * - Add traceID tags were necessary in approval-voting and availability-distribution - Always use block-hash tag in stead of relay-parent tag in approval-distribution * Remove schedule-wakeup span as it will duplicate spans on existing wakeups (which should be a no-op) * Remove a couple of warnings related to mutability * Fix failing tests in availability distribution * Add traceID tag to launch-approval and validation-result * Reshuffle the validation and validation result spans to where more appropriate and add block-hash tag * - Add tranche and should-trigger tag to process-wakeup span - Add candidate-hash and traceID to check-and-import-approval span * cargo fmt * - Adjustments after PR comments * Move span pruning after other pruning logic * Remove DerefMut - no longer needed * Relabel request-chunk spans * - Fix typo in span label - Add docs for drops * Add new approval-voting span pruning logic * Undo removal of ! * cargo fmt
-
- Mar 30, 2023
-
-
Bradley Olson authored
* First pass adding logs * fmt * Adjustments * Get rid of extra timers for running participations * fmt * Handling timer discards more elegantly
-
s0me0ne-unkn0wn authored
-
- Mar 29, 2023
-
-
s0me0ne-unkn0wn authored
* Check spawned worker version vs node version before PVF preparation * Address discussions * Propagate errors and shutdown preparation and execution pipelines properly * Add logs; Fix execution worker checks * Revert "Propagate errors and shutdown preparation and execution pipelines properly" This reverts commit b96cc3160ff58db5ff001d8ca0bfea9bd4bdd0f2. * Don't try to shut down; report the condition and exit worker * Get rid of `VersionMismatch` preparation error * Merge master * Add docs; Fix tests * Update Cargo.lock * Kill again, but only the main node process * Move unsafe code to a common safe function * Fix libc dependency error on MacOS * pvf spawning: Add some logging, add a small integration test * Minor fixes * Restart CI --------- Co-authored-by: Marcin S <[email protected]>
-
- Mar 24, 2023
-
-
Davide Galassi authored
* Companion for #13683 * Wraps trait is not required * update lockfile for {"substrate"} --------- Co-authored-by: parity-processbot <>
-
- Mar 23, 2023
-
-
Tsvetomir Dimitrov authored
-
Bastian Köcher authored
This ensures that both subsystems exit when the Overseer has exited because of some error.
-
- Mar 21, 2023
-
-
Bradley Olson authored
* Added participation and queue sizes metrics * First draft of all metric code * Tests pass * Changed Metrics to field on participation + queues * fmt * Improving naming * Refactor, placing timer in ParticipationRequest * fmt * Final cleanup * Revert "Final cleanup" This reverts commit 02e5608df64b2e0f7810905e4508673b2037d351. * Changing metric names * Implementing Eq only for unit tests * fmt * Removing Clone trait from ParticipationRequest * fmt * Moved clone functionality to tests helper * fmt * Fixing dropped timers on repeat requests * Keep older best effort timers * Removing comment redundency and explaining better * Updating queue() to use single mem read * fmt
-
- Mar 20, 2023
-
-
Mira Ressel authored
-
Vsevolod Stakhov authored
This reverts commit 5565bb9d.
-
- Mar 17, 2023
-
-
Davide Galassi authored
* Remove not required async calls * Fixed missing renaming * make_keystore can be sync * More fixes * Trivial nitpicks * Cherry pick test fix from master * Fixes after master merge * update lockfile for {"substrate"} --------- Co-authored-by: parity-processbot <>
-
Davide Galassi authored
-
- Mar 16, 2023
-
-
Bradley Olson authored
* First pass adding logs * fmt * Adjustments
-
Bradley Olson authored
* Added participation and queue sizes metrics * First draft of all metric code * Tests pass * Changed Metrics to field on participation + queues * fmt * Improving naming * Refactor, placing timer in ParticipationRequest * fmt * Final cleanup * Revert "Final cleanup" This reverts commit 02e5608df64b2e0f7810905e4508673b2037d351. * Changing metric names * Implementing Eq only for unit tests * fmt * Removing Clone trait from ParticipationRequest * fmt * Moved clone functionality to tests helper
-
- Mar 15, 2023
-
-
s0me0ne-unkn0wn authored
-
Marcin S. authored
* PVF: Document that preparation cannot lead to disputes * Add warning for deterministic errors * Fix warning
-
- Mar 14, 2023
-
-
Vsevolod Stakhov authored
* Test the recent orchestra changes * Update orchestra and metered-channel
-
- Mar 11, 2023
-
-
Bradley Olson authored
* Added participation and queue sizes metrics * First draft of all metric code * Tests pass * Changed Metrics to field on participation + queues * fmt * Improving naming * Refactor, placing timer in ParticipationRequest * fmt * Final cleanup * Revert "Final cleanup" This reverts commit 02e5608df64b2e0f7810905e4508673b2037d351. * Changing metric names * Implementing Eq only for unit tests * fmt
-
- Mar 08, 2023
-
-
s0me0ne-unkn0wn authored
* Move PVF timeouts to executor environment parameters * Typo Co-authored-by: Marcin S. <[email protected]> * Fix comments * Change handle_import_statements to FatalResult (#6820) * Changing dispute db errors to fatal * fmt * Change node-key for bootnodes (#6772) * Additional tracing in `provisioner`, `vote_selection` and `dispute-coordinator` (#6775) * Additional tracing in `provisioner`, `vote_selection` * Add `fetched_onchain_disputes` metric to provisioner * Some tracelines in dispute-coordinator TODO: cherry pick this in the initial branch!!! * Remove spammy logs * Remove some trace lines * Rename and fix things * Fix comments * Typo * Minor fixes * Add codec indexes; Remove macro --------- Co-authored-by: Marcin S. <[email protected]> Co-authored-by: Bradley Olson <[email protected]> Co-authored-by: Petr Mensik <[email protected]> Co-authored-by: Tsvetomir Dimitrov <[email protected]>
-
- Mar 06, 2023
-
-
Tsvetomir Dimitrov authored
* Additional tracing in `provisioner`, `vote_selection` * Add `fetched_onchain_disputes` metric to provisioner * Some tracelines in dispute-coordinator TODO: cherry pick this in the initial branch!!! * Remove spammy logs * Remove some trace lines
-
- Mar 04, 2023
-
-
Bradley Olson authored
* Changing dispute db errors to fatal * fmt
-
- Mar 02, 2023
-
-
eskimor authored
Co-authored-by: eskimor <[email protected]>
-
- Feb 24, 2023
-
-
Bradley Olson authored
* Initial pass * fmt * Lock changes * Implementers guide edit * Reasoning changes * typo Co-authored-by: Marcin S. <[email protected]> * Revert "Lock changes" This reverts commit 3131a5245c6349743789270e6d1ad7842dc4c9df. * Getting rid of cargo.lock changes * Argumentation update * fmt --------- Co-authored-by: Marcin S. <[email protected]>
-
Bastian Köcher authored
* Companion for Substrate#12520 https://github.com/paritytech/substrate/pull/12520 * Fix * update lockfile for {"substrate"} --------- Co-authored-by: parity-processbot <>
-
- Feb 20, 2023
-
-
s0me0ne-unkn0wn authored
-
s0me0ne-unkn0wn authored
-
- Feb 16, 2023
-
-
Tsvetomir Dimitrov authored
-
Bastian Köcher authored
* Change `MaxMemorySize` to `MaxMemoryPages` We should set the max memory for the executor in pages (64KiB) and not in bytes. The wasm memory is always a multiple of a page and we should use the same terminology. * FMT * Fix warning
-
Marcin S. authored
* Use a `BoundedVec` in `ValidationResult` > Use a `BoundedVec` for `upward_messages` and `horizontal_messages` in order to > limit the number of individual messages/memory allocations right at decoding > time. The reason for this is that the `ValidationResult` may contain a code > upgrade (including a full PVF binary), so the total size limit can't be set > too low and this limit will still allow several millions of upward messages, > which will (due to the memory allocator overhead) already have a > non-negligible memory footprint in decoded form. * List all fields when hashing so we don't miss one * Define types for `BoundedVec`s of messages * Fix test compile errors * Depend on `bounded-collections` 0.1.4 (fixes allocation issue) * Fix compilation issue * Derive `Hash` instead of manual `impl` * Avoid use of unwrap
-
- Feb 15, 2023
-
-
s0me0ne-unkn0wn authored
* Re-apply changes without Diener, rebase to the lastest master * Cache pruning * Bit-pack InstantiationStrategy * Move ExecutorParams version inside the structure itself * Rework runtime API and executor parameters storage * Pass executor parameters through backing subsystem * Update Cargo.lock * Introduce `ExecutorParams` to approval voting subsys * Introduce `ExecutorParams` to dispute coordinator * `cargo fmt` * Simplify requests from backing subsys * Fix tests * Replace manual config cloning with `.clone()` * Move constants to module * Parametrize executor performing PVF pre-check * Fix Malus * Fix test runtime * Introduce session executor params as a constant defined by session info pallet * Use Parity SCALE codec instead of hand-crafted binary encoding * Get rid of constants; Add docs * Get rid of constants * Minor typo * Fix Malus after rebase * `cargo fmt` * Use transparent SCALE encoding instead of explicit * Clean up * Get rid of relay parent to session index mapping * Join environment type and version in a single enum element * Use default execution parameters if running an old runtime * `unwrap()` -> `expect()` * Correct API version * Constants are back in town * Use constants for execution environment types * Artifact separation, first try * Get rid of explicit version * PVF execution queue worker separation * Worker handshake * Global renaming * Minor fixes resolving discussions * Two-stage requesting of executor params to make use of runtime API cache * Proper error handling in pvf-checker * Executor params storage bootstrapping * Propagate migration to v3 network runtimes * Fix storage versioning * Ensure `ExecutorParams` serialization determinism; Add comments * Rename constants to make things a bit more deterministic Get rid of stale code * Tidy up a structure of active PVFs * Minor formatting * Fix comment * Add try-runtime hooks * Add storage version write on upgrade Co-authored-by: Andronik <[email protected]> * Add pre- and post-upgrade assertions * Require to specify environment type; Remove redundant `impl`s * Add `ExecutorParamHash` creation from `H256` * Fix candidate validation subsys tests * Return splittable error from executor params request fn * Revert "Return splittable error from executor params request fn" This reverts commit a0b274177d8bb2f6e13c066741892ecd2e72a456. * Decompose approval voting metrics * Use more relevant errors * Minor formatting fix * Assert a valid environment type instead of checking * Fix `try-runtime` hooks * After-merge fixes * Add migration logs * Remove dead code * Fix tests * Fix tests * Back to the strongly typed implementation * Promote strong types to executor interface * Remove stale comment * Move executor params to `SessionInfo`: primitives and runtime * Move executor params to `SessionInfo`: node * Try to bump primitives and API version * Get rid of `MallocSizeOf` * Bump target API version to v4 * Make use of session index already in place * Back to v3 * Fix all the tests * Add migrations to all the runtimes * Make use of existing `SessionInfo` in approval voting subsys * Rename `TARGET` -> `LOG_TARGET` * Bump all the primitives to v3 * Fix Rococo ParachainHost API version * Use `RollingSessionWindow` to acquire `ExecutorParams` in disputes * Fix nits from discussions; add comments * Re-evaluate queue logic * Rework job assignment in execution queue * Add documentation * Use `RuntimeInfo` to obtain `SessionInfo` (with blackjack and caching) * Couple `Pvf` with `ExecutorParams` wherever possible * Put members of `PvfWithExecutorParams` under `Arc` for cheap cloning * Fix comment * Fix CI tests * Fix clippy warnings * Address nits from discussions * Add a placeholder for raw data * Fix non exhaustive match * Remove redundant reexports and fix imports * Keep only necessary semantic features, as discussed * Rework `RuntimeInfo` to support mock implementation for tests * Remove unneeded bound * `cargo fmt` * Revert "Remove unneeded bound" This reverts commit 932463f26b00ce290e1e61848eb9328632ef8a61. * Fix PVF host tests * Fix PVF checker tests * Fix overseer declarations * Simplify tests * `MAX_KEEP_WAITING` timeout based on `BACKGING_EXECUTION_TIMEOUT` * Add a unit test for varying executor parameters * Minor fixes from discussions * Add prechecking max. memory parameter (see paritytech/srlabs_findings#110) * Fix and improve a test * Remove `ExecutionEnvironment` and `RawData` * New primitives versioning in parachain host API * `disputes()` implementation for Kusama and Polkadot * Move `ExecutorParams` from `vstaging` to stable primitives * Move disputes from `vstaging` to stable implementation * Fix `try-runtime` * Fixes after merge * Move `ExecutorParams` to the bottom of `SessionInfo` * Revert "Move executor params to `SessionInfo`: primitives and runtime" This reverts commit dfcfb85fefd1c5be6c8a8f72dc09fd1809cfa9ce. * Always use fresh activated live hash in pvf precheck (re-apply 34b09a4c20de17e7926ed942cd0d657d18f743fa) * Fixing tests (broken commit) * Fix candidate validation tests * Fix PVF host test * Minor fixes * Address discussions * Restore migration * Fix `use` to only include what is needed instead of `*` * Add comment to never touch `DEFAULT_CONFIG` * Update migration to set default `ExecutorParams` for `dispute_period` sessions back * Use `earliest_stored_session` instead of calculations * Nit * Add logs * Treat any runtime error as `NotSupported` again * Always return default executor params if not available * Revert "Always return default executor params if not available" This reverts commit b58ac4482ef444c67a9852d5776550d08e312f30. * Add paritytech/substrate#9997 workaround * `cargo fmt` * Remove migration (again!) * Bump executor params to API v4 (backport from #6698) --------- Co-authored-by: Andronik <[email protected]>
-
- Feb 14, 2023
-
-
Marcin S. authored
* Refactor PVF preparation memory stats The original purpose of this change was to gate metrics that are unsupported by some systems behind conditional compilation directives (#[cfg]); see https://github.com/paritytech/polkadot/pull/6675#discussion_r1099996209. Then I started doing some random cleanups and simplifications and got a bit carried away.
🙈 The code should be overall tidier than before. Changes: - Don't register unsupported metrics (e.g. `max_rss` on non-Linux systems) - Introduce `PrepareStats` struct as an abstraction over the `Ok` values of `PrepareResult`. It is cleaner, and can be easily modified in the future. - Other small changes * Minor fixes to comments * Fix compile errors * Try to fix some Linux errors * Mep * Fix candidate-validation tests * Update docstring
-
- Feb 09, 2023
-
-
ordian authored
-
Anthony Alaribe authored
* Introduce jemalloc-stats feature flag * remove unneeded space * Update node/overseer/src/lib.rs Co-authored-by: Marcin S. <[email protected]> * Update Cargo.toml Co-authored-by: Marcin S. <[email protected]> * revert making tikv-jemallocator depend on jemalloc-stats * conditionally import memory_stats instead of using dead_code * fix test via expllicit import * Add jemalloc-stats feature to crates, propagate it from root * Apply `jemalloc-stats` feature to prepare mem stats; small refactor * effect changes recommended on PR * Update node/overseer/src/metrics.rs Co-authored-by: Marcin S. <[email protected]> * fix compile error on in pipeline for linux. missing import * Update node/overseer/src/lib.rs Co-authored-by: Bastian Köcher <[email protected]> * revert to defining collect_memory_stats inline --------- Co-authored-by: Marcin S. <[email protected]> Co-authored-by: Marcin S <[email protected]> Co-authored-by: Bastian Köcher <[email protected]>
-
- Feb 06, 2023
-
-
Marcin S. authored
* Add getrusage and memory tracker for precheck preparation * Log memory stats metrics after prechecking * Fix tests * Try to fix errors (linux-only so I'm relying on CI here) * Try to fix CI * Add module docs for `prepare/memory_stats.rs`; fix CI error * Report memory stats for all preparation jobs * Use `RUSAGE_SELF` instead of `RUSAGE_THREAD` Not sure why I did that -- was a brainfart on my end. * Revert last commit (RUSAGE_THREAD is correct) * Use exponential buckets * Use `RUSAGE_SELF` for `getrusage`; enable `max_rss` metric for MacOS * Increase poll interval * Revert "Use `RUSAGE_SELF` for `getrusage`; enable `max_rss` metric for MacOS" This reverts commit becf7a815409ab530fc61370abffcd1b97b9a777.
-