- Apr 25, 2024
-
-
s0me0ne-unkn0wn authored
Currently, PVFs are re-prepared if any execution environment parameter changes. As we've recently seen on Kusama and Polkadot, that may lead to a severe finality lag because every validator has to re-prepare every PVF. That cannot be avoided altogether; however, we could cease re-preparing PVFs when a change in the execution environment can't lead to a change in the artifact itself. For example, it's clear that changing the execution timeout cannot affect the artifact. In this PR, I'm introducing a separate hash for the subset of execution environment parameters that changes only if a preparation-related parameter changes. It introduces some minor code duplication, but without that, the scope of changes would be much bigger. TODO: - [x] Add a test to ensure the artifact is not re-prepared if non-preparation-related parameter is changed - [x] Add a test to ensure the artifact is re-prepared if a preparation-related parameter is changed - [x] Add comments, warnings, and, possibly, a test to ensure a new parameter ever added to the executor environment parameters will be evaluated by the author of changes with respect to its artifact preparation impact and added to the new hash preimage if needed. Closes #4132
-
- Apr 24, 2024
-
-
Alexandru Gheorghe authored
Part of https://github.com/paritytech/polkadot-sdk/issues/4126 we want to safely increase the execute_workers_max_num gradually from chain to chain and assess if there are any negative impacts. This PR performs the necessary plumbing to be able to increase it based on the chain id, it increase the number of execution workers from 2 to 4 on test network but lives kusama and polkadot unchanged until we gather more data. --------- Signed-off-by:
Alexandru Gheorghe <[email protected]>
-
- Mar 26, 2024
-
-
Dcompoze authored
**Update:** Pushed additional changes based on the review comments. **This pull request fixes various spelling mistakes in this repository.** Most of the changes are contained in the first **3** commits: - `Fix spelling mistakes in comments and docs` - `Fix spelling mistakes in test names` - `Fix spelling mistakes in error messages, panic messages, logs and tracing` Other source code spelling mistakes are separated into individual commits for easier reviewing: - `Fix the spelling of 'authority'` - `Fix the spelling of 'REASONABLE_HEADERS_IN_JUSTIFICATION_ANCESTRY'` - `Fix the spelling of 'prev_enqueud_messages'` - `Fix the spelling of 'endpoint'` - `Fix the spelling of 'children'` - `Fix the spelling of 'PenpalSiblingSovereignAccount'` - `Fix the spelling of 'PenpalSudoAccount'` - `Fix the spelling of 'insufficient'` - `Fix the spelling of 'PalletXcmExtrinsicsBenchmark'` - `Fix the spelling of 'subtracted'` - `Fix the spelling of 'CandidatePendingAvailability'` - `Fix the spelling of 'exclusive'` - `Fix the spelling of 'until'` - `Fix the spelling of 'discriminator'` - `Fix the spelling of 'nonexistent'` - `Fix the spelling of 'subsystem'` - `Fix the spelling of 'indices'` - `Fix the spelling of 'committed'` - `Fix the spelling of 'topology'` - `Fix the spelling of 'response'` - `Fix the spelling of 'beneficiary'` - `Fix the spelling of 'formatted'` - `Fix the spelling of 'UNKNOWN_PROOF_REQUEST'` - `Fix the spelling of 'succeeded'` - `Fix the spelling of 'reopened'` - `Fix the spelling of 'proposer'` - `Fix the spelling of 'InstantiationNonce'` - `Fix the spelling of 'depositor'` - `Fix the spelling of 'expiration'` - `Fix the spelling of 'phantom'` - `Fix the spelling of 'AggregatedKeyValue'` - `Fix the spelling of 'randomness'` - `Fix the spelling of 'defendant'` - `Fix the spelling of 'AquaticMammal'` - `Fix the spelling of 'transactions'` - `Fix the spelling of 'PassingTracingSubscriber'` - `Fix the spelling of 'TxSignaturePayload'` - `Fix the spelling of 'versioning'` - `Fix the spelling of 'descendant'` - `Fix the spelling of 'overridden'` - `Fix the spelling of 'network'` Let me know if this structure is adequate. **Note:** The usage of the words `Merkle`, `Merkelize`, `Merklization`, `Merkelization`, `Merkleization`, is somewhat inconsistent but I left it as it is. ~~**Note:** In some places the term `Receival` is used to refer to message reception, IMO `Reception` is the correct word here, but I left it as it is.~~ ~~**Note:** In some places the term `Overlayed` is used instead of the more acceptable version `Overlaid` but I also left it as it is.~~ ~~**Note:** In some places the term `Applyable` is used instead of the correct version `Applicable` but I also left it as it is.~~ **Note:** Some usage of British vs American english e.g. `judgement` vs `judgment`, `initialise` vs `initialize`, `optimise` vs `optimize` etc. are both present in different places, but I suppose that's understandable given the number of contributors. ~~**Note:** There is a spelling mistake in `.github/CODEOWNERS` but it triggers errors in CI when I make changes to it, so I left it as it is.~~
-
- Feb 28, 2024
-
-
maksimryndin authored
resolve https://github.com/paritytech/polkadot-sdk/issues/3139 - [x] use a distinguishable error for `execute_artifact` - [x] remove artifact in case of a `RuntimeConstruction` error during the execution - [x] augment the `validate_candidate_with_retry` of `ValidationBackend` with the case of retriable `RuntimeConstruction` error during the execution - [x] update the book (https://paritytech.github.io/polkadot-sdk/book/node/utility/pvf-host-and-workers.html#retrying-execution-requests ) - [x] add a test - [x] run zombienet tests --------- Co-authored-by:
s0me0ne-unkn0wn <[email protected]>
-
- Jan 21, 2024
-
-
jserrat authored
@mrcnski Done the change on the prepare worker, once the prepare worker part is good I'll do the same for the execute worker. This is based on https://github.com/koute/polkavm/blob/11beebd06276ce9b84f335350138479e714f6caf/crates/polkavm/src/sandbox/linux.rs#L711 . ## TODO - [x] Add a check for this capability at startup - [x] Add prdoc mentioning the new Secure Validator Mode (optional) requirement. ## Related Closes #2162 --------- Co-authored-by:
Marcin S <[email protected]>
-
- Jan 10, 2024
-
-
Marcin S. authored
Considering the complexity of https://github.com/paritytech/polkadot-sdk/pull/2871 and the discussion therein, as well as the further complexity introduced by the hardening in https://github.com/paritytech/polkadot-sdk/pull/2742, as well as the eventual replacement of wasmtime by PolkaVM, it seems best to remove this persistence as it is creating more problems than it solves. ## Related Closes https://github.com/paritytech/polkadot-sdk/issues/2863
-
- Dec 29, 2023
-
-
Marcin S. authored
Fixes a potential memory leak. `PR_SET_PDEATHSIG` is used to terminate children when the parent dies. Note that this is subject to a race. There seems to be a raceless alternative [here](https://stackoverflow.com/a/42498370/6085242), but the concern is small enough that a bit more complexity doesn't seem worth it. Left a bit more info in the code comment.
-
- Dec 13, 2023
-
-
Marcin S. authored
Co-authored-by:
Alexandru Vasile <[email protected]>
-
- Dec 05, 2023
-
-
Marcin S. authored
Co-authored-by:
Javier Viola <[email protected]>
-
- Nov 28, 2023
-
-
Marcin S. authored
-
- Nov 24, 2023
-
-
Julian Eager authored
closes #2194 cc @mrcnski --------- Co-authored-by:
Bastian Köcher <[email protected]>
-
- Nov 22, 2023
-
-
Marcin S. authored
-
- Nov 21, 2023
-
-
Marcin S. authored
-
Julian Eager authored
-
- Nov 19, 2023
-
-
Julian Eager authored
Co-authored-by:
Marcin S <[email protected]>
-
- Nov 14, 2023
-
-
jserrat authored
Co-authored-by:
Marcin S <[email protected]>
-
Marcin S. authored
-
- Nov 03, 2023
-
-
s0me0ne-unkn0wn authored
-
- Oct 31, 2023
-
-
Marcin S. authored
-
- Oct 24, 2023
-
-
Marcin S. authored
-
- Sep 28, 2023
-
-
Marcin S. authored
-
- Sep 11, 2023
-
-
s0me0ne-unkn0wn authored
Closes #583 After the separation of PVF worker binaries, dedicated puppet workers are not needed for tests anymore. The production workers can be used instead, avoiding some code duplication and decreasing complexity. The changes also make it possible to further refactor the code to isolate workers completely.
-
- Sep 01, 2023
-
-
s0me0ne-unkn0wn authored
* Import changes from archieved repo * Revert erroneous changes * Fix more tests * Resolve discussions * Fix MORE tests * approval-voting: launch_approval better interface (#1355) --------- Co-authored-by:
Javier Viola <[email protected]> Co-authored-by:
ordian <[email protected]> Co-authored-by:
ordian <[email protected]>
-
- Aug 31, 2023
-
-
Bastian Köcher authored
* Rename `polkadot-parachain` to `polkadot-parachain-primitives` While doing this it also fixes some last `rustdoc` issues and fixes another Cargo warning related to `pallet-paged-list`. * Fix compilation * ".git/.scripts/commands/fmt/fmt.sh" * Fix XCM docs --------- Co-authored-by: command-bot <>
-
- Aug 16, 2023
-
-
Oliver Tale-Yazdi authored
* Fix clippy check Signed-off-by:
Oliver Tale-Yazdi <[email protected]> * Autofix clippy Signed-off-by:
Oliver Tale-Yazdi <[email protected]> * Fix trivial Signed-off-by:
Oliver Tale-Yazdi <[email protected]> * fmt Signed-off-by:
Oliver Tale-Yazdi <[email protected]> * suppress warnings Signed-off-by:
Oliver Tale-Yazdi <[email protected]> * Quiet clippy
😌 Signed-off-by:Oliver Tale-Yazdi <[email protected]> --------- Signed-off-by:
Oliver Tale-Yazdi <[email protected]>
-
- Aug 14, 2023
-
-
jserrat authored
* pvf: use test-utils feature to export test only * adding comment to test-utils feature * make prepare-worker and execute-worker as optional dependencies and add comments to test-utils * remove doc hidden from pvf testing * add prepare worker and execute worker entrypoints to test-utils feature * pvf: add sp_tracing as optional dependency of test-utils * add test-utils for polkadot and malus * add test-utils feature to prepare and execute workers script * remove required features from prepare and executing * Try to trigger CI again to fix broken jobs --------- Co-authored-by:
Marcin S <[email protected]>
-
- Jul 31, 2023
-
-
Marcin S. authored
* [WIP] PVF: Split out worker binaries * Address compilation problems and re-design a bit * Reorganize once more, fix tests * Reformat with new nightly to make `cargo fmt` test happy * Address `clippy` warnings * Add temporary trace to debug zombienet tests * Fix zombienet node upgrade test * Fix malus and its CI * Fix building worker binaries with malus * More fixes for malus * Remove unneeded cli subcommands * Support placing auxiliary binaries to `/usr/libexec` * Fix spelling * Spelling Co-authored-by:
Marcin S. <[email protected]> * Implement review comments (mostly nits) * Fix worker node version flag * Rework getting the worker paths * Address a couple of review comments * Minor restructuring * Fix CI error * Add tests for worker binaries detection * Improve tests; try to fix CI * Move workers module into separate file * Try to fix failing test and workers not printing latest version - Tests were not finding the worker binaries - Workers were not being rebuilt when the version changed - Made some errors easier to read * Make a bunch of fixes * Rebuild nodes on version change * Fix more issues * Fix tests * Pass node version from node into dependencies to avoid recompiles - [X] get version in CLI - [X] pass it in to service - [X] pass version along to PVF - [X] remove rerun from service - [X] add rerun to CLI - [X] don’t rerun pvf/worker’s (these should be built by nodes which have rerun enabled) * Some more improvements for smoother tests - [X] Fix tests - [X] Make puppet workers pass None for version and remove rerun - [X] Make test collators self-contained * Add back rerun to PVF workers * Move worker binaries into files in cli crate As a final optimization I've separated out each worker binary from its own crate into the CLI crate. Before, the worker bin shared a crate with the worker lib, so when the binaries got recompiled so did the libs and everything transitively depending on the libs. This commit fixes this regression that was causing recompiles after every commit. * Fix bug (was passing worker version for node version) * Move workers out of cli into root src/bin/ dir - [X] Pass in node version from top-level (polkadot) - [X] Add build.rs with rerun-git-head to root dir * Add some sanity checks for workers to dockerfiles * Update malus + [X] Make it self-contained + [X] Undo multiple binary changes * Try to fix clippy errors * Address `cargo run` issue - [X] Add default-run for polkadot - [X] Add note about installation to error * Update readme (installation instructions) * Allow disabling external workers for local/testing setups + [X] cli flag to enable single-binary mode + [X] Add message to error * Revert unnecessary Cargo.lock changes * Remove unnecessary build scripts from collators * Add back missing malus commands (should fix failing ZN job) * Some minor fixes * Update Cargo.lock * Fix some build errors * Undo self-contained binaries; cli flag to disable version check + [X] Remove --dont-run-external-workers + [X] Add --disable-worker-version-check + [X] Remove PVF subcommands + [X] Redo malus changes * Try to fix failing job and add some docs for local tests --------- Co-authored-by:
Dmitry Sinyavin <[email protected]> Co-authored-by:
s0me0ne-unkn0wn <[email protected]> Co-authored-by: parity-processbot <>
-
- Jul 06, 2023
-
-
s0me0ne-unkn0wn authored
* Introduce `ci-only-tests` feature * Add comment
-
- Jun 02, 2023
-
-
Marcin S. authored
* PVF: Instantiate wasm in pre-checking * Move `runtime_construction_check` to prepare thread, use bytes * [minor] Update comment * Fix compile error * Update Cargo.lock * Update docs * Add some missing docs!
-
- May 25, 2023
-
-
Marcin S. authored
* PVF: Refactor workers into separate crates, remove host dependency * Fix compile error * Remove some leftover code * Fix compile errors * Update Cargo.lock * Remove worker main.rs files I accidentally copied these from the other PR. This PR isn't intended to introduce standalone workers yet. * Address review comments * cargo fmt * Update a couple of comments * Update log targets
-
- Apr 21, 2023
-
-
Marcin S. authored
* Move PVF workers into separate crate * Fix indentation * Fix compilation errors * Fix more compilation errors * Rename `worker.rs` files, make host interface to worker more clear * Fix more compilation errors * Fix more compilation errors * Add link to issue * Address review comments * Update comment
-
- Apr 20, 2023
-
-
Marcin S. authored
* PVF: Don't dispute on missing artifact A dispute should never be raised if the local cache doesn't provide a certain artifact. You can not dispute based on this reason, as it is a local hardware issue and not related to the candidate to check. Design: Currently we assume that if we prepared an artifact, it remains there on-disk until we prune it, i.e. we never check again if it's still there. We can change it so that instead of artifact-not-found triggering a dispute, we retry once (like we do for AmbiguousWorkerDeath, except we don't dispute if it still doesn't work). And when enqueuing an execute job, we check for the artifact on-disk, and start preparation if not found. Changes: - [x] Integration test (should fail without the following changes) - [x] Check if artifact exists when executing, prepare if not - [x] Return an internal error when file is missing - [x] Retry once on internal errors - [x] Document design (update impl guide) * Add some context to wasm error message (it is quite long) * Fix impl guide * Add check for missing/inaccessible file * Add comment referencing Substrate issue * Add test for retrying internal errors --------- Co-authored-by: parity-processbot <>
-
- Apr 08, 2023
-
-
s0me0ne-unkn0wn authored
* Happy New Year! * Remove year entierly Co-authored-by:
Oliver Tale-Yazdi <[email protected]> * Remove years from copyright notice in the entire repo --------- Co-authored-by:
Oliver Tale-Yazdi <[email protected]>
-
- Mar 29, 2023
-
-
s0me0ne-unkn0wn authored
* Check spawned worker version vs node version before PVF preparation * Address discussions * Propagate errors and shutdown preparation and execution pipelines properly * Add logs; Fix execution worker checks * Revert "Propagate errors and shutdown preparation and execution pipelines properly" This reverts commit b96cc3160ff58db5ff001d8ca0bfea9bd4bdd0f2. * Don't try to shut down; report the condition and exit worker * Get rid of `VersionMismatch` preparation error * Merge master * Add docs; Fix tests * Update Cargo.lock * Kill again, but only the main node process * Move unsafe code to a common safe function * Fix libc dependency error on MacOS * pvf spawning: Add some logging, add a small integration test * Minor fixes * Restart CI --------- Co-authored-by:
Marcin S <[email protected]>
-
- Mar 15, 2023
-
-
s0me0ne-unkn0wn authored
-
- Mar 08, 2023
-
-
s0me0ne-unkn0wn authored
* Move PVF timeouts to executor environment parameters * Typo Co-authored-by:
Marcin S. <[email protected]> * Fix comments * Change handle_import_statements to FatalResult (#6820) * Changing dispute db errors to fatal * fmt * Change node-key for bootnodes (#6772) * Additional tracing in `provisioner`, `vote_selection` and `dispute-coordinator` (#6775) * Additional tracing in `provisioner`, `vote_selection` * Add `fetched_onchain_disputes` metric to provisioner * Some tracelines in dispute-coordinator TODO: cherry pick this in the initial branch!!! * Remove spammy logs * Remove some trace lines * Rename and fix things * Fix comments * Typo * Minor fixes * Add codec indexes; Remove macro --------- Co-authored-by:
Marcin S. <[email protected]> Co-authored-by:
Bradley Olson <[email protected]> Co-authored-by:
Petr Mensik <[email protected]> Co-authored-by:
Tsvetomir Dimitrov <[email protected]>
-
- Feb 20, 2023
-
-
s0me0ne-unkn0wn authored
-
- Feb 15, 2023
-
-
s0me0ne-unkn0wn authored
* Re-apply changes without Diener, rebase to the lastest master * Cache pruning * Bit-pack InstantiationStrategy * Move ExecutorParams version inside the structure itself * Rework runtime API and executor parameters storage * Pass executor parameters through backing subsystem * Update Cargo.lock * Introduce `ExecutorParams` to approval voting subsys * Introduce `ExecutorParams` to dispute coordinator * `cargo fmt` * Simplify requests from backing subsys * Fix tests * Replace manual config cloning with `.clone()` * Move constants to module * Parametrize executor performing PVF pre-check * Fix Malus * Fix test runtime * Introduce session executor params as a constant defined by session info pallet * Use Parity SCALE codec instead of hand-crafted binary encoding * Get rid of constants; Add docs * Get rid of constants * Minor typo * Fix Malus after rebase * `cargo fmt` * Use transparent SCALE encoding instead of explicit * Clean up * Get rid of relay parent to session index mapping * Join environment type and version in a single enum element * Use default execution parameters if running an old runtime * `unwrap()` -> `expect()` * Correct API version * Constants are back in town * Use constants for execution environment types * Artifact separation, first try * Get rid of explicit version * PVF execution queue worker separation * Worker handshake * Global renaming * Minor fixes resolving discussions * Two-stage requesting of executor params to make use of runtime API cache * Proper error handling in pvf-checker * Executor params storage bootstrapping * Propagate migration to v3 network runtimes * Fix storage versioning * Ensure `ExecutorParams` serialization determinism; Add comments * Rename constants to make things a bit more deterministic Get rid of stale code * Tidy up a structure of active PVFs * Minor formatting * Fix comment * Add try-runtime hooks * Add storage version write on upgrade Co-authored-by:
Andronik <[email protected]> * Add pre- and post-upgrade assertions * Require to specify environment type; Remove redundant `impl`s * Add `ExecutorParamHash` creation from `H256` * Fix candidate validation subsys tests * Return splittable error from executor params request fn * Revert "Return splittable error from executor params request fn" This reverts commit a0b274177d8bb2f6e13c066741892ecd2e72a456. * Decompose approval voting metrics * Use more relevant errors * Minor formatting fix * Assert a valid environment type instead of checking * Fix `try-runtime` hooks * After-merge fixes * Add migration logs * Remove dead code * Fix tests * Fix tests * Back to the strongly typed implementation * Promote strong types to executor interface * Remove stale comment * Move executor params to `SessionInfo`: primitives and runtime * Move executor params to `SessionInfo`: node * Try to bump primitives and API version * Get rid of `MallocSizeOf` * Bump target API version to v4 * Make use of session index already in place * Back to v3 * Fix all the tests * Add migrations to all the runtimes * Make use of existing `SessionInfo` in approval voting subsys * Rename `TARGET` -> `LOG_TARGET` * Bump all the primitives to v3 * Fix Rococo ParachainHost API version * Use `RollingSessionWindow` to acquire `ExecutorParams` in disputes * Fix nits from discussions; add comments * Re-evaluate queue logic * Rework job assignment in execution queue * Add documentation * Use `RuntimeInfo` to obtain `SessionInfo` (with blackjack and caching) * Couple `Pvf` with `ExecutorParams` wherever possible * Put members of `PvfWithExecutorParams` under `Arc` for cheap cloning * Fix comment * Fix CI tests * Fix clippy warnings * Address nits from discussions * Add a placeholder for raw data * Fix non exhaustive match * Remove redundant reexports and fix imports * Keep only necessary semantic features, as discussed * Rework `RuntimeInfo` to support mock implementation for tests * Remove unneeded bound * `cargo fmt` * Revert "Remove unneeded bound" This reverts commit 932463f26b00ce290e1e61848eb9328632ef8a61. * Fix PVF host tests * Fix PVF checker tests * Fix overseer declarations * Simplify tests * `MAX_KEEP_WAITING` timeout based on `BACKGING_EXECUTION_TIMEOUT` * Add a unit test for varying executor parameters * Minor fixes from discussions * Add prechecking max. memory parameter (see paritytech/srlabs_findings#110) * Fix and improve a test * Remove `ExecutionEnvironment` and `RawData` * New primitives versioning in parachain host API * `disputes()` implementation for Kusama and Polkadot * Move `ExecutorParams` from `vstaging` to stable primitives * Move disputes from `vstaging` to stable implementation * Fix `try-runtime` * Fixes after merge * Move `ExecutorParams` to the bottom of `SessionInfo` * Revert "Move executor params to `SessionInfo`: primitives and runtime" This reverts commit dfcfb85fefd1c5be6c8a8f72dc09fd1809cfa9ce. * Always use fresh activated live hash in pvf precheck (re-apply 34b09a4c20de17e7926ed942cd0d657d18f743fa) * Fixing tests (broken commit) * Fix candidate validation tests * Fix PVF host test * Minor fixes * Address discussions * Restore migration * Fix `use` to only include what is needed instead of `*` * Add comment to never touch `DEFAULT_CONFIG` * Update migration to set default `ExecutorParams` for `dispute_period` sessions back * Use `earliest_stored_session` instead of calculations * Nit * Add logs * Treat any runtime error as `NotSupported` again * Always return default executor params if not available * Revert "Always return default executor params if not available" This reverts commit b58ac4482ef444c67a9852d5776550d08e312f30. * Add paritytech/substrate#9997 workaround * `cargo fmt` * Remove migration (again!) * Bump executor params to API v4 (backport from #6698) --------- Co-authored-by:
Andronik <[email protected]>
-
- Jan 10, 2023
-
-
Marcin S. authored
* Replace async-std with tokio in PVF subsystem * Rework workers to use `select!` instead of a mutex The improvement in code readability is more important than the thread overhead. * Remove unnecessary `fuse` * Add explanation for `expect()` * Update node/core/pvf/src/worker_common.rs Co-authored-by:
Bastian Köcher <[email protected]> * Update node/core/pvf/src/worker_common.rs Co-authored-by:
Bastian Köcher <[email protected]> * Address some review comments * Shutdown tokio runtime * Run cargo fmt * Add a small note about retries * Fix up merge * Rework `cpu_time_monitor_loop` to return when other thread finishes * Add error string to PrepareError::IoErr variant * Log when artifacts fail to prepare * Fix `cpu_time_monitor_loop`; fix test * Fix text * Fix a couple of potential minor data races. First data race was due to logging in the CPU monitor thread even if the job (other thread) finished. It can technically finish before or after the log. Maybe best would be to move this log to the `select!`s, where we are guaranteed to have chosen the timed-out branch, although there would be a bit of duplication. Also, it was possible for this thread to complete before we executed `finished_tx.send` in the other thread, which would trigger an error as the receiver has already been dropped. And right now, such a spurious error from `send` would be returned even if the job otherwise succeeded. * Update Cargo.lock Co-authored-by:
Bastian Köcher <[email protected]>
-
- Nov 30, 2022
-
-
Marcin S. authored
* Put in skeleton logic for CPU-time-preparation Still needed: - Flesh out logic - Refactor some spots - Tests * Continue filling in logic for prepare worker CPU time changes * Fix compiler errors * Update lenience factor * Fix some clippy lints for PVF module * Fix compilation errors * Address some review comments * Add logging * Add another log * Address some review comments; change Mutex to AtomicBool * Refactor handling response bytes * Add CPU clock timeout logic for execute jobs * Properly handle AtomicBool flag * Use `Ordering::Relaxed` * Refactor thread coordination logic * Fix bug * Add some timing information to execute tests * Add section about the mitigation to the IG * minor: Change more `Ordering`s to `Relaxed` * candidate-validation: Fix build errors
-