* Do not log PVF prunning every hour

This lowers the level of the PVF pruning.

Closes https://github.com/paritytech/polkadot/issues/4361

* Fix typo: ambigious -> ambiguous

The correct spelling is ambiguous
([dictionary](https://dictionary.cambridge.org/dictionary/english/ambiguous))

This lowers the level of the PVF pruning.

Closes https://github.com/paritytech/polkadot/issues/4361

* prepare worker: Catch unexpected unwinds

* Use more specific wording for unknown panic payload

* Increase preparation-timeout to 60 seconds

* Adapt `pvf_preparation_time` metric to the new value

* pvf host: store only compiled artifacts on disk

* Correctly handle failed artifacts

* Serialize result of PVF preparation uniquely

* Set the artifact state depending on the result

* Return the result of PVF preparation directly

* Move PrepareError to the error module

* Update doc comments

* Update misleading comment

* pvf host: turn off parallel compilation

* pvf host: implement precheck requests

* Fix warnings

* Unnecessary clone

* Add a note about timed out outcome

* Revert the pool outcome handling behavior

* Move the prepare result type into error mod

* Test prepare done

* fmt

* Add an explanation to wasmtime config

* Split pvf host test

* Add precheck to dictionary

Co-authored-by: Sergei Shulepov <[email protected]>

* Limit the number of PVF workers

In particular, limit the number of preparation workers to 1 (soft &
hard) and limit the number of execution workers to 2.

The reason why we are doing this is that it seems many workers launched
at the same time can cause problems. I.e. if there are more than 2
preparation workers, the time for preparation rises significantly to the
point of reaching the timeout.

This was mostly observed with parallel_compilation=true, so each worker
used `numcpu` threads and now we are looking to flip that parameter to
`false`. That said, we want to err on the safe side here and gradually
enable it later if our measurements show that we can do that safely.

* Adjust the test to accomodate the changed config value

* Remove light client companion

* Update substrate

* cargo fmt

* Fixed benches

* fmt

* SubsystemContext: add subsystem name str

Signed-off-by: Andrei Sandu <[email protected]>

* Overseer builder proc macro changes

* initilize SubsystemContext name field.
* Add subsystem name in TaskKind::launch_task()

Signed-off-by: Andrei Sandu <[email protected]>

* Update ToOverseer enum

Signed-off-by: Andrei Sandu <[email protected]>

* Assign subsystem names to orphan tasks

Signed-off-by: Andrei Sandu <[email protected]>

* cargo fmt

Signed-off-by: Andrei Sandu <[email protected]>

* SubsystemContext: add subsystem name str

Signed-off-by: Andrei Sandu <[email protected]>

* Overseer builder proc macro changes

* initilize SubsystemContext name field.
* Add subsystem name in TaskKind::launch_task()

Signed-off-by: Andrei Sandu <[email protected]>

* Update ToOverseer enum

Signed-off-by: Andrei Sandu <[email protected]>

* Assign subsystem names to orphan tasks

Signed-off-by: Andrei Sandu <[email protected]>

* cargo fmt

Signed-off-by: Andrei Sandu <[email protected]>

* Rebase changes for new spawn() group param

Signed-off-by: Andrei Sandu <[email protected]>

* Add subsystem constat in JobTrait

Signed-off-by: Andrei Sandu <[email protected]>

* Add subsystem string

Signed-off-by: Andrei Sandu <[email protected]>

* Fix tests

Signed-off-by: Andrei Sandu <[email protected]>

* Fix spawn() calls

Signed-off-by: Andrei Sandu <[email protected]>

* cargo fmt

Signed-off-by: Andrei Sandu <[email protected]>

* Fix

Signed-off-by: Andrei Sandu <[email protected]>

* Fix tests

Signed-off-by: Andrei Sandu <[email protected]>

* fix

Signed-off-by: Andrei Sandu <[email protected]>

* Fix more tests

Signed-off-by: Andrei Sandu <[email protected]>

* Address PR review feedback #1

Signed-off-by: Andrei Sandu <[email protected]>

* Address PR review round 2

Signed-off-by: Andrei Sandu <[email protected]>

* Fixes
- remove JobTrait::Subsystem
- fix tests

Signed-off-by: Andrei Sandu <[email protected]>

* update Cargo.lock

Co-authored-by: Andronik Ordian <[email protected]>

* pvf host: store only compiled artifacts on disk

* Correctly handle failed artifacts

* Serialize result of PVF preparation uniquely

* Set the artifact state depending on the result

* Return the result of PVF preparation directly

* Move PrepareError to the error module

* Update doc comments

* Update misleading comment

* Cleanup docs

* Conclude a test job with an error

Co-authored-by: Sergei Shulepov <[email protected]>

* Update wasmtime config to match new version

* update Substrate

Co-authored-by: parity-processbot <>

* pvf: make execution timeout configurable

* guide: add timeouts to candidate validation params

* add timeouts to candidate validation messages

* fmt

* port backing to use the backing pvf timeout

* port approval-voting to use the execution timeout

* port dispute participation to use the correct timeout

* fmt

* address grumbles & test failure

* introduce the OverseerConnector, use it

* introduce is_relay_chain to RelayChainSelection

* Update node/service/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

* avoid the deferred setting of `is_relay_chain` in `RelayChainSelection`

* positive assertion is not mandated, only the negative one, to avoid a stall

* cleanup: overseer residue

* spellcheck

* fixin

* groundwork to obsolete Overseer::new and AllSubsystemsGen proc-macro

* Now all malus & tests can be ported to the builder pattern.

Obsoletes `Overseer::new`, `AllSubsystemsGen` derive macro, `AllSubsystems`.

* spellcheck

* adjust tests, minor fixes

* remove derive macro AllSubsystemsGen

* add forgotten file dummy.rs

* remove residue

* good news everyone!

* spellcheck

* address review comments

* fixup imports

* make it conditional

* fixup docs

* reduce import

* chore: fmt

* chore: fmt

* chore: spellcheck / nlprules

* fixup malus variant-a

* fmt

* fix

* fixins

* pfmt

* fixins

* chore: fmt

* remove expanded overseer generation

* tracing version

* Update node/network/statement-distribution/src/lib.rs

Co-authored-by: Robert Habermeier <[email protected]>

* use future::ready instead

* silence warning

* chore: fmt

Co-authored-by: Andronik Ordian <[email protected]>
Co-authored-by: Robert Habermeier <[email protected]>

* add jemalloc memory statistics tracking

* chore: move Metronome in a separate file

* add meta flag spellcheck

* adjust metrics names

* account for new metrics in test

Co-authored-by: Bastian Köcher <[email protected]>

This is an artifact left by rustfmt which is not dare to remove the
comma being conservative.

* Add `max_memory_pages`

* update Substrate

Co-authored-by: Shawn Tabrizi <[email protected]>
Co-authored-by: parity-processbot <>

* cargo +nightly fmt

* add cargo-fmt check to ci

* update ci

* fmt

* fmt

* skip macro

* ignore bridges

* CI: add spellcheck

* revert me

* CI: explicit command for spellchecker

* spellcheck: edit misspells

* CI: run spellcheck on diff

* spellcheck: edits

* spellcheck: edit misspells

* spellcheck: add rules

* spellcheck: mv configs

* spellcheck: more edits

* spellcheck: chore

* spellcheck: one more thing

* spellcheck: and another one

* spellcheck: seems like it doesn't get to an end

* spellcheck: new words after rebase

* spellcheck: new words appearing out of nowhere

* chore

* review edits

* more review edits

* more edits

* wonky behavior

* wonky behavior 2

* wonky behavior 3

* change git behavior

* spellcheck: another bunch of new edits

* spellcheck: new words are koming out of nowhere

* CI: finding the master

* CI: fetching master implicitly

* CI: undebug

* new errors

* a bunch of new edits

* and some more

* Update node/core/approval-voting/src/approval_db/v1/mod.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Update xcm/xcm-executor/src/assets.rs

Co-authored-by: Andronik Ordian <[email protected]>

* Apply suggestions from code review

Co-authored-by: Andronik Ordian <[email protected]>

* Suggestions from the code review

* CI: scan only changed files

Co-authored-by: Andronik Ordian <[email protected]>

* update benchmark

* add host function

* update Substrate

* fix unused

Co-authored-by: parity-processbot <>

* Add `canonicalize_nans` and enable `stack_depth_metering`

* Update to the latest change in the substrate PR

* Explain why the numbers are what they are.

* Update Substrate to the latest master

Co-authored-by: parity-processbot <>

* PVF: unresponsive worker doesn't mean the candidate is bad

* s/if let Some/.is_some

This is useful for tests where something is failing :D

* Use wasm-builder from git

This brings new features like compressed runtimes out of the box.

* chore: update wasm builder dep. from 3.0.0 to master

* Fix tests

* Update node/core/pvf/tests/it/main.rs

Co-authored-by: Sergei Shulepov <[email protected]>

Co-authored-by: chevdor <[email protected]>
Co-authored-by: Sergei Shulepov <[email protected]>

* update apis

* fix api

* rename to `KillStorageResult`

* update Substrate

Co-authored-by: parity-processbot <>

* create newtype for ValidationCodeHash

* pvf: fix tests compilation

* primitives: fix test compilation

* Companion for PR 8913

polkadot companion: https://github.com/paritytech/substrate/pull/8913

* update Substrate

Co-authored-by: parity-processbot <>

* Put WIP artifacts next to ready ones

Fixes #3044

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>

* Register ReadRuntimeVersionExt

* Update Substrate

This includes the fix of compilation for macOS platforms.

* issue with ArtifactId::path

* fix artifacts path

* Wrap an overly long line

Co-authored-by: Sergei Shulepov <[email protected]>

* Implement PVF validation host

* WIP: Diener

* Increase the alloted compilation time

* Add more comments

* Minor clean up

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* Fix pruning artifact removal

* Fix formatting and newlines

* Fix the thread pool

* Update node/core/pvf/src/executor_intf.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Remove redundant test declaration

* Don't convert the path into an intermediate string

* Try to workaround the test failure

* Use the puppet_worker trick again

* Fix a blip

* Move `ensure_wasmtime_version` under the tests mod

* Add a macro for puppet_workers

* fix build for not real-overseer

* Rename the puppet worker for adder collator

* play it safe with the name of adder puppet worker

* Typo: triggered

* Add more comments

* Do not kill exec worker on every error

* Plumb Duration for timeouts

* typo: critical

* Add proofs

* Clean unused imports

* Revert "WIP: Diener"

This reverts commit b9f54e513366c7a6dfdd117ac19fbdc46b900b4d.

* Sync version of wasmtime

* Update cargo.lock

* Update Substrate

* Merge fixes still

* Update wasmtime version in test

* bastifmt

Co-authored-by: Bastian Köcher <[email protected]>

* Squash spaces

* Trailing new line for testing.rs

* Remove controversial code

* comment about biasing

* Fix suggestion

* Add comments

* make it more clear why unwrap_err

* tmpfile retry

* proper proofs for claim_idle

* Remove mutex from ValidationHost

* Add some more logging

* Extract exec timeout into a constant

* Add some clarifying logging

* Use blake2_256

* Clean up the merge

Specifically the leftovers after removing real-overseer

* Update parachain/test-parachains/adder/collator/Cargo.toml

Co-authored-by: Andronik Ordian <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Andronik Ordian <[email protected]>