Skip to content
  1. Feb 16, 2023
  3. Feb 15, 2023
    • s0me0ne-unkn0wn's avatar
      Executor Environment parameterization (#6161) · dd0a5566
      s0me0ne-unkn0wn authored 
      

* Re-apply changes without Diener, rebase to the lastest master

* Cache pruning

* Bit-pack InstantiationStrategy

* Move ExecutorParams version inside the structure itself

* Rework runtime API and executor parameters storage

* Pass executor parameters through backing subsystem

* Update Cargo.lock

* Introduce `ExecutorParams` to approval voting subsys

* Introduce `ExecutorParams` to dispute coordinator

* `cargo fmt`

* Simplify requests from backing subsys

* Fix tests

* Replace manual config cloning with `.clone()`

* Move constants to module

* Parametrize executor performing PVF pre-check

* Fix Malus

* Fix test runtime

* Introduce session executor params as a constant defined by session info
pallet

* Use Parity SCALE codec instead of hand-crafted binary encoding

* Get rid of constants; Add docs

* Get rid of constants

* Minor typo

* Fix Malus after rebase

* `cargo fmt`

* Use transparent SCALE encoding instead of explicit

* Clean up

* Get rid of relay parent to session index mapping

* Join environment type and version in a single enum element

* Use default execution parameters if running an old runtime

* `unwrap()` -> `expect()`

* Correct API version

* Constants are back in town

* Use constants for execution environment types

* Artifact separation, first try

* Get rid of explicit version

* PVF execution queue worker separation

* Worker handshake

* Global renaming

* Minor fixes resolving discussions

* Two-stage requesting of executor params to make use of runtime API cache

* Proper error handling in pvf-checker

* Executor params storage bootstrapping

* Propagate migration to v3 network runtimes

* Fix storage versioning

* Ensure `ExecutorParams` serialization determinism; Add comments

* Rename constants to make things a bit more deterministic
Get rid of stale code

* Tidy up a structure of active PVFs

* Minor formatting

* Fix comment

* Add try-runtime hooks

* Add storage version write on upgrade

Co-authored-by: default avatarAndronik <[email protected]>

* Add pre- and post-upgrade assertions

* Require to specify environment type; Remove redundant `impl`s

* Add `ExecutorParamHash` creation from `H256`

* Fix candidate validation subsys tests

* Return splittable error from executor params request fn

* Revert "Return splittable error from executor params request fn"

This reverts commit a0b274177d8bb2f6e13c066741892ecd2e72a456.

* Decompose approval voting metrics

* Use more relevant errors

* Minor formatting fix

* Assert a valid environment type instead of checking

* Fix `try-runtime` hooks

* After-merge fixes

* Add migration logs

* Remove dead code

* Fix tests

* Fix tests

* Back to the strongly typed implementation

* Promote strong types to executor interface

* Remove stale comment

* Move executor params to `SessionInfo`: primitives and runtime

* Move executor params to `SessionInfo`: node

* Try to bump primitives and API version

* Get rid of `MallocSizeOf`

* Bump target API version to v4

* Make use of session index already in place

* Back to v3

* Fix all the tests

* Add migrations to all the runtimes

* Make use of existing `SessionInfo` in approval voting subsys

* Rename `TARGET` -> `LOG_TARGET`

* Bump all the primitives to v3

* Fix Rococo ParachainHost API version

* Use `RollingSessionWindow` to acquire `ExecutorParams` in disputes

* Fix nits from discussions; add comments

* Re-evaluate queue logic

* Rework job assignment in execution queue

* Add documentation

* Use `RuntimeInfo` to obtain `SessionInfo` (with blackjack and caching)

* Couple `Pvf` with `ExecutorParams` wherever possible

* Put members of `PvfWithExecutorParams` under `Arc` for cheap cloning

* Fix comment

* Fix CI tests

* Fix clippy warnings

* Address nits from discussions

* Add a placeholder for raw data

* Fix non exhaustive match

* Remove redundant reexports and fix imports

* Keep only necessary semantic features, as discussed

* Rework `RuntimeInfo` to support mock implementation for tests

* Remove unneeded bound

* `cargo fmt`

* Revert "Remove unneeded bound"

This reverts commit 932463f26b00ce290e1e61848eb9328632ef8a61.

* Fix PVF host tests

* Fix PVF checker tests

* Fix overseer declarations

* Simplify tests

* `MAX_KEEP_WAITING` timeout based on `BACKGING_EXECUTION_TIMEOUT`

* Add a unit test for varying executor parameters

* Minor fixes from discussions

* Add prechecking max. memory parameter (see paritytech/srlabs_findings#110)

* Fix and improve a test

* Remove `ExecutionEnvironment` and `RawData`

* New primitives versioning in parachain host API

* `disputes()` implementation for Kusama and Polkadot

* Move `ExecutorParams` from `vstaging` to stable primitives

* Move disputes from `vstaging` to stable implementation

* Fix `try-runtime`

* Fixes after merge

* Move `ExecutorParams` to the bottom of `SessionInfo`

* Revert "Move executor params to `SessionInfo`: primitives and runtime"

This reverts commit dfcfb85fefd1c5be6c8a8f72dc09fd1809cfa9ce.

* Always use fresh activated live hash in pvf precheck
(re-apply 34b09a4c20de17e7926ed942cd0d657d18f743fa)

* Fixing tests (broken commit)

* Fix candidate validation tests

* Fix PVF host test

* Minor fixes

* Address discussions

* Restore migration

* Fix `use` to only include what is needed instead of `*`

* Add comment to never touch `DEFAULT_CONFIG`

* Update migration to set default `ExecutorParams` for `dispute_period`
sessions back

* Use `earliest_stored_session` instead of calculations

* Nit

* Add logs

* Treat any runtime error as `NotSupported` again

* Always return default executor params if not available

* Revert "Always return default executor params if not available"

This reverts commit b58ac4482ef444c67a9852d5776550d08e312f30.

* Add paritytech/substrate#9997 workaround

* `cargo fmt`

* Remove migration (again!)

* Bump executor params to API v4 (backport from #6698)

---------

Co-authored-by: default avatarAndronik <[email protected]>
      dd0a5566
  5. Feb 14, 2023
    • Sam Elamin's avatar
      add warp sync params to enable parachain warp sync (#6334) · 7f6b8e6d
      Sam Elamin authored 
      * add warp sync params to enable parachain warp sync

* update lockfile for {"substrate"}

---------

Co-authored-by: parity-processbot <>
      7f6b8e6d
    • Marcin S.'s avatar
      Refactor PVF preparation memory stats (#6693) · fd70d012
      Marcin S. authored 
      * Refactor PVF preparation memory stats

The original purpose of this change was to gate metrics that are unsupported by
some systems behind conditional compilation directives (#[cfg]); see
https://github.com/paritytech/polkadot/pull/6675#discussion_r1099996209.

Then I started doing some random cleanups and simplifications and got a bit
carried away. 🙈 The code should be overall tidier than before.

Changes:
- Don't register unsupported metrics (e.g. `max_rss` on non-Linux systems)
- Introduce `PrepareStats` struct as an abstraction over the `Ok` values of
  `PrepareResult`. It is cleaner, and can be easily modified in the future.
- Other small changes

* Minor fixes to comments

* Fix compile errors

* Try to fix some Linux errors

* Mep

* Fix candidate-validation tests

* Update docstring
      fd70d012
    • Gavin Wood's avatar
      v3::Junction supports small (32-byte max) "vecs". (#6716) · 9c357630
      Gavin Wood authored 
      * v3::Junction supports small (32-byte max) "vecs".

* Formatting

* Add test
      9c357630
  7. Feb 13, 2023
  9. Feb 10, 2023
  11. Feb 09, 2023
  13. Feb 08, 2023
  15. Feb 07, 2023
  17. Feb 06, 2023
    • Marcin S.'s avatar
      pvf: Log memory metrics from preparation (#6565) · f317115b
      Marcin S. authored 
      * Add getrusage and memory tracker for precheck preparation

* Log memory stats metrics after prechecking

* Fix tests

* Try to fix errors (linux-only so I'm relying on CI here)

* Try to fix CI

* Add module docs for `prepare/memory_stats.rs`; fix CI error

* Report memory stats for all preparation jobs

* Use `RUSAGE_SELF` instead of `RUSAGE_THREAD`

Not sure why I did that -- was a brainfart on my end.

* Revert last commit (RUSAGE_THREAD is correct)

* Use exponential buckets

* Use `RUSAGE_SELF` for `getrusage`; enable `max_rss` metric for MacOS

* Increase poll interval

* Revert "Use `RUSAGE_SELF` for `getrusage`; enable `max_rss` metric for MacOS"

This reverts commit becf7a815409ab530fc61370abffcd1b97b9a777.
      f317115b
    • Joakim Nyman's avatar
      Added Dwellir bootnodes with non-ssl. (#6544) · e3930760
      Joakim Nyman authored
      e3930760
    • dependabot[bot]'s avatar
      Bump tokio from 1.24.1 to 1.24.2 (#6671) · 36f014d3
      dependabot[bot] authored 
      

Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.24.1 to 1.24.2.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/commits)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
...

Signed-off-by: default avatardependabot[bot] <[email protected]>
Co-authored-by: default avatardependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
      36f014d3
  19. Feb 03, 2023
  21. Feb 02, 2023
  23. Feb 01, 2023
    • ordian's avatar
      Some late short-term fixes for dispute slashing (#6249) · 2fa8565f
      ordian authored 
      

* disputes/slashing: slash only backers for ForInvalid

* add an assertion in mock impl

* fix tests

* do not slash backers on onconcluded disputes

* slash an intersection of backers and losers

* zombienet/disputes: check for offence only for invalid disputes

* add backing votes to disputes bench builder

* Update runtime/parachains/src/builder.rs

* Brad implementers guide revisions 2 (#6239)

* Add disputes subsystems fix

* Updated dispute approval vote import reasoning

* Improved wording of my changes

* Resolving issues brought up in comments

* Update disputes prioritisation in `dispute-coordinator` (#6130)

* Scraper processes CandidateBacked events

* Change definition of best-effort

* Fix `dispute-coordinator` tests

* Unit test for dispute filtering

* Clarification comment

* Add tests

* Fix logic

If a dispute is not backed, not included and not confirmed we
don't participate but we do import votes.

* Add metrics for refrained participations

* Revert "Add tests"

This reverts commit 7b8391a087922ced942cde9cd2b50ff3f633efc0.

* Revert "Unit test for dispute filtering"

This reverts commit 92ba5fe678214ab360306313a33c781338e600a0.

* fix dispute-coordinator tests

* Fix scraping

* new tests

* Small fixes in guide

* Apply suggestions from code review

Co-authored-by: default avatarAndrei Sandu <[email protected]>

* Fix some comments and remove a pointless test

* Code review feedback

* Clarification comment in tests

* Some tests

* Reference counted `CandidateHash` in scraper

* Proper handling for Backed and Included candidates in scraper

Backed candidates which are not included should be kept for a
predetermined window of finalized blocks. E.g. if a candidate is backed
but not included in block 2, and the window size is 2, the same
candidate should be cleaned after block 4 is finalized.

Add reference counting for candidates in scraper. A candidate can be
added on multiple block heights so we have to make sure we don't clean
it prematurely from the scraper.

Add tests.

* Update comments in tests

* Guide update

* Fix cleanup logic for `backed_candidates_by_block_number`

* Simplify cleanup

* Make spellcheck happy

* Update tests

* Extract candidate backing logic in separate struct

* Code review feedback

* Treat  backed and included candidates in the same fashion

* Update some comments

* Small improvements in test

* spell check

* Fix some more comments

* clean -> prune

* Code review feedback

* Reword comment

* spelling

Co-authored-by: default avatarAndrei Sandu <[email protected]>

* approval-voting: remove redundant validation check (#6266)

* approval-voting: remove a redundant check

* candidate-validation: remove unreachable check

* remove fill_block (#6200)

Co-authored-by: parity-processbot <>

* fix a compilation warning (#6279)

Fixes #6277.

* Only report concluded if there is an actual dispute. (#6270)

* Only report concluded if there is an actual dispute.

Hence no "non"-disputes will be added to disputes anymore.

* Fix redundant check.

* Test for no onesided disputes.

Co-authored-by: default avatareskimor <[email protected]>

* [ci] fix buildah image (#6281)

* Revert special casing of Kusama for grandpa rounds. (#6217)

Co-authored-by: default avatareskimor <[email protected]>

* Fixes "for loop over an `Option`" warnings (#6291)

Was seeing these warnings when running `cargo check --all`:

```
warning: for loop over an `Option`. This is more readably written as an `if let` statement
    --> node/core/approval-voting/src/lib.rs:1147:21
     |
1147 |             for activated in update.activated {
     |                              ^^^^^^^^^^^^^^^^
     |
     = note: `#[warn(for_loops_over_fallibles)]` on by default
help: to check pattern in a loop use `while let`
     |
1147 |             while let Some(activated) = update.activated {
     |             ~~~~~~~~~~~~~~~         ~~~
help: consider using `if let` to clear intent
     |
1147 |             if let Some(activated) = update.activated {
     |             ~~~~~~~~~~~~         ~~~
```

My guess is that `activated` used to be a SmallVec or similar, as is
`deactivated`. It was changed to an `Option`, the `for` still compiled (it's
technically correct, just weird), and the compiler didn't catch it until now.

* companion for #12599 (#6290)

* companion for #12599

* update Cargo.lock

* use cargo path instead of diener

* update lockfile for {"substrate"}

Co-authored-by: parity-processbot <>

* remove the runtime check and test

* append keys on past-session slashing

* runtime/disputes: allow importing backing votes after explicit for

* explicit MaliciousBacker error and a test

* update an outdated comment

* Revert "update an outdated comment"

This reverts commit 7c4c3f5a848f16e2b61435e981d814f00333ed41.

* Revert "remove the runtime check and test"

This reverts commit a5bff0c75e77effb5b7d3a1691de1b14bcdbd648.

* incremental punishment post conclusion + test

* punish backers post FOR vote

* remove unnecessary lifetime annotation

* add a comment to zombinet test

* typo

* fmt

* post merge test fixes

* fix test after changes in master

* address review nits

---------

Co-authored-by: default avatarBradley Olson <[email protected]>
Co-authored-by: default avatarTsvetomir Dimitrov <[email protected]>
Co-authored-by: default avatarAndrei Sandu <[email protected]>
Co-authored-by: default avatarSergej Sakac <[email protected]>
Co-authored-by: default avatareskimor <[email protected]>
Co-authored-by: default avatareskimor <[email protected]>
Co-authored-by: default avatarAlexander Samusev <[email protected]>
Co-authored-by: default avatarMarcin S <[email protected]>
Co-authored-by: default avatarNiklas Adolfsson <[email protected]>
      2fa8565f
  25. Jan 31, 2023
  27. Jan 30, 2023
  29. Jan 29, 2023
  31. Jan 28, 2023
  33. Jan 27, 2023