Skip to content
  1. Feb 20, 2023
  3. Feb 17, 2023
  5. Feb 16, 2023
    • Tsvetomir Dimitrov's avatar
      Fix a metric description (#6732) · 2853f349
      Tsvetomir Dimitrov authored
      2853f349
    • Bastian Köcher's avatar
      Change `MaxMemorySize` to `MaxMemoryPages` (#6728) · 83eb8453
      Bastian Köcher authored 
      * Change `MaxMemorySize` to `MaxMemoryPages`

We should set the max memory for the executor in pages (64KiB) and not in bytes.
The wasm memory is always a multiple of a page and we should use the
same terminology.

* FMT

* Fix warning
      83eb8453
    • Marcin S.'s avatar
      Use a `BoundedVec` in `ValidationResult` (#6603) · d5a79914
      Marcin S. authored 
      * Use a `BoundedVec` in `ValidationResult`

> Use a `BoundedVec` for `upward_messages` and `horizontal_messages` in order to
> limit the number of individual messages/memory allocations right at decoding
> time. The reason for this is that the `ValidationResult` may contain a code
> upgrade (including a full PVF binary), so the total size limit can't be set
> too low and this limit will still allow several millions of upward messages,
> which will (due to the memory allocator overhead) already have a
> non-negligible memory footprint in decoded form.

* List all fields when hashing so we don't miss one

* Define types for  `BoundedVec`s of messages

* Fix test compile errors

* Depend on `bounded-collections` 0.1.4 (fixes allocation issue)

* Fix compilation issue

* Derive `Hash` instead of manual `impl`

* Avoid use of unwrap
      d5a79914
    • Adrian Catangiu's avatar
      Companion for substrate #12910: BEEFY metrics (#6706) · 686f6972
      Adrian Catangiu authored 
      * service: provide prometheus registry for beefy metrics

* update lockfile for {"substrate"}

---------

Co-authored-by: parity-processbot <>
      686f6972
    • JP's avatar
      Update release-related templates for the new changelogerator output format (#6702) · 9a5e01a3
      JP authored 
      

* update templates to upcoming context.json format

* fix merging error

* more fixes

* fix templates

* fix

* print "changelogerator --version"

* print changelogerator version as debug

* remove excluded debug info

* WIP

* Switch to changelogerator v0.10.1

* Update bundle deps

* fix

---------

Co-authored-by: default avatarWilfried Kopp <[email protected]>
      9a5e01a3
  7. Feb 15, 2023
    • s0me0ne-unkn0wn's avatar
      Executor Environment parameterization (#6161) · dd0a5566
      s0me0ne-unkn0wn authored 
      

* Re-apply changes without Diener, rebase to the lastest master

* Cache pruning

* Bit-pack InstantiationStrategy

* Move ExecutorParams version inside the structure itself

* Rework runtime API and executor parameters storage

* Pass executor parameters through backing subsystem

* Update Cargo.lock

* Introduce `ExecutorParams` to approval voting subsys

* Introduce `ExecutorParams` to dispute coordinator

* `cargo fmt`

* Simplify requests from backing subsys

* Fix tests

* Replace manual config cloning with `.clone()`

* Move constants to module

* Parametrize executor performing PVF pre-check

* Fix Malus

* Fix test runtime

* Introduce session executor params as a constant defined by session info
pallet

* Use Parity SCALE codec instead of hand-crafted binary encoding

* Get rid of constants; Add docs

* Get rid of constants

* Minor typo

* Fix Malus after rebase

* `cargo fmt`

* Use transparent SCALE encoding instead of explicit

* Clean up

* Get rid of relay parent to session index mapping

* Join environment type and version in a single enum element

* Use default execution parameters if running an old runtime

* `unwrap()` -> `expect()`

* Correct API version

* Constants are back in town

* Use constants for execution environment types

* Artifact separation, first try

* Get rid of explicit version

* PVF execution queue worker separation

* Worker handshake

* Global renaming

* Minor fixes resolving discussions

* Two-stage requesting of executor params to make use of runtime API cache

* Proper error handling in pvf-checker

* Executor params storage bootstrapping

* Propagate migration to v3 network runtimes

* Fix storage versioning

* Ensure `ExecutorParams` serialization determinism; Add comments

* Rename constants to make things a bit more deterministic
Get rid of stale code

* Tidy up a structure of active PVFs

* Minor formatting

* Fix comment

* Add try-runtime hooks

* Add storage version write on upgrade

Co-authored-by: default avatarAndronik <[email protected]>

* Add pre- and post-upgrade assertions

* Require to specify environment type; Remove redundant `impl`s

* Add `ExecutorParamHash` creation from `H256`

* Fix candidate validation subsys tests

* Return splittable error from executor params request fn

* Revert "Return splittable error from executor params request fn"

This reverts commit a0b274177d8bb2f6e13c066741892ecd2e72a456.

* Decompose approval voting metrics

* Use more relevant errors

* Minor formatting fix

* Assert a valid environment type instead of checking

* Fix `try-runtime` hooks

* After-merge fixes

* Add migration logs

* Remove dead code

* Fix tests

* Fix tests

* Back to the strongly typed implementation

* Promote strong types to executor interface

* Remove stale comment

* Move executor params to `SessionInfo`: primitives and runtime

* Move executor params to `SessionInfo`: node

* Try to bump primitives and API version

* Get rid of `MallocSizeOf`

* Bump target API version to v4

* Make use of session index already in place

* Back to v3

* Fix all the tests

* Add migrations to all the runtimes

* Make use of existing `SessionInfo` in approval voting subsys

* Rename `TARGET` -> `LOG_TARGET`

* Bump all the primitives to v3

* Fix Rococo ParachainHost API version

* Use `RollingSessionWindow` to acquire `ExecutorParams` in disputes

* Fix nits from discussions; add comments

* Re-evaluate queue logic

* Rework job assignment in execution queue

* Add documentation

* Use `RuntimeInfo` to obtain `SessionInfo` (with blackjack and caching)

* Couple `Pvf` with `ExecutorParams` wherever possible

* Put members of `PvfWithExecutorParams` under `Arc` for cheap cloning

* Fix comment

* Fix CI tests

* Fix clippy warnings

* Address nits from discussions

* Add a placeholder for raw data

* Fix non exhaustive match

* Remove redundant reexports and fix imports

* Keep only necessary semantic features, as discussed

* Rework `RuntimeInfo` to support mock implementation for tests

* Remove unneeded bound

* `cargo fmt`

* Revert "Remove unneeded bound"

This reverts commit 932463f26b00ce290e1e61848eb9328632ef8a61.

* Fix PVF host tests

* Fix PVF checker tests

* Fix overseer declarations

* Simplify tests

* `MAX_KEEP_WAITING` timeout based on `BACKGING_EXECUTION_TIMEOUT`

* Add a unit test for varying executor parameters

* Minor fixes from discussions

* Add prechecking max. memory parameter (see paritytech/srlabs_findings#110)

* Fix and improve a test

* Remove `ExecutionEnvironment` and `RawData`

* New primitives versioning in parachain host API

* `disputes()` implementation for Kusama and Polkadot

* Move `ExecutorParams` from `vstaging` to stable primitives

* Move disputes from `vstaging` to stable implementation

* Fix `try-runtime`

* Fixes after merge

* Move `ExecutorParams` to the bottom of `SessionInfo`

* Revert "Move executor params to `SessionInfo`: primitives and runtime"

This reverts commit dfcfb85fefd1c5be6c8a8f72dc09fd1809cfa9ce.

* Always use fresh activated live hash in pvf precheck
(re-apply 34b09a4c20de17e7926ed942cd0d657d18f743fa)

* Fixing tests (broken commit)

* Fix candidate validation tests

* Fix PVF host test

* Minor fixes

* Address discussions

* Restore migration

* Fix `use` to only include what is needed instead of `*`

* Add comment to never touch `DEFAULT_CONFIG`

* Update migration to set default `ExecutorParams` for `dispute_period`
sessions back

* Use `earliest_stored_session` instead of calculations

* Nit

* Add logs

* Treat any runtime error as `NotSupported` again

* Always return default executor params if not available

* Revert "Always return default executor params if not available"

This reverts commit b58ac4482ef444c67a9852d5776550d08e312f30.

* Add paritytech/substrate#9997 workaround

* `cargo fmt`

* Remove migration (again!)

* Bump executor params to API v4 (backport from #6698)

---------

Co-authored-by: default avatarAndronik <[email protected]>
      dd0a5566
  9. Feb 14, 2023
    • Sam Elamin's avatar
      add warp sync params to enable parachain warp sync (#6334) · 7f6b8e6d
      Sam Elamin authored 
      * add warp sync params to enable parachain warp sync

* update lockfile for {"substrate"}

---------

Co-authored-by: parity-processbot <>
      7f6b8e6d
    • Marcin S.'s avatar
      Refactor PVF preparation memory stats (#6693) · fd70d012
      Marcin S. authored 
      * Refactor PVF preparation memory stats

The original purpose of this change was to gate metrics that are unsupported by
some systems behind conditional compilation directives (#[cfg]); see
https://github.com/paritytech/polkadot/pull/6675#discussion_r1099996209.

Then I started doing some random cleanups and simplifications and got a bit
carried away. 🙈 The code should be overall tidier than before.

Changes:
- Don't register unsupported metrics (e.g. `max_rss` on non-Linux systems)
- Introduce `PrepareStats` struct as an abstraction over the `Ok` values of
  `PrepareResult`. It is cleaner, and can be easily modified in the future.
- Other small changes

* Minor fixes to comments

* Fix compile errors

* Try to fix some Linux errors

* Mep

* Fix candidate-validation tests

* Update docstring
      fd70d012
    • Gavin Wood's avatar
      v3::Junction supports small (32-byte max) "vecs". (#6716) · 9c357630
      Gavin Wood authored 
      * v3::Junction supports small (32-byte max) "vecs".

* Formatting

* Add test
      9c357630
  11. Feb 13, 2023
  13. Feb 10, 2023
  15. Feb 09, 2023
  17. Feb 08, 2023
  19. Feb 07, 2023
  21. Feb 06, 2023
    • Marcin S.'s avatar
      pvf: Log memory metrics from preparation (#6565) · f317115b
      Marcin S. authored 
      * Add getrusage and memory tracker for precheck preparation

* Log memory stats metrics after prechecking

* Fix tests

* Try to fix errors (linux-only so I'm relying on CI here)

* Try to fix CI

* Add module docs for `prepare/memory_stats.rs`; fix CI error

* Report memory stats for all preparation jobs

* Use `RUSAGE_SELF` instead of `RUSAGE_THREAD`

Not sure why I did that -- was a brainfart on my end.

* Revert last commit (RUSAGE_THREAD is correct)

* Use exponential buckets

* Use `RUSAGE_SELF` for `getrusage`; enable `max_rss` metric for MacOS

* Increase poll interval

* Revert "Use `RUSAGE_SELF` for `getrusage`; enable `max_rss` metric for MacOS"

This reverts commit becf7a815409ab530fc61370abffcd1b97b9a777.
      f317115b
    • Joakim Nyman's avatar
      Added Dwellir bootnodes with non-ssl. (#6544) · e3930760
      Joakim Nyman authored
      e3930760
    • dependabot[bot]'s avatar
      Bump tokio from 1.24.1 to 1.24.2 (#6671) · 36f014d3
      dependabot[bot] authored 
      

Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.24.1 to 1.24.2.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/commits)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
...

Signed-off-by: default avatardependabot[bot] <[email protected]>
Co-authored-by: default avatardependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
      36f014d3
  23. Feb 03, 2023
  25. Feb 02, 2023