Skip to content
  1. Feb 23, 2024
    • Andrei Sandu's avatar
      Runtime: allow backing multiple candidates of same parachain on different cores (#3231) · 2431001e
      Andrei Sandu authored 
      Fixes https://github.com/paritytech/polkadot-sdk/issues/3144

Builds on top of https://github.com/paritytech/polkadot-sdk/pull/3229

### Summary
Some preparations for Runtime to support elastic scaling, guarded by
config node features bit `FeatureIndex::ElasticScalingMVP`. This PR
introduces a per-candidate `CoreIndex` but does it in a hacky way to
avoid changing `CandidateCommitments`, `CandidateReceipts` primitives
and networking protocols.

#### Including `CoreIndex` in `BackedCandidate`
If the `ElasticScalingMVP` feature bit is enabled then
`BackedCandidate::validator_indices` is extended by 8 bits.
The value stored in these bits represents the assumed core index for the
candidate.

It is temporary solution which works by creating a mapping from
`BackedCandidate` to `CoreIndex` by assuming the `CoreIndex` can be
discovered by checking in which validator group the validator that
signed the statement is.

TODO:
- [x] fix tests
- [x] add new tests
- [x] Bump runtime API for Kusama, so we have that node features thing!
-> https://github.com/polkadot-fellows/runtimes/pull/194



---------

Signed-off-by: default avatarAndrei Sandu <[email protected]>
Signed-off-by: default avataralindima <[email protected]>
Co-authored-by: default avataralindima <[email protected]>
      2431001e
    • Sebastian Kunert's avatar
      PoV Reclaim Runtime Side (#3002) · 3386377b
      Sebastian Kunert authored 
      

# Runtime side for PoV Reclaim

## Implementation Overview
- Hostfunction to fetch the storage proof size has been added to the
PVF. It uses the size tracking recorder that was introduced in my
previous PR.
- Mechanisms to use the reclaim HostFunction have been introduced.
- 1. A SignedExtension that checks the node-reported proof size before
and after application of an extrinsic. Then it reclaims the difference.
- 2. A manual helper to make reclaiming easier when manual interaction
is required, for example in `on_idle` or other hooks.
- In order to utilize the manual reclaiming, I modified `WeightMeter` to
support the reduction of consumed weight, at least for storage proof
size.

## How to use
To enable the general functionality for a parachain:
1. Add the SignedExtension to your parachain runtime. 
2. Provide the HostFunction to the node
3. Enable proof recording during block import

## TODO
- [x] PRDoc

---------

Co-authored-by: default avatarDmitry Markin <[email protected]>
Co-authored-by: default avatarDavide Galassi <[email protected]>
Co-authored-by: default avatarBastian Köcher <[email protected]>
      3386377b
  3. Feb 20, 2024
    • Niklas Adolfsson's avatar
      rpc server: make possible to disable/enable batch requests (#3364) · fee810a5
      Niklas Adolfsson authored 
      The rationale behind this, is that it may be useful for some users
actually disable RPC batch requests or limit them by length instead of
the total size bytes of the batch.

This PR adds two new CLI options:

```
--rpc-disable-batch-requests - disable batch requests on the server
--rpc-max-batch-request-len <LEN> - limit batches to LEN on the server.
```
      fee810a5
  5. Feb 18, 2024
  7. Feb 16, 2024
  9. Feb 15, 2024
  11. Feb 14, 2024
  13. Feb 12, 2024
  15. Feb 09, 2024
  17. Feb 06, 2024
    • Koute's avatar
      Build more runtimes targeting PolkaVM (#3209) · 402b64ca
      Koute authored 
      This PR improves compatibility with RISC-V and PolkaVM, allowing more
runtimes to successfully compile.

In particular, it makes the following changes:

- The `sp-mmr-primitives` and `sp-consensus-beefy` crates
unconditionally required an `std`-only dependency; now they only require
those dependencies when the `std` feature is actually enabled. (Our
RISC-V target is, unlike WASM, a true `no_std` target where you can't
accidentally use stuff from `std` anymore.)
- One of our dependencies (the `bitvec` trace) uses a crate called
`radium` which doesn't compile under RISC-V due to incomplete
autodetection logic in their `build.rs` file. The good news is that this
is already fixed in the newest upstream version of `radium`, and the
newest version of `bitvec` uses it. The bad news is that the newest
version of `bitvec` is not currently released on crates.io, so we can't
use it. I've [created an
issue](https://github.com/ferrilab/ferrilab/issues/5) asking for a new
release, but in the meantime I forked the currently used `radium` 0.7,
[fixed the faulty
logic](https://github.com/paritytech/radium-0.7-fork/commit/ed66c8a294b138c67f93499644051d97d4c7fbda)
and used cargo's patching capabilities to use it for the RISC-V runtime
builds. This might be a little hacky, but it is the least intrusive way
to fix the problem, doesn't affect WASM builds at all, and we can
trivially remove it once a new `bitvec` is released.
- The new runtimes are added to the CI to make sure their compilation
doesn't break.
      402b64ca
  19. Feb 03, 2024
    • Koute's avatar
      Initial support for building RISC-V runtimes targeting PolkaVM (#3179) · e349fc9e
      Koute authored 
      This PR adds initial support for building RISC-V runtimes targeting
PolkaVM.

- Setting the `SUBSTRATE_RUNTIME_TARGET=riscv` environment variable will
now build a RISC-V runtime instead of a WASM runtime.
- This only adds support for *building* runtimes; running them will need
a PolkaVM-based executor, which I will add in a future PR.
- Only building the minimal runtime is supported (building the Polkadot
runtime doesn't work *yet* due to one of the dependencies).
- The builder now sets a `substrate_runtime` cfg flag when building the
runtimes, with the idea being that instead of doing `#[cfg(not(feature =
"std"))]` or `#[cfg(target_arch = "wasm32")]` to detect that we're
building a runtime you'll do `#[cfg(substrate_runtime)]`. (Switching the
whole codebase to use this will be done in a future PR; I deliberately
didn't do this here to keep this PR minimal and reviewable.)
- Further renaming of things (e.g. types, environment variables and proc
macro attributes having "wasm" in their name) to be target-agnostic will
also be done in a future refactoring PR (while keeping backwards
compatibility where it makes sense; I don't intend to break anyone's
workflow or create unnecessary churn).
- This PR also fixes two bugs in the `wasm-builder` crate:
* The `RUSTC` environment variable is now removed when invoking the
compiler. This prevents the toolchain version from being overridden when
called from a `build.rs` script.
* When parsing the `rustup toolchain list` output the `(default)` is now
properly stripped and not treated as part of the version.
- I've also added a minimal CI job that makes sure this doesn't break in
the future. (cc @paritytech/ci)

cc @athei



------

Also, just a fun little tidbit: quickly comparing the size of the built
runtimes it seems that the PolkaVM runtime is slightly smaller than the
WASM one. (`production` build, with the `names` section substracted from
the WASM's size to keep things fair, since for the PolkaVM runtime we're
currently stripping out everything)

- `.wasm`: 625505 bytes
- `.wasm` (after wasm-opt -O3): 563205 bytes
- `.wasm` (after wasm-opt -Os): 562987 bytes
- `.wasm` (after wasm-opt -Oz): 536852 bytes
- `.polkavm`: ~~580338 bytes~~ 550476 bytes (after enabling extra target
features; I'll add those in another PR once we have an executor working)

---------

Co-authored-by: default avatarBastian Köcher <[email protected]>
      e349fc9e
  21. Feb 02, 2024
  23. Jan 31, 2024
  25. Jan 26, 2024
    • Svyatoslav Nikolsky's avatar
      Remove bridges zombienet tests from CI (#3071) · acd043bc
      Svyatoslav Nikolsky authored 
      closes https://github.com/paritytech/parity-bridges-common/issues/2796

This partially reverts the #2439 - there are some changes (unrelated to
CI) that we still want to keep. The reason of that removal is that with
async backing enabled for Rococo AH (and for other chains in the near
future), we see a lot of issues there (because we run `14` nodes +
additional standalone process within a same container and it causes a
lot of timeouts). There's no way known to me to fix it right now, so
we're removing those tests hopefully temporarily to keep CI green
      acd043bc
  27. Jan 15, 2024
  29. Jan 12, 2024
    • Javier Viola's avatar
      Bump zombienet version `v1.3.91` (#2912) · c421b879
      Javier Viola authored 
      This version includes
- Performance improvements.
- Minor fixes.
      c421b879
    • Svyatoslav Nikolsky's avatar
      Run bridges zombienet tests on CI (#2439) · 5ed0a75f
      Svyatoslav Nikolsky authored 
      Brridges zombienet tests are non-standard - zombienet currently missing
multiple relay chains support (see e.g.
https://github.com/paritytech/zombienet/pull/796), so we need to go live
with two relay networks, their parachains + custom test runner (which
e.g. doesn't shutdown net when its tests are finished and instead waits
for both networks tests to complete). So we are stuck with native
zombienet provider => this PR is an attempt to gather everything in a
single docker container and run tests there ~Draft, because it is far
from finishing - what I want now is to see how it works on CI~
      5ed0a75f
  31. Jan 11, 2024
  33. Jan 10, 2024
    • ordian's avatar
      statement-distribution: validator disabling (#1841) · a4195326
      ordian authored 
      

Closes #1591.

The purpose of this PR is filter out backing statements from the network
signed by disabled validators. This is just an optimization, since we
will do filtering in the runtime in #1863 to avoid nodes to filter
garbage out at block production time.

- [x] Ensure it's ok to fiddle with the mask of manifests
- [x] Write more unit tests
- [x] Test locally
- [x] simple zombienet test
- [x] PRDoc

---------

Co-authored-by: default avatarTsvetomir Dimitrov <[email protected]>
      a4195326
  35. Jan 08, 2024
  37. Dec 31, 2023
  39. Dec 20, 2023
    • Dónal Murray's avatar
      Fix clippy lints behind feature gates and add new CI step all features (#2569) · d68868f6
      Dónal Murray authored 
      

Many clippy lints usually enforced by `-Dcomplexity` and `-Dcorrectness`
are not caught by CI as they are gated by `features`, like
`runtime-benchmarks`, while the clippy CI job runs with only the default
features for all targets.

This PR also adds a CI step to run clippy with `--all-features` to
ensure the code quality is maintained behind feature gates from now on.

To improve local development, clippy lints are downgraded to warnings,
but they still will result in an error at CI due to the `-Dwarnings`
rustflag.

---------

Co-authored-by: default avatarLiam Aharon <[email protected]>
      d68868f6
  41. Dec 19, 2023
  43. Dec 18, 2023
  45. Dec 15, 2023
  47. Dec 13, 2023
    • Squirrel's avatar
      Set clippy lints in workspace (requires rust 1.74) (#2390) · be8e6268
      Squirrel authored 
      

We currently use a bit of a hack in `.cargo/config` to make sure that
clippy isn't too annoying by specifying the list of lints.

There is now a stable way to define lints for a workspace. The only down
side is that every crate seems to have to opt into this so there's a
*few* files modified in this PR.

Dependencies:

- [x] PR that upgrades CI to use rust 1.74 is merged.

---------

Co-authored-by: default avatarjoe petrowski <[email protected]>
Co-authored-by: default avatarBranislav Kontur <[email protected]>
Co-authored-by: default avatarLiam Aharon <[email protected]>
      be8e6268
    • Alexandru Gheorghe's avatar
      Approve multiple candidates with a single signature (#1191) · a84dd0db
      Alexandru Gheorghe authored 
      Initial implementation for the plan discussed here: https://github.com/paritytech/polkadot-sdk/issues/701
Built on top of https://github.com/paritytech/polkadot-sdk/pull/1178
v0: https://github.com/paritytech/polkadot/pull/7554,

## Overall idea

When approval-voting checks a candidate and is ready to advertise the
approval, defer it in a per-relay chain block until we either have
MAX_APPROVAL_COALESCE_COUNT candidates to sign or a candidate has stayed
MAX_APPROVALS_COALESCE_TICKS in the queue, in both cases we sign what
candidates we have available.

This should allow us to reduce the number of approvals messages we have
to create/send/verify. The parameters are configurable, so we should
find some values that balance:

- Security of the network: Delaying broadcasting of an approval
shouldn't but the finality at risk and to make sure that never happens
we won't delay sending a vote if we are past 2/3 from the no-show time.
- Scalability of the network: MAX_APPROVAL_COALESCE_COUNT = 1 &
MAX_APPROVALS_COALESCE_TICKS =0, is what we have now and we know from
the measurements we did on versi, it bottlenecks
approval-distribution/approval-voting when increase significantly the
number of validators and parachains
- Block storage: In case of disputes we have to import this votes on
chain and that increase the necessary storage with
MAX_APPROVAL_COALESCE_COUNT * CandidateHash per vote. Given that
disputes are not the normal way of the network functioning and we will
limit MAX_APPROVAL_COALESCE_COUNT in the single digits numbers, this
should be good enough. Alternatively, we could try to create a better
way to store this on-chain through indirection, if that's needed.

## Other fixes:
- Fixed the fact that we were sending random assignments to
non-validators, that was wrong because those won't do anything with it
and they won't gossip it either because they do not have a grid topology
set, so we would waste the random assignments.
- Added metrics to be able to debug potential no-shows and
mis-processing of approvals/assignments.

## TODO:
- [x] Get feedback, that this is moving in the right direction. @ordian
@sandreim @eskimor @burdges, let me know what you think.
- [x] More and more testing.
- [x]  Test in versi.
- [x] Make MAX_APPROVAL_COALESCE_COUNT &
MAX_APPROVAL_COALESCE_WAIT_MILLIS a parachain host configuration.
- [x] Make sure the backwards compatibility works correctly
- [x] Make sure this direction is compatible with other streams of work:
https://github.com/paritytech/polkadot-sdk/issues/635 &
https://github.com/paritytech/polkadot-sdk/issues/742


- [x] Final versi burn-in before merging

---------

Signed-off-by: default avatarAlexandru Gheorghe <[email protected]>
      a84dd0db
  49. Dec 12, 2023
  51. Dec 06, 2023
  53. Dec 05, 2023
  55. Dec 01, 2023
  57. Nov 30, 2023
  59. Nov 29, 2023
  61. Nov 28, 2023