- Feb 09, 2024
-
-
Eugen Snitko authored
Add [forklift caching](https://gitlab.parity.io/parity/infrastructure/ci_cd/forklift/forklift) to remainig jobs by .sh and .py scripts: - cargo-check-each-crate x6 (`.gitlab/check-each-crate.py`) - build-linux-stable (`polkadot/scripts/build-only-wasm.sh`) by before_script: - build-linux-substrate - build-subkey-linux (with `.build-subkey` job) - cargo-check-benches x2 **To disable feature set FORKLIFT_BYPASS variable to true in [project settings in gitlab](https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/settings/ci_cd)** (forklift now handles FORKLIFT_BYPASS by itself)
-
- Feb 06, 2024
-
-
Koute authored
This PR improves compatibility with RISC-V and PolkaVM, allowing more runtimes to successfully compile. In particular, it makes the following changes: - The `sp-mmr-primitives` and `sp-consensus-beefy` crates unconditionally required an `std`-only dependency; now they only require those dependencies when the `std` feature is actually enabled. (Our RISC-V target is, unlike WASM, a true `no_std` target where you can't accidentally use stuff from `std` anymore.) - One of our dependencies (the `bitvec` trace) uses a crate called `radium` which doesn't compile under RISC-V due to incomplete autodetection logic in their `build.rs` file. The good news is that this is already fixed in the newest upstream version of `radium`, and the newest version of `bitvec` uses it. The bad news is that the newest version of `bitvec` is not currently released on crates.io, so we can't use it. I've [created an issue](https://github.com/ferrilab/ferrilab/issues/5) asking for a new release, but in the meantime I forked the currently used `radium` 0.7, [fixed the faulty logic](https://github.com/paritytech/radium-0.7-fork/commit/ed66c8a294b138c67f93499644051d97d4c7fbda) and used cargo's patching capabilities to use it for the RISC-V runtime builds. This might be a little hacky, but it is the least intrusive way to fix the problem, doesn't affect WASM builds at all, and we can trivially remove it once a new `bitvec` is released. - The new runtimes are added to the CI to make sure their compilation doesn't break.
-
- Feb 03, 2024
-
-
Koute authored
This PR adds initial support for building RISC-V runtimes targeting PolkaVM. - Setting the `SUBSTRATE_RUNTIME_TARGET=riscv` environment variable will now build a RISC-V runtime instead of a WASM runtime. - This only adds support for *building* runtimes; running them will need a PolkaVM-based executor, which I will add in a future PR. - Only building the minimal runtime is supported (building the Polkadot runtime doesn't work *yet* due to one of the dependencies). - The builder now sets a `substrate_runtime` cfg flag when building the runtimes, with the idea being that instead of doing `#[cfg(not(feature = "std"))]` or `#[cfg(target_arch = "wasm32")]` to detect that we're building a runtime you'll do `#[cfg(substrate_runtime)]`. (Switching the whole codebase to use this will be done in a future PR; I deliberately didn't do this here to keep this PR minimal and reviewable.) - Further renaming of things (e.g. types, environment variables and proc macro attributes having "wasm" in their name) to be target-agnostic will also be done in a future refactoring PR (while keeping backwards compatibility where it makes sense; I don't intend to break anyone's workflow or create unnecessary churn). - This PR also fixes two bugs in the `wasm-builder` crate: * The `RUSTC` environment variable is now removed when invoking the compiler. This prevents the toolchain version from being overridden when called from a `build.rs` script. * When parsing the `rustup toolchain list` output the `(default)` is now properly stripped and not treated as part of the version. - I've also added a minimal CI job that makes sure this doesn't break in the future. (cc @paritytech/ci) cc @athei ------ Also, just a fun little tidbit: quickly comparing the size of the built runtimes it seems that the PolkaVM runtime is slightly smaller than the WASM one. (`production` build, with the `names` section substracted from the WASM's size to keep things fair, since for the PolkaVM runtime we're currently stripping out everything) - `.wasm`: 625505 bytes - `.wasm` (after wasm-opt -O3): 563205 bytes - `.wasm` (after wasm-opt -Os): 562987 bytes - `.wasm` (after wasm-opt -Oz): 536852 bytes - `.polkavm`: ~~580338 bytes~~ 550476 bytes (after enabling extra target features; I'll add those in another PR once we have an executor working) --------- Co-authored-by:
Bastian Köcher <[email protected]>
-
- Feb 02, 2024
-
-
Alexandru Gheorghe authored
Fixes: https://github.com/paritytech/polkadot-sdk/issues/3165 --------- Signed-off-by:
Alexandru Gheorghe <[email protected]>
-
- Jan 31, 2024
-
-
Branislav Kontur authored
Found it when trying to run: ``` bot bench-all pallet --pallet=pallet_balances ``` https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5081585 ## TODO - [x] check/fix command bot for `people-westend-dev` / `people-rococo-dev` https://github.com/paritytech/command-bot-scripts/pull/67
-
- Jan 26, 2024
-
-
Svyatoslav Nikolsky authored
closes https://github.com/paritytech/parity-bridges-common/issues/2796 This partially reverts the #2439 - there are some changes (unrelated to CI) that we still want to keep. The reason of that removal is that with async backing enabled for Rococo AH (and for other chains in the near future), we see a lot of issues there (because we run `14` nodes + additional standalone process within a same container and it causes a lot of timeouts). There's no way known to me to fix it right now, so we're removing those tests hopefully temporarily to keep CI green
-
- Jan 15, 2024
-
-
Serban Iorga authored
Closes https://github.com/paritytech/polkadot-sdk/issues/2787 Regenerating the DB snapshot after fixing: https://github.com/paritytech/polkadot-sdk/pull/2919 Now we can add some BEEFY checks to `0002-validators-warp-sync`
-
- Jan 12, 2024
-
-
Javier Viola authored
This version includes - Performance improvements. - Minor fixes.
-
Svyatoslav Nikolsky authored
Brridges zombienet tests are non-standard - zombienet currently missing multiple relay chains support (see e.g. https://github.com/paritytech/zombienet/pull/796), so we need to go live with two relay networks, their parachains + custom test runner (which e.g. doesn't shutdown net when its tests are finished and instead waits for both networks tests to complete). So we are stuck with native zombienet provider => this PR is an attempt to gather everything in a single docker container and run tests there ~Draft, because it is far from finishing - what I want now is to see how it works on CI~
-
- Jan 11, 2024
-
-
Alexander Samusev authored
<del>PR custom review is deprecated. Now `review-bot` performs these functions.</del> PR removes unused ci jobs and adjusts zombienet jobs for merge queues cc @Bullrich
-
Serban Iorga authored
Part of https://github.com/paritytech/polkadot-sdk/issues/2787 This is an initial PR that adds some basic BEEFY checks to the warp sync zombienet tests. To be more specific, it does the following: - Changes the snapshot used by the warp sync zombienet tests to one built from an updated version of the kitchensink runtime, that supports BEEFY - Adds some basic BEEFY checks to the warp sync zombienet tests - Deduplicates some params of the warp sync zombienet tests, making them easier to extend
-
- Jan 10, 2024
-
-
ordian authored
Closes #1591. The purpose of this PR is filter out backing statements from the network signed by disabled validators. This is just an optimization, since we will do filtering in the runtime in #1863 to avoid nodes to filter garbage out at block production time. - [x] Ensure it's ok to fiddle with the mask of manifests - [x] Write more unit tests - [x] Test locally - [x] simple zombienet test - [x] PRDoc --------- Co-authored-by:
Tsvetomir Dimitrov <[email protected]>
-
- Jan 08, 2024
-
-
Bastian Köcher authored
This adds a Zombienet test for Coretime. Requires: https://github.com/paritytech/polkadot-sdk/pull/2862 --------- Co-authored-by:
Javier Viola <[email protected]> Co-authored-by:
Javier Viola <[email protected]>
-
PG Herveou authored
- Keep feature name short s/riscv-experimental/riscv - Add to feature to CI build --------- Co-authored-by:
Bastian Köcher <[email protected]>
-
- Dec 31, 2023
-
-
André Silva authored
-
- Dec 20, 2023
-
-
Dónal Murray authored
Many clippy lints usually enforced by `-Dcomplexity` and `-Dcorrectness` are not caught by CI as they are gated by `features`, like `runtime-benchmarks`, while the clippy CI job runs with only the default features for all targets. This PR also adds a CI step to run clippy with `--all-features` to ensure the code quality is maintained behind feature gates from now on. To improve local development, clippy lints are downgraded to warnings, but they still will result in an error at CI due to the `-Dwarnings` rustflag. --------- Co-authored-by:
Liam Aharon <[email protected]>
-
- Dec 19, 2023
-
-
joe petrowski authored
New runtimes for the Coretime Chain (a.k.a. "Broker Chain") described in RFC-1. Replaces https://github.com/paritytech/cumulus/pull/2889 - [x] Add Agile Coretime pallet https://github.com/paritytech/substrate/pull/14568 - [x] Generate chain specs for local and testnets - [x] Deploy parachain on Rococo - Done: [rococo-coretime-rpc.polkadot.io](https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Frococo-coretime-rpc.polkadot.io#/explorer) DevOps issue for Aura keygen: https://github.com/paritytech/devops/issues/2725 Edit (Dónal): This PR is mainly for Rococo, the Westend runtime is a shell with no `Broker` pallet. The Rococo runtime has the broker calls filtered for initial deployment. --------- Co-authored-by:
Dónal Murray <[email protected]> Co-authored-by:
0xmovses <[email protected]> Co-authored-by:
Liam Aharon <[email protected]> Co-authored-by:
Bastian Köcher <[email protected]> Co-authored-by:
Marcin S. <[email protected]> Co-authored-by:
Bastian Köcher <[email protected]> Co-authored-by: command-bot <> Co-authored-by:
Branislav Kontur <[email protected]>
-
- Dec 18, 2023
-
-
Alexander Samusev authored
After GitLab update it started to remove artifacts from master if PR has more than 20 commits. PR fixes this for `node-bench-regression-guard` job cc https://github.com/paritytech/ci_cd/issues/915
-
- Dec 15, 2023
-
-
Javier Viola authored
Remove old version for `cli_args`, since this was fixed in the latest version of zombienet and the `latest` version of polkadot introduce the new flag `--insecure-validator-i-know-what-i-do`. Fix jobs like https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4726174 Thx!
-
- Dec 13, 2023
-
-
Squirrel authored
We currently use a bit of a hack in `.cargo/config` to make sure that clippy isn't too annoying by specifying the list of lints. There is now a stable way to define lints for a workspace. The only down side is that every crate seems to have to opt into this so there's a *few* files modified in this PR. Dependencies: - [x] PR that upgrades CI to use rust 1.74 is merged. --------- Co-authored-by:
joe petrowski <[email protected]> Co-authored-by:
Branislav Kontur <[email protected]> Co-authored-by:
Liam Aharon <[email protected]>
-
Alexandru Gheorghe authored
Initial implementation for the plan discussed here: https://github.com/paritytech/polkadot-sdk/issues/701 Built on top of https://github.com/paritytech/polkadot-sdk/pull/1178 v0: https://github.com/paritytech/polkadot/pull/7554, ## Overall idea When approval-voting checks a candidate and is ready to advertise the approval, defer it in a per-relay chain block until we either have MAX_APPROVAL_COALESCE_COUNT candidates to sign or a candidate has stayed MAX_APPROVALS_COALESCE_TICKS in the queue, in both cases we sign what candidates we have available. This should allow us to reduce the number of approvals messages we have to create/send/verify. The parameters are configurable, so we should find some values that balance: - Security of the network: Delaying broadcasting of an approval shouldn't but the finality at risk and to make sure that never happens we won't delay sending a vote if we are past 2/3 from the no-show time. - Scalability of the network: MAX_APPROVAL_COALESCE_COUNT = 1 & MAX_APPROVALS_COALESCE_TICKS =0, is what we have now and we know from the measurements we did on versi, it bottlenecks approval-distribution/approval-voting when increase significantly the number of validators and parachains - Block storage: In case of disputes we have to import this votes on chain and that increase the necessary storage with MAX_APPROVAL_COALESCE_COUNT * CandidateHash per vote. Given that disputes are not the normal way of the network functioning and we will limit MAX_APPROVAL_COALESCE_COUNT in the single digits numbers, this should be good enough. Alternatively, we could try to create a better way to store this on-chain through indirection, if that's needed. ## Other fixes: - Fixed the fact that we were sending random assignments to non-validators, that was wrong because those won't do anything with it and they won't gossip it either because they do not have a grid topology set, so we would waste the random assignments. - Added metrics to be able to debug potential no-shows and mis-processing of approvals/assignments. ## TODO: - [x] Get feedback, that this is moving in the right direction. @ordian @sandreim @eskimor @burdges, let me know what you think. - [x] More and more testing. - [x] Test in versi. - [x] Make MAX_APPROVAL_COALESCE_COUNT & MAX_APPROVAL_COALESCE_WAIT_MILLIS a parachain host configuration. - [x] Make sure the backwards compatibility works correctly - [x] Make sure this direction is compatible with other streams of work: https://github.com/paritytech/polkadot-sdk/issues/635 & https://github.com/paritytech/polkadot-sdk/issues/742 - [x] Final versi burn-in before merging --------- Signed-off-by:
Alexandru Gheorghe <[email protected]>
-
- Dec 12, 2023
-
-
Branislav Kontur authored
## Summary This PR turns on `-D warnings` for `cargo-check-each-crate job` job to fail on warnings e.g. like this: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4673130 Before this PR, there was a warning and `cargo-check-each-crate` job did not fail: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4641444 ``` warning: unused import: `ToTokens` --> substrate/primitives/api/proc-macro/src/utils.rs:22:34 | 22 | use quote::{format_ident, quote, ToTokens}; | ^^^^^^^^ | = note: `#[warn(unused_imports)]` on by default warning: `sp-api-proc-macro` (lib) generated 1 warning (run `cargo fix --lib -p sp-api-proc-macro` to apply 1 suggestion) ``` Fixes on the way: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4641444 https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4673265 https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4673410 https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4673681 https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4673836 https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4673941 https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4674256 https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4679328 ## Questions - [ ] why does this check triggers only `cargo check --locked`? `--all-features` or `--all-targets` are not needed? Or aren't they avoided intentionally? --------- Co-authored-by: command-bot <>
-
- Dec 06, 2023
-
-
Alexander Samusev authored
-
- Dec 05, 2023
-
-
Marcin S. authored
Co-authored-by:
Javier Viola <[email protected]>
-
juangirini authored
This PR is a continuation of https://github.com/paritytech/polkadot-sdk/pull/2102 and part of an initiative started here https://hackmd.io/@romanp/rJ318ZCEp What has been done: - The content under `docs/*` (with the exception of `docs/mermaid`) has been moved to `docs/contributor/` - Developer Hub has been renamed to Polkadot SDK Docs, and the crate has been renamed from `developer-hub` to `polkadot-sdk-docs` - The content under `developer-hub/*` has been moved to `docs/sdk` --- Original PR https://github.com/paritytech/polkadot-sdk/pull/2565 , it has been close due to too many rebase conflicts --------- Co-authored-by:
Serban Iorga <[email protected]> Co-authored-by:
Chevdor <[email protected]> Co-authored-by:
Egor_P <[email protected]> Co-authored-by:
Bastian Köcher <[email protected]>
-
Vladimir Istyufeev authored
-
- Dec 01, 2023
-
-
Liam Aharon authored
Using taplo, fixes all our broken and inconsistent toml formatting and adds CI to keep them tidy. If people want we can customise the format rules as described here https://taplo.tamasfe.dev/configuration/formatter-options.html @ggwpez , I suggest zepter is used only for checking features are propagated, and leave formatting for taplo to avoid duplicate work and conflicts. TODO - [x] Use `exclude = [...]` syntax in taplo file to ignore zombienet tests instead of deleting the dir --------- Signed-off-by:
Oliver Tale-Yazdi <[email protected]> Co-authored-by:
Oliver Tale-Yazdi <[email protected]> Co-authored-by:
Bastian Köcher <[email protected]>
-
- Nov 30, 2023
-
-
Kian Paimani authored
This PR introduces the new crate `developer_hub` into the polkadot-sdk repo. The vision for the developer-hub crate is detailed in [this document](https://docs.google.com/document/d/1XLLkFNE8v8HLvZpI2rzsa8N2IN1FcKntc8q-Sc4xBAk/edit?usp=sharing). <img width="1128" alt="Screenshot 2023-11-02 at 10 45 48" src="https://github.com/paritytech/polkadot-sdk/assets/5588131/1e12b60f-fef5-42c4-8503-a3ba234077c3"> Other than adding the new crate, it also does the following: * Remove the `substrate` crate, as there is now a unique umbrella crate for multiple things in `developer_hub::polkadot_sdk`. * (backport candidate) A minor change to `frame-support` macros that allows `T::RuntimeOrigin` to also be acceptable as the origin type. * (backport candidate) A minor change to `frame-system` that allows us to deposit events at genesis because now the real genesis config is generated via wasm, and we can safely assume `cfg!(feature = "std")` means only testing. related to #62. * (backport candidate) Introduces a small `read_events_for_pallet` to `frame_system` for easier event reading in tests. * From https://github.com/paritytech/polkadot-sdk-docs/issues/31, it takes action on improving the `pallet::call` docs. * From https://github.com/paritytech/polkadot-sdk-docs/issues/31, it takes action on improving the `UncheckedExtrinsic` docs. ## Way Forward First, a version of this is deployed temporarily [here](https://blog.kianenigma.nl/polkadot-sdk/developer_hub/index.html). I will keep it up to date on a daily basis. ### This Pull Request I see two ways forward: 1. We acknowledge that everything in `developer-hub` is going to be WIP, and merge this asap. We should not yet use links to this crate anywhere. 2. We make this be the feature branch, make PRs against this, and either gradually backport it, or only merge to master once it is done. I am personally in favor of option 1. If we stick to option 2, we need a better way to deploy a staging version of this to gh-pages. ### Issue Tracking The main issues related to the future of `developer_hub` are: - https://github.com/paritytech/polkadot-sdk-docs/issues/31 - https://github.com/paritytech/polkadot-sdk-docs/issues/4 - https://github.com/paritytech/polkadot-sdk-docs/issues/26 - https://github.com/paritytech/polkadot-sdk-docs/issues/32 - https://github.com/paritytech/polkadot-sdk-docs/issues/36 ### After This Pull Request - [ ] create a redirect for https://paritytech.github.io/polkadot-sdk/master/substrate/ - [x] analytics - [ ] link checker - [ ] the matter of publishing, and how all of these relative links for when we do, that is still an open question. There is section on this in the landing page. - [ ] updated https://paritytech.github.io/ --------- Co-authored-by:
Liam Aharon <[email protected]> Co-authored-by:
Juan Girini <[email protected]> Co-authored-by:
bader y <[email protected]> Co-authored-by:
Sebastian Kunert <[email protected]> Co-authored-by:
James Wilson <[email protected]> Co-authored-by:
Michal Kucharczyk <[email protected]>
-
- Nov 29, 2023
-
-
Vladimir Istyufeev authored
-
Dónal Murray authored
Since the Polkadot and Kusama runtimes are no longer in the repo, the relevant systems parachains runtimes also need to be removed. More context [here](https://github.com/paritytech/polkadot-sdk/issues/603) and [here](https://github.com/paritytech/polkadot-sdk/pull/1731). Removes the following: - `asset-hub-kusama` and `asset-hub-polkadot` - `bridge-hub-kusama` and `bridge-hub-polkadot` - `collectives-polkadot` - `glutton-kusama` Partially solves #603 and adds to #1731.
-
- Nov 28, 2023
-
-
s0me0ne-unkn0wn authored
Co-authored-by:
ordian <[email protected]> Co-authored-by:
Vladimir Istyufeev <[email protected]>
-
Vladimir Istyufeev authored
-
- Nov 27, 2023
-
-
Maciej authored
**Overview:** Adding an extra malus variant focusing on disputing finalized blocks. It will: - wrap around approval-voting - listen to `OverseerSignal::BlockFinalized` and when encountered start a dispute for the `dispute_offset`th ancestor - simply pass through all other messages and signals Add zombienet tests testing various edgecases: - disputing freshly finalized blocks - disputing stale finalized blocks - disputing eagerly pruned finalized blocks (might be separate PR) **TODO:** - [x] Register new malus variant - [x] Simple pass through wrapper (approval-voting) - [x] Simple network definition - [x] Listen to block finalizations - [x] Fetch ancestor hash - [x] Fetch session index - [x] Fetch candidate - [x] Construct and send dispute message - [x] zndsl test 1 checking that disputes on fresh finalizations resolve valid Closes #1365 - [x] zndsl test 2 checking that disputes for too old finalized blocks are not possible Closes #1364 - [ ] zndsl test 3 checking that disputes for candidates with eagerly pruned relay parent state are handled correctly #1359 (deferred to a separate PR) - [x] Unit tests for new malus variant (testing cli etc) - [x] Clean/streamline error handling - [ ] ~~Ensure it tests properly on session boundaries~~ --------- Co-authored-by:
Javier Viola <[email protected]> Co-authored-by:
Marcin S. <[email protected]> Co-authored-by:
Tsvetomir Dimitrov <[email protected]>
-
- Nov 25, 2023
-
-
Marcin S. authored
-
- Nov 24, 2023
-
-
Javier Viola authored
Add `build-push-image-polkadot-debug` job to needs since we changed to use the polkadot-debug image from the current branch for cumulus test and we need to be sure that the image is ready. Fix issues like https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4481059 cc: @bkchr Co-authored-by:
Bastian Köcher <[email protected]>
-
Javier Viola authored
Include fix for `0002-parachains-upgrade-smoke-test` test.
-
- Nov 23, 2023
-
-
Liam Aharon authored
CI: Disable runtime upgrade spec name check on Westend Asset Hub and fix Staking pallet migration (#2447) Westend Asset Hub currently failing because the spec name is being changed next runtime upgrade (https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4413125). This also fixes an idempotency issue with a staking pallet migration. Similar issues will be caught automatically now that we've also updated to try-runtime-cli v0.5.0 which checks for idempotency issues. This also enables try-state checks running in the CI.
-
- Nov 22, 2023
-
-
Michal Kucharczyk authored
The out-dated version (bad tag) of [polkadot image](https://github.com/paritytech/polkadot-sdk/blob/ede4a362622dfa69eb167eaa876246b1289f4b41/.gitlab/pipeline/zombienet/cumulus.yml#L31) ([docker info](https://hub.docker.com/layers/paritypr/polkadot-debug/master/images/sha256:adb1658052cf671b50c90d5cece5c7a131efa1a95978249bd5cb85a5ad654f7a?context=explore)) was used. This PR fixes this. See also: https://github.com/paritytech/polkadot-sdk/pull/2411#issuecomment-1822632724 Also adds an abstraction that allows asynchronous backends to be passed to `ChainApiSubsystem` --------- Co-authored-by:
Sebastian Kunert <[email protected]>
-
Marcin S. authored
-
Alexander Samusev authored
Docker images from paritypr are also used in testnets. PR reverts docker tags naming to a more convenient. cc @PierreBesson
-