- Jan 10, 2024
-
-
dependabot[bot] authored
Bumps [parking_lot](https://github.com/Amanieu/parking_lot) from 0.11.2 to 0.12.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md">parking_lot's changelog</a>.</em></p> <blockquote> <h2>parking_lot 0.12.1 (2022-05-31)</h2> <ul> <li>Fixed incorrect memory ordering in <code>RwLock</code>. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/344">#344</a>)</li> <li>Added <code>Condvar::wait_while</code> convenience methods (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/343">#343</a>)</li> </ul> <h2>parking_lot_core 0.9.3 (2022-04-30)</h2> <ul> <li>Bump windows-sys dependency to 0.36. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/339">#339</a>)</li> </ul> <h2>parking_lot_core 0.9.2, lock_api 0.4.7 (2022-03-25)</h2> <ul> <li>Enable const new() on lock types on stable. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/325">#325</a>)</li> <li>Added <code>MutexGuard::leak</code> function. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/333">#333</a>)</li> <li>Bump windows-sys dependency to 0.34. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/331">#331</a>)</li> <li>Bump petgraph dependency to 0.6. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/326">#326</a>)</li> <li>Don't use pthread attributes on the espidf platform. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/319">#319</a>)</li> </ul> <h2>parking_lot_core 0.9.1 (2022-02-06)</h2> <ul> <li>Bump windows-sys dependency to 0.32. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/316">#316</a>)</li> </ul> <h2>parking_lot 0.12.0, parking_lot_core 0.9.0, lock_api 0.4.6 (2022-01-28)</h2> <ul> <li>The MSRV is bumped to 1.49.0.</li> <li>Disabled eventual fairness on wasm32-unknown-unknown. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/302">#302</a>)</li> <li>Added a rwlock method to report if lock is held exclusively. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/303">#303</a>)</li> <li>Use new <code>asm!</code> macro. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/304">#304</a>)</li> <li>Use windows-rs instead of winapi for faster builds. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/311">#311</a>)</li> <li>Moved hardware lock elision support to a separate Cargo feature. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/313">#313</a>)</li> <li>Removed used of deprecated <code>spin_loop_hint</code>. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/314">#314</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/Amanieu/parking_lot/commit/336a9b31ff385728d00eb7ef173e4d054584b787"><code>336a9b3</code></a> Release parking_lot 0.12.1</li> <li><a href="https://github.com/Amanieu/parking_lot/commit/b69a0547ce6c92b164528c65daafa48e8f1c08bb"><code>b69a054</code></a> Merge pull request <a href="https://redirect.github.com/Amanieu/parking_lot/issues/343">#343</a> from bryanhitc/master</li> <li><a href="https://github.com/Amanieu/parking_lot/commit/3fe7233ee05fbae9bc94a831d4e66df0257880a2"><code>3fe7233</code></a> Merge pull request <a href="https://redirect.github.com/Amanieu/parking_lot/issues/344">#344</a> from Amanieu/fix_rwlock_ordering</li> <li><a href="https://github.com/Amanieu/parking_lot/commit/ef12b00daf0dbc6dd025098ec2cd0517bb9f737c"><code>ef12b00</code></a> small test update</li> <li><a href="https://github.com/Amanieu/parking_lot/commit/fdb063cd4e33d9c44cc08aea0c341ffec7e7b282"><code>fdb063c</code></a> wait_while can't timeout fix</li> <li><a href="https://github.com/Amanieu/parking_lot/commit/26e19dced4ae01000abc0dfca251f56695b1b8aa"><code>26e19dc</code></a> Remove WaitWhileResult</li> <li><a href="https://github.com/Amanieu/parking_lot/commit/d26c284fe8413bdeeed3ef4f70e23b5919a9ffeb"><code>d26c284</code></a> Fix incorrect memory ordering in RwLock</li> <li><a href="https://github.com/Amanieu/parking_lot/commit/045828381a5facff71b1aad3c01af15801959458"><code>0458283</code></a> Use saturating_add for WaitWhileResult</li> <li><a href="https://github.com/Amanieu/parking_lot/commit/686db4755913947219f83a04b21885032685d246"><code>686db47</code></a> Add Condvar::wait_while convenience methods</li> <li><a href="https://github.com/Amanieu/parking_lot/commit/6f6e021ced6f67fef8002442aa97c4a08ed5c3e4"><code>6f6e021</code></a> Merge pull request <a href="https://redirect.github.com/Amanieu/parking_lot/issues/342">#342</a> from MarijnS95/patch-1</li> <li>Additional commits viewable in <a href="https://github.com/Amanieu/parking_lot/compare/0.11.2...0.12.1">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=parking_lot&package-manager=cargo&previous-version=0.11.2&new-version=0.12.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores ) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by:
dependabot[bot] <[email protected]> Co-authored-by:
dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
-
ordian authored
Closes #1591. The purpose of this PR is filter out backing statements from the network signed by disabled validators. This is just an optimization, since we will do filtering in the runtime in #1863 to avoid nodes to filter garbage out at block production time. - [x] Ensure it's ok to fiddle with the mask of manifests - [x] Write more unit tests - [x] Test locally - [x] simple zombienet test - [x] PRDoc --------- Co-authored-by:
Tsvetomir Dimitrov <[email protected]>
-
- Dec 19, 2023
-
-
André Silva authored
-
- Dec 18, 2023
-
-
Branislav Kontur authored
This PR contains just a few clippy fixes and nits, which are, however, relaxed by workspace clippy settings here: https://github.com/paritytech/polkadot-sdk/blob/master/Cargo.toml#L483-L506 --------- Co-authored-by:
Dmitry Sinyavin <[email protected]> Co-authored-by:
ordian <[email protected]> Co-authored-by: command-bot <> Co-authored-by:
Bastian Köcher <[email protected]>
-
dependabot[bot] authored
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.73 to 0.1.74. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/dtolnay/async-trait/releases">async-trait's releases</a>.</em></p> <blockquote> <h2>0.1.74</h2> <ul> <li>Documentation improvements</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/dtolnay/async-trait/commit/265979b07a9af573e1edd3b2a9b179533cfa7a6c"><code>265979b</code></a> Release 0.1.74</li> <li><a href="https://github.com/dtolnay/async-trait/commit/5e677097d2e67f7a5c5e3023e2f3b99b36a9e132"><code>5e67709</code></a> Fix doc test when async fn in trait is natively supported</li> <li><a href="https://github.com/dtolnay/async-trait/commit/ef144aed28b636eb65759505b2323afc4c753fbe"><code>ef144ae</code></a> Update ui test suite to nightly-2023-10-15</li> <li><a href="https://github.com/dtolnay/async-trait/commit/9398a28d6fc977ccf8c286bd85b4b87a883f92ac"><code>9398a28</code></a> Test docs.rs documentation build in CI</li> <li><a href="https://github.com/dtolnay/async-trait/commit/8737173dafa371e5e9e491d736513be1baf697f4"><code>8737173</code></a> Update ui test suite to nightly-2023-09-24</li> <li><a href="https://github.com/dtolnay/async-trait/commit/5ba643c001a55f70c4a44690e040cdfab873ba56"><code>5ba643c</code></a> Test dyn Trait containing async fn</li> <li><a href="https://github.com/dtolnay/async-trait/commit/247c8e7b0b3ff69c9518ebf93e69fe74d47f17b6"><code>247c8e7</code></a> Add ui test testing the recommendation to use async-trait</li> <li><a href="https://github.com/dtolnay/async-trait/commit/799db66a84834c403860df4a8c0227d8fb7f9d9d"><code>799db66</code></a> Update ui test suite to nightly-2023-09-23</li> <li><a href="https://github.com/dtolnay/async-trait/commit/0e60248011f751d8ccf58219d0a79aacfe9619f1"><code>0e60248</code></a> Update actions/checkout@v3 -> v4</li> <li><a href="https://github.com/dtolnay/async-trait/commit/7fcbc83993d5ef483d048c271a7f6c4ac8c98388"><code>7fcbc83</code></a> Update ui test suite to nightly-2023-08-29</li> <li>See full diff in <a href="https://github.com/dtolnay/async-trait/compare/0.1.73...0.1.74">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=async-trait&package-manager=cargo&previous-version=0.1.73&new-version=0.1.74)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores ) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by:
dependabot[bot] <[email protected]> Co-authored-by:
dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
-
- Dec 14, 2023
-
-
Andrei Sandu authored
This tool makes it easy to run parachain consensus stress/performance testing on your development machine or in CI. ## Motivation The parachain consensus node implementation spans across many modules which we call subsystems. Each subsystem is responsible for a small part of logic of the parachain consensus pipeline, but in general the most load and performance issues are localized in just a few core subsystems like `availability-recovery`, `approval-voting` or `dispute-coordinator`. In the absence of such a tool, we would run large test nets to load/stress test these parts of the system. Setting up and making sense of the amount of data produced by such a large test is very expensive, hard to orchestrate and is a huge development time sink. ## PR contents - CLI tool - Data Availability Read test - reusable mockups and components needed so far - Documentation on how to get started ### Data Availability Read test An overseer is built with using a real `availability-recovery` susbsytem instance while dependent subsystems like `av-store`, `network-bridge` and `runtime-api` are mocked. The network bridge will emulate all the network peers and their answering to requests. The test is going to be run for a number of blocks. For each block it will generate send a “RecoverAvailableData” request for an arbitrary number of candidates. We wait for the subsystem to respond to all requests before moving to the next block. At the same time we collect the usual subsystem metrics and task CPU metrics and show some nice progress reports while running. ### Here is how the CLI looks like: ``` [2023-11-28T13:06:27Z INFO subsystem_bench::core::display] n_validators = 1000, n_cores = 20, pov_size = 5120 - 5120, error = 3, latency = Some(PeerLatency { min_latency: 1ms, max_latency: 100ms }) [2023-11-28T13:06:27Z INFO subsystem-bench::availability] Generating template candidate index=0 pov_size=5242880 [2023-11-28T13:06:27Z INFO subsystem-bench::availability] Created test environment. [2023-11-28T13:06:27Z INFO subsystem-bench::availability] Pre-generating 60 candidates. [2023-11-28T13:06:30Z INFO subsystem-bench::core] Initializing network emulation for 1000 peers. [2023-11-28T13:06:30Z INFO subsystem-bench::availability] Current block 1/3 [2023-11-28T13:06:30Z INFO substrate_prometheus_endpoint]
〽 ️ Prometheus exporter started at 127.0.0.1:9999 [2023-11-28T13:06:30Z INFO subsystem_bench::availability] 20 recoveries pending [2023-11-28T13:06:37Z INFO subsystem_bench::availability] Block time 6262ms [2023-11-28T13:06:37Z INFO subsystem-bench::availability] Sleeping till end of block (0ms) [2023-11-28T13:06:37Z INFO subsystem-bench::availability] Current block 2/3 [2023-11-28T13:06:37Z INFO subsystem_bench::availability] 20 recoveries pending [2023-11-28T13:06:43Z INFO subsystem_bench::availability] Block time 6369ms [2023-11-28T13:06:43Z INFO subsystem-bench::availability] Sleeping till end of block (0ms) [2023-11-28T13:06:43Z INFO subsystem-bench::availability] Current block 3/3 [2023-11-28T13:06:43Z INFO subsystem_bench::availability] 20 recoveries pending [2023-11-28T13:06:49Z INFO subsystem_bench::availability] Block time 6194ms [2023-11-28T13:06:49Z INFO subsystem-bench::availability] Sleeping till end of block (0ms) [2023-11-28T13:06:49Z INFO subsystem_bench::availability] All blocks processed in 18829ms [2023-11-28T13:06:49Z INFO subsystem_bench::availability] Throughput: 102400 KiB/block [2023-11-28T13:06:49Z INFO subsystem_bench::availability] Block time: 6276 ms [2023-11-28T13:06:49Z INFO subsystem_bench::availability] Total received from network: 415 MiB Total sent to network: 724 KiB Total subsystem CPU usage 24.00s CPU usage per block 8.00s Total test environment CPU usage 0.15s CPU usage per block 0.05s ``` ### Prometheus/Grafana stack in action <img width="1246" alt="Screenshot 2023-11-28 at 15 11 10" src="https://github.com/paritytech/polkadot-sdk/assets/54316454/eaa47422-4a5e-4a3a-aaef-14ca644c1574"> <img width="1246" alt="Screenshot 2023-11-28 at 15 12 01" src="https://github.com/paritytech/polkadot-sdk/assets/54316454/237329d6-1710-4c27-8f67-5fb11d7f66ea"> <img width="1246" alt="Screenshot 2023-11-28 at 15 12 38" src="https://github.com/paritytech/polkadot-sdk/assets/54316454/a07119e8-c9f1-4810-a1b3-f1b7b01cf357 "> --------- Signed-off-by:Andrei Sandu <[email protected]>
-
- Dec 13, 2023
-
-
Squirrel authored
We currently use a bit of a hack in `.cargo/config` to make sure that clippy isn't too annoying by specifying the list of lints. There is now a stable way to define lints for a workspace. The only down side is that every crate seems to have to opt into this so there's a *few* files modified in this PR. Dependencies: - [x] PR that upgrades CI to use rust 1.74 is merged. --------- Co-authored-by:
joe petrowski <[email protected]> Co-authored-by:
Branislav Kontur <[email protected]> Co-authored-by:
Liam Aharon <[email protected]>
-
Alexandru Gheorghe authored
Initial implementation for the plan discussed here: https://github.com/paritytech/polkadot-sdk/issues/701 Built on top of https://github.com/paritytech/polkadot-sdk/pull/1178 v0: https://github.com/paritytech/polkadot/pull/7554, ## Overall idea When approval-voting checks a candidate and is ready to advertise the approval, defer it in a per-relay chain block until we either have MAX_APPROVAL_COALESCE_COUNT candidates to sign or a candidate has stayed MAX_APPROVALS_COALESCE_TICKS in the queue, in both cases we sign what candidates we have available. This should allow us to reduce the number of approvals messages we have to create/send/verify. The parameters are configurable, so we should find some values that balance: - Security of the network: Delaying broadcasting of an approval shouldn't but the finality at risk and to make sure that never happens we won't delay sending a vote if we are past 2/3 from the no-show time. - Scalability of the network: MAX_APPROVAL_COALESCE_COUNT = 1 & MAX_APPROVALS_COALESCE_TICKS =0, is what we have now and we know from the measurements we did on versi, it bottlenecks approval-distribution/approval-voting when increase significantly the number of validators and parachains - Block storage: In case of disputes we have to import this votes on chain and that increase the necessary storage with MAX_APPROVAL_COALESCE_COUNT * CandidateHash per vote. Given that disputes are not the normal way of the network functioning and we will limit MAX_APPROVAL_COALESCE_COUNT in the single digits numbers, this should be good enough. Alternatively, we could try to create a better way to store this on-chain through indirection, if that's needed. ## Other fixes: - Fixed the fact that we were sending random assignments to non-validators, that was wrong because those won't do anything with it and they won't gossip it either because they do not have a grid topology set, so we would waste the random assignments. - Added metrics to be able to debug potential no-shows and mis-processing of approvals/assignments. ## TODO: - [x] Get feedback, that this is moving in the right direction. @ordian @sandreim @eskimor @burdges, let me know what you think. - [x] More and more testing. - [x] Test in versi. - [x] Make MAX_APPROVAL_COALESCE_COUNT & MAX_APPROVAL_COALESCE_WAIT_MILLIS a parachain host configuration. - [x] Make sure the backwards compatibility works correctly - [x] Make sure this direction is compatible with other streams of work: https://github.com/paritytech/polkadot-sdk/issues/635 & https://github.com/paritytech/polkadot-sdk/issues/742 - [x] Final versi burn-in before merging --------- Signed-off-by:
Alexandru Gheorghe <[email protected]>
-
- Dec 11, 2023
-
-
Alexandru Gheorghe authored
Scaffold everything, so that we can enable v2 assignments via a node feature bit, once all nodes have upgraded to the new protocol. Implements: https://github.com/paritytech/polkadot-sdk/issues/628 --------- Signed-off-by:
Alexandru Gheorghe <[email protected]>
-
- Dec 06, 2023
-
-
André Silva authored
This will make more sense after https://github.com/paritytech/polkadot-sdk/pull/2524 since the schnorrkel type for VRF outputs is also renamed in the latest version. Can be reviewed independently though. Can be merged after https://github.com/paritytech/polkadot-sdk/pull/1577 so that there is less pain for @davxy . --------- Co-authored-by:
Bastian Köcher <[email protected]>
-
- Dec 05, 2023
-
-
Marcin S. authored
-
- Dec 01, 2023
-
-
Liam Aharon authored
Using taplo, fixes all our broken and inconsistent toml formatting and adds CI to keep them tidy. If people want we can customise the format rules as described here https://taplo.tamasfe.dev/configuration/formatter-options.html @ggwpez , I suggest zepter is used only for checking features are propagated, and leave formatting for taplo to avoid duplicate work and conflicts. TODO - [x] Use `exclude = [...]` syntax in taplo file to ignore zombienet tests instead of deleting the dir --------- Signed-off-by:
Oliver Tale-Yazdi <[email protected]> Co-authored-by:
Oliver Tale-Yazdi <[email protected]> Co-authored-by:
Bastian Köcher <[email protected]>
-
- Nov 30, 2023
-
-
asynchronous rob authored
In gossip-support, we shuffled the list of authorities using the `rand::seq::SliceRandom::shuffle`. This function's behavior is unspecified beyond being `O(n)` and could change in the future, leading to network issues between nodes using different shuffling algorithms. In practice, the implementation was a Fisher-Yates shuffle. This PR replaces the call with a re-implementation of Fisher-Yates and adds a test to ensure the behavior is the same between the two at the moment.
-
Kian Paimani authored
This PR introduces the new crate `developer_hub` into the polkadot-sdk repo. The vision for the developer-hub crate is detailed in [this document](https://docs.google.com/document/d/1XLLkFNE8v8HLvZpI2rzsa8N2IN1FcKntc8q-Sc4xBAk/edit?usp=sharing). <img width="1128" alt="Screenshot 2023-11-02 at 10 45 48" src="https://github.com/paritytech/polkadot-sdk/assets/5588131/1e12b60f-fef5-42c4-8503-a3ba234077c3"> Other than adding the new crate, it also does the following: * Remove the `substrate` crate, as there is now a unique umbrella crate for multiple things in `developer_hub::polkadot_sdk`. * (backport candidate) A minor change to `frame-support` macros that allows `T::RuntimeOrigin` to also be acceptable as the origin type. * (backport candidate) A minor change to `frame-system` that allows us to deposit events at genesis because now the real genesis config is generated via wasm, and we can safely assume `cfg!(feature = "std")` means only testing. related to #62. * (backport candidate) Introduces a small `read_events_for_pallet` to `frame_system` for easier event reading in tests. * From https://github.com/paritytech/polkadot-sdk-docs/issues/31, it takes action on improving the `pallet::call` docs. * From https://github.com/paritytech/polkadot-sdk-docs/issues/31, it takes action on improving the `UncheckedExtrinsic` docs. ## Way Forward First, a version of this is deployed temporarily [here](https://blog.kianenigma.nl/polkadot-sdk/developer_hub/index.html). I will keep it up to date on a daily basis. ### This Pull Request I see two ways forward: 1. We acknowledge that everything in `developer-hub` is going to be WIP, and merge this asap. We should not yet use links to this crate anywhere. 2. We make this be the feature branch, make PRs against this, and either gradually backport it, or only merge to master once it is done. I am personally in favor of option 1. If we stick to option 2, we need a better way to deploy a staging version of this to gh-pages. ### Issue Tracking The main issues related to the future of `developer_hub` are: - https://github.com/paritytech/polkadot-sdk-docs/issues/31 - https://github.com/paritytech/polkadot-sdk-docs/issues/4 - https://github.com/paritytech/polkadot-sdk-docs/issues/26 - https://github.com/paritytech/polkadot-sdk-docs/issues/32 - https://github.com/paritytech/polkadot-sdk-docs/issues/36 ### After This Pull Request - [ ] create a redirect for https://paritytech.github.io/polkadot-sdk/master/substrate/ - [x] analytics - [ ] link checker - [ ] the matter of publishing, and how all of these relative links for when we do, that is still an open question. There is section on this in the landing page. - [ ] updated https://paritytech.github.io/ --------- Co-authored-by:
Liam Aharon <[email protected]> Co-authored-by:
Juan Girini <[email protected]> Co-authored-by:
bader y <[email protected]> Co-authored-by:
Sebastian Kunert <[email protected]> Co-authored-by:
James Wilson <[email protected]> Co-authored-by:
Michal Kucharczyk <[email protected]>
-
- Nov 28, 2023
-
-
Aaro Altonen authored
This commit introduces a new concept called `NotificationService` which allows Polkadot protocols to communicate with the underlying notification protocol implementation directly, without routing events through `NetworkWorker`. This implies that each protocol has its own service which it uses to communicate with remote peers and that each `NotificationService` is unique with respect to the underlying notification protocol, meaning `NotificationService` for the transaction protocol can only be used to send and receive transaction-related notifications. The `NotificationService` concept introduces two additional benefits: * allow protocols to start using custom handshakes * allow protocols to accept/reject inbound peers Previously the validation of inbound connections was solely the responsibility of `ProtocolController`. This caused issues with light peers and `SyncingEngine` as `ProtocolController` would accept more peers than `SyncingEngine` could accept which caused peers to have differing views of their own states. `SyncingEngine` would reject excess peers but these rejections were not properly communicated to those peers causing them to assume that they were accepted. With `NotificationService`, the local handshake is not sent to remote peer if peer is rejected which allows it to detect that it was rejected. This commit also deprecates the use of `NetworkEventStream` for all notification-related events and going forward only DHT events are provided through `NetworkEventStream`. If protocols wish to follow each other's events, they must introduce additional abtractions, as is done for GRANDPA and transactions protocols by following the syncing protocol through `SyncEventStream`. Fixes https://github.com/paritytech/polkadot-sdk/issues/512 Fixes https://github.com/paritytech/polkadot-sdk/issues/514 Fixes https://github.com/paritytech/polkadot-sdk/issues/515 Fixes https://github.com/paritytech/polkadot-sdk/issues/554 Fixes https://github.com/paritytech/polkadot-sdk/issues/556 --- These changes are transferred from https://github.com/paritytech/substrate/pull/14197 but there are no functional changes compared to that PR --------- Co-authored-by:
Dmitry Markin <[email protected]> Co-authored-by:
Alexandru Vasile <[email protected]>
-
- Nov 14, 2023
-
-
Kristian Sosnin authored
Fixes #1437 Co-authored-by:
Sophia Gold <[email protected]>
-
Alin Dima authored
Collators were previously reencoding the available data and checking the erasure root. Replace that with just checking the PoV hash, which consumes much less CPU and takes less time. We also don't need to check the `PersistedValidationData` hash, as collators don't use it. Reason: https://github.com/paritytech/polkadot-sdk/issues/575#issuecomment-1806572230 After systematic chunks recovery is merged, collators will no longer do any reed-solomon encoding/decoding, which has proven to be a great CPU consumer. Signed-off-by:
alindima <[email protected]>
-
- Nov 09, 2023
-
-
Oliver Tale-Yazdi authored
Missing descriptions (47): - [x] `cumulus/client/collator/Cargo.toml` - [x] `cumulus/client/relay-chain-inprocess-interface/Cargo.toml` - [x] `cumulus/client/cli/Cargo.toml` - [x] `cumulus/client/service/Cargo.toml` - [x] `cumulus/client/relay-chain-rpc-interface/Cargo.toml` - [x] `cumulus/client/relay-chain-interface/Cargo.toml` - [x] `cumulus/client/relay-chain-minimal-node/Cargo.toml` - [x] `cumulus/parachains/pallets/parachain-info/Cargo.toml` - [x] `cumulus/parachains/pallets/ping/Cargo.toml` - [x] `cumulus/primitives/utility/Cargo.toml` - [x] `cumulus/primitives/aura/Cargo.toml` - [x] `cumulus/primitives/core/Cargo.toml` - [x] `cumulus/primitives/parachain-inherent/Cargo.toml` - [x] `cumulus/test/relay-sproof-builder/Cargo.toml` - [x] `cumulus/pallets/xcmp-queue/Cargo.toml` - [x] `cumulus/pallets/dmp-queue/Cargo.toml` - [x] `cumulus/pallets/xcm/Cargo.toml` - [x] `polkadot/erasure-coding/Cargo.toml` - [x] `polkadot/statement-table/Cargo.toml` - [x] `polkadot/primitives/Cargo.toml` - [x] `polkadot/rpc/Cargo.toml` - [x] `polkadot/node/service/Cargo.toml` - [x] `polkadot/node/core/parachains-inherent/Cargo.toml` - [x] `polkadot/node/core/approval-voting/Cargo.toml` - [x] `polkadot/node/core/dispute-coordinator/Cargo.toml` - [x] `polkadot/node/core/av-store/Cargo.toml` - [x] `polkadot/node/core/chain-api/Cargo.toml` - [x] `polkadot/node/core/prospective-parachains/Cargo.toml` - [x] `polkadot/node/core/backing/Cargo.toml` - [x] `polkadot/node/core/provisioner/Cargo.toml` - [x] `polkadot/node/core/runtime-api/Cargo.toml` - [x] `polkadot/node/core/bitfield-signing/Cargo.toml` - [x] `polkadot/node/network/dispute-distribution/Cargo.toml` - [x] `polkadot/node/network/bridge/Cargo.toml` - [x] `polkadot/node/network/collator-protocol/Cargo.toml` - [x] `polkadot/node/network/approval-distribution/Cargo.toml` - [x] `polkadot/node/network/availability-distribution/Cargo.toml` - [x] `polkadot/node/network/bitfield-distribution/Cargo.toml` - [x] `polkadot/node/network/gossip-support/Cargo.toml` - [x] `polkadot/node/network/availability-recovery/Cargo.toml` - [x] `polkadot/node/collation-generation/Cargo.toml` - [x] `polkadot/node/overseer/Cargo.toml` - [x] `polkadot/runtime/parachains/Cargo.toml` - [x] `polkadot/runtime/common/slot_range_helper/Cargo.toml` - [x] `polkadot/runtime/metrics/Cargo.toml` - [x] `polkadot/xcm/pallet-xcm-benchmarks/Cargo.toml` - [x] `polkadot/utils/generate-bags/Cargo.toml` - [x] `substrate/bin/minimal/runtime/Cargo.toml` --------- Signed-off-by:
Oliver Tale-Yazdi <[email protected]> Signed-off-by:
alindima <[email protected]> Co-authored-by:
ordian <[email protected]> Co-authored-by:
Tsvetomir Dimitrov <[email protected]> Co-authored-by:
Marcin S <[email protected]> Co-authored-by:
alindima <[email protected]> Co-authored-by:
Sebastian Kunert <[email protected]> Co-authored-by:
Dmitry Markin <[email protected]> Co-authored-by:
joe petrowski <[email protected]> Co-authored-by:
Liam Aharon <[email protected]>
-
- Nov 06, 2023
-
-
Andrei Sandu authored
**_PR migrated from https://github.com/paritytech/polkadot/pull/6782_** This PR will upgrade the network protocol to version 3 -> VStaging which will later be renamed to V3. This version introduces a new kind of assignment certificate that will be used for tranche0 assignments. Instead of issuing/importing one tranche0 assignment per candidate, there will be just one certificate per relay chain block per validator. However, we will not be sending out the new assignment certificates, yet. So everything should work exactly as before. Once the majority of the validators have been upgraded to the new protocol version we will enable the new certificates (starting at a specific relay chain block) with a new client update. There are still a few things that need to be done: - [x] Use bitfield instead of Vec<CandidateIndex>: https://github.com/paritytech/polkadot/pull/6802 - [x] Fix existing approval-distribution and approval-voting tests - [x] Fix bitfield-distribution and statement-distribution tests - [x] Fix network bridge tests - [x] Implement todos in the code - [x] Add tests to cover new code - [x] Update metrics - [x] Remove the approval distribution aggression levels: TBD PR - [x] Parachains DB migration - [x] Test network protocol upgrade on Versi - [x] Versi Load test - [x] Add Zombienet test - [x] Documentation updates - [x] Fix for sending DistributeAssignment for each candidate claimed by a v2 assignment (warning: Importing locally an already known assignment) - [x] Fix AcceptedDuplicate - [x] Fix DB migration so that we can still keep old data. - [x] Final Versi burn in --------- Signed-off-by:
Andrei Sandu <[email protected]> Signed-off-by:
Alexandru Gheorghe <[email protected]> Co-authored-by:
Alexandru Gheorghe <[email protected]>
-
Alexandru Gheorghe authored
... see https://github.com/paritytech/polkadot-sdk/issues/2138 for why is not good, until we fix it let's add a warning to understand if this is happening in the wild. --------- Signed-off-by:
Alexandru Gheorghe <[email protected]> Co-authored-by:
Bastian Köcher <[email protected]>
-
- Oct 27, 2023
-
-
Alexandru Gheorghe authored
While investigating some db migrations that make the node startup fail, I noticed that the node wasn't exiting and that the log file were growing exponentially, until my whole system was freezing and that makes it really hard to actually find why it was failing in the first place. E.g: ``` ls -lh /tmp/zombie-01a04c2a2c0265d85f6440cf01c0f44a_-51319-uyggzuD4wEpV/bob.log 32,6G oct 27 11:16 /tmp/zombie-01a04c2a2c0265d85f6440cf01c0f44a_-51319-uyggzuD4wEpV/bob.log ``` This was happening because the following errors were being printed continously without the subsystem main loop exiting: From dispute-coordinator: ``` WARN tokio-runtime-worker parachain::dispute-coordinator: error=Subsystem(Generated(Context("Signal channel is terminated and empty."))) ``` From availability recovery: ``` Erasure task channel closed. Node shutting down ? ``` Signed-off-by:
Alexandru Gheorghe <[email protected]>
-
- Oct 23, 2023
-
-
ordian authored
Fixes #768.
-
- Oct 21, 2023
-
-
asynchronous rob authored
in-progress PR adding new tests and solving bugs --------- Co-authored-by:
Bradley Olson <[email protected]> Co-authored-by:
eskimor <[email protected]> Co-authored-by:
eskimor <[email protected]> Co-authored-by:
Andrei Sandu <[email protected]>
-
- Oct 19, 2023
-
-
Bastian Köcher authored
The validators are checking if async backing is enabled by checking the version of the runtime api. If the runtime api is upgraded by a runtime upgrade, the validators start to also enable the async backing logic. However, just because async backing is enabled, it doesn't mean that all collators and parachain runtimes have upgraded. This pull request fixes an issue about advertising collations to the relay chain when it has async backing enabled, but the collator is still using the old networking protocol. The implementation is actually backwards compatible as we can not expect that everyone directly upgrades. However, the collation advertisement logic was requiring V2 networking messages after async backing was enabled, which was wrong. This is now fixed by this pull request. Closes: https://github.com/paritytech/polkadot-sdk/issues/1923 --------- Co-authored-by:
eskimor <[email protected]>
-
- Oct 12, 2023
-
-
Anton Vilhelm Ásgeirsson authored
# Description In a couple of cases, there were links pointing to the w3f github pages domain. In other instances, there were links pointing to the old polkadot repo's github pages. Both of these are now pointing to the relevant links in https://paritytech.github.io/polkadot-sdk/book/index.html. These changes were made specifically because the w3f github pages returns a 404, and while fixing the links, the old polkadot repo links were touched up as well even if they do redirect properly. This shouldn't affect anything as these are documentation link changes only.
-
- Sep 27, 2023
-
-
Chris Sosnin authored
- Async-backing related primitives are stable `primitives::v6` - Async-backing API is now part of `api_version(7)` - It's enabled on Rococo and Westend runtimes --------- Signed-off-by:
Andrei Sandu <[email protected]> Co-authored-by:
Andrei Sandu <[email protected]>
-
- Sep 20, 2023
-
-
Alin Dima authored
Refactors availability-recovery strategies to allow for easily adding new hotpaths and failover mechanisms. The new interface allows for chaining multiple `RecoveryStrategy`-es together, to cleanly express the relationship between them and share state and code where neccessary/possible: This was done in order to aid in implementing new hotpaths like [systematic chunks recovery](https://github.com/paritytech/polkadot-sdk/issues/598) and [fetching from approval checkers](https://github.com/paritytech/polkadot-sdk/issues/575). Thanks to this design, intermediate state can be shared between the strategies. For example, if the systematic chunks recovery retrieved less than the needed amount of chunks, pass them over to the next FetchChunks strategy, which will only need to recover the remaining number of chunks. Draft example of how a systematic chunk recovery strategy would look: https://github.com/paritytech/polkadot-sdk/commit/667d870bdf1470525d66c13929d5eac7249dd995 (notice how easy it was to add and reuse code) Note that this PR doesn't itself add any new strategy, it should fully preserve backwards compatiblity in terms of functionality. Follow-up PRs to add new strategies will come.
-
- Sep 18, 2023
-
-
Vsevolod Stakhov authored
Futures channels that are used by default has a side effect of `Sender::Clone` that efficiently increases the capacity of the bounded channel by one. This PR fixes the undesired backpressure removal that was caused by the #1409. This issue has been discovered by @sandreim during Versi testing and needs to be treated as critical that should not be included in any release without this reversion. This PR reverts the original behaviour.
-
- Sep 11, 2023
-
-
Vsevolod Stakhov authored
This PR addresses multiple issues pending: * [x] Update orchestra to the recent version and test how the node performs * [x] Add some useful metrics for outbound network bridge * [x] Try to send incoming network requests to all subsystems without blocking on some particular subsystem in that loop * [x] Fix all incompatibilities between orchestra and polkadot code (e.g. malus node)
-
- Sep 08, 2023
-
-
Bradley Olson authored
* Removing unnecessary log * Drop unneeded log * Handling logs conditionally * Changing unexpected message to warn * Back to debug * fmt
-
- Sep 07, 2023
-
-
ordian authored
* polkadot: propagate UnpinHandle to ActiveLeafUpdate Also extract the leaf creation for tests into a common function. * dispute-coordinator: try pinned blocks for slashin * apparently 1.72 is smarter than 1.70 * address nits * rename fresh_leaf to new_leaf
-
- Sep 05, 2023
-
-
dependabot[bot] authored
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.47 to 1.0.48. - [Release notes](https://github.com/dtolnay/thiserror/releases) - [Commits](https://github.com/dtolnay/thiserror/compare/1.0.47...1.0.48 ) --- updated-dependencies: - dependency-name: thiserror dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by:
dependabot[bot] <[email protected]> Co-authored-by:
dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by:
Liam Aharon <[email protected]>
-
- Sep 01, 2023
-
-
s0me0ne-unkn0wn authored
* Import changes from archieved repo * Revert erroneous changes * Fix more tests * Resolve discussions * Fix MORE tests * approval-voting: launch_approval better interface (#1355) --------- Co-authored-by:
Javier Viola <[email protected]> Co-authored-by:
ordian <[email protected]> Co-authored-by:
ordian <[email protected]>
-
- Aug 31, 2023
-
-
Bastian Köcher authored
* Rename `polkadot-parachain` to `polkadot-parachain-primitives` While doing this it also fixes some last `rustdoc` issues and fixes another Cargo warning related to `pallet-paged-list`. * Fix compilation * ".git/.scripts/commands/fmt/fmt.sh" * Fix XCM docs --------- Co-authored-by: command-bot <>
-
Alin Dima authored
* move min backing votes const to runtime also cache it per-session in the backing subsystem Signed-off-by:
alindima <[email protected]> * add runtime migration * introduce api versioning for min_backing votes also enable it for rococo/versi for testing * also add min_backing_votes runtime calls to statement-distribution this dependency has been recently introduced by async backing * remove explicit version runtime API call this is not needed, as the RuntimeAPISubsystem already takes care of versioning and will return NotSupported if the version is not right. * address review comments - parametrise backing votes runtime API with session index - remove RuntimeInfo usage in backing subsystem, as runtime API caches the min backing votes by session index anyway. - move the logic for adjusting the configured needed backing votes with the size of the backing group to a primitives helper. - move the legacy min backing votes value to a primitives helper. - mark JoinMultiple error as fatal, since the Canceled (non-multiple) counterpart is also fatal. - make backing subsystem handle fatal errors for new leaves update. - add HostConfiguration consistency check for zeroed backing votes threshold - add cumulus accompanying change * fix cumulus test compilation * fix tests * more small fixes * fix merge * bump runtime api version for westend and rollback version for rococo --------- Signed-off-by:
alindima <[email protected]> Co-authored-by:
Javier Viola <[email protected]>
-
- Aug 30, 2023
-
-
Przemek Rzad authored
* Add missing Cumulus licenses * Typo * Add missing Substrate licenses * Single job checking the sub-repos in steps * Remove dates * Remove dates * Add missing (C) * Update FRAME UI tests Signed-off-by:
Oliver Tale-Yazdi <[email protected]> * Update more UI tests Signed-off-by:
Oliver Tale-Yazdi <[email protected]> --------- Signed-off-by:
Oliver Tale-Yazdi <[email protected]> Co-authored-by:
Oliver Tale-Yazdi <[email protected]>
-
- Aug 29, 2023
-
-
Bastian Köcher authored
* Fix `test-rustdoc` * ".git/.scripts/commands/fmt/fmt.sh" --------- Co-authored-by: command-bot <>
-
Oliver Tale-Yazdi authored
* Fix build profiles Closes https://github.com/paritytech/polkadot-sdk/issues/1155 Signed-off-by:
Oliver Tale-Yazdi <[email protected]> * Manually set version to 1.0.0 Signed-off-by:
Oliver Tale-Yazdi <[email protected]> * Use workspace repo Signed-off-by:
Oliver Tale-Yazdi <[email protected]> * 'Authors and Edition from workspace Signed-off-by:
Oliver Tale-Yazdi <[email protected]> --------- Signed-off-by:
Oliver Tale-Yazdi <[email protected]>
-
- Aug 28, 2023
-
-
ordian authored
* deps: replace lru with schnellru * bring the peace to the galaxy
-
- Aug 25, 2023
-
-
Alexander authored
Signed-off-by:
alvicsam <[email protected]>
-