- Feb 12, 2024
-
-
Oliver Tale-Yazdi authored
Changes (partial https://github.com/paritytech/polkadot-sdk/issues/994): - Set log to `0.4.20` everywhere - Lift `log` to the workspace Starting with a simpler one after seeing https://github.com/paritytech/polkadot-sdk/pull/2065 from @jsdw . This sets the `default-features` to `false` in the root and then overwrites that in each create to its original value. This is necessary since otherwise the `default` features are additive and its impossible to disable them in the crate again once they are enabled in the workspace. I am using a tool to do this, so its mostly a test to see that it works as expected. --------- Signed-off-by: Oliver Tale-Yazdi <[email protected]>
-
- Feb 05, 2024
-
-
Alexandru Gheorghe authored
## Summary Built on top of the tooling and ideas introduced in https://github.com/paritytech/polkadot-sdk/pull/2528, this PR introduces a synthetic benchmark for measuring and assessing the performance characteristics of the approval-voting and approval-distribution subsystems. Currently this allows, us to simulate the behaviours of these systems based on the following dimensions: ``` TestConfiguration: # Test 1 - objective: !ApprovalsTest last_considered_tranche: 89 min_coalesce: 1 max_coalesce: 6 enable_assignments_v2: true send_till_tranche: 60 stop_when_approved: false coalesce_tranche_diff: 12 workdir_prefix: "/tmp" num_no_shows_per_candidate: 0 approval_distribution_expected_tof: 6.0 approval_distribution_cpu_ms: 3.0 approval_voting_cpu_ms: 4.30 n_validators: 500 n_cores: 100 n_included_candidates: 100 min_pov_size: 1120 max_pov_size: 5120 peer_bandwidth: 524288000000 bandwidth: 524288000000 latency: min_latency: secs: 0 nanos: 1000000 max_latency: secs: 0 nanos: 100000000 error: 0 num_blocks: 10 ``` ## The approach 1. We build a real overseer with the real implementations for approval-voting and approval-distribution subsystems. 2. For a given network size, for each validator we pre-computed all potential assignments and approvals it would send, because this a computation heavy operation this will be cached on a file on disk and be re-used if the generation parameters don't change. 3. The messages will be sent accordingly to the configured parameters and those are split into 3 main benchmarking scenarios. ## Benchmarking scenarios ### Best case scenario *approvals_throughput_best_case.yaml* It send to the approval-distribution only the minimum required tranche to gathered the needed_approvals, so that a candidate is approved. ### Behaviour in the presence of no-shows *approvals_no_shows.yaml* It sends the tranche needed to approve a candidate when we have a maximum of *num_no_shows_per_candidate* tranches with no-shows for each candidate. ### Maximum throughput *approvals_throughput.yaml* It sends all the tranches for each block and measures the used CPU and necessary network bandwidth. by the approval-voting and approval-distribution subsystem. ## How to run it ``` cargo run -p polkadot-subsystem-bench --release -- test-sequence --path polkadot/node/subsystem-bench/examples/approvals_throughput.yaml ``` ## Evaluating performance ### Use the real subsystems metrics If you follow the steps in https://github.com/paritytech/polkadot-sdk/tree/master/polkadot/node/subsystem-bench#install-grafana for installing locally prometheus and grafana, all real metrics for the `approval-distribution`, `approval-voting` and overseer are available. E.g: <img width="2149" alt="Screenshot 2023-12-05 at 11 07 46" src="https://github.com/paritytech/polkadot-sdk/assets/49718502/cb8ae2dd-178b-4922-bfa4-dc37e572ed38"> <img width="2551" alt="Screenshot 2023-12-05 at 11 09 42" src="https://github.com/paritytech/polkadot-sdk/assets/49718502/8b4542ba-88b9-46f9-9b70-cc345366081b"> <img width="2154" alt="Screenshot 2023-12-05 at 11 10 15" src="https://github.com/paritytech/polkadot-sdk/assets/49718502/b8874d8d-632e-443a-9840-14ad8e90c54f"> <img width="2535" alt="Screenshot 2023-12-05 at 11 10 52" src="https://github.com/paritytech/polkadot-sdk/assets/49718502/779a439f-fd18-4985-bb80-85d5afad78e2"> ### Profile with pyroscope 1. Setup pyroscope following the steps in https://github.com/paritytech/polkadot-sdk/tree/master/polkadot/node/subsystem-bench#install-pyroscope, then run any of the benchmark scenario with `--profile` as the arguments. 2. Open the pyroscope dashboard in grafana, e.g: <img width="2544" alt="Screenshot 2024-01-09 at 17 09 58" src="https://github.com/paritytech/polkadot-sdk/assets/49718502/58f50c99-a910-4d20-951a-8b16639303d9"> ### Useful logs 1. Network bandwidth requirements: ``` Payload bytes received from peers: 503993 KiB total, 50399 KiB/block Payload bytes sent to peers: 629971 KiB total, 62997 KiB/block ``` 2. Cpu usage by the approval-distribution/approval-voting subsystems. ``` approval-distribution CPU usage 84.061s approval-distribution CPU usage per block 8.406s approval-voting CPU usage 96.532s approval-voting CPU usage per block 9.653s ``` 3. Time passed until a given block is approved ``` Chain selection approved after 3500 ms hash=0x0101010101010101010101010101010101010101010101010101010101010101 Chain selection approved after 4500 ms hash=0x0202020202020202020202020202020202020202020202020202020202020202 ``` ### Using benchmark to quantify improvements from https://github.com/paritytech/polkadot-sdk/pull/1178 + https://github.com/paritytech/polkadot-sdk/pull/1191 Using a versi-node we compare the scenarios where all new optimisations are disabled with a scenarios where tranche0 assignments are sent in a single message and a conservative simulation where the coalescing of approvals gives us just 50% reduction in the number of messages we send. Overall, what we see is a speedup of around 30-40% in the time it takes to process the necessary messages and a 30-40% reduction in the necessary bandwidth. #### Best case scenario comparison(minimum required tranches sent). Unoptimised ``` Number of blocks: 10 Payload bytes received from peers: 53289 KiB total, 5328 KiB/block Payload bytes sent to peers: 52489 KiB total, 5248 KiB/block approval-distribution CPU usage 6.732s approval-distribution CPU usage per block 0.673s approval-voting CPU usage 9.523s approval-voting CPU usage per block 0.952s ``` vs Optimisation enabled ``` Number of blocks: 10 Payload bytes received from peers: 32141 KiB total, 3214 KiB/block Payload bytes sent to peers: 37314 KiB total, 3731 KiB/block approval-distribution CPU usage 4.658s approval-distribution CPU usage per block 0.466s approval-voting CPU usage 6.236s approval-voting CPU usage per block 0.624s ``` #### Worst case all tranches sent, very unlikely happens when sharding breaks. Unoptimised ``` Number of blocks: 10 Payload bytes received from peers: 746393 KiB total, 74639 KiB/block Payload bytes sent to peers: 729151 KiB total, 72915 KiB/block approval-distribution CPU usage 118.681s approval-distribution CPU usage per block 11.868s approval-voting CPU usage 124.118s approval-voting CPU usage per block 12.412s ``` vs optimised ``` Number of blocks: 10 Payload bytes received from peers: 503993 KiB total, 50399 KiB/block Payload bytes sent to peers: 629971 KiB total, 62997 KiB/block approval-distribution CPU usage 84.061s approval-distribution CPU usage per block 8.406s approval-voting CPU usage 96.532s approval-voting CPU usage per block 9.653s ``` ## TODOs [x] Polish implementation. [x] Use what we have so far to evaluate https://github.com/paritytech/polkadot-sdk/pull/1191 before merging. [x] List of features and additional dimensions we want to use for benchmarking. [x] Run benchmark on hardware similar with versi and kusama nodes. [ ] Add benchmark to be run in CI for catching regression in performance. [ ] Rebase on latest changes for network emulation. --------- Signed-off-by: Andrei Sandu <[email protected]> Signed-off-by: Alexandru Gheorghe <[email protected]> Co-authored-by: Andrei Sandu <[email protected]> Co-authored-by: Andrei Sandu <[email protected]>
-
- Jan 26, 2024
-
-
Liam Aharon authored
Related https://github.com/paritytech/polkadot-sdk/issues/3032 --- Using https://github.com/liamaharon/cargo-workspace-version-tools/ `cargo run -- sync --path ../polkadot-sdk` --------- Signed-off-by: Oliver Tale-Yazdi <[email protected]> Co-authored-by: Oliver Tale-Yazdi <[email protected]>
-
- Jan 22, 2024
-
-
Alexandru Gheorghe authored
... it was like that before it was accidentally changed in https://github.com/paritytech/polkadot-sdk/commit/a84dd0dba58d51503b8942360aa4fb30a5a96af5 Signed-off-by: Alexandru Gheorghe <[email protected]>
-
- Jan 18, 2024
-
-
Bastian Köcher authored
There is no need to spawn operators with this debug log. Closes: https://github.com/paritytech/polkadot-sdk/issues/2974
-
- Jan 10, 2024
-
-
dependabot[bot] authored
Bumps [parking_lot](https://github.com/Amanieu/parking_lot) from 0.11.2 to 0.12.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md">parking_lot's changelog</a>.</em></p> <blockquote> <h2>parking_lot 0.12.1 (2022-05-31)</h2> <ul> <li>Fixed incorrect memory ordering in <code>RwLock</code>. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/344">#344</a>)</li> <li>Added <code>Condvar::wait_while</code> convenience methods (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/343">#343</a>)</li> </ul> <h2>parking_lot_core 0.9.3 (2022-04-30)</h2> <ul> <li>Bump windows-sys dependency to 0.36. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/339">#339</a>)</li> </ul> <h2>parking_lot_core 0.9.2, lock_api 0.4.7 (2022-03-25)</h2> <ul> <li>Enable const new() on lock types on stable. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/325">#325</a>)</li> <li>Added <code>MutexGuard::leak</code> function. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/333">#333</a>)</li> <li>Bump windows-sys dependency to 0.34. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/331">#331</a>)</li> <li>Bump petgraph dependency to 0.6. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/326">#326</a>)</li> <li>Don't use pthread attributes on the espidf platform. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/319">#319</a>)</li> </ul> <h2>parking_lot_core 0.9.1 (2022-02-06)</h2> <ul> <li>Bump windows-sys dependency to 0.32. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/316">#316</a>)</li> </ul> <h2>parking_lot 0.12.0, parking_lot_core 0.9.0, lock_api 0.4.6 (2022-01-28)</h2> <ul> <li>The MSRV is bumped to 1.49.0.</li> <li>Disabled eventual fairness on wasm32-unknown-unknown. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/302">#302</a>)</li> <li>Added a rwlock method to report if lock is held exclusively. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/303">#303</a>)</li> <li>Use new <code>asm!</code> macro. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/304">#304</a>)</li> <li>Use windows-rs instead of winapi for faster builds. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/311">#311</a>)</li> <li>Moved hardware lock elision support to a separate Cargo feature. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/313">#313</a>)</li> <li>Removed used of deprecated <code>spin_loop_hint</code>. (<a href="https://redirect.github.com/Amanieu/parking_lot/issues/314">#314</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/Amanieu/parking_lot/commit/336a9b31ff385728d00eb7ef173e4d054584b787"><code>336a9b3</code></a> Release parking_lot 0.12.1</li> <li><a href="https://github.com/Amanieu/parking_lot/commit/b69a0547ce6c92b164528c65daafa48e8f1c08bb"><code>b69a054</code></a> Merge pull request <a href="https://redirect.github.com/Amanieu/parking_lot/issues/343">#343</a> from bryanhitc/master</li> <li><a href="https://github.com/Amanieu/parking_lot/commit/3fe7233ee05fbae9bc94a831d4e66df0257880a2"><code>3fe7233</code></a> Merge pull request <a href="https://redirect.github.com/Amanieu/parking_lot/issues/344">#344</a> from Amanieu/fix_rwlock_ordering</li> <li><a href="https://github.com/Amanieu/parking_lot/commit/ef12b00daf0dbc6dd025098ec2cd0517bb9f737c"><code>ef12b00</code></a> small test update</li> <li><a href="https://github.com/Amanieu/parking_lot/commit/fdb063cd4e33d9c44cc08aea0c341ffec7e7b282"><code>fdb063c</code></a> wait_while can't timeout fix</li> <li><a href="https://github.com/Amanieu/parking_lot/commit/26e19dced4ae01000abc0dfca251f56695b1b8aa"><code>26e19dc</code></a> Remove WaitWhileResult</li> <li><a href="https://github.com/Amanieu/parking_lot/commit/d26c284fe8413bdeeed3ef4f70e23b5919a9ffeb"><code>d26c284</code></a> Fix incorrect memory ordering in RwLock</li> <li><a href="https://github.com/Amanieu/parking_lot/commit/045828381a5facff71b1aad3c01af15801959458"><code>0458283</code></a> Use saturating_add for WaitWhileResult</li> <li><a href="https://github.com/Amanieu/parking_lot/commit/686db4755913947219f83a04b21885032685d246"><code>686db47</code></a> Add Condvar::wait_while convenience methods</li> <li><a href="https://github.com/Amanieu/parking_lot/commit/6f6e021ced6f67fef8002442aa97c4a08ed5c3e4"><code>6f6e021</code></a> Merge pull request <a href="https://redirect.github.com/Amanieu/parking_lot/issues/342">#342</a> from MarijnS95/patch-1</li> <li>Additional commits viewable in <a href="https://github.com/Amanieu/parking_lot/compare/0.11.2...0.12.1">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=parking_lot&package-manager=cargo&previous-version=0.11.2&new-version=0.12.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores ) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
-
- Dec 19, 2023
-
-
André Silva authored
-
- Dec 18, 2023
-
-
dependabot[bot] authored
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.73 to 0.1.74. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/dtolnay/async-trait/releases">async-trait's releases</a>.</em></p> <blockquote> <h2>0.1.74</h2> <ul> <li>Documentation improvements</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/dtolnay/async-trait/commit/265979b07a9af573e1edd3b2a9b179533cfa7a6c"><code>265979b</code></a> Release 0.1.74</li> <li><a href="https://github.com/dtolnay/async-trait/commit/5e677097d2e67f7a5c5e3023e2f3b99b36a9e132"><code>5e67709</code></a> Fix doc test when async fn in trait is natively supported</li> <li><a href="https://github.com/dtolnay/async-trait/commit/ef144aed28b636eb65759505b2323afc4c753fbe"><code>ef144ae</code></a> Update ui test suite to nightly-2023-10-15</li> <li><a href="https://github.com/dtolnay/async-trait/commit/9398a28d6fc977ccf8c286bd85b4b87a883f92ac"><code>9398a28</code></a> Test docs.rs documentation build in CI</li> <li><a href="https://github.com/dtolnay/async-trait/commit/8737173dafa371e5e9e491d736513be1baf697f4"><code>8737173</code></a> Update ui test suite to nightly-2023-09-24</li> <li><a href="https://github.com/dtolnay/async-trait/commit/5ba643c001a55f70c4a44690e040cdfab873ba56"><code>5ba643c</code></a> Test dyn Trait containing async fn</li> <li><a href="https://github.com/dtolnay/async-trait/commit/247c8e7b0b3ff69c9518ebf93e69fe74d47f17b6"><code>247c8e7</code></a> Add ui test testing the recommendation to use async-trait</li> <li><a href="https://github.com/dtolnay/async-trait/commit/799db66a84834c403860df4a8c0227d8fb7f9d9d"><code>799db66</code></a> Update ui test suite to nightly-2023-09-23</li> <li><a href="https://github.com/dtolnay/async-trait/commit/0e60248011f751d8ccf58219d0a79aacfe9619f1"><code>0e60248</code></a> Update actions/checkout@v3 -> v4</li> <li><a href="https://github.com/dtolnay/async-trait/commit/7fcbc83993d5ef483d048c271a7f6c4ac8c98388"><code>7fcbc83</code></a> Update ui test suite to nightly-2023-08-29</li> <li>See full diff in <a href="https://github.com/dtolnay/async-trait/compare/0.1.73...0.1.74">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=async-trait&package-manager=cargo&previous-version=0.1.73&new-version=0.1.74)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores ) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
-
- Dec 13, 2023
-
-
Squirrel authored
We currently use a bit of a hack in `.cargo/config` to make sure that clippy isn't too annoying by specifying the list of lints. There is now a stable way to define lints for a workspace. The only down side is that every crate seems to have to opt into this so there's a *few* files modified in this PR. Dependencies: - [x] PR that upgrades CI to use rust 1.74 is merged. --------- Co-authored-by: joe petrowski <[email protected]> Co-authored-by: Branislav Kontur <[email protected]> Co-authored-by: Liam Aharon <[email protected]>
-
Alexandru Gheorghe authored
Initial implementation for the plan discussed here: https://github.com/paritytech/polkadot-sdk/issues/701 Built on top of https://github.com/paritytech/polkadot-sdk/pull/1178 v0: https://github.com/paritytech/polkadot/pull/7554, ## Overall idea When approval-voting checks a candidate and is ready to advertise the approval, defer it in a per-relay chain block until we either have MAX_APPROVAL_COALESCE_COUNT candidates to sign or a candidate has stayed MAX_APPROVALS_COALESCE_TICKS in the queue, in both cases we sign what candidates we have available. This should allow us to reduce the number of approvals messages we have to create/send/verify. The parameters are configurable, so we should find some values that balance: - Security of the network: Delaying broadcasting of an approval shouldn't but the finality at risk and to make sure that never happens we won't delay sending a vote if we are past 2/3 from the no-show time. - Scalability of the network: MAX_APPROVAL_COALESCE_COUNT = 1 & MAX_APPROVALS_COALESCE_TICKS =0, is what we have now and we know from the measurements we did on versi, it bottlenecks approval-distribution/approval-voting when increase significantly the number of validators and parachains - Block storage: In case of disputes we have to import this votes on chain and that increase the necessary storage with MAX_APPROVAL_COALESCE_COUNT * CandidateHash per vote. Given that disputes are not the normal way of the network functioning and we will limit MAX_APPROVAL_COALESCE_COUNT in the single digits numbers, this should be good enough. Alternatively, we could try to create a better way to store this on-chain through indirection, if that's needed. ## Other fixes: - Fixed the fact that we were sending random assignments to non-validators, that was wrong because those won't do anything with it and they won't gossip it either because they do not have a grid topology set, so we would waste the random assignments. - Added metrics to be able to debug potential no-shows and mis-processing of approvals/assignments. ## TODO: - [x] Get feedback, that this is moving in the right direction. @ordian @sandreim @eskimor @burdges, let me know what you think. - [x] More and more testing. - [x] Test in versi. - [x] Make MAX_APPROVAL_COALESCE_COUNT & MAX_APPROVAL_COALESCE_WAIT_MILLIS a parachain host configuration. - [x] Make sure the backwards compatibility works correctly - [x] Make sure this direction is compatible with other streams of work: https://github.com/paritytech/polkadot-sdk/issues/635 & https://github.com/paritytech/polkadot-sdk/issues/742 - [x] Final versi burn-in before merging --------- Signed-off-by: Alexandru Gheorghe <[email protected]>
-
- Dec 11, 2023
-
-
Alexandru Gheorghe authored
Scaffold everything, so that we can enable v2 assignments via a node feature bit, once all nodes have upgraded to the new protocol. Implements: https://github.com/paritytech/polkadot-sdk/issues/628 --------- Signed-off-by: Alexandru Gheorghe <[email protected]>
-
- Dec 06, 2023
-
-
André Silva authored
This will make more sense after https://github.com/paritytech/polkadot-sdk/pull/2524 since the schnorrkel type for VRF outputs is also renamed in the latest version. Can be reviewed independently though. Can be merged after https://github.com/paritytech/polkadot-sdk/pull/1577 so that there is less pain for @davxy . --------- Co-authored-by: Bastian Köcher <[email protected]>
-
- Dec 01, 2023
-
-
Liam Aharon authored
Using taplo, fixes all our broken and inconsistent toml formatting and adds CI to keep them tidy. If people want we can customise the format rules as described here https://taplo.tamasfe.dev/configuration/formatter-options.html @ggwpez , I suggest zepter is used only for checking features are propagated, and leave formatting for taplo to avoid duplicate work and conflicts. TODO - [x] Use `exclude = [...]` syntax in taplo file to ignore zombienet tests instead of deleting the dir --------- Signed-off-by: Oliver Tale-Yazdi <[email protected]> Co-authored-by: Oliver Tale-Yazdi <[email protected]> Co-authored-by: Bastian Köcher <[email protected]>
-
- Nov 20, 2023
-
-
jserrat authored
Hey guys, as discussed I've changed the name to a more general one `PvfExecKind`, is this good or too general? Creating this as a draft, I still have to fix the tests. Closes #1585 Kusama address: FkB6QEo8VnV3oifugNj5NeVG3Mvq1zFbrUu4P5YwRoe5mQN --------- Co-authored-by: command-bot <> Co-authored-by: Marcin S <[email protected]>
-
- Nov 09, 2023
-
-
Oliver Tale-Yazdi authored
Missing descriptions (47): - [x] `cumulus/client/collator/Cargo.toml` - [x] `cumulus/client/relay-chain-inprocess-interface/Cargo.toml` - [x] `cumulus/client/cli/Cargo.toml` - [x] `cumulus/client/service/Cargo.toml` - [x] `cumulus/client/relay-chain-rpc-interface/Cargo.toml` - [x] `cumulus/client/relay-chain-interface/Cargo.toml` - [x] `cumulus/client/relay-chain-minimal-node/Cargo.toml` - [x] `cumulus/parachains/pallets/parachain-info/Cargo.toml` - [x] `cumulus/parachains/pallets/ping/Cargo.toml` - [x] `cumulus/primitives/utility/Cargo.toml` - [x] `cumulus/primitives/aura/Cargo.toml` - [x] `cumulus/primitives/core/Cargo.toml` - [x] `cumulus/primitives/parachain-inherent/Cargo.toml` - [x] `cumulus/test/relay-sproof-builder/Cargo.toml` - [x] `cumulus/pallets/xcmp-queue/Cargo.toml` - [x] `cumulus/pallets/dmp-queue/Cargo.toml` - [x] `cumulus/pallets/xcm/Cargo.toml` - [x] `polkadot/erasure-coding/Cargo.toml` - [x] `polkadot/statement-table/Cargo.toml` - [x] `polkadot/primitives/Cargo.toml` - [x] `polkadot/rpc/Cargo.toml` - [x] `polkadot/node/service/Cargo.toml` - [x] `polkadot/node/core/parachains-inherent/Cargo.toml` - [x] `polkadot/node/core/approval-voting/Cargo.toml` - [x] `polkadot/node/core/dispute-coordinator/Cargo.toml` - [x] `polkadot/node/core/av-store/Cargo.toml` - [x] `polkadot/node/core/chain-api/Cargo.toml` - [x] `polkadot/node/core/prospective-parachains/Cargo.toml` - [x] `polkadot/node/core/backing/Cargo.toml` - [x] `polkadot/node/core/provisioner/Cargo.toml` - [x] `polkadot/node/core/runtime-api/Cargo.toml` - [x] `polkadot/node/core/bitfield-signing/Cargo.toml` - [x] `polkadot/node/network/dispute-distribution/Cargo.toml` - [x] `polkadot/node/network/bridge/Cargo.toml` - [x] `polkadot/node/network/collator-protocol/Cargo.toml` - [x] `polkadot/node/network/approval-distribution/Cargo.toml` - [x] `polkadot/node/network/availability-distribution/Cargo.toml` - [x] `polkadot/node/network/bitfield-distribution/Cargo.toml` - [x] `polkadot/node/network/gossip-support/Cargo.toml` - [x] `polkadot/node/network/availability-recovery/Cargo.toml` - [x] `polkadot/node/collation-generation/Cargo.toml` - [x] `polkadot/node/overseer/Cargo.toml` - [x] `polkadot/runtime/parachains/Cargo.toml` - [x] `polkadot/runtime/common/slot_range_helper/Cargo.toml` - [x] `polkadot/runtime/metrics/Cargo.toml` - [x] `polkadot/xcm/pallet-xcm-benchmarks/Cargo.toml` - [x] `polkadot/utils/generate-bags/Cargo.toml` - [x] `substrate/bin/minimal/runtime/Cargo.toml` --------- Signed-off-by: Oliver Tale-Yazdi <[email protected]> Signed-off-by: alindima <[email protected]> Co-authored-by: ordian <[email protected]> Co-authored-by: Tsvetomir Dimitrov <[email protected]> Co-authored-by: Marcin S <[email protected]> Co-authored-by: alindima <[email protected]> Co-authored-by: Sebastian Kunert <[email protected]> Co-authored-by: Dmitry Markin <[email protected]> Co-authored-by: joe petrowski <[email protected]> Co-authored-by: Liam Aharon <[email protected]>
-
- Nov 08, 2023
-
-
s0me0ne-unkn0wn authored
-
- Nov 06, 2023
-
-
Andrei Sandu authored
**_PR migrated from https://github.com/paritytech/polkadot/pull/6782_** This PR will upgrade the network protocol to version 3 -> VStaging which will later be renamed to V3. This version introduces a new kind of assignment certificate that will be used for tranche0 assignments. Instead of issuing/importing one tranche0 assignment per candidate, there will be just one certificate per relay chain block per validator. However, we will not be sending out the new assignment certificates, yet. So everything should work exactly as before. Once the majority of the validators have been upgraded to the new protocol version we will enable the new certificates (starting at a specific relay chain block) with a new client update. There are still a few things that need to be done: - [x] Use bitfield instead of Vec<CandidateIndex>: https://github.com/paritytech/polkadot/pull/6802 - [x] Fix existing approval-distribution and approval-voting tests - [x] Fix bitfield-distribution and statement-distribution tests - [x] Fix network bridge tests - [x] Implement todos in the code - [x] Add tests to cover new code - [x] Update metrics - [x] Remove the approval distribution aggression levels: TBD PR - [x] Parachains DB migration - [x] Test network protocol upgrade on Versi - [x] Versi Load test - [x] Add Zombienet test - [x] Documentation updates - [x] Fix for sending DistributeAssignment for each candidate claimed by a v2 assignment (warning: Importing locally an already known assignment) - [x] Fix AcceptedDuplicate - [x] Fix DB migration so that we can still keep old data. - [x] Final Versi burn in --------- Signed-off-by: Andrei Sandu <[email protected]> Signed-off-by: Alexandru Gheorghe <[email protected]> Co-authored-by: Alexandru Gheorghe <[email protected]>
-
- Sep 07, 2023
-
-
ordian authored
* polkadot: propagate UnpinHandle to ActiveLeafUpdate Also extract the leaf creation for tests into a common function. * dispute-coordinator: try pinned blocks for slashin * apparently 1.72 is smarter than 1.70 * address nits * rename fresh_leaf to new_leaf
-
- Sep 05, 2023
-
-
dependabot[bot] authored
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.47 to 1.0.48. - [Release notes](https://github.com/dtolnay/thiserror/releases) - [Commits](https://github.com/dtolnay/thiserror/compare/1.0.47...1.0.48 ) --- updated-dependencies: - dependency-name: thiserror dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Liam Aharon <[email protected]>
-
- Sep 04, 2023
-
-
ordian authored
-
- Sep 01, 2023
-
-
s0me0ne-unkn0wn authored
* Import changes from archieved repo * Revert erroneous changes * Fix more tests * Resolve discussions * Fix MORE tests * approval-voting: launch_approval better interface (#1355) --------- Co-authored-by: Javier Viola <[email protected]> Co-authored-by: ordian <[email protected]> Co-authored-by: ordian <[email protected]>
-
- Aug 29, 2023
-
-
Oliver Tale-Yazdi authored
* Fix build profiles Closes https://github.com/paritytech/polkadot-sdk/issues/1155 Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Manually set version to 1.0.0 Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Use workspace repo Signed-off-by: Oliver Tale-Yazdi <[email protected]> * 'Authors and Edition from workspace Signed-off-by: Oliver Tale-Yazdi <[email protected]> --------- Signed-off-by: Oliver Tale-Yazdi <[email protected]>
-
- Aug 28, 2023
-
-
ordian authored
* deps: replace lru with schnellru * bring the peace to the galaxy
-
- Aug 25, 2023
-
-
Alexander authored
Signed-off-by: alvicsam <[email protected]>
-
- Aug 16, 2023
-
-
Aaro Altonen authored
* Revert "[Substrate companion] update libp2p to 0.52.0 (#7472)" This reverts commit 7a1d96e1. * Update dependencies * update lockfile for {"substrate"} --------- Co-authored-by: parity-processbot <>
-
- Aug 15, 2023
-
-
Bastian Köcher authored
Remove superflous parameter `overseer_enable_anyways` and make parachain node type more explicit (#7617) * Remove superflous parameter `overseer_enable_anyways` We don't need this flag, as we don't need the overseer enabled when the node isn't a collator or validator. * Rename `IsCollator` to `IsParachainNode` `IsParachainNode` is more expressive and also encapsulates the state of the parachain node being a full node. Some functionality like the overseer needs to run always when the node runs alongside a parachain node. The parachain node needs the overseer to e.g. recover PoVs. Other things like candidate validation or pvf checking are only required for when the node is running as validator. * FMT * Fix CI
-
- Aug 14, 2023
-
-
Oliver Tale-Yazdi authored
* Use same rustfmt.toml as Substrate Signed-off-by: Oliver Tale-Yazdi <[email protected]> * format format file Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Format with new config Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Add Substrate Clippy config Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Print Clippy version in CI Otherwise its difficult to reproduce locally. Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Make fmt happy Signed-off-by: Oliver Tale-Yazdi <[email protected]> * Update node/core/pvf/src/error.rs Co-authored-by: Tsvetomir Dimitrov <[email protected]> * Update node/core/pvf/src/error.rs Co-authored-by: Tsvetomir Dimitrov <[email protected]> --------- Signed-off-by: Oliver Tale-Yazdi <[email protected]> Co-authored-by: Tsvetomir Dimitrov <[email protected]>
-
- Aug 10, 2023
-
-
Andrei Eres authored
* Add counter for unapproved candidates * Update metrics * Split metrics * Remove depth metric * Print only the oldest unapproved candidates * Update logging condition * Fix logging condition * Update logging * Update node/core/approval-voting/src/lib.rs Co-authored-by: Andrei Sandu <[email protected]> --------- Co-authored-by: Andrei Sandu <[email protected]>
-
- Aug 04, 2023
-
-
Lulu authored
* Add license to crates This is required to publish to crates.io * Add more licenses
-
- Jul 25, 2023
-
-
Anton authored
* update tinyvec to 1.6.0 * update once_cell to 1.18.0 * update data-encoding to 2.4.0 * update libc * update js-sys * update wasm-bindgen-futures * update pin-project * update tokio * update syn * p2p protocol now contains `PeerId` not multihash * update arrayvec to 0.7.4 * update sha2 * update smallvec * updates to Cargo.lock after merge * redo dep updates * update lru in attempt to compile polkadot * update lockfile for {"substrate"} --------- Co-authored-by: parity-processbot <>
-
- Jul 04, 2023
-
-
Andrei Sandu authored
* Move expensive computations to blocking thread Signed-off-by: Andrei Sandu <[email protected]> * fix test Signed-off-by: Andrei Sandu <[email protected]> * add internal error and fix dependent subystems Signed-off-by: Andrei Sandu <[email protected]> * fmt Signed-off-by: Andrei Sandu <[email protected]> * fix test fix Signed-off-by: Andrei Sandu <[email protected]> * minor refactor and TODOs Signed-off-by: Andrei Sandu <[email protected]> * Impl Feedback for Review Signed-off-by: Andrei Sandu <[email protected]> * review feedback Signed-off-by: Andrei Sandu <[email protected]> * More docs Signed-off-by: Andrei Sandu <[email protected]> * add some example timings in comments Signed-off-by: Andrei Sandu <[email protected]> --------- Signed-off-by: Andrei Sandu <[email protected]>
-
- Jun 21, 2023
-
-
Koute authored
-
- May 30, 2023
-
-
Tsvetomir Dimitrov authored
* Replace `RollingSessionWindow` with `RuntimeInfo` - initial commit * Fix tests in import * Fix the rest of the tests * Remove dead code * Fix todos * Simplify session caching * Comments for `SessionInfoProvider` * Separate `SessionInfoProvider` from `State` * `cache_session_info_for_head` becomes freestanding function * Remove unneeded `mut` usage * fn session_info -> fn get_session_info() to avoid name clashes. The function also tries to initialize `SessionInfoProvider` * Fix SessionInfo retrieval * Code cleanup * Don't wrap `SessionInfoProvider` in an `Option` * Remove `earliest_session()` * Remove pre-caching -> wip * Fix some tests and code cleanup * Fix all tests * Fixes in tests * Fix comments, variable names and small style changes * Fix a warning * impl From<SessionWindowSize> for NonZeroUsize * Fix logging for `get_session_info` - remove redundant logs and decrease log level to DEBUG * Code review feedback * Storage migration removing `COL_SESSION_WINDOW_DATA` from parachains db * Remove `col_session_data` usages * Storage migration clearing columns w/o removing them * Remove session data column usages from `approval-voting` and `dispute-coordinator` tests * Add some test cases from `RollingSessionWindow` to `dispute-coordinator` tests * Fix formatting in initialized.rs * Fix a corner case in `SessionInfo` caching for `dispute-coordinator` * Remove `RollingSessionWindow` ;( * Revert "Fix formatting in initialized.rs" This reverts commit 0f94664ec9f3a7e3737a30291195990e1e7065fc. * v2 to v3 migration drops `COL_DISPUTE_COORDINATOR_DATA` instead of clearing it * Fix `NUM_COLUMNS` in `approval-voting` * Use `columns::v3::NUM_COLUMNS` when opening db * Update node/service/src/parachains_db/upgrade.rs Co-authored-by: Andrei Sandu <[email protected]> * Don't write in `COL_DISPUTE_COORDINATOR_DATA` for `test_rocksdb_migrate_2_to_3` * Fix `NUM+COLUMNS` in approval_voting * Fix formatting * Fix columns usage * Clarification comments about the different db versions --------- Co-authored-by: Andrei Sandu <[email protected]>
-
- May 12, 2023
-
-
Tsvetomir Dimitrov authored
* Replace `RollingSessionWindow` with `RuntimeInfo` - initial commit * Fix tests in import * Fix the rest of the tests * Remove dead code * Fix todos * Simplify session caching * Comments for `SessionInfoProvider` * Separate `SessionInfoProvider` from `State` * `cache_session_info_for_head` becomes freestanding function * Remove unneeded `mut` usage * fn session_info -> fn get_session_info() to avoid name clashes. The function also tries to initialize `SessionInfoProvider` * Fix SessionInfo retrieval * Code cleanup * Don't wrap `SessionInfoProvider` in an `Option` * Remove `earliest_session()` * Remove pre-caching -> wip * Fix some tests and code cleanup * Fix all tests * Fixes in tests * Fix comments, variable names and small style changes * Fix a warning * impl From<SessionWindowSize> for NonZeroUsize * Fix logging for `get_session_info` - remove redundant logs and decrease log level to DEBUG * Code review feedback
-
- May 04, 2023
-
-
Davide Galassi authored
* Companion for substrate #14036 * Rollback of some trivial renamings * update lockfile
-
- Apr 19, 2023
-
-
Davide Galassi authored
* Companion for substrate #13889 * Remove leftover * Remove removed dependency * Remove sp-consensus-vrf from lock * Revert "Remove sp-consensus-vrf from lock" This reverts commit 2269ca1e32df89272e8fd4544292204db387f436. * Fix after substrate modifications * Fix tests to use new VRF Signature type * Don't rely of Deref trait * Fix test * Further code simplification * Reuse garbage_vrf_signature * update lockfile for {"substrate"} --------- Co-authored-by: parity-processbot <>
-
- Apr 11, 2023
-
-
Mira Ressel authored
* Simplify &(ref foo, _) tuple pattern matches * Drop unneccessary .clone() calls * Replace .position(cond).is_some() by .any(cond) * Drop unneccessary lifetime specs
-
- Apr 08, 2023
-
-
s0me0ne-unkn0wn authored
* Happy New Year! * Remove year entierly Co-authored-by: Oliver Tale-Yazdi <[email protected]> * Remove years from copyright notice in the entire repo --------- Co-authored-by: Oliver Tale-Yazdi <[email protected]>
-
- Mar 31, 2023
-
-
Mattia L.V. Bradascio authored
* Pass the PerLeafSpan as mutable reference to handle_new_head function * cargo +nightly fmt --all * Add mock span for test * cargo +nightly fmt --all * add new-blocks-hashes to span * ref span in match statement, set span to disabled if not passed * remove second match clause, make handle_new_head_span mutable * cargo +nightly fmt --all * improve tag on error and warning * add imported blocks and info span * cargo +nightly fmt --all * Improve error for imported_blocks_and_info trace * format tags on get_header_span * add lost-to-finality tag * add missing bracket * - Add bitfield child span - Add block db insertion span * - fix update-bitfield span tag * - Fix type conversion to u64 - Add missing argument * - Cargo fmt * - Test add_follows_from * - Revert as relationship between spans not working correctly * - use drop to test if parent-child relationship can be re-established * - remove bitfield span, check if parent-child relationship can be reestablished * - Remove dangling bitfield span which is not used, to see if parent-child relationship can be re-established * Another dangling bitfield span * cargo fmt * - add imported blocks and info span - add candidate span per candidate * add tags before moving block_header to push scope * - Add db-insertion span * cargo fmt * fix types * * Pass mutable reference to span in handle_new_head * Change get-header-span tags in handle_new_head * Create cache-session-info span in handle_new_head * Create optional argument in determine_new_blocks * Pass mutable reference to handle_new_head_span in determine_new_blocks in handle_new_head function * Add candidate-hash, candidate-number, lost-to-finality tags to candidate_span in handle_new_head function * Manually drop db_insertion_span and remove superfluous tags to it, only keeping approved-bitfields tag * Add ApprovalVoting stage in jaeger * * Pass mutable reference to jaeger::Span in stead of PerLeafSpan * Add block-import span * *Pass optional_span (optional argument) to determine_new_blocks util function * * Add num-candidates int tag to block_import_span * * Add head tag to cache_session_span * * Create PerLeafSpan in handle_from_overseer (this is required to establish parent-child relationship between approval-voting span, and leaf-activated root span) * * Add candidate-import-span as child of block-import-span * Add candidate-hash and num-approval tags to candidate-import-span * * Fix num-candidate tag to bitvec-len tag in candidate-import-span * *Fix imported_blocKs_and_info span to create new-block-span as not dealing with candidates * Consider the future::select! block * Use HashMap<Hash, jaeger::PerLeafSpan> * Remove Stage 9 * Add missing spans * cargo +nightly fmt --all * Remove optional span argument for determine_new_blocks * * Remove no-longer needed default PerLeafSpan implementation * Remove no-longer necessary mock span given re-factoring of handle_new_head() no longer neeing mutable span * Split validation-result and request-data (availability and validation code) spans into two by dropping request_validation_data_spans * Remove drop statements for cache_session_info_span * * Remove unnecessary span * Remove another excessively spammy span * Add missing spans from State in import tests * Use functional approach to get spans * - Add functional approach for the approval-voting span - Add doc on block_numbers given labelling ambiguity - Add span pruning logic - Use .add_para_id on validation_result_span * Replace for hash_set in hash_set_iter with map closure * cargo +nightly fmt --all * Change from unconsumed `map` to `.for_each` * cargo +nightly fmt --all * Refactor add_para_id to validation_result_span * cargo +nightly fmt --all * Remove duplicate tag * Add missing tag to handle-approved-ancestor span * Refactor span pruning to only invoke retain once * Typo in span name * - Replace unwrap_or with unwrap_or_else due to lazy evaluation of trace-identifier in polkadot_node_jaeger - Remove some redundant spans * Add approval-distribution spans * - Add unwrap_or_else on note-approved-in-chain-selection - Use child_with_trace_id to add traceID string tag on span (note this does not change the traceID, but just adds a tag) * cargo +nightly fmt --all * - Add traceID tags were necessary in approval-voting and availability-distribution - Always use block-hash tag in stead of relay-parent tag in approval-distribution * Remove schedule-wakeup span as it will duplicate spans on existing wakeups (which should be a no-op) * Remove a couple of warnings related to mutability * Fix failing tests in availability distribution * Add traceID tag to launch-approval and validation-result * Reshuffle the validation and validation result spans to where more appropriate and add block-hash tag * - Add tranche and should-trigger tag to process-wakeup span - Add candidate-hash and traceID to check-and-import-approval span * cargo fmt * - Adjustments after PR comments * Move span pruning after other pruning logic * Remove DerefMut - no longer needed * Relabel request-chunk spans * - Fix typo in span label - Add docs for drops * Add new approval-voting span pruning logic * Undo removal of ! * cargo fmt
-
- Mar 23, 2023
-
-
Tsvetomir Dimitrov authored
-