Skip to content
  1. Feb 27, 2024
    • Petr Mensik's avatar
      Add Polkadotters bootnoders per IBP application (#3423) · 0cc9b900
      Petr Mensik authored
      Hey everyone,
      
      this PR will replace existing Polkadotters bootnodes for Polkadot,
      Kusama and Westend and add Paseo bootnode to the relay chain suite. At
      the same time, it will add new bootnodes for all the system parachains,
      including People on Westend. This PR is a part of our membership in the
      IBP, meaning that all the bootnodes are hosted on our hardware housed in
      the data center in Christchurch, New Zealand.
      All the bootnodes were tested with an empty chain spec file with the
      following command yielding 1 peer.
      
      The test commands used are as follows:
      ```
      ./polkadot --base-path /tmp/node --reserved-only --chain paseo --reserved-nodes "/dns/paseo.bootnodes.polkadotters.com/tcp/30540/wss/p2p/12D3KooWPbbFy4TefEGTRF5eTYhq8LEzc4VAHdNUVCbY4nAnhqPP"
      ./polkadot --base-path /tmp/node --reserved-only --chain westend --reserved-nodes "/dns/westend.bootnodes.polkadotters.com/tcp/30310/wss/p2p/12D3KooWHPHb64jXMtSRJDrYFATWeLnvChL8NtWVttY67DCH1eC5"
      ./polkadot --base-path /tmp/node --reserved-only --chain kusama --reserved-nodes "/dns/kusama.bootnodes.polkadotters.com/tcp/30313/wss/p2p/12D3KooWHB5rTeNkQdXNJ9ynvGz8Lpnmsctt7Tvp7mrYv6bcwbPG"
      ./polkadot --base-path /tmp/node --no-hardware-benchmarks --reserved-only --chain polkadot --reserved-nodes "/dns/polkadot.bootnodes.polkadotters.com/tcp/30316/wss/p2p/12D3KooWPAVUgBaBk6n8SztLrMk8ESByncbAfRKUdxY1nygb9zG3"
      ./polkadot-parachain --base-path /tmp/node --reserved-only --chain asset-hub-kusama --reserved-nodes "/dns/asset-hub-kusama.bootnodes.polkadotters.com/tcp/30513/wss/p2p/12D3KooWDpk7wVH7RgjErEvbvAZ2kY5VeaAwRJP5ojmn1e8b8UbU"
      ./polkadot-parachain --base-path /tmp/node --reserved-only --chain asset-hub-polkadot --reserved-nodes "/dns/asset-hub-polkadot.bootnodes.polkadotters.com/tcp/30510/wss/p2p/12D3KooWKbfY9a9oywxMJKiALmt7yhrdQkjXMtvxhhDDN23vG93R"
      ./polkadot-parachain --base-path /tmp/node --reserved-only --chain asset-hub-westend --reserved-nodes "/dns/asset-hub-westend.bootnodes.polkadotters.com/tcp/30516/wss/p2p/12D3KooWNFYysCqmojxqjjaTfD2VkWBNngfyUKWjcR4WFixfHNTk"
      ./polkadot-parachain --base-path /tmp/node --reserved-only --chain bridge-hub-kusama --reserved-nodes "/dns/bridge-hub-kusama.bootnodes.polkadotters.com/tcp/30522/wss/p2p/12D3KooWH3pucezRRS5esoYyzZsUkKWcPSByQxEvmM819QL1HPLV"
      ./polkadot-parachain --base-path /tmp/node --reserved-only --chain bridge-hub-kusama --reserved-nodes "/dns/bridge-hub-kusama.bootnodes.polkadotters.com/tcp/30522/wss/p2p/12D3KooWH3pucezRRS5esoYyzZsUkKWcPSByQxEvmM819QL1HPLV"
      ./polkadot-parachain --base-path /tmp/node --reserved-only --chain bridge-hub-westend --reserved-nodes "/dns/bridge-hub-westend.bootnodes.polkadotters.com/tcp/30525/wss/p2p/12D3KooWPkwgJofp4GeeRwNgXqkp2aFwdLkCWv3qodpBJLwK43Jj"
      ./polkadot-parachain --base-path /tmp/node --reserved-only --chain collectives-polkadot --reserved-nodes "/dns/collectives-polkadot.bootnodes.polkadotters.com/tcp/30528/wss/p2p/12D3KooWNohUjvJtGKUa8Vhy8C1ZBB5N8JATB6e7rdLVCioeb3ff"
      ./polkadot-parachain --base-path /tmp/node --reserved-only --chain collectives-westend --reserved-nodes "/dns/collectives-westend.bootnodes.polkadotters.com/tcp/30531/wss/p2p/12D3KooWAFkXNSBfyPduZVgfS7pj5NuVpbU8Ee5gHeF8wvos7Yqn"
      ./polkadot-parachain --base-path /tmp/node --reserved-only --chain people-westend --reserved-nodes "/dns/identity-westend.bootnodes.polkadotters.com/tcp/30534/wss/p2p/12D3KooWKr9San6KTM7REJ95cBaDoiciGcWnW8TTftEJgxGF5Ehb"
      ```
      
      Best regards,
      
      Petr, Polkadotters
      0cc9b900
  2. Feb 26, 2024
  3. Feb 23, 2024
    • Andrei Sandu's avatar
      Runtime: allow backing multiple candidates of same parachain on different cores (#3231) · 2431001e
      Andrei Sandu authored
      
      
      Fixes https://github.com/paritytech/polkadot-sdk/issues/3144
      
      Builds on top of https://github.com/paritytech/polkadot-sdk/pull/3229
      
      ### Summary
      Some preparations for Runtime to support elastic scaling, guarded by
      config node features bit `FeatureIndex::ElasticScalingMVP`. This PR
      introduces a per-candidate `CoreIndex` but does it in a hacky way to
      avoid changing `CandidateCommitments`, `CandidateReceipts` primitives
      and networking protocols.
      
      #### Including `CoreIndex` in `BackedCandidate`
      If the `ElasticScalingMVP` feature bit is enabled then
      `BackedCandidate::validator_indices` is extended by 8 bits.
      The value stored in these bits represents the assumed core index for the
      candidate.
      
      It is temporary solution which works by creating a mapping from
      `BackedCandidate` to `CoreIndex` by assuming the `CoreIndex` can be
      discovered by checking in which validator group the validator that
      signed the statement is.
      
      TODO:
      - [x] fix tests
      - [x] add new tests
      - [x] Bump runtime API for Kusama, so we have that node features thing!
      -> https://github.com/polkadot-fellows/runtimes/pull/194
      
      ---------
      
      Signed-off-by: default avatarAndrei Sandu <[email protected]>
      Signed-off-by: default avataralindima <[email protected]>
      Co-authored-by: default avataralindima <[email protected]>
      2431001e
    • Ignacio Palacios's avatar
      Fix `DepositReserveAsset` fees payment (#3340) · 11b5354f
      Ignacio Palacios authored
      The `fee` should be calculated with the reanchored asset, otherwise it
      could lead to a failure where the set aside fee ends up not being
      enough.
      
      @acatangiu
      
      
      
      ---------
      
      Co-authored-by: default avatarAdrian Catangiu <[email protected]>
      11b5354f
  4. Feb 22, 2024
    • Bastian Köcher's avatar
      Check that the validation code matches the parachain code (#3433) · 9bf1a5e2
      Bastian Köcher authored
      This introduces a check to ensure that the parachain code matches the
      validation code stored in the relay chain state. If not, it will print a
      warning. This should be mainly useful for parachain builders to make
      sure they have setup everything correctly.
      9bf1a5e2
    • Andrei Sandu's avatar
      Elastic scaling: use an assumed `CoreIndex` in `candidate-backing` (#3229) · 60e537b9
      Andrei Sandu authored
      
      
      First step in implementing
      https://github.com/paritytech/polkadot-sdk/issues/3144
      
      ### Summary of changes
      - switch statement `Table` candidate mapping from `ParaId` to
      `CoreIndex`
      - introduce experimental `InjectCoreIndex`  node feature.
      - determine and assume a `CoreIndex` for a candidate based on statement
      validator index. If the signature is valid it means validator controls
      the validator that index and we can easily map it to a validator
      group/core.
      - introduce a temporary provisioner fix until we fully enable elastic
      scaling in the subystem. The fix ensures we don't fetch the same
      backable candidate when calling `get_backable_candidate` for each core.
      
      TODO:
      - [x] fix backing tests
      - [x] fix statement table tests
      - [x] add new test
      
      ---------
      
      Signed-off-by: default avatarAndrei Sandu <[email protected]>
      Signed-off-by: default avataralindima <[email protected]>
      Co-authored-by: default avataralindima <[email protected]>
      60e537b9
  5. Feb 20, 2024
    • Niklas Adolfsson's avatar
      rpc server: make possible to disable/enable batch requests (#3364) · fee810a5
      Niklas Adolfsson authored
      The rationale behind this, is that it may be useful for some users
      actually disable RPC batch requests or limit them by length instead of
      the total size bytes of the batch.
      
      This PR adds two new CLI options:
      
      ```
      --rpc-disable-batch-requests - disable batch requests on the server
      --rpc-max-batch-request-len <LEN> - limit batches to LEN on the server.
      ```
      fee810a5
    • Oliver Tale-Yazdi's avatar
      Lift dependencies to the workspace (Part 2/x) (#3366) · e89d0fca
      Oliver Tale-Yazdi authored
      
      
      Lifting some more dependencies to the workspace. Just using the
      most-often updated ones for now.
      It can be reproduced locally.
      
      ```sh
      # First you can check if there would be semver incompatible bumps (looks good in this case):
      $ zepter transpose dependency lift-to-workspace --ignore-errors syn quote thiserror "regex:^serde.*"
      
      # Then apply the changes:
      $ zepter transpose dependency lift-to-workspace --version-resolver=highest syn quote thiserror "regex:^serde.*" --fix
      
      # And format the changes:
      $ taplo format --config .config/taplo.toml
      ```
      
      ---------
      
      Signed-off-by: default avatarOliver Tale-Yazdi <[email protected]>
      e89d0fca
    • Bastian Köcher's avatar
      Downgrade log message to `trace` (#3405) · ef6ac94f
      Bastian Köcher authored
      This spams logs in `Debug` with no useful information.
      ef6ac94f
  6. Feb 19, 2024
    • Alexandru Gheorghe's avatar
      gossip-support: add unittests for update authorities (#3258) · 197c6cf9
      Alexandru Gheorghe authored
      
      
      ~The previous fix was actually incomplete because we update the
      authorties only on the situation where we decided to reconnect because
      we had a low connectivity issue. Now the problem is that
      update_authority_ids use the list of connected peers, so on restart that
      does contain anything, so calling immediately after
      issue_connection_request won't detect all authorities, so we need to
      also check every block as the comment said, but that did not match the
      code.~
      
      Actually the fix was correct the flow is follow if more than 1/3 of the
      authorities can not be resolved we set last_failure and call
      `ConnectToResolvedValidators`.
      
      We will call UpdateAuthorities for all the authorities already connected
      and for which we already know the address and for the ones that will
      connect later on `PeerConnected` will have the AuthorityId field set,
      because it is already known, so approval-distribution will update its
      cache topology.
      
      ---------
      
      Signed-off-by: default avatarAlexandru Gheorghe <[email protected]>
      Co-authored-by: default avatarAlexander Samusev <[email protected]>
      197c6cf9
    • dependabot[bot]'s avatar
      Bump the known_good_semver group with 1 update (#3379) · 3168ed4d
      dependabot[bot] authored
      
      
      Bumps the known_good_semver group with 1 update:
      [clap](https://github.com/clap-rs/clap).
      
      Updates `clap` from 4.5.0 to 4.5.1
      <details>
      <summary>Release notes</summary>
      <p><em>Sourced from <a
      href="https://github.com/clap-rs/clap/releases">clap's
      releases</a>.</em></p>
      <blockquote>
      <h2>v4.5.1</h2>
      <h2>[4.5.1] - 2024-02-16</h2>
      <h3>Fixes</h3>
      <ul>
      <li><em>(error)</em> Include suggestion to add <code>--</code> even if
      there is a &quot;did you mean&quot; so long as <code>last</code> or
      <code>trailing_var_arg</code> is used</li>
      </ul>
      </blockquote>
      </details>
      <details>
      <summary>Changelog</summary>
      <p><em>Sourced from <a
      href="https://github.com/clap-rs/clap/blob/master/CHANGELOG.md">clap's
      changelog</a>.</em></p>
      <blockquote>
      <h2>[4.5.1] - 2024-02-16</h2>
      <h3>Fixes</h3>
      <ul>
      <li><em>(error)</em> Include suggestion to add <code>--</code> even if
      there is a &quot;did you mean&quot; so long as <code>last</code> or
      <code>trailing_var_arg</code> is used</li>
      </ul>
      </blockquote>
      </details>
      <details>
      <summary>Commits</summary>
      <ul>
      <li><a
      href="https://github.com/clap-rs/clap/commit/0c01b5558de0a1a513a2af429981099e550d1a78"><code>0c01b55</code></a>
      chore: Release</li>
      <li><a
      href="https://github.com/clap-rs/clap/commit/08e0b5bde4b45567be98d12e7d394762fa8da6a4"><code>08e0b5b</code></a>
      docs: Update changelog</li>
      <li><a
      href="https://github.com/clap-rs/clap/commit/f2c4e6ec740cce29cea3fd7232e4d2716f089fc2"><code>f2c4e6e</code></a>
      Merge pull request <a
      href="https://redirect.github.com/clap-rs/clap/issues/5359">#5359</a>
      from poliorcetics/ab/push-szymvyzpmnqx</li>
      <li><a
      href="https://github.com/clap-rs/clap/commit/e782775229081697e47d340a155d6b087eeb46f8"><code>e782775</code></a>
      fix(complete): Handle newlines in command/arg descriptions</li>
      <li><a
      href="https://github.com/clap-rs/clap/commit/fba7c8597bf3028400d6f6a22d83cedb7a2bd5a5"><code>fba7c85</code></a>
      test(complete): Show newline issue</li>
      <li><a
      href="https://github.com/clap-rs/clap/commit/8a7a13a5618cfdc4ff328624a5266e7b4d88649a"><code>8a7a13a</code></a>
      chore: Release</li>
      <li><a
      href="https://github.com/clap-rs/clap/commit/7b3a3e1e5eee57fc37110343b980cbe8170d20ad"><code>7b3a3e1</code></a>
      docs: Update changelog</li>
      <li><a
      href="https://github.com/clap-rs/clap/commit/7b624ca74336f6f14f01007b9039990d53acda0f"><code>7b624ca</code></a>
      Merge pull request <a
      href="https://redirect.github.com/clap-rs/clap/issues/5356">#5356</a>
      from epage/escape</li>
      <li><a
      href="https://github.com/clap-rs/clap/commit/446328a8d3cdaac28884bf8fdfcc85f35c8b5134"><code>446328a</code></a>
      fix(error): Include -- in more cases</li>
      <li><a
      href="https://github.com/clap-rs/clap/commit/7de6df878238ca8e3d9723bb9650f7fe9470d8bd"><code>7de6df8</code></a>
      test(error): Show existing last behavior</li>
      <li>Additional commits viewable in <a
      href="https://github.com/clap-rs/clap/compare/clap_complete-v4.5.0...clap_complete-v4.5.1">compare
      view</a></li>
      </ul>
      </details>
      <br />
      
      
      [![Dependabot compatibility
      score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=clap&package-manager=cargo&previous-version=4.5.0&new-version=4.5.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
      
      Dependabot will resolve any conflicts with this PR as long as you don't
      alter it yourself. You can also trigger a rebase manually by commenting
      `@dependabot rebase`.
      
      [//]: # (dependabot-automerge-start)
      [//]: # (dependabot-automerge-end)
      
      ---
      
      <details>
      <summary>Dependabot commands and options</summary>
      <br />
      
      You can trigger Dependabot actions by commenting on this PR:
      - `@dependabot rebase` will rebase this PR
      - `@dependabot recreate` will recreate this PR, overwriting any edits
      that have been made to it
      - `@dependabot merge` will merge this PR after your CI passes on it
      - `@dependabot squash and merge` will squash and merge this PR after
      your CI passes on it
      - `@dependabot cancel merge` will cancel a previously requested merge
      and block automerging
      - `@dependabot reopen` will reopen this PR if it is closed
      - `@dependabot close` will close this PR and stop Dependabot recreating
      it. You can achieve the same result by closing it manually
      - `@dependabot show <dependency name> ignore conditions` will show all
      of the ignore conditions of the specified dependency
      - `@dependabot ignore <dependency name> major version` will close this
      group update PR and stop Dependabot creating any more for the specific
      dependency's major version (unless you unignore this specific
      dependency's major version or upgrade to it yourself)
      - `@dependabot ignore <dependency name> minor version` will close this
      group update PR and stop Dependabot creating any more for the specific
      dependency's minor version (unless you unignore this specific
      dependency's minor version or upgrade to it yourself)
      - `@dependabot ignore <dependency name>` will close this group update PR
      and stop Dependabot creating any more for the specific dependency
      (unless you unignore this specific dependency or upgrade to it yourself)
      - `@dependabot unignore <dependency name>` will remove all of the ignore
      conditions of the specified dependency
      - `@dependabot unignore <dependency name> <ignore condition>` will
      remove the ignore condition of the specified dependency and ignore
      conditions
      
      
      </details>
      
      Signed-off-by: default avatardependabot[bot] <[email protected]>
      Co-authored-by: default avatardependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
      3168ed4d
  7. Feb 17, 2024
  8. Feb 16, 2024
    • dependabot[bot]'s avatar
      Bump the known_good_semver group with 6 updates (#3347) · 4b484e6a
      dependabot[bot] authored
      
      
      Bumps the known_good_semver group with 6 updates:
      
      | Package | From | To |
      | --- | --- | --- |
      | [serde](https://github.com/serde-rs/serde) | `1.0.195` | `1.0.196` |
      | [serde_json](https://github.com/serde-rs/json) | `1.0.111` | `1.0.113`
      |
      | [clap](https://github.com/clap-rs/clap) | `4.4.18` | `4.5.0` |
      | [syn](https://github.com/dtolnay/syn) | `2.0.48` | `2.0.49` |
      | [serde_yaml](https://github.com/dtolnay/serde-yaml) | `0.9.30` |
      `0.9.31` |
      | [serde_derive](https://github.com/serde-rs/serde) | `1.0.195` |
      `1.0.196` |
      
      Updates `serde` from 1.0.195 to 1.0.196
      <details>
      <summary>Release notes</summary>
      <p><em>Sourced from <a
      href="https://github.com/serde-rs/serde/releases">serde's
      releases</a>.</em></p>
      <blockquote>
      <h2>v1.0.196</h2>
      <ul>
      <li>Improve formatting of &quot;invalid type&quot; error messages
      involving floats (<a
      href="https://redirect.github.com/serde-rs/serde/issues/2682">#2682</a>)</li>
      </ul>
      </blockquote>
      </details>
      <details>
      <summary>Commits</summary>
      <ul>
      <li><a
      href="https://github.com/serde-rs/serde/commit/ede9762a583c3cc3b87c10a53551828fad339525"><code>ede9762</code></a>
      Release 1.0.196</li>
      <li><a
      href="https://github.com/serde-rs/serde/commit/d438c2d67bf30e3edab31c2272c4829c12cf4cb5"><code>d438c2d</code></a>
      Merge pull request <a
      href="https://redirect.github.com/serde-rs/serde/issues/2682">#2682</a>
      from dtolnay/decimalpoint</li>
      <li><a
      href="https://github.com/serde-rs/serde/commit/bef110b92a171ac568a47339f5bd97938a8c9da2"><code>bef110b</code></a>
      Format Unexpected::Float with decimal point</li>
      <li><a
      href="https://github.com/serde-rs/serde/commit/b971ef11d1b53da7673e0c8199e87509c003c1a8"><code>b971ef1</code></a>
      Merge pull request <a
      href="https://redirect.github.com/serde-rs/serde/issues/2681">#2681</a>
      from dtolnay/workspacedeps</li>
      <li><a
      href="https://github.com/serde-rs/serde/commit/29d9f693996d199748136d5561a971ed68626724"><code>29d9f69</code></a>
      Fix workspace.dependencies default-features future compat warning</li>
      <li><a
      href="https://github.com/serde-rs/serde/commit/aecb4083bde754155752f5d7d442b64eb7dc636f"><code>aecb408</code></a>
      Sort workspace dependencies</li>
      <li><a
      href="https://github.com/serde-rs/serde/commit/1c675ab3a38e46df4e62465800970f8b20a2055d"><code>1c675ab</code></a>
      Merge pull request <a
      href="https://redirect.github.com/serde-rs/serde/issues/2678">#2678</a>
      from rodoufu/workspaceDependencies</li>
      <li><a
      href="https://github.com/serde-rs/serde/commit/dd619630a337139424725697ccd9a9f7596a2d3a"><code>dd61963</code></a>
      Adding workspace dependencies</li>
      <li><a
      href="https://github.com/serde-rs/serde/commit/111803ab0768d010c606f2fc0d0add12750d5eef"><code>111803a</code></a>
      Merge pull request <a
      href="https://redirect.github.com/serde-rs/serde/issues/2673">#2673</a>
      from Sky9x/msrv-badge</li>
      <li><a
      href="https://github.com/serde-rs/serde/commit/0024f74f34fbbdc44a7b22457faebe36c5cbe7f8"><code>0024f74</code></a>
      Use shields.io's MSRV badges</li>
      <li>See full diff in <a
      href="https://github.com/serde-rs/serde/compare/v1.0.195...v1.0.196">compare
      view</a></li>
      </ul>
      </details>
      <br />
      
      Updates `serde_json` from 1.0.111 to 1.0.113
      <details>
      <summary>Release notes</summary>
      <p><em>Sourced from <a
      href="https://github.com/serde-rs/json/releases">serde_json's
      releases</a>.</em></p>
      <blockquote>
      <h2>v1.0.113</h2>
      <ul>
      <li>Add <code>swap_remove</code> and <code>shift_remove</code> methods
      on Map (<a
      href="https://redirect.github.com/serde-rs/json/issues/1109">#1109</a>)</li>
      </ul>
      <h2>v1.0.112</h2>
      <ul>
      <li>Improve formatting of &quot;invalid type&quot; error messages
      involving floats (<a
      href="https://redirect.github.com/serde-rs/json/issues/1107">#1107</a>)</li>
      </ul>
      </blockquote>
      </details>
      <details>
      <summary>Commits</summary>
      <ul>
      <li><a
      href="https://github.com/serde-rs/json/commit/09d865b34b9701be52764dc9bf571b1a16e9d3dc"><code>09d865b</code></a>
      Release 1.0.113</li>
      <li><a
      href="https://github.com/serde-rs/json/commit/5aeab4eaf69d7959f013f8081865c264d6c00551"><code>5aeab4e</code></a>
      Merge pull request <a
      href="https://redirect.github.com/serde-rs/json/issues/1109">#1109</a>
      from serde-rs/remove</li>
      <li><a
      href="https://github.com/serde-rs/json/commit/ca3c2ca3696cab79b8b279be7569ee1647250f1e"><code>ca3c2ca</code></a>
      Add swap_remove and shift_remove methods on Map</li>
      <li><a
      href="https://github.com/serde-rs/json/commit/7fece969e3b480ec620419d65c2aeb08776bebcb"><code>7fece96</code></a>
      Release 1.0.112</li>
      <li><a
      href="https://github.com/serde-rs/json/commit/6a6d2bbd9e8b8bd72573b863f12a4ec991f33232"><code>6a6d2bb</code></a>
      Merge pull request <a
      href="https://redirect.github.com/serde-rs/json/issues/1107">#1107</a>
      from serde-rs/unexpectedfloat</li>
      <li><a
      href="https://github.com/serde-rs/json/commit/83d7bad54ba5db3a44198d6df0ff2e81621683fa"><code>83d7bad</code></a>
      Format f64 in error messages using ryu</li>
      <li><a
      href="https://github.com/serde-rs/json/commit/107c2d1c42817f0d71f07a4d5b0ea2f29dbce8b8"><code>107c2d1</code></a>
      Merge pull request <a
      href="https://redirect.github.com/serde-rs/json/issues/1106">#1106</a>
      from serde-rs/invalidvalue</li>
      <li><a
      href="https://github.com/serde-rs/json/commit/62ca3e4c01c2e62cd5c2a32e9104f386e5ce7808"><code>62ca3e4</code></a>
      Handle Unexpected::Unit in Error::invalid_value</li>
      <li><a
      href="https://github.com/serde-rs/json/commit/296fafb8f32e8442ef8e4d5725c15ffca726b288"><code>296fafb</code></a>
      Factor out JSON-specific Display impl for serde::de::Unexpected</li>
      <li><a
      href="https://github.com/serde-rs/json/commit/e56cc696bd7c112e5dd4ccfa23d094c3a1c1c1ff"><code>e56cc69</code></a>
      Merge pull request <a
      href="https://redirect.github.com/serde-rs/json/issues/1105">#1105</a>
      from keienWang/master</li>
      <li>Additional commits viewable in <a
      href="https://github.com/serde-rs/json/compare/v1.0.111...v1.0.113">compare
      view</a></li>
      </ul>
      </details>
      <br />
      
      Updates `clap` from 4.4.18 to 4.5.0
      <details>
      <summary>Changelog</summary>
      <p><em>Sourced from <a
      href="https://github.com/clap-rs/clap/blob/master/CHANGELOG.md">clap's
      changelog</a>.</em></p>
      <blockquote>
      <h2>[4.5.0] - 2024-02-08</h2>
      <h3>Compatibility</h3>
      <ul>
      <li>Update MSRV to 1.74</li>
      </ul>
      </blockquote>
      </details>
      <details>
      <summary>Commits</summary>
      <ul>
      <li><a
      href="https://github.com/clap-rs/clap/commit/a751c5fe65cd33cb09e85ff3039b4fd0182cdb6e"><code>a751c5f</code></a>
      chore: Release</li>
      <li><a
      href="https://github.com/clap-rs/clap/commit/9ec6c942b81c370a8c14652e42295933244555ac"><code>9ec6c94</code></a>
      chore: Release</li>
      <li><a
      href="https://github.com/clap-rs/clap/commit/0735119775c2d27fef6b3c232cb9ef2fcbbd963f"><code>0735119</code></a>
      docs: Update changelog</li>
      <li><a
      href="https://github.com/clap-rs/clap/commit/c4d3959506f5068f86ffb7ab34b622bd2da40dd8"><code>c4d3959</code></a>
      Merge pull request <a
      href="https://redirect.github.com/clap-rs/clap/issues/5344">#5344</a>
      from epage/encode</li>
      <li><a
      href="https://github.com/clap-rs/clap/commit/f750e577789e1dd34c6950d8c8fe16d1bfd1f49c"><code>f750e57</code></a>
      fix(lex): Use new-ish OsStr API</li>
      <li><a
      href="https://github.com/clap-rs/clap/commit/1d9a554cdfddffcb92d197706b9720e2760cb443"><code>1d9a554</code></a>
      Merge pull request <a
      href="https://redirect.github.com/clap-rs/clap/issues/5343">#5343</a>
      from epage/msrv</li>
      <li><a
      href="https://github.com/clap-rs/clap/commit/4b45d361b1b96eb1f37b7d1db2684203ca828a94"><code>4b45d36</code></a>
      chore: Update MSRV to 1.74</li>
      <li><a
      href="https://github.com/clap-rs/clap/commit/55b1f945157d4d0e480dea6c492e04d0b541088a"><code>55b1f94</code></a>
      Merge pull request <a
      href="https://redirect.github.com/clap-rs/clap/issues/5342">#5342</a>
      from epage/divan</li>
      <li><a
      href="https://github.com/clap-rs/clap/commit/ea77b98da3f321069ae136a241a2d266c2b28bd7"><code>ea77b98</code></a>
      test(complete): Make it order independent</li>
      <li><a
      href="https://github.com/clap-rs/clap/commit/b0fea2bac60819bdb042e10c032817fded67f815"><code>b0fea2b</code></a>
      test(bench): Switch to divan</li>
      <li>Additional commits viewable in <a
      href="https://github.com/clap-rs/clap/compare/v4.4.18...clap_complete-v4.5.0">compare
      view</a></li>
      </ul>
      </details>
      <br />
      
      Updates `syn` from 2.0.48 to 2.0.49
      <details>
      <summary>Release notes</summary>
      <p><em>Sourced from <a
      href="https://github.com/dtolnay/syn/releases">syn's
      releases</a>.</em></p>
      <blockquote>
      <h2>2.0.49</h2>
      <ul>
      <li>Improve error location when parsing from an empty string literal
      using <code>LitStr::parse</code> (<a
      href="https://redirect.github.com/dtolnay/syn/issues/1590">#1590</a>)</li>
      </ul>
      </blockquote>
      </details>
      <details>
      <summary>Commits</summary>
      <ul>
      <li><a
      href="https://github.com/dtolnay/syn/commit/e64c0636042bcd8cf56f9e6c3e828a49cb434649"><code>e64c063</code></a>
      Release 2.0.49</li>
      <li><a
      href="https://github.com/dtolnay/syn/commit/981359c5f46f25f3db42d1b341cc607f634688e7"><code>981359c</code></a>
      Merge pull request <a
      href="https://redirect.github.com/dtolnay/syn/issues/1590">#1590</a>
      from dtolnay/streof</li>
      <li><a
      href="https://github.com/dtolnay/syn/commit/51298d40a447b9e5b3cd8c898c2ea9d24b31cd6d"><code>51298d4</code></a>
      Improve error location at eof in LitStr::parse</li>
      <li><a
      href="https://github.com/dtolnay/syn/commit/270c63384a9d891f65880a03a5a92f0bf4605bbf"><code>270c633</code></a>
      Update test suite to nightly-2024-02-13</li>
      <li><a
      href="https://github.com/dtolnay/syn/commit/dc9cf16b9b0601d2d393d059c46a8f8e0220cc21"><code>dc9cf16</code></a>
      Remove FilterAttrs trait when unused</li>
      <li><a
      href="https://github.com/dtolnay/syn/commit/7dcfac79eda716fa806d68cc0aa811f3be717dc0"><code>7dcfac7</code></a>
      Ignore dead_code warning in test</li>
      <li><a
      href="https://github.com/dtolnay/syn/commit/98318441089f9a9bb596fc010ed14c3b593d4bda"><code>9831844</code></a>
      Update signature of Emitter::emit_diagnostic in nightly-2024-02-07</li>
      <li><a
      href="https://github.com/dtolnay/syn/commit/9e8033f63da93dfb6a3e698449c6edff5e56727d"><code>9e8033f</code></a>
      Update test suite to nightly-2024-02-07</li>
      <li><a
      href="https://github.com/dtolnay/syn/commit/cb3348cd94be50995054796facdfa055b6e0e9e8"><code>cb3348c</code></a>
      Update test suite to nightly-2024-01-23</li>
      <li><a
      href="https://github.com/dtolnay/syn/commit/15b9dbcd67bc56f3da7d33bbede46f7380fc7164"><code>15b9dbc</code></a>
      Update test suite to nightly-2024-01-19</li>
      <li>Additional commits viewable in <a
      href="https://github.com/dtolnay/syn/compare/2.0.48...2.0.49">compare
      view</a></li>
      </ul>
      </details>
      <br />
      
      Updates `serde_yaml` from 0.9.30 to 0.9.31
      <details>
      <summary>Release notes</summary>
      <p><em>Sourced from <a
      href="https://github.com/dtolnay/serde-yaml/releases">serde_yaml's
      releases</a>.</em></p>
      <blockquote>
      <h2>0.9.31</h2>
      <ul>
      <li>Add <code>swap_remove</code> and <code>shift_remove</code> methods
      on Mapping (<a
      href="https://redirect.github.com/dtolnay/serde-yaml/issues/408">#408</a>)</li>
      </ul>
      </blockquote>
      </details>
      <details>
      <summary>Commits</summary>
      <ul>
      <li><a
      href="https://github.com/dtolnay/serde-yaml/commit/2a77483b23897115bdd69af32ae34d593268d555"><code>2a77483</code></a>
      Release 0.9.31</li>
      <li><a
      href="https://github.com/dtolnay/serde-yaml/commit/d8d1a839cf42ef4627c66bebd743f5fecd24e3ed"><code>d8d1a83</code></a>
      Merge pull request <a
      href="https://redirect.github.com/dtolnay/serde-yaml/issues/408">#408</a>
      from dtolnay/remove</li>
      <li><a
      href="https://github.com/dtolnay/serde-yaml/commit/f8a99a496836ccfa6c547af9136986d13011be98"><code>f8a99a4</code></a>
      Add swap_remove and shift_remove methods on Mapping</li>
      <li><a
      href="https://github.com/dtolnay/serde-yaml/commit/8b26413e3307e39329dc68a96b065058aec38f9a"><code>8b26413</code></a>
      Work around dead_code warning in tests</li>
      <li>See full diff in <a
      href="https://github.com/dtolnay/serde-yaml/compare/0.9.30...0.9.31">compare
      view</a></li>
      </ul>
      </details>
      <br />
      
      Updates `serde_derive` from 1.0.195 to 1.0.196
      <details>
      <summary>Release notes</summary>
      <p><em>Sourced from <a
      href="https://github.com/serde-rs/serde/releases">serde_derive's
      releases</a>.</em></p>
      <blockquote>
      <h2>v1.0.196</h2>
      <ul>
      <li>Improve formatting of &quot;invalid type&quot; error messages
      involving floats (<a
      href="https://redirect.github.com/serde-rs/serde/issues/2682">#2682</a>)</li>
      </ul>
      </blockquote>
      </details>
      <details>
      <summary>Commits</summary>
      <ul>
      <li><a
      href="https://github.com/serde-rs/serde/commit/ede9762a583c3cc3b87c10a53551828fad339525"><code>ede9762</code></a>
      Release 1.0.196</li>
      <li><a
      href="https://github.com/serde-rs/serde/commit/d438c2d67bf30e3edab31c2272c4829c12cf4cb5"><code>d438c2d</code></a>
      Merge pull request <a
      href="https://redirect.github.com/serde-rs/serde/issues/2682">#2682</a>
      from dtolnay/decimalpoint</li>
      <li><a
      href="https://github.com/serde-rs/serde/commit/bef110b92a171ac568a47339f5bd97938a8c9da2"><code>bef110b</code></a>
      Format Unexpected::Float with decimal point</li>
      <li><a
      href="https://github.com/serde-rs/serde/commit/b971ef11d1b53da7673e0c8199e87509c003c1a8"><code>b971ef1</code></a>
      Merge pull request <a
      href="https://redirect.github.com/serde-rs/serde/issues/2681">#2681</a>
      from dtolnay/workspacedeps</li>
      <li><a
      href="https://github.com/serde-rs/serde/commit/29d9f693996d199748136d5561a971ed68626724"><code>29d9f69</code></a>
      Fix workspace.dependencies default-features future compat warning</li>
      <li><a
      href="https://github.com/serde-rs/serde/commit/aecb4083bde754155752f5d7d442b64eb7dc636f"><code>aecb408</code></a>
      Sort workspace dependencies</li>
      <li><a
      href="https://github.com/serde-rs/serde/commit/1c675ab3a38e46df4e62465800970f8b20a2055d"><code>1c675ab</code></a>
      Merge pull request <a
      href="https://redirect.github.com/serde-rs/serde/issues/2678">#2678</a>
      from rodoufu/workspaceDependencies</li>
      <li><a
      href="https://github.com/serde-rs/serde/commit/dd619630a337139424725697ccd9a9f7596a2d3a"><code>dd61963</code></a>
      Adding workspace dependencies</li>
      <li><a
      href="https://github.com/serde-rs/serde/commit/111803ab0768d010c606f2fc0d0add12750d5eef"><code>111803a</code></a>
      Merge pull request <a
      href="https://redirect.github.com/serde-rs/serde/issues/2673">#2673</a>
      from Sky9x/msrv-badge</li>
      <li><a
      href="https://github.com/serde-rs/serde/commit/0024f74f34fbbdc44a7b22457faebe36c5cbe7f8"><code>0024f74</code></a>
      Use shields.io's MSRV badges</li>
      <li>See full diff in <a
      href="https://github.com/serde-rs/serde/compare/v1.0.195...v1.0.196">compare
      view</a></li>
      </ul>
      </details>
      <br />
      
      
      Dependabot will resolve any conflicts with this PR as long as you don't
      alter it yourself. You can also trigger a rebase manually by commenting
      `@dependabot rebase`.
      
      [//]: # (dependabot-automerge-start)
      [//]: # (dependabot-automerge-end)
      
      ---
      
      <details>
      <summary>Dependabot commands and options</summary>
      <br />
      
      You can trigger Dependabot actions by commenting on this PR:
      - `@dependabot rebase` will rebase this PR
      - `@dependabot recreate` will recreate this PR, overwriting any edits
      that have been made to it
      - `@dependabot merge` will merge this PR after your CI passes on it
      - `@dependabot squash and merge` will squash and merge this PR after
      your CI passes on it
      - `@dependabot cancel merge` will cancel a previously requested merge
      and block automerging
      - `@dependabot reopen` will reopen this PR if it is closed
      - `@dependabot close` will close this PR and stop Dependabot recreating
      it. You can achieve the same result by closing it manually
      - `@dependabot show <dependency name> ignore conditions` will show all
      of the ignore conditions of the specified dependency
      - `@dependabot ignore <dependency name> major version` will close this
      group update PR and stop Dependabot creating any more for the specific
      dependency's major version (unless you unignore this specific
      dependency's major version or upgrade to it yourself)
      - `@dependabot ignore <dependency name> minor version` will close this
      group update PR and stop Dependabot creating any more for the specific
      dependency's minor version (unless you unignore this specific
      dependency's minor version or upgrade to it yourself)
      - `@dependabot ignore <dependency name>` will close this group update PR
      and stop Dependabot creating any more for the specific dependency
      (unless you unignore this specific dependency or upgrade to it yourself)
      - `@dependabot unignore <dependency name>` will remove all of the ignore
      conditions of the specified dependency
      - `@dependabot unignore <dependency name> <ignore condition>` will
      remove the ignore condition of the specified dependency and ignore
      conditions
      
      
      </details>
      
      Signed-off-by: default avatardependabot[bot] <[email protected]>
      Co-authored-by: default avatardependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
      4b484e6a
  9. Feb 12, 2024
    • Oliver Tale-Yazdi's avatar
      Lift dependencies to the workspace (Part 1) (#2070) · e80c2473
      Oliver Tale-Yazdi authored
      Changes (partial https://github.com/paritytech/polkadot-sdk/issues/994):
      - Set log to `0.4.20` everywhere
      - Lift `log` to the workspace
      
      Starting with a simpler one after seeing
      https://github.com/paritytech/polkadot-sdk/pull/2065 from @jsdw
      
      .
      This sets the `default-features` to `false` in the root and then
      overwrites that in each create to its original value. This is necessary
      since otherwise the `default` features are additive and its impossible
      to disable them in the crate again once they are enabled in the
      workspace.
      
      I am using a tool to do this, so its mostly a test to see that it works
      as expected.
      
      ---------
      
      Signed-off-by: default avatarOliver Tale-Yazdi <[email protected]>
      e80c2473
    • Alexandru Gheorghe's avatar
      statement-distribution: Fix CostMinor("Unexpected Statement") (#3223) · 8362a681
      Alexandru Gheorghe authored
      
      
      On grid distribution messages have two paths of reaching a node, so
      there is the possiblity of a race when two peers send each other the
      same statement around the same time. Statement local_knowledge will tell
      us that the peer should have not send the statement because we sent it
      to it.
      
      Fix it by also keeping track only of the statement we received from a
      given peer and penalize it only if it sends it to us more than once.
      
      Fixes: https://github.com/paritytech/polkadot-sdk/issues/2346
      
      Additionally, also use different Cost labels for different paths to make
      it easier to debug things.
      
      ---------
      
      Signed-off-by: default avatarAlexandru Gheorghe <[email protected]>
      8362a681
    • Andrei Eres's avatar
      subsystem-bench: polish imports (#3262) · cbd68467
      Andrei Eres authored
      cbd68467
  10. Feb 11, 2024
    • maksimryndin's avatar
      refactor pvf security module (#3047) · 4883e144
      maksimryndin authored
      
      
      resolve https://github.com/paritytech/polkadot-sdk/issues/2321
      
      - [x] refactor `security` module into a conditionally compiled
      - [x] rename `amd64` into x86-64 for consistency with conditional
      compilation guards and remove reference to a particular vendor
      - [x] run unit tests and zombienet
      
      ---------
      
      Co-authored-by: default avatars0me0ne-unkn0wn <[email protected]>
      4883e144
  11. Feb 09, 2024
  12. Feb 08, 2024
  13. Feb 06, 2024
    • Andrei Eres's avatar
      subsystem-bench: Prepare CI output (#3158) · 9e6298e7
      Andrei Eres authored
      
      
      1. Benchmark results are collected in a single struct.
      2. The output of the results is prettified.
      3. The result struct used to save the output as a yaml and store it in
      artifacts in a CI job.
      
      ```
      $ cargo run -p polkadot-subsystem-bench --release -- test-sequence --path polkadot/node/subsystem-bench/examples/availability_read.yaml | tee output.txt
      $ cat output.txt
      
      polkadot/node/subsystem-bench/examples/availability_read.yaml #1
      
      Network usage, KiB                     total   per block
      Received from peers               510796.000  170265.333
      Sent to peers                        221.000      73.667
      
      CPU usage, s                           total   per block
      availability-recovery                 38.671      12.890
      Test environment                       0.255       0.085
      
      
      polkadot/node/subsystem-bench/examples/availability_read.yaml #2
      
      Network usage, KiB                     total   per block
      Received from peers               413633.000  137877.667
      Sent to peers                        353.000     117.667
      
      CPU usage, s                           total   per block
      availability-recovery                 52.630      17.543
      Test environment                       0.271       0.090
      
      
      polkadot/node/subsystem-bench/examples/availability_read.yaml #3
      
      Network usage, KiB                     total   per block
      Received from peers               424379.000  141459.667
      Sent to peers                        703.000     234.333
      
      CPU usage, s                           total   per block
      availability-recovery                 51.128      17.043
      Test environment                       0.502       0.167
      
      ```
      
      ```
      $ cargo run -p polkadot-subsystem-bench --release -- --ci test-sequence --path polkadot/node/subsystem-bench/examples/availability_read.yaml | tee output.txt
      $ cat output.txt
      - benchmark_name: 'polkadot/node/subsystem-bench/examples/availability_read.yaml #1'
        network:
        - resource: Received from peers
          total: 509011.0
          per_block: 169670.33333333334
        - resource: Sent to peers
          total: 220.0
          per_block: 73.33333333333333
        cpu:
        - resource: availability-recovery
          total: 31.845848445
          per_block: 10.615282815
        - resource: Test environment
          total: 0.23582828799999941
          per_block: 0.07860942933333313
      
      - benchmark_name: 'polkadot/node/subsystem-bench/examples/availability_read.yaml #2'
        network:
        - resource: Received from peers
          total: 411738.0
          per_block: 137246.0
        - resource: Sent to peers
          total: 351.0
          per_block: 117.0
        cpu:
        - resource: availability-recovery
          total: 18.93596025099999
          per_block: 6.31198675033333
        - resource: Test environment
          total: 0.2541994199999979
          per_block: 0.0847331399999993
      
      - benchmark_name: 'polkadot/node/subsystem-bench/examples/availability_read.yaml #3'
        network:
        - resource: Received from peers
          total: 424548.0
          per_block: 141516.0
        - resource: Sent to peers
          total: 703.0
          per_block: 234.33333333333334
        cpu:
        - resource: availability-recovery
          total: 16.54178526900001
          per_block: 5.513928423000003
        - resource: Test environment
          total: 0.43960946299999537
          per_block: 0.14653648766666513
      ```
      
      ---------
      
      Co-authored-by: default avatarAndrei Sandu <[email protected]>
      9e6298e7
    • Alin Dima's avatar
      prospective-parachains: respond with multiple backable candidates (#3160) · 7df1ae3b
      Alin Dima authored
      Fixes https://github.com/paritytech/polkadot-sdk/issues/3129
      7df1ae3b
  14. Feb 05, 2024
    • Alexandru Gheorghe's avatar
      Introduce approval-voting/distribution benchmark (#2621) · f9f88688
      Alexandru Gheorghe authored
      
      
      ## Summary
      Built on top of the tooling and ideas introduced in
      https://github.com/paritytech/polkadot-sdk/pull/2528, this PR introduces
      a synthetic benchmark for measuring and assessing the performance
      characteristics of the approval-voting and approval-distribution
      subsystems.
      
      Currently this allows, us to simulate the behaviours of these systems
      based on the following dimensions:
      ```
      TestConfiguration:
      # Test 1
      - objective: !ApprovalsTest
          last_considered_tranche: 89
          min_coalesce: 1
          max_coalesce: 6
          enable_assignments_v2: true
          send_till_tranche: 60
          stop_when_approved: false
          coalesce_tranche_diff: 12
          workdir_prefix: "/tmp"
          num_no_shows_per_candidate: 0
          approval_distribution_expected_tof: 6.0
          approval_distribution_cpu_ms: 3.0
          approval_voting_cpu_ms: 4.30
        n_validators: 500
        n_cores: 100
        n_included_candidates: 100
        min_pov_size: 1120
        max_pov_size: 5120
        peer_bandwidth: 524288000000
        bandwidth: 524288000000
        latency:
          min_latency:
            secs: 0
            nanos: 1000000
          max_latency:
            secs: 0
            nanos: 100000000
        error: 0
        num_blocks: 10
      ```
      
      ## The approach
      1. We build a real overseer with the real implementations for
      approval-voting and approval-distribution subsystems.
      2. For a given network size, for each validator we pre-computed all
      potential assignments and approvals it would send, because this a
      computation heavy operation this will be cached on a file on disk and be
      re-used if the generation parameters don't change.
      3. The messages will be sent accordingly to the configured parameters
      and those are split into 3 main benchmarking scenarios.
      
      ## Benchmarking scenarios
      
      ### Best case scenario *approvals_throughput_best_case.yaml*
      It send to the approval-distribution only the minimum required tranche
      to gathered the needed_approvals, so that a candidate is approved.
      
      ### Behaviour in the presence of no-shows *approvals_no_shows.yaml*
      It sends the tranche needed to approve a candidate when we have a
      maximum of *num_no_shows_per_candidate* tranches with no-shows for each
      candidate.
      
      ### Maximum throughput *approvals_throughput.yaml*
      It sends all the tranches for each block and measures the used CPU and
      necessary network bandwidth. by the approval-voting and
      approval-distribution subsystem.
      
      ## How to run it
      ```
      cargo run -p polkadot-subsystem-bench --release -- test-sequence --path polkadot/node/subsystem-bench/examples/approvals_throughput.yaml
      ```
      
      ## Evaluating performance
      ### Use the real subsystems metrics
      If you follow the steps in
      https://github.com/paritytech/polkadot-sdk/tree/master/polkadot/node/subsystem-bench#install-grafana
      for installing locally prometheus and grafana, all real metrics for the
      `approval-distribution`, `approval-voting` and overseer are available.
      E.g:
      <img width="2149" alt="Screenshot 2023-12-05 at 11 07 46"
      src="https://github.com/paritytech/polkadot-sdk/assets/49718502/cb8ae2dd-178b-4922-bfa4-dc37e572ed38">
      
      <img width="2551" alt="Screenshot 2023-12-05 at 11 09 42"
      src="https://github.com/paritytech/polkadot-sdk/assets/49718502/8b4542ba-88b9-46f9-9b70-cc345366081b">
      
      <img width="2154" alt="Screenshot 2023-12-05 at 11 10 15"
      src="https://github.com/paritytech/polkadot-sdk/assets/49718502/b8874d8d-632e-443a-9840-14ad8e90c54f">
      
      <img width="2535" alt="Screenshot 2023-12-05 at 11 10 52"
      src="https://github.com/paritytech/polkadot-sdk/assets/49718502/779a439f-fd18-4985-bb80-85d5afad78e2">
      
      ### Profile with pyroscope
      1. Setup pyroscope following the steps in
      https://github.com/paritytech/polkadot-sdk/tree/master/polkadot/node/subsystem-bench#install-pyroscope,
      then run any of the benchmark scenario with `--profile` as the
      arguments.
      2. Open the pyroscope dashboard in grafana, e.g:
      <img width="2544" alt="Screenshot 2024-01-09 at 17 09 58"
      src="https://github.com/paritytech/polkadot-sdk/assets/49718502/58f50c99-a910-4d20-951a-8b16639303d9">
      
      
      
      ### Useful  logs
      1. Network bandwidth requirements:
      ```
      Payload bytes received from peers: 503993 KiB total, 50399 KiB/block
      Payload bytes sent to peers: 629971 KiB total, 62997 KiB/block
      ```
      
      2. Cpu usage by the approval-distribution/approval-voting subsystems.
      ```
      approval-distribution CPU usage 84.061s
      approval-distribution CPU usage per block 8.406s
      approval-voting CPU usage 96.532s
      approval-voting CPU usage per block 9.653s
      ```
      
      3. Time passed until a given block is approved
      ```
       Chain selection approved  after 3500 ms hash=0x0101010101010101010101010101010101010101010101010101010101010101
      Chain selection approved  after 4500 ms hash=0x0202020202020202020202020202020202020202020202020202020202020202
      ```
      
      ### Using benchmark to quantify improvements from
      https://github.com/paritytech/polkadot-sdk/pull/1178 +
      https://github.com/paritytech/polkadot-sdk/pull/1191
      
      Using a versi-node we compare the scenarios where all new optimisations
      are disabled with a scenarios where tranche0 assignments are sent in a
      single message and a conservative simulation where the coalescing of
      approvals gives us just 50% reduction in the number of messages we send.
      
      Overall, what we see is a speedup of around 30-40% in the time it takes
      to process the necessary messages and a 30-40% reduction in the
      necessary bandwidth.
      
      #### Best case scenario comparison(minimum required tranches sent).
      Unoptimised
      ```
          Number of blocks: 10
          Payload bytes received from peers: 53289 KiB total, 5328 KiB/block
          Payload bytes sent to peers: 52489 KiB total, 5248 KiB/block
          approval-distribution CPU usage 6.732s
          approval-distribution CPU usage per block 0.673s
          approval-voting CPU usage 9.523s
          approval-voting CPU usage per block 0.952s
      ```
      
      vs Optimisation enabled
      ```
         Number of blocks: 10
         Payload bytes received from peers: 32141 KiB total, 3214 KiB/block
         Payload bytes sent to peers: 37314 KiB total, 3731 KiB/block
         approval-distribution CPU usage 4.658s
         approval-distribution CPU usage per block 0.466s
         approval-voting CPU usage 6.236s
         approval-voting CPU usage per block 0.624s
      ```
      
      #### Worst case all tranches sent, very unlikely happens when sharding
      breaks.
      
      Unoptimised
      ```
         Number of blocks: 10
         Payload bytes received from peers: 746393 KiB total, 74639 KiB/block
         Payload bytes sent to peers: 729151 KiB total, 72915 KiB/block
         approval-distribution CPU usage 118.681s
         approval-distribution CPU usage per block 11.868s
         approval-voting CPU usage 124.118s
         approval-voting CPU usage per block 12.412s
      ```
      
      vs optimised
      ```
          Number of blocks: 10
          Payload bytes received from peers: 503993 KiB total, 50399 KiB/block
          Payload bytes sent to peers: 629971 KiB total, 62997 KiB/block
          approval-distribution CPU usage 84.061s
          approval-distribution CPU usage per block 8.406s
          approval-voting CPU usage 96.532s
          approval-voting CPU usage per block 9.653s
      ```
      
      
      ## TODOs
      [x] Polish implementation.
      [x] Use what we have so far to evaluate
      https://github.com/paritytech/polkadot-sdk/pull/1191 before merging.
      [x] List of features and additional dimensions we want to use for
      benchmarking.
      [x] Run benchmark on hardware similar with versi and kusama nodes.
      [ ] Add benchmark to be run in CI for catching regression in
      performance.
      [ ] Rebase on latest changes for network emulation.
      
      ---------
      
      Signed-off-by: default avatarAndrei Sandu <[email protected]>
      Signed-off-by: default avatarAlexandru Gheorghe <[email protected]>
      Co-authored-by: default avatarAndrei Sandu <[email protected]>
      Co-authored-by: default avatarAndrei Sandu <[email protected]>
      f9f88688
  15. Feb 02, 2024
  16. Jan 31, 2024
  17. Jan 30, 2024
  18. Jan 29, 2024
  19. Jan 26, 2024
  20. Jan 25, 2024
    • Andrei Sandu's avatar
      Add subsystem benchmarks for `availability-distribution` and... · 47e46d17
      Andrei Sandu authored
      
      Add subsystem benchmarks for `availability-distribution` and `biftield-distribution` (availability write) (#2970)
      
      Introduce a new test objective : `DataAvailabilityWrite`.
      
      The new benchmark measures the network and cpu usage of
      `availability-distribution`, `biftield-distribution` and
      `availability-store` subsystems from the perspective of a validator node
      during the process when candidates are made available.
      
      Additionally I refactored the networking emulation to support bandwidth
      acounting and limits of incoming and outgoing requests.
      
      Screenshot of succesful run
      
      
      <img width="1293" alt="Screenshot 2024-01-17 at 19 17 44"
      src="https://github.com/paritytech/polkadot-sdk/assets/54316454/fde11280-e25b-4dc3-9dc9-d4b9752f9b7a">
      
      ---------
      
      Signed-off-by: default avatarAndrei Sandu <[email protected]>
      47e46d17
    • Alexandru Gheorghe's avatar
      approval-distribution: Update topology if authorities are discovered later (#2981) · a6952c74
      Alexandru Gheorghe authored
      
      
      Fixes: https://github.com/paritytech/polkadot-sdk/issues/2138.
      
      Especially on restart AuthorithyDiscovery cache is not populated so we
      create an invalid topology and messages won't be routed correctly for
      the entire session. This PR proposes to try to fix this by updating the
      topology as soon as we now the Authority/PeerId mapping, that should
      impact the situation dramatically.
      
      
      [This issue was hit
      yesterday](https://grafana.teleport.parity.io/goto/o9q2625Sg?orgId=1),
      on Westend and resulted in stalling the finality.
      
      
      # TODO
      
      - [x] Unit tests
      - [x] Test impact on versi
      
      ---------
      
      Signed-off-by: default avatarAlexandru Gheorghe <[email protected]>
      a6952c74
  21. Jan 24, 2024
  22. Jan 23, 2024
    • Andrei Sandu's avatar
      approval-distribution: aggresion must target unfinalized chain rather than unapproved chain (#2988) · b4dfad83
      Andrei Sandu authored
      
      
      Found the issue while investigating the recent finality stall on Westend
      after upgrading to 1.6.0. Approval distribution aggression is supposed
      to trade off bandwidth and re-send assignemnts/approvals until enough
      approvals are be received by at least 2/3 validators. This is supposed
      to be a catch all mechanism when network connectivity goes south or many
      validators reboot at the same time.
      
      This fix ensures that we always resend approvals starting with the first
      unfinalized block even in the case when it appears approved from the
      node's perspective.
      
      TODO:
      - [x] Versi test
      
      ---------
      
      Signed-off-by: default avatarAndrei Sandu <[email protected]>
      b4dfad83
    • Niklas Adolfsson's avatar
      rpc: backpressured RPC server (bump jsonrpsee 0.20) (#1313) · e16ef086
      Niklas Adolfsson authored
      This is a rather big change in jsonrpsee, the major things in this bump
      are:
      - Server backpressure (the subscription impls are modified to deal with
      that)
      - Allow custom error types / return types (remove jsonrpsee::core::Error
      and jsonrpee::core::CallError)
      - Bug fixes (graceful shutdown in particular not used by substrate
      anyway)
         - Less dependencies for the clients in particular
         - Return type requires Clone in method call responses
         - Moved to tokio channels
         - Async subscription API (not used in this PR)
      
      Major changes in this PR:
      - The subscriptions are now bounded and if subscription can't keep up
      with the server it is dropped
      - CLI: add parameter to configure the jsonrpc server bounded message
      buffer (default is 64)
      - Add our own subscription helper to deal with the unbounded streams in
      substrate
      
      The most important things in this PR to review is the added helpers
      functions in `substrate/client/rpc/src/utils.rs` and the rest is pretty
      much chore.
      
      Regarding the "bounded buffer limit" it may cause the server to handle
      the JSON-RPC calls
      slower than before.
      
      The message size limit is bounded by "--rpc-response-size" thus "by
      default 10MB * 64 = 640MB"
      but the subscription message size is not covered by this limit and could
      be capped as well.
      
      Hopefully the last release prior to 1.0, sorry in advance for a big PR
      
      Previous attempt: https://github.com/paritytech/substrate/pull/13992
      
      Resolves https://github.com/paritytech/polkadot-sdk/issues/748, resolves
      https://github.com/paritytech/polkadot-sdk/issues/627
      e16ef086