1. Apr 10, 2024
  2. Apr 09, 2024
    • Sebastian Kunert's avatar
      Move cumulus zombienet tests to aura & async backing (#3568) · df818d29
      Sebastian Kunert authored
      Cumulus test-parachain node and test runtime were still using relay
      chain consensus and 12s blocktimes. With async backing around the corner
      on the major chains we should switch our tests too.
      
      Also needed to nicely test the changes coming to collators in #3168.
      
      ### Changes Overview
      - Followed the [migration
      guide](https://wiki.polkadot.network/docs/maintain-guides-async-backing)
      for async backing for the cumulus-test-runtime
      - Adjusted the cumulus-test-service to use the correct import-queue,
      lookahead collator etc.
      - The block validation function now uses the Aura Ext Executor so that
      the seal of the block is validated
      - Previous point requires that we seal block before calling into
      `validate_block`, I introduced a helper function for that
      - Test client adjusted to provide a slot to the relay chain proof and
      the aura pre-digest
      df818d29
    • Eugen Snitko's avatar
      Check for bundled forklift (#4055) · cb192d10
      Eugen Snitko authored
      Added check if forklift already exists in ci image as forklift binary is
      now bundled with the ci-unified.
      This is a temporary check for the transition period
      cb192d10
    • Alexandru Vasile's avatar
      rpc-v2/transaction: Generate `Invalid` events and add tests (#3784) · 598e9557
      Alexandru Vasile authored
      This PR ensures that the transaction API generates an `Invalid` events
      for transaction bytes that fail to decode.
      
      The spec mentioned the `Invalid` event at the jsonrpc error section,
      however this spec PR makes things clearer:
      - https://github.com/paritytech/json-rpc-interface-spec/pull/146
      
      While at it have discovered an inconsistency with the generated events.
      The drop event from the transaction pool was incorrectly mapped to the
      `invalid` event.
      
      Added tests for the API stabilize the API soon:
      - https://github.com/paritytech/json-rpc-interface-spec/pull/144
      
      
      Closes: https://github.com/paritytech/polkadot-sdk/issues/3083
      
      
      
      
      cc @paritytech/subxt-team
      
      ---------
      
      Signed-off-by: default avatarAlexandru Vasile <[email protected]>
      598e9557
    • Dmitry Markin's avatar
      Detect closed notification substreams instead of evicting all peers (#3983) · a26d25d5
      Dmitry Markin authored
      This PR brings the fix
      https://github.com/paritytech/substrate/pull/13396 to polkadot-sdk.
      
      In the past, due to insufficient inbound slot count on polkadot &
      kusama, this fix led to low peer count. The situation has improved since
      then after changing the default ratio between `--in-peers` &
      `--out-peers`.
      
      Nevertheless, it's expected that the reported total peer count with this
      fix is going to be lower than without it. This should be seen as the
      correct number of working connections reported, as opposed to also
      reporting already closed connections, and not as lower count of working
      connections with peers.
      
      This PR also removes the peer eviction mechanism, as closed substream
      detection is a more granular way of detecting peers that stopped syncing
      with us.
      
      The burn-in has been already performed as part of testing these changes
      in https://github.com/paritytech/polkadot-sdk/pull/3426
      
      .
      
      ---------
      
      Co-authored-by: default avatarAaro Altonen <[email protected]>
      a26d25d5
    • Adrian Catangiu's avatar
      test runtimes: allow local origins to execute arbitrary XCMs (#3930) · 74a42ceb
      Adrian Catangiu authored
      We are exploring [allowing this for
      Kusama](https://github.com/polkadot-fellows/runtimes/pull/261) as well,
      disallowing on test chains seems unnecessarily limiting.
      74a42ceb
    • Alexandru Vasile's avatar
      peer_store: Increase peer ban time until escapes banned threshold (#4031) · b1c9209a
      Alexandru Vasile authored
      This is a tiny PR to increase the time a peer remains banned.
      
      A peer is banned when the reputation drops below a threshold.
      With every second, the peer reputation is exponentially decayed towards
      zero.
      
      For the previous setup:
      - decaying to zero from (i32::MAX or i32::MIN) would take 948 seconds
      (15mins 48seconds)
      - from i32::MIN to escaping the banned threshold would take 10 seconds
      This means we are decaying reputation a bit too aggressive and
      misbehaving peers can misbehave again in 10 seconds.
      Another side effect of this is that we have encountered multiple
      warnings caused by a few misbehaving peers.
      
      In the new setup:
      - decaying to zero from (i32::MAX or i32::MIN) would take 3544 seconds
      (59 minutes)
      - from i32::MIN to escaping the banned threshold would take ~69 seconds
      
      This is a followup of:
      - https://github.com/paritytech/polkadot-sdk/pull/4000.
      
      ### Testing Done
      - Created a misbehaving client with
      [subp2p-explorer](https://github.com/lexnv/subp2p-explorer
      
      ), the client
      is banned for approx 69seconds until it is allowed to connect again.
      
      cc @paritytech/networking
      
      ---------
      
      Signed-off-by: default avatarAlexandru Vasile <[email protected]>
      b1c9209a
    • Facundo Farall's avatar
      Upgrade `trie-db` from `0.28.0` to `0.29.0` (#3982) · 4e73c0fc
      Facundo Farall authored
      
      
      # Description
      - What does this PR do?
      1. Upgrades `trie-db`'s version to the latest release. This release
      includes, among others, an implementation of `DoubleEndedIterator` for
      the `TrieDB` struct, allowing to iterate both backwards and forwards
      within the leaves of a trie.
      2. Upgrades `trie-bench` to `0.39.0` for compatibility.
      3. Upgrades `criterion` to `0.5.1` for compatibility.
      - Why are these changes needed?
      Besides keeping up with the upgrade of `trie-db`, this specifically adds
      the functionality of iterating back on the leafs of a trie, with
      `sp-trie`. In a project we're currently working on, this comes very
      handy to verify a Merkle proof that is the response to a challenge. The
      challenge is a random hash that (most likely) will not be an existing
      leaf in the trie. So the challenged user, has to provide a Merkle proof
      of the previous and next existing leafs in the trie, that surround the
      random challenged hash.
      
      Without having DoubleEnded iterators, we're forced to iterate until we
      find the first existing leaf, like so:
      ```rust
              // ************* VERIFIER (RUNTIME) *************
              // Verify proof. This generates a partial trie based on the proof and
              // checks that the root hash matches the `expected_root`.
              let (memdb, root) = proof.to_memory_db(Some(&root)).unwrap();
              let trie = TrieDBBuilder::<LayoutV1<RefHasher>>::new(&memdb, &root).build();
      
              // Print all leaf node keys and values.
              println!("\nPrinting leaf nodes of partial tree...");
              for key in trie.key_iter().unwrap() {
                  if key.is_ok() {
                      println!("Leaf node key: {:?}", key.clone().unwrap());
      
                      let val = trie.get(&key.unwrap());
      
                      if val.is_ok() {
                          println!("Leaf node value: {:?}", val.unwrap());
                      } else {
                          println!("Leaf node value: None");
                      }
                  }
              }
      
              println!("RECONSTRUCTED TRIE {:#?}", trie);
      
              // Create an iterator over the leaf nodes.
              let mut iter = trie.iter().unwrap();
      
              // First element with a value should be the previous existing leaf to the challenged hash.
              let mut prev_key = None;
              for element in &mut iter {
                  if element.is_ok() {
                      let (key, _) = element.unwrap();
                      prev_key = Some(key);
                      break;
                  }
              }
              assert!(prev_key.is_some());
      
              // Since hashes are `Vec<u8>` ordered in big-endian, we can compare them directly.
              assert!(prev_key.unwrap() <= challenge_hash.to_vec());
      
              // The next element should exist (meaning there is no other existing leaf between the
              // previous and next leaf) and it should be greater than the challenged hash.
              let next_key = iter.next().unwrap().unwrap().0;
              assert!(next_key >= challenge_hash.to_vec());
      ```
      
      With DoubleEnded iterators, we can avoid that, like this:
      ```rust
              // ************* VERIFIER (RUNTIME) *************
              // Verify proof. This generates a partial trie based on the proof and
              // checks that the root hash matches the `expected_root`.
              let (memdb, root) = proof.to_memory_db(Some(&root)).unwrap();
              let trie = TrieDBBuilder::<LayoutV1<RefHasher>>::new(&memdb, &root).build();
      
              // Print all leaf node keys and values.
              println!("\nPrinting leaf nodes of partial tree...");
              for key in trie.key_iter().unwrap() {
                  if key.is_ok() {
                      println!("Leaf node key: {:?}", key.clone().unwrap());
      
                      let val = trie.get(&key.unwrap());
      
                      if val.is_ok() {
                          println!("Leaf node value: {:?}", val.unwrap());
                      } else {
                          println!("Leaf node value: None");
                      }
                  }
              }
      
              // println!("RECONSTRUCTED TRIE {:#?}", trie);
              println!("\nChallenged key: {:?}", challenge_hash);
      
              // Create an iterator over the leaf nodes.
              let mut double_ended_iter = trie.into_double_ended_iter().unwrap();
      
              // First element with a value should be the previous existing leaf to the challenged hash.
              double_ended_iter.seek(&challenge_hash.to_vec()).unwrap();
              let next_key = double_ended_iter.next_back().unwrap().unwrap().0;
              let prev_key = double_ended_iter.next_back().unwrap().unwrap().0;
      
              // Since hashes are `Vec<u8>` ordered in big-endian, we can compare them directly.
              println!("Prev key: {:?}", prev_key);
              assert!(prev_key <= challenge_hash.to_vec());
      
              println!("Next key: {:?}", next_key);
              assert!(next_key >= challenge_hash.to_vec());
      ```
      - How were these changes implemented and what do they affect?
      All that is needed for this functionality to be exposed is changing the
      version number of `trie-db` in all the `Cargo.toml`s applicable, and
      re-exporting some additional structs from `trie-db` in `sp-trie`.
      
      ---------
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      4e73c0fc
    • PG Herveou's avatar
      Contracts: Refactor API to use WeightMeter (#2943) · b6231c79
      PG Herveou authored
      
      
      Update the Contracts API to use `WeightMeter`, as it simplifies the code
      and makes it easier to reason about, rather than taking a mutable weight
      or returning a tuple with the weight consumed
      
      ---------
      
      Co-authored-by: default avatarAlexander Theißen <[email protected]>
      b6231c79
    • Ankan's avatar
      Nomination pool configurations can be managed by custom origin (#3959) · 10ed7643
      Ankan authored
      closes https://github.com/paritytech/polkadot-sdk/issues/3894
      
      
      
      Allows Nomination Pool configuration to be set by a custom origin
      instead of root.
      
      In runtimes, we would set this to be `StakingAdmin`, same as for
      pallet-staking.
      
      ---------
      
      Co-authored-by: default avatarLiam Aharon <[email protected]>
      10ed7643
    • Alexander Samusev's avatar
    • Bastian Köcher's avatar
      parachain-template: Simplify it (#3801) · 22b95a8a
      Bastian Köcher authored
      Also while doing this, move slot duration fetching into the AURA code.
      22b95a8a
    • Branislav Kontur's avatar
    • gupnik's avatar
      Fixes validation for `SkipCheckIfFeeless` extension (#3993) · 0e27b881
      gupnik authored
      During validation, `SkipCheckIfFeeless` should check if the call is
      `feeless` and delegate to the wrapped extension if not.
      0e27b881
  3. Apr 08, 2024
    • Léa Narzis's avatar
    • Aaro Altonen's avatar
      Integrate litep2p into Polkadot SDK (#2944) · 80616f6d
      Aaro Altonen authored
      [litep2p](https://github.com/altonen/litep2p) is a libp2p-compatible P2P
      networking library. It supports all of the features of `rust-libp2p`
      that are currently being utilized by Polkadot SDK.
      
      Compared to `rust-libp2p`, `litep2p` has a quite different architecture
      which is why the new `litep2p` network backend is only able to use a
      little of the existing code in `sc-network`. The design has been mainly
      influenced by how we'd wish to structure our networking-related code in
      Polkadot SDK: independent higher-levels protocols directly communicating
      with the network over links that support bidirectional backpressure. A
      good example would be `NotificationHandle`/`RequestResponseHandle`
      abstractions which allow, e.g., `SyncingEngine` to directly communicate
      with peers to announce/request blocks.
      
      I've tried running `polkadot --network-backend litep2p` with a few
      different peer configurations and there is a noticeable reduction in
      networking CPU usage. For high load (`--out-peers 200`), networking CPU
      usage goes down from ~110% to ~30% (80 pp) and for normal load
      (`--out-peers 40`), the usage goes down from ~55% to ~18% (37 pp).
      
      These should not be taken as final numbers because:
      
      a) there are still some low-hanging optimization fruits, such as
      enabling [receive window
      auto-tuning](https://github.com/libp2p/rust-yamux/pull/176
      
      ), integrating
      `Peerset` more closely with `litep2p` or improving memory usage of the
      WebSocket transport
      b) fixing bugs/instabilities that incorrectly cause `litep2p` to do less
      work will increase the networking CPU usage
      c) verification in a more diverse set of tests/conditions is needed
      
      Nevertheless, these numbers should give an early estimate for CPU usage
      of the new networking backend.
      
      This PR consists of three separate changes:
      * introduce a generic `PeerId` (wrapper around `Multihash`) so that we
      don't have use `NetworkService::PeerId` in every part of the code that
      uses a `PeerId`
      * introduce `NetworkBackend` trait, implement it for the libp2p network
      stack and make Polkadot SDK generic over `NetworkBackend`
        * implement `NetworkBackend` for litep2p
      
      The new library should be considered experimental which is why
      `rust-libp2p` will remain as the default option for the time being. This
      PR currently depends on the master branch of `litep2p` but I'll cut a
      new release for the library once all review comments have been
      addresses.
      
      ---------
      
      Signed-off-by: default avatarAlexandru Vasile <[email protected]>
      Co-authored-by: default avatarDmitry Markin <[email protected]>
      Co-authored-by: default avatarAlexandru Vasile <[email protected]>
      Co-authored-by: default avatarAlexandru Vasile <[email protected]>
      80616f6d
    • Oliver Tale-Yazdi's avatar
      [FRAME] Runtime Omni Bencher (#3512) · 9543d314
      Oliver Tale-Yazdi authored
      This MR contains two major changes and some maintenance cleanup.  
      
      ## 1. Free Standing Pallet Benchmark Runner
      
      Closes https://github.com/paritytech/polkadot-sdk/issues/3045, depends
      on your runtime exposing the `GenesisBuilderApi` (like
      https://github.com/paritytech/polkadot-sdk/pull/1492).
      
      Introduces a new binary crate: `frame-omni-bencher`.  
      It allows to directly benchmark a WASM blob - without needing a node or
      chain spec.
      
      This makes it much easier to generate pallet weights and should allow us
      to remove bloaty code from the node.
      It should work for all FRAME runtimes that dont use 3rd party host calls
      or non `BlakeTwo256` block hashing (basically all polkadot parachains
      should work).
      
      It is 100% backwards compatible with the old CLI args, when the `v1`
      compatibility command is used. This is done to allow for forwards
      compatible addition of new commands.
      
      ### Example (full example in the Rust docs)
      
      Installing the CLI:
      ```sh
      cargo install --locked --path substrate/utils/frame/omni-bencher
      frame-omni-bencher --help
      ```
      
      Building the Westend runtime:
      ```sh
      cargo build -p westend-runtime --release --features runtime-benchmarks
      ```
      
      Benchmarking the runtime:
      ```sh
      frame-omni-bencher v1 benchmark pallet --runtime target/release/wbuild/westend-runtime/westend_runtime.compact.compressed.wasm --all
      ```
      
      ## 2. Building the Benchmark Genesis State in the Runtime
      
      Closes https://github.com/paritytech/polkadot-sdk/issues/2664
      
      This adds `--runtime` and `--genesis-builder=none|runtime|spec`
      arguments to the `benchmark pallet` command to make it possible to
      generate the genesis storage by the runtime. This can be used with both
      the node and the freestanding benchmark runners. It utilizes the new
      `GenesisBuilder` RA and depends on having
      https://github.com/paritytech/polkadot-sdk/pull/3412
      
       deployed.
      
      ## 3. Simpler args for `PalletCmd::run`
      
      You can do three things here to integrate the changes into your node:
      - nothing: old code keeps working as before but emits a deprecated
      warning
      - delete: remove the pallet benchmarking code from your node and use the
      omni-bencher instead
      - patch: apply the patch below and keep using as currently. This emits a
      deprecated warning at runtime, since it uses the old way to generate a
      genesis state, but is the smallest change.
      
      ```patch
      runner.sync_run(|config| cmd
      -    .run::<HashingFor<Block>, ReclaimHostFunctions>(config)
      +    .run_with_spec::<HashingFor<Block>, ReclaimHostFunctions>(Some(config.chain_spec))
      )
      ```
      
      ## 4. Maintenance Change
      - `pallet-nis` get a `BenchmarkSetup` config item to prepare its
      counterparty asset.
      - Add percent progress print when running benchmarks.
      - Dont immediately exit on benchmark error but try to run as many as
      possible and print errors last.
      
      ---------
      
      Signed-off-by: default avatarOliver Tale-Yazdi <[email protected]>
      Co-authored-by: default avatarLiam Aharon <[email protected]>
      9543d314
    • Egor_P's avatar
      Github workflow to automate release draft creation (#3978) · 216509db
      Egor_P authored
      
      
      This PR introduces the github flow which will create a release draft
      automatically when the rc tag is pushed. The flow contains the following
      steps:
      
      - Gets the info about rust version used to build the node
      - Builds the runtimes using `srtool`
      - Extracts the info about each runtime 
      - Aggregates the changelog from the prdocs
      - Creates the release draft containing all the info related to the
      release (changelog, runtimes, rust versions)
      - Attaches the runtimes to the draft
      - Posts the message to the RelEng internal channel to inform that the
      build is done.
      
      Related to the #3295
      
      ---------
      
      Co-authored-by: default avatarOliver Tale-Yazdi <[email protected]>
      216509db
    • Sebastian Kunert's avatar
      Add best block indicator to informant message + print parent block on import message (#4021) · fdb1dba2
      Sebastian Kunert authored
      Sometimes you need to debug some issues just by the logs and reconstruct
      what happened.
      In these scenarios it would be nice to know if a block was imported as
      best block, and what it parent was.
      So here I propose to change the output of the informant to this:
      
      ```
      2024-04-05 20:38:22.004  INFO ⋮substrate: [Parachain]  Imported #18 (0xe7b3…4555 -> 0xbd6f…ced7)    
      2024-04-05 20:38:24.005  INFO ⋮substrate: [Parachain]  Imported #19 (0xbd6f…ced7 -> 0x4dd0…d81f)    
      2024-04-05 20:38:24.011  INFO ⋮substrate: [jobless-children-5352] 🌟 Imported #42 (0xed2e…27fc -> 0x718f…f30e)    
      2024-04-05 20:38:26.005  INFO ⋮substrate: [Parachain]  Imported #20 (0x4dd0…d81f -> 0x6e85…e2b8)    
      2024-04-05 20:38:28.004  INFO ⋮substrate: [Parachain] 🌟 Imported #21 (0x6e85…e2b8 -> 0xad53…2a97)    
      2024-04-05 20:38:30.004  INFO ⋮substrate: [Parachain] 🌟
      
       Imported #22 (0xad53…2a97 -> 0xa874…890f)    
      ```
      
      ---------
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      fdb1dba2
    • Andrei Sandu's avatar
      adder-collator: add velocity measurement and make elastic scaling test more robust (#4016) · 039d183b
      Andrei Sandu authored
      
      
      Improves `adder-collator` to also compute the parachain velocity. The
      velocity is defined as number of parachain blocks progressing per relay
      chain block.
      
      In this test we're asserting that the elastic parachain always
      progresses by 3 blocks per RCB, while the non-elastic parachain
      progresses normally - 1 block per RCB.
      
      ---------
      
      Signed-off-by: default avatarAndrei Sandu <[email protected]>
      Co-authored-by: default avatarordian <[email protected]>
      039d183b
    • Bastian Köcher's avatar
      sc-beefy-consensus: Remove unneeded stream. (#4015) · c1063a53
      Bastian Köcher authored
      The stream was just used to communicate from the validator the peer
      reports back to the gossip engine. Internally the gossip engine just
      forwards these reports to the networking engine. So, we can just do this
      directly.
      
      The reporting stream was also pumped [in the worker behind the
      engine](https://github.com/paritytech/polkadot-sdk/blob/9d6261892814fa27c97881c0321c008d7340b54b/substrate/client/consensus/beefy/src/worker.rs#L939).
      This means if there was a lot of data incoming over the engine, the
      reporting stream was almost never processed and thus, it could have
      started to grow and we have seen issues around this.
      
      Partly Closes: https://github.com/paritytech/polkadot-sdk/issues/3945
      c1063a53
    • Tsvetomir Dimitrov's avatar
      Deprecate `para_id()` from `CoreState` in polkadot primitives (#3979) · 59f868d1
      Tsvetomir Dimitrov authored
      With Coretime enabled we can no longer assume there is a static 1:1
      mapping between core index and para id. This mapping should be obtained
      from the scheduler/claimqueue on block by block basis.
      
      This PR modifies `para_id()` (from `CoreState`) to return the scheduled
      `ParaId` for occupied cores and removes its usages in the code.
      
      Closes https://github.com/paritytech/polkadot-sdk/issues/3948
      
      
      
      ---------
      
      Co-authored-by: default avatarAndrei Sandu <[email protected]>
      59f868d1
    • HongKuang's avatar
      Fix some typos (#4018) · bd4471b4
      HongKuang authored
      
      
      Signed-off-by: default avatarhongkuang <[email protected]>
      bd4471b4
  4. Apr 06, 2024
  5. Apr 05, 2024
  6. Apr 04, 2024