1. Apr 25, 2024
  2. Apr 24, 2024
  3. Apr 23, 2024
  4. Apr 22, 2024
  5. Apr 19, 2024
  6. Apr 17, 2024
    • Alexandru Vasile's avatar
      chainHead: Report unique hashes for pruned blocks (#3667) · bfbf7f5d
      Alexandru Vasile authored
      This PR ensures that the reported pruned blocks are unique.
      
      While at it, ensure that the best block event is properly generated when
      the last best block is a fork that will be pruned in the future.
      
      To achieve this, the chainHead keeps a LRU set of reported pruned blocks
      to ensure the following are not reported twice:
      
      ```bash
      	 finalized -> block 1 -> block 2 -> block 3
      	
      	                      -> block 2 -> block 4 -> block 5
      	
      	           -> block 1 -> block 2_f -> block 6 -> block 7 -> block 8
      ```
      
      When block 7 is finalized the branch [block 2; block 3] is reported as
      pruned.
      When block 8 is finalized the branch [block 2; block 4; block 5] should
      be reported as pruned, however block 2 was already reported as pruned at
      the previous step.
      
      This is a side-effect of the pruned blocks being reported at level N -
      1. For example, if all pruned forks would be reported with the first
      encounter (when block 6 is finalized we know that block 3 and block 5
      are stale), we would not need the LRU cache.
      
      cc @paritytech/subxt-team  
      
      Closes https://github.com/paritytech/polkadot-sdk/issues/3658
      
      
      
      ---------
      
      Signed-off-by: default avatarAlexandru Vasile <[email protected]>
      Co-authored-by: default avatarSebastian Kunert <[email protected]>
      bfbf7f5d
    • Muharem Ismailov's avatar
      Asset Conversion: Pool Account ID derivation with additional Pallet ID seed (#3250) · 4e10d3b0
      Muharem Ismailov authored
      Introduce `PalletId` as an additional seed parameter for pool's account
      id derivation.
      
      The PR also introduces the `pallet_asset_conversion_ops` pallet with a
      call to migrate a given pool to thew new account. Additionally
      `fungibles::lifetime::ResetTeam` and `fungible::lifetime::Refund`
      traits, to facilitate the migration of pools.
      
      ---------
      
      Co-authored-by: command-bot <>
      4e10d3b0
    • Sergej Sakac's avatar
      XCM coretime region transfers (#3455) · e6f3106d
      Sergej Sakac authored
      This PR introduces changes enabling the transfer of coretime regions via
      XCM.
      
      TL;DR: There are two primary issues that are resolved in this PR:
      
      1. The `mint` and `burn` functions were not implemented for coretime
      regions. These operations are essential for moving assets to and from
      the XCM holding register.
      2. The transfer of non-fungible assets through XCM was previously
      disallowed. This was due to incorrectly benchmarking non-fungible asset
      transfers via XCM, which led to assigning it a weight of `Weight::Max`,
      effectively preventing its execution.
      
      ### `mint_into` and `burn` implementation
      
      This PR addresses the issue with cross-chain transferring regions back
      to the Coretime chain. Remote reserve transfers are performed by
      withdrawing and depositing the asset to and from the holding registry.
      This requires the asset to support burning and minting functionality.
      
      This PR adds burning and minting; however, they work a bit differently
      than usual so that the associated region record is not lost when
      burning. Instead of removing all the data, burning will set the owner of
      the region to `None`, and when minting it back, it will set it to an
      actual value. So, when cross-chain transferring, withdrawing into the
      registry will remove the region from its original owner, and when
      depositing it from the registry, it will set its owner to another
      account
      
      This was originally implemented in this PR: #3455, however we decided to
      move all of it to this single PR
      (https://github.com/paritytech/polkadot-sdk/pull/3455#discussion_r1547324892)
      
      ### Fixes made in this PR
      
      - Update the `XcmReserveTransferFilter` on coretime chain since it is
      meant as a reserve chain for coretime regions.
      - Update the XCM benchmark to use `AssetTransactor` instead of assuming
      `pallet-balances` for fungible transfers.
      - Update the XCM benchmark to properly measure weight consumption for
      nonfungible reserve asset transfers. ATM reserve transfers via the
      extrinsic do not work since the weight for it is set to `Weight::max()`.
      
      Closes: https://github.com/paritytech/polkadot-sdk/issues/865
      
      
      
      ---------
      
      Co-authored-by: default avatarBranislav Kontur <[email protected]>
      Co-authored-by: default avatarFrancisco Aguirre <[email protected]>
      Co-authored-by: default avatarDónal Murray <[email protected]>
      e6f3106d
  7. Apr 16, 2024
  8. Apr 15, 2024
  9. Apr 13, 2024
  10. Apr 12, 2024
    • Branislav Kontur's avatar
      Remove `xcm::v3` from `assets-common` nits (#4037) · 5601f286
      Branislav Kontur authored
      This PR mainly removes `xcm::v3` stuff from `assets-common` to make it
      more generic and facilitate the transition to newer XCM versions. Some
      of the implementations here used hard-coded `xcm::v3::Location`, but now
      it's up to the runtime to configure according to its needs.
      
      Additional/consequent changes:
      - `penpal` runtime uses now `xcm::latest::Location` for `pallet_assets`
      as `AssetId`, because we don't care about migrations here
      - it pretty much simplify xcm-emulator integration tests, where we don't
      need now a lots of boilerplate conversions:
            ```
            v3::Location::try_from(...).expect("conversion works")`
            ```
      - xcm-emulator tests
      - split macro `impl_assets_helpers_for_parachain` to the
      `impl_assets_helpers_for_parachain` and
      `impl_foreign_assets_helpers_for_parachain` (avoids using hard-coded
      `xcm::v3::Location`)
      5601f286
  11. Apr 11, 2024
  12. Apr 10, 2024
  13. Apr 09, 2024
    • Sebastian Kunert's avatar
      Move cumulus zombienet tests to aura & async backing (#3568) · df818d29
      Sebastian Kunert authored
      Cumulus test-parachain node and test runtime were still using relay
      chain consensus and 12s blocktimes. With async backing around the corner
      on the major chains we should switch our tests too.
      
      Also needed to nicely test the changes coming to collators in #3168.
      
      ### Changes Overview
      - Followed the [migration
      guide](https://wiki.polkadot.network/docs/maintain-guides-async-backing)
      for async backing for the cumulus-test-runtime
      - Adjusted the cumulus-test-service to use the correct import-queue,
      lookahead collator etc.
      - The block validation function now uses the Aura Ext Executor so that
      the seal of the block is validated
      - Previous point requires that we seal block before calling into
      `validate_block`, I introduced a helper function for that
      - Test client adjusted to provide a slot to the relay chain proof and
      the aura pre-digest
      df818d29
    • Facundo Farall's avatar
      Upgrade `trie-db` from `0.28.0` to `0.29.0` (#3982) · 4e73c0fc
      Facundo Farall authored
      
      
      # Description
      - What does this PR do?
      1. Upgrades `trie-db`'s version to the latest release. This release
      includes, among others, an implementation of `DoubleEndedIterator` for
      the `TrieDB` struct, allowing to iterate both backwards and forwards
      within the leaves of a trie.
      2. Upgrades `trie-bench` to `0.39.0` for compatibility.
      3. Upgrades `criterion` to `0.5.1` for compatibility.
      - Why are these changes needed?
      Besides keeping up with the upgrade of `trie-db`, this specifically adds
      the functionality of iterating back on the leafs of a trie, with
      `sp-trie`. In a project we're currently working on, this comes very
      handy to verify a Merkle proof that is the response to a challenge. The
      challenge is a random hash that (most likely) will not be an existing
      leaf in the trie. So the challenged user, has to provide a Merkle proof
      of the previous and next existing leafs in the trie, that surround the
      random challenged hash.
      
      Without having DoubleEnded iterators, we're forced to iterate until we
      find the first existing leaf, like so:
      ```rust
              // ************* VERIFIER (RUNTIME) *************
              // Verify proof. This generates a partial trie based on the proof and
              // checks that the root hash matches the `expected_root`.
              let (memdb, root) = proof.to_memory_db(Some(&root)).unwrap();
              let trie = TrieDBBuilder::<LayoutV1<RefHasher>>::new(&memdb, &root).build();
      
              // Print all leaf node keys and values.
              println!("\nPrinting leaf nodes of partial tree...");
              for key in trie.key_iter().unwrap() {
                  if key.is_ok() {
                      println!("Leaf node key: {:?}", key.clone().unwrap());
      
                      let val = trie.get(&key.unwrap());
      
                      if val.is_ok() {
                          println!("Leaf node value: {:?}", val.unwrap());
                      } else {
                          println!("Leaf node value: None");
                      }
                  }
              }
      
              println!("RECONSTRUCTED TRIE {:#?}", trie);
      
              // Create an iterator over the leaf nodes.
              let mut iter = trie.iter().unwrap();
      
              // First element with a value should be the previous existing leaf to the challenged hash.
              let mut prev_key = None;
              for element in &mut iter {
                  if element.is_ok() {
                      let (key, _) = element.unwrap();
                      prev_key = Some(key);
                      break;
                  }
              }
              assert!(prev_key.is_some());
      
              // Since hashes are `Vec<u8>` ordered in big-endian, we can compare them directly.
              assert!(prev_key.unwrap() <= challenge_hash.to_vec());
      
              // The next element should exist (meaning there is no other existing leaf between the
              // previous and next leaf) and it should be greater than the challenged hash.
              let next_key = iter.next().unwrap().unwrap().0;
              assert!(next_key >= challenge_hash.to_vec());
      ```
      
      With DoubleEnded iterators, we can avoid that, like this:
      ```rust
              // ************* VERIFIER (RUNTIME) *************
              // Verify proof. This generates a partial trie based on the proof and
              // checks that the root hash matches the `expected_root`.
              let (memdb, root) = proof.to_memory_db(Some(&root)).unwrap();
              let trie = TrieDBBuilder::<LayoutV1<RefHasher>>::new(&memdb, &root).build();
      
              // Print all leaf node keys and values.
              println!("\nPrinting leaf nodes of partial tree...");
              for key in trie.key_iter().unwrap() {
                  if key.is_ok() {
                      println!("Leaf node key: {:?}", key.clone().unwrap());
      
                      let val = trie.get(&key.unwrap());
      
                      if val.is_ok() {
                          println!("Leaf node value: {:?}", val.unwrap());
                      } else {
                          println!("Leaf node value: None");
                      }
                  }
              }
      
              // println!("RECONSTRUCTED TRIE {:#?}", trie);
              println!("\nChallenged key: {:?}", challenge_hash);
      
              // Create an iterator over the leaf nodes.
              let mut double_ended_iter = trie.into_double_ended_iter().unwrap();
      
              // First element with a value should be the previous existing leaf to the challenged hash.
              double_ended_iter.seek(&challenge_hash.to_vec()).unwrap();
              let next_key = double_ended_iter.next_back().unwrap().unwrap().0;
              let prev_key = double_ended_iter.next_back().unwrap().unwrap().0;
      
              // Since hashes are `Vec<u8>` ordered in big-endian, we can compare them directly.
              println!("Prev key: {:?}", prev_key);
              assert!(prev_key <= challenge_hash.to_vec());
      
              println!("Next key: {:?}", next_key);
              assert!(next_key >= challenge_hash.to_vec());
      ```
      - How were these changes implemented and what do they affect?
      All that is needed for this functionality to be exposed is changing the
      version number of `trie-db` in all the `Cargo.toml`s applicable, and
      re-exporting some additional structs from `trie-db` in `sp-trie`.
      
      ---------
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      4e73c0fc
    • Branislav Kontur's avatar
  14. Apr 08, 2024
    • Aaro Altonen's avatar
      Integrate litep2p into Polkadot SDK (#2944) · 80616f6d
      Aaro Altonen authored
      [litep2p](https://github.com/altonen/litep2p) is a libp2p-compatible P2P
      networking library. It supports all of the features of `rust-libp2p`
      that are currently being utilized by Polkadot SDK.
      
      Compared to `rust-libp2p`, `litep2p` has a quite different architecture
      which is why the new `litep2p` network backend is only able to use a
      little of the existing code in `sc-network`. The design has been mainly
      influenced by how we'd wish to structure our networking-related code in
      Polkadot SDK: independent higher-levels protocols directly communicating
      with the network over links that support bidirectional backpressure. A
      good example would be `NotificationHandle`/`RequestResponseHandle`
      abstractions which allow, e.g., `SyncingEngine` to directly communicate
      with peers to announce/request blocks.
      
      I've tried running `polkadot --network-backend litep2p` with a few
      different peer configurations and there is a noticeable reduction in
      networking CPU usage. For high load (`--out-peers 200`), networking CPU
      usage goes down from ~110% to ~30% (80 pp) and for normal load
      (`--out-peers 40`), the usage goes down from ~55% to ~18% (37 pp).
      
      These should not be taken as final numbers because:
      
      a) there are still some low-hanging optimization fruits, such as
      enabling [receive window
      auto-tuning](https://github.com/libp2p/rust-yamux/pull/176
      
      ), integrating
      `Peerset` more closely with `litep2p` or improving memory usage of the
      WebSocket transport
      b) fixing bugs/instabilities that incorrectly cause `litep2p` to do less
      work will increase the networking CPU usage
      c) verification in a more diverse set of tests/conditions is needed
      
      Nevertheless, these numbers should give an early estimate for CPU usage
      of the new networking backend.
      
      This PR consists of three separate changes:
      * introduce a generic `PeerId` (wrapper around `Multihash`) so that we
      don't have use `NetworkService::PeerId` in every part of the code that
      uses a `PeerId`
      * introduce `NetworkBackend` trait, implement it for the libp2p network
      stack and make Polkadot SDK generic over `NetworkBackend`
        * implement `NetworkBackend` for litep2p
      
      The new library should be considered experimental which is why
      `rust-libp2p` will remain as the default option for the time being. This
      PR currently depends on the master branch of `litep2p` but I'll cut a
      new release for the library once all review comments have been
      addresses.
      
      ---------
      
      Signed-off-by: default avatarAlexandru Vasile <[email protected]>
      Co-authored-by: default avatarDmitry Markin <[email protected]>
      Co-authored-by: default avatarAlexandru Vasile <[email protected]>
      Co-authored-by: default avatarAlexandru Vasile <[email protected]>
      80616f6d
    • Oliver Tale-Yazdi's avatar
      [FRAME] Runtime Omni Bencher (#3512) · 9543d314
      Oliver Tale-Yazdi authored
      This MR contains two major changes and some maintenance cleanup.  
      
      ## 1. Free Standing Pallet Benchmark Runner
      
      Closes https://github.com/paritytech/polkadot-sdk/issues/3045, depends
      on your runtime exposing the `GenesisBuilderApi` (like
      https://github.com/paritytech/polkadot-sdk/pull/1492).
      
      Introduces a new binary crate: `frame-omni-bencher`.  
      It allows to directly benchmark a WASM blob - without needing a node or
      chain spec.
      
      This makes it much easier to generate pallet weights and should allow us
      to remove bloaty code from the node.
      It should work for all FRAME runtimes that dont use 3rd party host calls
      or non `BlakeTwo256` block hashing (basically all polkadot parachains
      should work).
      
      It is 100% backwards compatible with the old CLI args, when the `v1`
      compatibility command is used. This is done to allow for forwards
      compatible addition of new commands.
      
      ### Example (full example in the Rust docs)
      
      Installing the CLI:
      ```sh
      cargo install --locked --path substrate/utils/frame/omni-bencher
      frame-omni-bencher --help
      ```
      
      Building the Westend runtime:
      ```sh
      cargo build -p westend-runtime --release --features runtime-benchmarks
      ```
      
      Benchmarking the runtime:
      ```sh
      frame-omni-bencher v1 benchmark pallet --runtime target/release/wbuild/westend-runtime/westend_runtime.compact.compressed.wasm --all
      ```
      
      ## 2. Building the Benchmark Genesis State in the Runtime
      
      Closes https://github.com/paritytech/polkadot-sdk/issues/2664
      
      This adds `--runtime` and `--genesis-builder=none|runtime|spec`
      arguments to the `benchmark pallet` command to make it possible to
      generate the genesis storage by the runtime. This can be used with both
      the node and the freestanding benchmark runners. It utilizes the new
      `GenesisBuilder` RA and depends on having
      https://github.com/paritytech/polkadot-sdk/pull/3412
      
       deployed.
      
      ## 3. Simpler args for `PalletCmd::run`
      
      You can do three things here to integrate the changes into your node:
      - nothing: old code keeps working as before but emits a deprecated
      warning
      - delete: remove the pallet benchmarking code from your node and use the
      omni-bencher instead
      - patch: apply the patch below and keep using as currently. This emits a
      deprecated warning at runtime, since it uses the old way to generate a
      genesis state, but is the smallest change.
      
      ```patch
      runner.sync_run(|config| cmd
      -    .run::<HashingFor<Block>, ReclaimHostFunctions>(config)
      +    .run_with_spec::<HashingFor<Block>, ReclaimHostFunctions>(Some(config.chain_spec))
      )
      ```
      
      ## 4. Maintenance Change
      - `pallet-nis` get a `BenchmarkSetup` config item to prepare its
      counterparty asset.
      - Add percent progress print when running benchmarks.
      - Dont immediately exit on benchmark error but try to run as many as
      possible and print errors last.
      
      ---------
      
      Signed-off-by: default avatarOliver Tale-Yazdi <[email protected]>
      Co-authored-by: default avatarLiam Aharon <[email protected]>
      9543d314
  15. Apr 06, 2024
    • Squirrel's avatar
      Major bump of tracing-subscriber version (#3891) · 99400385
      Squirrel authored
      
      
      I don't think there are any more releases to the 0.2.x versions, so best
      we're on the 0.3.x release.
      
      No change on the benchmarks, fast local time is still just as fast as
      before:
      
      new version bench:
      ```
      fast_local_time         time:   [30.551 ns 30.595 ns 30.668 ns]
      ```
      
      old version bench:
      ```
      fast_local_time         time:   [30.598 ns 30.646 ns 30.723 ns]
      ```
      
      ---------
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      99400385
  16. Apr 05, 2024
  17. Apr 04, 2024
  18. Apr 03, 2024
  19. Apr 02, 2024
  20. Apr 01, 2024
  21. Mar 31, 2024
  22. Mar 28, 2024
  23. Mar 27, 2024
  24. Mar 26, 2024
    • Pavel Orlov's avatar
      XCM Fee Payment Runtime API (#3607) · 3c972fc1
      Pavel Orlov authored
      The PR provides API for obtaining:
      - the weight required to execute an XCM message,
      - a list of acceptable `AssetId`s for message execution payment,
      - the cost of the weight in the specified acceptable `AssetId`.
      
      It is meant to address an issue where one has to guess how much fee to
      pay for execution. Also, at the moment, a client has to guess which
      assets are acceptable for fee execution payment.
      See the related issue
      https://github.com/paritytech/polkadot-sdk/issues/690.
      With this API, a client is supposed to query the list of the supported
      asset IDs (in the XCM version format the client understands), weigh the
      XCM program the client wants to execute and convert the weight into one
      of the acceptable assets. Note that the client is supposed to know what
      program will be executed on what chains. However, having a small
      companion JS library for the pallet-xcm and xtokens should be enough to
      determine what XCM programs will be executed and where (since these
      pallets compose a known small set of programs).
      ```Rust
      pub trait XcmPaymentApi<Call>
      	where
      		Call: Codec,
      	{
      		/// Returns a list of acceptable payment assets.
      		///
      		/// # Arguments
      		///
      		/// * `xcm_version`: Version.
      		fn query_acceptable_payment_assets(xcm_version: Version) -> Result<Vec<VersionedAssetId>, Error>;
      		/// Returns a weight needed to execute a XCM.
      		///
      		/// # Arguments
      		///
      		/// * `message`: `VersionedXcm`.
      		fn query_xcm_weight(message: VersionedXcm<Call>) -> Result<Weight, Error>;
      		/// Converts a weight into a fee for the specified `AssetId`.
      		///
      		/// # Arguments
      		///
      		/// * `weight`: convertible `Weight`.
      		/// * `asset`: `VersionedAssetId`.
      		fn query_weight_to_asset_fee(weight: Weight, asset: VersionedAssetId) -> Result<u128, Error>;
      		/// Get delivery fees for sending a specific `message` to a `destination`.
      		/// These always come in a specific asset, defined by the chain.
      		///
      		/// # Arguments
      		/// * `message`: The message that'll be sent, necessary because most delivery fees are based on the
      		///   size of the message.
      		/// * `destination`: The destination to send the message to. Different destinations may use
      		///   different senders that charge different fees.
      		fn query_delivery_fees(destination: VersionedLocation, message: VersionedXcm<()>) -> Result<VersionedAssets, Error>;
      	}
      ```
      An
      [example](https://gist.github.com/PraetorP/4bc323ff85401abe253897ba990ec29d
      
      )
      of a client side code.
      
      ---------
      
      Co-authored-by: default avatarFrancisco Aguirre <[email protected]>
      Co-authored-by: default avatarAdrian Catangiu <[email protected]>
      Co-authored-by: default avatarDaniel Shiposha <[email protected]>
      3c972fc1