Skip to content
  1. Apr 08, 2024
    • Aaro Altonen's avatar
      Integrate litep2p into Polkadot SDK (#2944) · 80616f6d
      Aaro Altonen authored 
      [litep2p](https://github.com/altonen/litep2p) is a libp2p-compatible P2P
networking library. It supports all of the features of `rust-libp2p`
that are currently being utilized by Polkadot SDK.

Compared to `rust-libp2p`, `litep2p` has a quite different architecture
which is why the new `litep2p` network backend is only able to use a
little of the existing code in `sc-network`. The design has been mainly
influenced by how we'd wish to structure our networking-related code in
Polkadot SDK: independent higher-levels protocols directly communicating
with the network over links that support bidirectional backpressure. A
good example would be `NotificationHandle`/`RequestResponseHandle`
abstractions which allow, e.g., `SyncingEngine` to directly communicate
with peers to announce/request blocks.

I've tried running `polkadot --network-backend litep2p` with a few
different peer configurations and there is a noticeable reduction in
networking CPU usage. For high load (`--out-peers 200`), networking CPU
usage goes down from ~110% to ~30% (80 pp) and for normal load
(`--out-peers 40`), the usage goes down from ~55% to ~18% (37 pp).

These should not be taken as final numbers because:

a) there are still some low-hanging optimization fruits, such as
enabling [receive window
auto-tuning](https://github.com/libp2p/rust-yamux/pull/176

), integrating
`Peerset` more closely with `litep2p` or improving memory usage of the
WebSocket transport
b) fixing bugs/instabilities that incorrectly cause `litep2p` to do less
work will increase the networking CPU usage
c) verification in a more diverse set of tests/conditions is needed

Nevertheless, these numbers should give an early estimate for CPU usage
of the new networking backend.

This PR consists of three separate changes:
* introduce a generic `PeerId` (wrapper around `Multihash`) so that we
don't have use `NetworkService::PeerId` in every part of the code that
uses a `PeerId`
* introduce `NetworkBackend` trait, implement it for the libp2p network
stack and make Polkadot SDK generic over `NetworkBackend`
  * implement `NetworkBackend` for litep2p

The new library should be considered experimental which is why
`rust-libp2p` will remain as the default option for the time being. This
PR currently depends on the master branch of `litep2p` but I'll cut a
new release for the library once all review comments have been
addresses.

---------

Signed-off-by: default avatarAlexandru Vasile <[email protected]>
Co-authored-by: default avatarDmitry Markin <[email protected]>
Co-authored-by: default avatarAlexandru Vasile <[email protected]>
Co-authored-by: default avatarAlexandru Vasile <[email protected]>
      80616f6d
    • Oliver Tale-Yazdi's avatar
      [FRAME] Runtime Omni Bencher (#3512) · 9543d314
      Oliver Tale-Yazdi authored 
      This MR contains two major changes and some maintenance cleanup.  

## 1. Free Standing Pallet Benchmark Runner

Closes https://github.com/paritytech/polkadot-sdk/issues/3045, depends
on your runtime exposing the `GenesisBuilderApi` (like
https://github.com/paritytech/polkadot-sdk/pull/1492).

Introduces a new binary crate: `frame-omni-bencher`.  
It allows to directly benchmark a WASM blob - without needing a node or
chain spec.

This makes it much easier to generate pallet weights and should allow us
to remove bloaty code from the node.
It should work for all FRAME runtimes that dont use 3rd party host calls
or non `BlakeTwo256` block hashing (basically all polkadot parachains
should work).

It is 100% backwards compatible with the old CLI args, when the `v1`
compatibility command is used. This is done to allow for forwards
compatible addition of new commands.

### Example (full example in the Rust docs)

Installing the CLI:
```sh
cargo install --locked --path substrate/utils/frame/omni-bencher
frame-omni-bencher --help
```

Building the Westend runtime:
```sh
cargo build -p westend-runtime --release --features runtime-benchmarks
```

Benchmarking the runtime:
```sh
frame-omni-bencher v1 benchmark pallet --runtime target/release/wbuild/westend-runtime/westend_runtime.compact.compressed.wasm --all
```

## 2. Building the Benchmark Genesis State in the Runtime

Closes https://github.com/paritytech/polkadot-sdk/issues/2664

This adds `--runtime` and `--genesis-builder=none|runtime|spec`
arguments to the `benchmark pallet` command to make it possible to
generate the genesis storage by the runtime. This can be used with both
the node and the freestanding benchmark runners. It utilizes the new
`GenesisBuilder` RA and depends on having
https://github.com/paritytech/polkadot-sdk/pull/3412

 deployed.

## 3. Simpler args for `PalletCmd::run`

You can do three things here to integrate the changes into your node:
- nothing: old code keeps working as before but emits a deprecated
warning
- delete: remove the pallet benchmarking code from your node and use the
omni-bencher instead
- patch: apply the patch below and keep using as currently. This emits a
deprecated warning at runtime, since it uses the old way to generate a
genesis state, but is the smallest change.

```patch
runner.sync_run(|config| cmd
-    .run::<HashingFor<Block>, ReclaimHostFunctions>(config)
+    .run_with_spec::<HashingFor<Block>, ReclaimHostFunctions>(Some(config.chain_spec))
)
```

## 4. Maintenance Change
- `pallet-nis` get a `BenchmarkSetup` config item to prepare its
counterparty asset.
- Add percent progress print when running benchmarks.
- Dont immediately exit on benchmark error but try to run as many as
possible and print errors last.

---------

Signed-off-by: default avatarOliver Tale-Yazdi <[email protected]>
Co-authored-by: default avatarLiam Aharon <[email protected]>
      9543d314
  3. Apr 06, 2024
    • Squirrel's avatar
      Major bump of tracing-subscriber version (#3891) · 99400385
      Squirrel authored 
      

I don't think there are any more releases to the 0.2.x versions, so best
we're on the 0.3.x release.

No change on the benchmarks, fast local time is still just as fast as
before:

new version bench:
```
fast_local_time         time:   [30.551 ns 30.595 ns 30.668 ns]
```

old version bench:
```
fast_local_time         time:   [30.598 ns 30.646 ns 30.723 ns]
```

---------

Co-authored-by: default avatarBastian Köcher <[email protected]>
      99400385
  5. Apr 05, 2024
  7. Apr 04, 2024
  9. Apr 03, 2024
  11. Apr 02, 2024
  13. Apr 01, 2024
  15. Mar 31, 2024
  17. Mar 28, 2024
  19. Mar 27, 2024
  21. Mar 26, 2024
    • Pavel Orlov's avatar
      XCM Fee Payment Runtime API (#3607) · 3c972fc1
      Pavel Orlov authored 
      The PR provides API for obtaining:
- the weight required to execute an XCM message,
- a list of acceptable `AssetId`s for message execution payment,
- the cost of the weight in the specified acceptable `AssetId`.

It is meant to address an issue where one has to guess how much fee to
pay for execution. Also, at the moment, a client has to guess which
assets are acceptable for fee execution payment.
See the related issue
https://github.com/paritytech/polkadot-sdk/issues/690.
With this API, a client is supposed to query the list of the supported
asset IDs (in the XCM version format the client understands), weigh the
XCM program the client wants to execute and convert the weight into one
of the acceptable assets. Note that the client is supposed to know what
program will be executed on what chains. However, having a small
companion JS library for the pallet-xcm and xtokens should be enough to
determine what XCM programs will be executed and where (since these
pallets compose a known small set of programs).
```Rust
pub trait XcmPaymentApi<Call>
	where
		Call: Codec,
	{
		/// Returns a list of acceptable payment assets.
		///
		/// # Arguments
		///
		/// * `xcm_version`: Version.
		fn query_acceptable_payment_assets(xcm_version: Version) -> Result<Vec<VersionedAssetId>, Error>;
		/// Returns a weight needed to execute a XCM.
		///
		/// # Arguments
		///
		/// * `message`: `VersionedXcm`.
		fn query_xcm_weight(message: VersionedXcm<Call>) -> Result<Weight, Error>;
		/// Converts a weight into a fee for the specified `AssetId`.
		///
		/// # Arguments
		///
		/// * `weight`: convertible `Weight`.
		/// * `asset`: `VersionedAssetId`.
		fn query_weight_to_asset_fee(weight: Weight, asset: VersionedAssetId) -> Result<u128, Error>;
		/// Get delivery fees for sending a specific `message` to a `destination`.
		/// These always come in a specific asset, defined by the chain.
		///
		/// # Arguments
		/// * `message`: The message that'll be sent, necessary because most delivery fees are based on the
		///   size of the message.
		/// * `destination`: The destination to send the message to. Different destinations may use
		///   different senders that charge different fees.
		fn query_delivery_fees(destination: VersionedLocation, message: VersionedXcm<()>) -> Result<VersionedAssets, Error>;
	}
```
An
[example](https://gist.github.com/PraetorP/4bc323ff85401abe253897ba990ec29d

)
of a client side code.

---------

Co-authored-by: default avatarFrancisco Aguirre <[email protected]>
Co-authored-by: default avatarAdrian Catangiu <[email protected]>
Co-authored-by: default avatarDaniel Shiposha <[email protected]>
      3c972fc1
    • Andrei Eres's avatar
      [subsystem-benchmarks] Save results to json (#3829) · fd79b3b0
      Andrei Eres authored 
      Here we add the ability to save subsystem benchmark results in JSON
format to display them as graphs

To draw graphs, CI team will use
[github-action-benchmark](https://github.com/benchmark-action/github-action-benchmark).
Since we are using custom benchmarks, we need to prepare [a specific
data
type](https://github.com/benchmark-action/github-action-benchmark?tab=readme-ov-file#examples):
```
[
    {
        "name": "CPU Load",
        "unit": "Percent",
        "value": 50
    }
]
```

Then we'll get graphs like this: 

![example](https://raw.githubusercontent.com/rhysd/ss/master/github-action-benchmark/main.png)

[A live page with
graphs](https://benchmark-action.github.io/github-action-benchmark/dev/bench/

)

---------

Co-authored-by: default avatarordian <[email protected]>
      fd79b3b0
  23. Mar 22, 2024
  25. Mar 20, 2024
  27. Mar 19, 2024
  29. Mar 18, 2024
    • Squirrel's avatar
      sp-std removal from substrate/primitives (#3274) · 1b5f4243
      Squirrel authored 
      

This PR removes sp-std crate from substrate/primitives sub-directories.

For now crates that have `pub use` of sp-std or export macros that would
necessitate users of the macros to `extern crate alloc` have been
excluded from this PR.

There should be no breaking changes in this PR.

---------

Co-authored-by: default avatarKoute <[email protected]>
      1b5f4243
  31. Mar 17, 2024
  33. Mar 15, 2024
  35. Mar 14, 2024
    • Ignacio Palacios's avatar
      Improve Penpal runtime + emulated tests (#3543) · cfc4050d
      Ignacio Palacios authored 
      Issues addressed in this PR:
- Improve *Penpal* runtime:
- Properly handled received assets. Previously, it treated `(1, Here)`
as the local native currency, whereas it should be treated as a
`ForeignAsset`. This wasn't a great example of standard Parachain
behaviour, as no Parachain treats the system asset as the local
currency.
- Remove `AllowExplicitUnpaidExecutionFrom` the system. Again, this
wasn't a great example of standard Parachain behaviour.
- Move duplicated
`ForeignAssetFeeAsExistentialDepositMultiplierFeeCharger` to
`assets_common` crate.
- Improve emulated tests:
  - Update *Penpal* tests to new runtime.
- To simplify tests, register the reserve transferred, teleported, and
system assets in *Penpal* and *AssetHub* genesis. This saves us from
having to create the assets repeatedly for each test
- Add missing test case:
`reserve_transfer_assets_from_para_to_system_para`.
  - Cleanup.
- Prevent integration tests crates imports from being re-exported, as
they were polluting the `polkadot-sdk` docs.

There is still a test case missing for reserve transfers:
- Reserve transfer of system asset from *Parachain* to *Parachain*
trough *AssetHub*.
- This is not yet possible with `pallet-xcm` due to the reasons
explained in https://github.com/paritytech/polkadot-sdk/pull/3339

---------

Co-authored-by: command-bot <>
      cfc4050d
  37. Mar 13, 2024
  39. Mar 12, 2024
  41. Mar 11, 2024
  43. Mar 09, 2024
  45. Mar 07, 2024