1. Apr 15, 2024
    • Alexandru Gheorghe's avatar
      Prevent accidental change of network-key for active authorities (#3852) · 2bc4ed11
      Alexandru Gheorghe authored
      As discovered during investigation of
      https://github.com/paritytech/polkadot-sdk/issues/3314 and
      https://github.com/paritytech/polkadot-sdk/issues/3673 there are active
      validators which accidentally might change their network key during
      restart, that's not a safe operation when you are in the active set
      because of distributed nature of DHT, so the old records would still
      exist in the network until they expire 36h, so unless they have a good
      reason validators should avoid changing their key when they restart
      their nodes.
      
      There is an effort in parallel to improve this situation
      https://github.com/paritytech/polkadot-sdk/pull/3786
      
      , but those changes
      are way more intrusive and will need more rigorous testing, additionally
      they will reduce the time to less than 36h, but the propagation won't be
      instant anyway, so not changing your network during restart should be
      the safest way to run your node, unless you have a really good reason to
      change it.
      
      ## Proposal
      1. Do not auto-generate the network if the network file does not exist
      in the provided path. Nodes where the key file does not exist will get
      the following error:
      ```
      Error: 
         0: Starting an authorithy without network key in /home/alexggh/.local/share/polkadot/chains/ksmcc3/network/secret_ed25519.
            
             This is not a safe operation because the old identity still lives in the dht for 36 hours.
            
             Because of it your node might suffer from not being properly connected to other nodes for validation purposes.
            
             If it is the first time running your node you could use one of the following methods.
            
             1. Pass --unsafe-force-node-key-generation and make sure you remove it for subsequent node restarts
            
             2. Separetly generate the key with: polkadot key generate-node-key --file <YOUR_PATH_TO_NODE_KEY>
      ```
      
      2. Add an explicit parameters for nodes that do want to change their
      network despite the warnings or if they run the node for the first time.
      `--unsafe-force-node-key-generation`
      
      3. For `polkadot key generate-node-key` add two new mutually exclusive
      parameters `base_path` and `default_base_path` to help with the key
      generation in the same path the polkadot main command would expect it.
       
      4. Modify the installation scripts to auto-generate a key in default
      path if one was not present already there, this should help with making
      the executable work out of the box after an instalation.
      
      ## Notes
      
      Nodes that do not have already the key persisted will fail to start
      after this change, however I do consider that better than the current
      situation where they start but they silently hide that they might not be
      properly connected to their peers.
      
      ## TODO
      - [x] Make sure only nodes that are authorities on producation chains
      will be affected by this restrictions.
      - [x] Proper PRDOC, to make sure node operators are aware this is
      coming.
      
      ---------
      
      Signed-off-by: default avatarAlexandru Gheorghe <[email protected]>
      Co-authored-by: default avatarDmitry Markin <[email protected]>
      Co-authored-by: default avatars0me0ne-unkn0wn <[email protected]>
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      2bc4ed11
  2. Apr 14, 2024
  3. Apr 13, 2024
  4. Apr 12, 2024
    • Vedhavyas Singareddi's avatar
      define block hash provider and default impl using frame_system (#4080) · 5b513cc0
      Vedhavyas Singareddi authored
      This PR introduces `BlockHashProvider` into `pallet_mmr::Config`
      This type is used to get `block_hash` for a given `block_number` rather
      than directly using `frame_system::Pallet::block_hash`
      
      The `DefaultBlockHashProvider` uses `frame_system::Pallet::block_hash`
      to get the `block_hash`
      
      Closes: #4062
      5b513cc0
    • Branislav Kontur's avatar
      Remove `xcm::v3` from `assets-common` nits (#4037) · 5601f286
      Branislav Kontur authored
      This PR mainly removes `xcm::v3` stuff from `assets-common` to make it
      more generic and facilitate the transition to newer XCM versions. Some
      of the implementations here used hard-coded `xcm::v3::Location`, but now
      it's up to the runtime to configure according to its needs.
      
      Additional/consequent changes:
      - `penpal` runtime uses now `xcm::latest::Location` for `pallet_assets`
      as `AssetId`, because we don't care about migrations here
      - it pretty much simplify xcm-emulator integration tests, where we don't
      need now a lots of boilerplate conversions:
            ```
            v3::Location::try_from(...).expect("conversion works")`
            ```
      - xcm-emulator tests
      - split macro `impl_assets_helpers_for_parachain` to the
      `impl_assets_helpers_for_parachain` and
      `impl_foreign_assets_helpers_for_parachain` (avoids using hard-coded
      `xcm::v3::Location`)
      5601f286
    • Adrian Catangiu's avatar
      pallet-xcm: add new extrinsic for asset transfers using explicit XCM transfer types (#3695) · 1e971b8d
      Adrian Catangiu authored
      
      
      # Description
      
      Add `transfer_assets_using()` for transferring assets from local chain
      to destination chain using explicit XCM transfer types such as:
      - `TransferType::LocalReserve`: transfer assets to sovereign account of
      destination chain and forward a notification XCM to `dest` to mint and
      deposit reserve-based assets to `beneficiary`.
      - `TransferType::DestinationReserve`: burn local assets and forward a
      notification to `dest` chain to withdraw the reserve assets from this
      chain's sovereign account and deposit them to `beneficiary`.
      - `TransferType::RemoteReserve(reserve)`: burn local assets, forward XCM
      to `reserve` chain to move reserves from this chain's SA to `dest`
      chain's SA, and forward another XCM to `dest` to mint and deposit
      reserve-based assets to `beneficiary`. Typically the remote `reserve` is
      Asset Hub.
      - `TransferType::Teleport`: burn local assets and forward XCM to `dest`
      chain to mint/teleport assets and deposit them to `beneficiary`.
      
      By default, an asset's reserve is its origin chain. But sometimes we may
      want to explicitly use another chain as reserve (as long as allowed by
      runtime `IsReserve` filter).
      
      This is very helpful for transferring assets with multiple configured
      reserves (such as Asset Hub ForeignAssets), when the transfer strictly
      depends on the used reserve.
      
      E.g. For transferring Foreign Assets over a bridge, Asset Hub must be
      used as the reserve location.
      
      # Example usage scenarios
      
      ## Transfer bridged ethereum ERC20-tokenX between ecosystem parachains.
      
      ERC20-tokenX is registered on AssetHub as a ForeignAsset by the
      Polkadot<>Ethereum bridge (Snowbridge). Its asset_id is something like
      `(parents:2, (GlobalConsensus(Ethereum), Address(tokenX_contract)))`.
      Its _original_ reserve is Ethereum (only we can't use Ethereum as a
      reserve in local transfers); but, since tokenX is also registered on
      AssetHub as a ForeignAsset, we can use AssetHub as a reserve.
      
      With this PR we can transfer tokenX from ParaA to ParaB while using
      AssetHub as a reserve.
      
      ## Transfer AssetHub ForeignAssets between parachains
      
      AssetA created on ParaA but also registered as foreign asset on Asset
      Hub. Can use AssetHub as a reserve.
      
      And all of the above can be done while still controlling transfer type
      for `fees` so mixing assets in same transfer is supported.
      
      # Tests
      
      Added integration tests for showcasing:
      - transferring local (not bridged) assets from parachain over bridge
      using local Asset Hub reserve,
      - transferring foreign assets from parachain to Asset Hub,
      - transferring foreign assets from Asset Hub to parachain,
      - transferring foreign assets from parachain to parachain using local
      Asset Hub reserve.
      
      ---------
      
      Co-authored-by: default avatarBranislav Kontur <[email protected]>
      Co-authored-by: command-bot <>
      1e971b8d
    • Andrei Sandu's avatar
      Runtime API: introduce `candidates_pending_availability` (#4027) · 2dfe5f74
      Andrei Sandu authored
      Fixes https://github.com/paritytech/polkadot-sdk/issues/3576
      
      
      
      Required by elastic scaling collators.
      Deprecates old API: `candidate_pending_availability`.
      
      TODO:
      - [x] PRDoc
      
      ---------
      
      Signed-off-by: default avatarAndrei Sandu <[email protected]>
      2dfe5f74
    • Xiliang Chen's avatar
      establish_channel_with_system (#3721) · b1db5f3a
      Xiliang Chen authored
      Implements https://github.com/polkadot-fellows/RFCs/issues/82
      
      
      Allow any parachain to have bidirectional channel with any system
      parachains
      
      Looking for initial feedbacks before continue finish it
      
      TODOs:
      - [x] docs
      - [x] benchmarks
      - [x] tests
      
      ---------
      
      Co-authored-by: default avatarjoe petrowski <[email protected]>
      Co-authored-by: default avatarAdrian Catangiu <[email protected]>
      b1db5f3a
    • Andrei Sandu's avatar
      Adjust zombienet test resources and logic (#4032) · 13ca339e
      Andrei Sandu authored
      
      
      One more try to make this test robust from a resource perspective.
      
      ---------
      
      Signed-off-by: default avatarAndrei Sandu <[email protected]>
      Co-authored-by: default avatarJavier Viola <[email protected]>
      13ca339e
  5. Apr 11, 2024
  6. Apr 10, 2024
  7. Apr 09, 2024
  8. Apr 08, 2024
    • Aaro Altonen's avatar
      Integrate litep2p into Polkadot SDK (#2944) · 80616f6d
      Aaro Altonen authored
      [litep2p](https://github.com/altonen/litep2p) is a libp2p-compatible P2P
      networking library. It supports all of the features of `rust-libp2p`
      that are currently being utilized by Polkadot SDK.
      
      Compared to `rust-libp2p`, `litep2p` has a quite different architecture
      which is why the new `litep2p` network backend is only able to use a
      little of the existing code in `sc-network`. The design has been mainly
      influenced by how we'd wish to structure our networking-related code in
      Polkadot SDK: independent higher-levels protocols directly communicating
      with the network over links that support bidirectional backpressure. A
      good example would be `NotificationHandle`/`RequestResponseHandle`
      abstractions which allow, e.g., `SyncingEngine` to directly communicate
      with peers to announce/request blocks.
      
      I've tried running `polkadot --network-backend litep2p` with a few
      different peer configurations and there is a noticeable reduction in
      networking CPU usage. For high load (`--out-peers 200`), networking CPU
      usage goes down from ~110% to ~30% (80 pp) and for normal load
      (`--out-peers 40`), the usage goes down from ~55% to ~18% (37 pp).
      
      These should not be taken as final numbers because:
      
      a) there are still some low-hanging optimization fruits, such as
      enabling [receive window
      auto-tuning](https://github.com/libp2p/rust-yamux/pull/176
      
      ), integrating
      `Peerset` more closely with `litep2p` or improving memory usage of the
      WebSocket transport
      b) fixing bugs/instabilities that incorrectly cause `litep2p` to do less
      work will increase the networking CPU usage
      c) verification in a more diverse set of tests/conditions is needed
      
      Nevertheless, these numbers should give an early estimate for CPU usage
      of the new networking backend.
      
      This PR consists of three separate changes:
      * introduce a generic `PeerId` (wrapper around `Multihash`) so that we
      don't have use `NetworkService::PeerId` in every part of the code that
      uses a `PeerId`
      * introduce `NetworkBackend` trait, implement it for the libp2p network
      stack and make Polkadot SDK generic over `NetworkBackend`
        * implement `NetworkBackend` for litep2p
      
      The new library should be considered experimental which is why
      `rust-libp2p` will remain as the default option for the time being. This
      PR currently depends on the master branch of `litep2p` but I'll cut a
      new release for the library once all review comments have been
      addresses.
      
      ---------
      
      Signed-off-by: default avatarAlexandru Vasile <[email protected]>
      Co-authored-by: default avatarDmitry Markin <[email protected]>
      Co-authored-by: default avatarAlexandru Vasile <[email protected]>
      Co-authored-by: default avatarAlexandru Vasile <[email protected]>
      80616f6d
    • Oliver Tale-Yazdi's avatar
      [FRAME] Runtime Omni Bencher (#3512) · 9543d314
      Oliver Tale-Yazdi authored
      This MR contains two major changes and some maintenance cleanup.  
      
      ## 1. Free Standing Pallet Benchmark Runner
      
      Closes https://github.com/paritytech/polkadot-sdk/issues/3045, depends
      on your runtime exposing the `GenesisBuilderApi` (like
      https://github.com/paritytech/polkadot-sdk/pull/1492).
      
      Introduces a new binary crate: `frame-omni-bencher`.  
      It allows to directly benchmark a WASM blob - without needing a node or
      chain spec.
      
      This makes it much easier to generate pallet weights and should allow us
      to remove bloaty code from the node.
      It should work for all FRAME runtimes that dont use 3rd party host calls
      or non `BlakeTwo256` block hashing (basically all polkadot parachains
      should work).
      
      It is 100% backwards compatible with the old CLI args, when the `v1`
      compatibility command is used. This is done to allow for forwards
      compatible addition of new commands.
      
      ### Example (full example in the Rust docs)
      
      Installing the CLI:
      ```sh
      cargo install --locked --path substrate/utils/frame/omni-bencher
      frame-omni-bencher --help
      ```
      
      Building the Westend runtime:
      ```sh
      cargo build -p westend-runtime --release --features runtime-benchmarks
      ```
      
      Benchmarking the runtime:
      ```sh
      frame-omni-bencher v1 benchmark pallet --runtime target/release/wbuild/westend-runtime/westend_runtime.compact.compressed.wasm --all
      ```
      
      ## 2. Building the Benchmark Genesis State in the Runtime
      
      Closes https://github.com/paritytech/polkadot-sdk/issues/2664
      
      This adds `--runtime` and `--genesis-builder=none|runtime|spec`
      arguments to the `benchmark pallet` command to make it possible to
      generate the genesis storage by the runtime. This can be used with both
      the node and the freestanding benchmark runners. It utilizes the new
      `GenesisBuilder` RA and depends on having
      https://github.com/paritytech/polkadot-sdk/pull/3412
      
       deployed.
      
      ## 3. Simpler args for `PalletCmd::run`
      
      You can do three things here to integrate the changes into your node:
      - nothing: old code keeps working as before but emits a deprecated
      warning
      - delete: remove the pallet benchmarking code from your node and use the
      omni-bencher instead
      - patch: apply the patch below and keep using as currently. This emits a
      deprecated warning at runtime, since it uses the old way to generate a
      genesis state, but is the smallest change.
      
      ```patch
      runner.sync_run(|config| cmd
      -    .run::<HashingFor<Block>, ReclaimHostFunctions>(config)
      +    .run_with_spec::<HashingFor<Block>, ReclaimHostFunctions>(Some(config.chain_spec))
      )
      ```
      
      ## 4. Maintenance Change
      - `pallet-nis` get a `BenchmarkSetup` config item to prepare its
      counterparty asset.
      - Add percent progress print when running benchmarks.
      - Dont immediately exit on benchmark error but try to run as many as
      possible and print errors last.
      
      ---------
      
      Signed-off-by: default avatarOliver Tale-Yazdi <[email protected]>
      Co-authored-by: default avatarLiam Aharon <[email protected]>
      9543d314
    • Andrei Sandu's avatar
      adder-collator: add velocity measurement and make elastic scaling test more robust (#4016) · 039d183b
      Andrei Sandu authored
      
      
      Improves `adder-collator` to also compute the parachain velocity. The
      velocity is defined as number of parachain blocks progressing per relay
      chain block.
      
      In this test we're asserting that the elastic parachain always
      progresses by 3 blocks per RCB, while the non-elastic parachain
      progresses normally - 1 block per RCB.
      
      ---------
      
      Signed-off-by: default avatarAndrei Sandu <[email protected]>
      Co-authored-by: default avatarordian <[email protected]>
      039d183b
    • Tsvetomir Dimitrov's avatar
      Deprecate `para_id()` from `CoreState` in polkadot primitives (#3979) · 59f868d1
      Tsvetomir Dimitrov authored
      With Coretime enabled we can no longer assume there is a static 1:1
      mapping between core index and para id. This mapping should be obtained
      from the scheduler/claimqueue on block by block basis.
      
      This PR modifies `para_id()` (from `CoreState`) to return the scheduled
      `ParaId` for occupied cores and removes its usages in the code.
      
      Closes https://github.com/paritytech/polkadot-sdk/issues/3948
      
      
      
      ---------
      
      Co-authored-by: default avatarAndrei Sandu <[email protected]>
      59f868d1
    • HongKuang's avatar
      Fix some typos (#4018) · bd4471b4
      HongKuang authored
      
      
      Signed-off-by: default avatarhongkuang <[email protected]>
      bd4471b4
  9. Apr 05, 2024
  10. Apr 04, 2024
  11. Apr 03, 2024
  12. Apr 02, 2024
    • Dastan's avatar
      migrations: prevent accidentally using unversioned migrations instead of... · e5427969
      Dastan authored
      migrations: prevent accidentally using unversioned migrations instead of `VersionedMigration` (#3835)
      
      closes #1324 
      
      #### Problem
      Currently, it is possible to accidentally use inner unversioned
      migration instead of `VersionedMigration` since both implement
      `OnRuntimeUpgrade`.
      
      #### Solution
      
      With this change, we make it clear that value of `Inner` is not intended
      to be used directly. It is achieved by bounding `Inner` to new trait
      `UncheckedOnRuntimeUpgrade`, which has the same interface (except
      `unchecked_` prefix) as `OnRuntimeUpgrade`.
      
      #### `try-runtime` functions
      
      Since developers can implement `try-runtime` for `Inner` value in
      `VersionedMigration` and have custom logic for it, I added the same
      `try-runtime` functions to `UncheckedOnRuntimeUpgrade`. I looked for a
      ways to not duplicate functions, but couldn't find anything that doesn't
      significantly change the codebase. So I would appreciate If you have any
      suggestions to improve this
      
      cc @liamaharon
      
       @xlc 
      
      polkadot address: 16FqwPZ8GRC5U5D4Fu7W33nA55ZXzXGWHwmbnE1eT6pxuqcT
      
      ---------
      
      Co-authored-by: default avatarLiam Aharon <[email protected]>
      e5427969
    • Serban Iorga's avatar
      Align dependencies with `parity-bridges-common` (#3937) · 8e95a3e1
      Serban Iorga authored
      Working towards migrating the `parity-bridges-common` repo inside
      `polkadot-sdk`. This PR upgrades some dependencies in order to align
      them with the versions used in `parity-bridges-common`
      
      Related to
      https://github.com/paritytech/parity-bridges-common/issues/2538
      8e95a3e1
    • Adrian Catangiu's avatar
      beefy: error logs for validators with dummy keys (#3939) · 5eff3f94
      Adrian Catangiu authored
      This outputs:
      ```
      2024-04-02 14:36:02.135 ERROR tokio-runtime-worker beefy: 🥩 for session starting at block 21990151
      no BEEFY authority key found in store, you must generate valid session keys
      (https://wiki.polkadot.network/docs/maintain-guides-how-to-validate-polkadot#generating-the-session-keys
      
      )
      ```
      error log entry, once every session, for nodes running with
      `Role::Authority` that have no public BEEFY key in their keystore
      
      ---------
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      5eff3f94
    • Bastian Köcher's avatar
      Fix parachain upgrade scheduling when done by the owner/root (#3341) · 12eb285d
      Bastian Köcher authored
      When using `schedule_code_upgrade` to change the code of a parachain in
      the relay chain runtime, we had already fixed to not set the `GoAhead`
      signal. This was done to not brick any parachain after the upgrade,
      because they were seeing the signal without having any upgrade prepared.
      The remaining problem is that the parachain code is only upgraded after
      a parachain header was enacted, aka the parachain made some progress.
      However, this is quite complicated if the parachain is bricked (which is
      the most common scenario why to manually schedule a code upgrade). Thus,
      this pull request replaces `SetGoAhead` with `UpgradeStrategy` to signal
      to the logic kind of strategy want to use. The strategies are either
      `SetGoAheadSignal` or `ApplyAtExpectedBlock`. `SetGoAheadSignal` sets
      the go ahead signal as before and awaits a parachain block.
      `ApplyAtExpectedBlock` schedules the upgrade and applies it directly at
      the `expected_block` without waiting for the parachain to make any kind
      of progress.
      12eb285d
    • Adrian Catangiu's avatar
      pallet-xcm: fix weights for all XTs and deprecate unlimited weight ones (#3927) · d0ebb850
      Adrian Catangiu authored
      
      
      Fix "double-weights" for extrinsics, use only the ones benchmarked in
      the runtime.
      
      Deprecate extrinsics that don't specify WeightLimit, remove their usage
      across the repo.
      
      ---------
      
      Signed-off-by: default avatarAdrian Catangiu <[email protected]>
      Co-authored-by: command-bot <>
      d0ebb850
  13. Apr 01, 2024