Co-authored-by: Bastian Köcher <[email protected]>

follow up of https://github.com/paritytech/polkadot-sdk/pull/2604
closes https://github.com/paritytech/polkadot-sdk/pull/2604

- [x] take relevant changes from Marcin's PR 
- [x] extract common duplicate code for workers (low-hanging fruits)

~Some unpassed ci problems are more general and should be fixed in
master (see https://github.com/paritytech/polkadot-sdk/pull/4074)~

Proposed labels: **T0-node**, **R0-silent**, **I4-refactor**

-----

kusama address: FZXVQLqLbFV2otNXs6BMnNch54CFJ1idpWwjMb3Z8fTLQC6

---------

Co-authored-by: s0me0ne-unkn0wn <[email protected]>

Related to https://github.com/paritytech/polkadot-sdk/issues/4126
discussion

Currently all preparations have same priority and this is not ideal in
all cases. This change should improve the finality time in the context
of on-demand parachains and when `ExecutorParams` are updated on-chain
and a rebuild of all artifacts is required. The desired effect is to
speed up approval and dispute PVF executions which require preparation
and delay backing executions which require preparation.

---------

Signed-off-by: Andrei Sandu <[email protected]>

- Returned latency (with it, results are more stable)
- The threshold is weakened
- Increased number of runs

approval-voting: Make sure we always mark approved candidates approved in a different relay chain context (#4153)

... see for more detail why this is needed

https://github.com/paritytech/polkadot-sdk/issues/4149#issuecomment-2058472444

## TODO:
- [x] Unittests
- [x] Replicate scenario from
https://github.com/paritytech/polkadot-sdk/issues/4149 and confirm this
fixes it: https://github.com/paritytech/polkadot-sdk/issues/4149 [
Replicated on a zombienet with some hacked nodes, that we can end up in
this state where no-wake is schedule and the nodes are pending new
assignments]

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>

The `next_retry_time` gets populated when a request receives an error
timeout or any other error, after thatn next_retry would check all
requests in the queue returns the smallest one, which then gets used to
move the main loop by creating a Delay
```
futures_timer::Delay::new(instant.saturating_duration_since(Instant::now())).await,
```

However when we retry a task for the first time we still keep it in the
queue an mark it as in flight so its next_retry_time would be the oldest
and it would be small than `now`, so the Delay will always triggers, so
that would make the main loop essentially busy wait untill we received a
response for the retry request.

Fix this by excluding the tasks that are already in-flight.

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>

Block reversion of the current block is technically possible as can be
seen from

https://github.com/paritytech/polkadot-sdk/blob/39b1f50f/polkadot/runtime/parachains/src/disputes.rs#L1215-L1223

- [x] Fix the test

cc https://github.com/paritytech/ci_cd/issues/974

---------

Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <[email protected]>

# Description
- Link to issue: https://github.com/paritytech/polkadot-sdk/issues/3651

polkadot address: 19nSqFQorfF2HxD3oBzWM3oCh4SaCRKWt1yvmgaPYGCo71J

Preparation for https://github.com/paritytech/polkadot-sdk/pull/3935

Changes:
- Add some `default-features = false` for the case that a crate and that
dependency both support nostd builds.
- Shuffle files around of some benchmarking-only crates. These
conditionally disabled the `cfg_attr` for nostd and pulled in libstd.
Example [here](https://github.com/ggwpez/zepter/pull/95). The actual
logic is moved into a `inner.rs` to preserve nostd capability of the
crate in case the benchmarking feature is disabled.
- Add some `use sp_std::vec` where needed.
- Remove some `optional = true` in cases where it was not optional.
- Removed one superfluous `cfg_attr(not(feature = "std"), no_std..`.

All in all this should be logical no-op.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

This PR introduces changes enabling the transfer of coretime regions via
XCM.

TL;DR: There are two primary issues that are resolved in this PR:

1. The `mint` and `burn` functions were not implemented for coretime
regions. These operations are essential for moving assets to and from
the XCM holding register.
2. The transfer of non-fungible assets through XCM was previously
disallowed. This was due to incorrectly benchmarking non-fungible asset
transfers via XCM, which led to assigning it a weight of `Weight::Max`,
effectively preventing its execution.

### `mint_into` and `burn` implementation

This PR addresses the issue with cross-chain transferring regions back
to the Coretime chain. Remote reserve transfers are performed by
withdrawing and depositing the asset to and from the holding registry.
This requires the asset to support burning and minting functionality.

This PR adds burning and minting; however, they work a bit differently
than usual so that the associated region record is not lost when
burning. Instead of removing all the data, burning will set the owner of
the region to `None`, and when minting it back, it will set it to an
actual value. So, when cross-chain transferring, withdrawing into the
registry will remove the region from its original owner, and when
depositing it from the registry, it will set its owner to another
account

This was originally implemented in this PR: #3455, however we decided to
move all of it to this single PR
(https://github.com/paritytech/polkadot-sdk/pull/3455#discussion_r1547324892)

### Fixes made in this PR

- Update the `XcmReserveTransferFilter` on coretime chain since it is
meant as a reserve chain for coretime regions.
- Update the XCM benchmark to use `AssetTransactor` instead of assuming
`pallet-balances` for fungible transfers.
- Update the XCM benchmark to properly measure weight consumption for
nonfungible reserve asset transfers. ATM reserve transfers via the
extrinsic do not work since the weight for it is set to `Weight::max()`.

Closes: https://github.com/paritytech/polkadot-sdk/issues/865

---------

Co-authored-by: Branislav Kontur <[email protected]>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Dónal Murray <[email protected]>

Introduce types to define 1:1 balance conversion for different relative
asset ids/locations of native asset.

Examples:
native asset on Asset Hub presented as `VersionedLocatableAsset` type in
the context of Relay Chain is
```
{
  `location`: (0, Parachain(1000)),
  `asset_id`: (1, Here),
}
```
and it's balance should be converted 1:1 by implementations of
`ConversionToAssetBalance` trait.

---------

Co-authored-by: Branislav Kontur <[email protected]>

Will make https://github.com/paritytech/polkadot-sdk/pull/4035 easier to
review (the mentioned PR already does this move so the diff will be
clearer).

Also called out as part of:
https://github.com/paritytech/polkadot-sdk/pull/3233#discussion_r1490867383

As discovered during investigation of
https://github.com/paritytech/polkadot-sdk/issues/3314 and
https://github.com/paritytech/polkadot-sdk/issues/3673 there are active
validators which accidentally might change their network key during
restart, that's not a safe operation when you are in the active set
because of distributed nature of DHT, so the old records would still
exist in the network until they expire 36h, so unless they have a good
reason validators should avoid changing their key when they restart
their nodes.

There is an effort in parallel to improve this situation
https://github.com/paritytech/polkadot-sdk/pull/3786, but those changes
are way more intrusive and will need more rigorous testing, additionally
they will reduce the time to less than 36h, but the propagation won't be
instant anyway, so not changing your network during restart should be
the safest way to run your node, unless you have a really good reason to
change it.

## Proposal
1. Do not auto-generate the network if the network file does not exist
in the provided path. Nodes where the key file does not exist will get
the following error:
```
Error: 
   0: Starting an authorithy without network key in /home/alexggh/.local/share/polkadot/chains/ksmcc3/network/secret_ed25519.
      
       This is not a safe operation because the old identity still lives in the dht for 36 hours.
      
       Because of it your node might suffer from not being properly connected to other nodes for validation purposes.
      
       If it is the first time running your node you could use one of the following methods.
      
       1. Pass --unsafe-force-node-key-generation and make sure you remove it for subsequent node restarts
      
       2. Separetly generate the key with: polkadot key generate-node-key --file <YOUR_PATH_TO_NODE_KEY>
```

2. Add an explicit parameters for nodes that do want to change their
network despite the warnings or if they run the node for the first time.
`--unsafe-force-node-key-generation`

3. For `polkadot key generate-node-key` add two new mutually exclusive
parameters `base_path` and `default_base_path` to help with the key
generation in the same path the polkadot main command would expect it.
 
4. Modify the installation scripts to auto-generate a key in default
path if one was not present already there, this should help with making
the executable work out of the box after an instalation.

## Notes

Nodes that do not have already the key persisted will fail to start
after this change, however I do consider that better than the current
situation where they start but they silently hide that they might not be
properly connected to their peers.

## TODO
- [x] Make sure only nodes that are authorities on producation chains
will be affected by this restrictions.
- [x] Proper PRDOC, to make sure node operators are aware this is
coming.

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>
Co-authored-by: Dmitry Markin <[email protected]>
Co-authored-by: s0me0ne-unkn0wn <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Verified by running a node using `--reserved-only` and
`--reserved-nodes`.

Changes:
- Add pallet-parameters to Rococo to configure the NIS and preimage
pallet.
- Fix names of expanded dynamic params. Apparently, `to_class_case`
removes suffix `s`, and `Nis` becomes `Ni` 😑

. Now using
`to_pascal_case`.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Alessandro Siniscalchi <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: command-bot <>

This PR introduces `BlockHashProvider` into `pallet_mmr::Config`
This type is used to get `block_hash` for a given `block_number` rather
than directly using `frame_system::Pallet::block_hash`

The `DefaultBlockHashProvider` uses `frame_system::Pallet::block_hash`
to get the `block_hash`

Closes: #4062

This PR mainly removes `xcm::v3` stuff from `assets-common` to make it
more generic and facilitate the transition to newer XCM versions. Some
of the implementations here used hard-coded `xcm::v3::Location`, but now
it's up to the runtime to configure according to its needs.

Additional/consequent changes:
- `penpal` runtime uses now `xcm::latest::Location` for `pallet_assets`
as `AssetId`, because we don't care about migrations here
- it pretty much simplify xcm-emulator integration tests, where we don't
need now a lots of boilerplate conversions:
      ```
      v3::Location::try_from(...).expect("conversion works")`
      ```
- xcm-emulator tests
- split macro `impl_assets_helpers_for_parachain` to the
`impl_assets_helpers_for_parachain` and
`impl_foreign_assets_helpers_for_parachain` (avoids using hard-coded
`xcm::v3::Location`)

# Description

Add `transfer_assets_using()` for transferring assets from local chain
to destination chain using explicit XCM transfer types such as:
- `TransferType::LocalReserve`: transfer assets to sovereign account of
destination chain and forward a notification XCM to `dest` to mint and
deposit reserve-based assets to `beneficiary`.
- `TransferType::DestinationReserve`: burn local assets and forward a
notification to `dest` chain to withdraw the reserve assets from this
chain's sovereign account and deposit them to `beneficiary`.
- `TransferType::RemoteReserve(reserve)`: burn local assets, forward XCM
to `reserve` chain to move reserves from this chain's SA to `dest`
chain's SA, and forward another XCM to `dest` to mint and deposit
reserve-based assets to `beneficiary`. Typically the remote `reserve` is
Asset Hub.
- `TransferType::Teleport`: burn local assets and forward XCM to `dest`
chain to mint/teleport assets and deposit them to `beneficiary`.

By default, an asset's reserve is its origin chain. But sometimes we may
want to explicitly use another chain as reserve (as long as allowed by
runtime `IsReserve` filter).

This is very helpful for transferring assets with multiple configured
reserves (such as Asset Hub ForeignAssets), when the transfer strictly
depends on the used reserve.

E.g. For transferring Foreign Assets over a bridge, Asset Hub must be
used as the reserve location.

# Example usage scenarios

## Transfer bridged ethereum ERC20-tokenX between ecosystem parachains.

ERC20-tokenX is registered on AssetHub as a ForeignAsset by the
Polkadot<>Ethereum bridge (Snowbridge). Its asset_id is something like
`(parents:2, (GlobalConsensus(Ethereum), Address(tokenX_contract)))`.
Its _original_ reserve is Ethereum (only we can't use Ethereum as a
reserve in local transfers); but, since tokenX is also registered on
AssetHub as a ForeignAsset, we can use AssetHub as a reserve.

With this PR we can transfer tokenX from ParaA to ParaB while using
AssetHub as a reserve.

## Transfer AssetHub ForeignAssets between parachains

AssetA created on ParaA but also registered as foreign asset on Asset
Hub. Can use AssetHub as a reserve.

And all of the above can be done while still controlling transfer type
for `fees` so mixing assets in same transfer is supported.

# Tests

Added integration tests for showcasing:
- transferring local (not bridged) assets from parachain over bridge
using local Asset Hub reserve,
- transferring foreign assets from parachain to Asset Hub,
- transferring foreign assets from Asset Hub to parachain,
- transferring foreign assets from parachain to parachain using local
Asset Hub reserve.

---------

Co-authored-by: Branislav Kontur <[email protected]>
Co-authored-by: command-bot <>

Fixes https://github.com/paritytech/polkadot-sdk/issues/3576

Required by elastic scaling collators.
Deprecates old API: `candidate_pending_availability`.

TODO:
- [x] PRDoc

---------

Signed-off-by: Andrei Sandu <[email protected]>

Implements https://github.com/polkadot-fellows/RFCs/issues/82
Allow any parachain to have bidirectional channel with any system
parachains

Looking for initial feedbacks before continue finish it

TODOs:
- [x] docs
- [x] benchmarks
- [x] tests

---------

Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: Adrian Catangiu <[email protected]>

One more try to make this test robust from a resource perspective.

---------

Signed-off-by: Andrei Sandu <[email protected]>
Co-authored-by: Javier Viola <[email protected]>

Completes the removal of `try-runtime-cli` logic from `polkadot-sdk`.

Implements the idea from
https://github.com/paritytech/polkadot-sdk/pull/3899
- Removed latencies
- Number of runs reduced from 50 to 5, according to local runs it's
quite enough
- Network message is always sent in a spawned task, even if latency is
zero. Without it, CPU time sometimes spikes.
- Removed the `testnet` profile because we probably don't need that
debug additions.

After the local tests I can't say that it brings a significant
improvement in the stability of the results. However, I belive it is
worth trying and looking at the results over time.

fixes https://github.com/paritytech/polkadot-sdk/issues/4067

Also add an early bail out for look ahead collator such that we don't
waste time if a CollatorFn is not set.

TODO:
- [x] add test.
- [x] Polkadot System Parachain burn-in.

---------

Signed-off-by: Andrei Sandu <[email protected]>

Same issue but about av-cores was fixed in
https://github.com/paritytech/polkadot-sdk/pull/1403

Signed-off-by: Andrei Sandu <[email protected]>

This tiny PR extends the `on_validated_block_announce` log with the bad
PeerID.
Used to identify if the peerID is malicious by correlating with other
logs (ie peer-set).

While at it, have removed the `\n` from a multiline log, which did not
play well with
[sub-triage-logs](https://github.com/lexnv/sub-triage-logs/tree/master).

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Part of #3326 

@Kianenigma @ggwpez

 

polkadot address: 12poSUQPtcF1HUPQGY3zZu2P8emuW9YnsPduA4XG3oCEfJVp

---------

Signed-off-by: Matteo Muraca <[email protected]>
Co-authored-by: ordian <[email protected]>

This PR backports `spec_version`, `node_version` bumps and reordering of
the prdocs from the 1.10.0 release branch

We are exploring [allowing this for
Kusama](https://github.com/polkadot-fellows/runtimes/pull/261) as well,
disallowing on test chains seems unnecessarily limiting.

closes https://github.com/paritytech/polkadot-sdk/issues/3894

Allows Nomination Pool configuration to be set by a custom origin
instead of root.

In runtimes, we would set this to be `StakingAdmin`, same as for
pallet-staking.

---------

Co-authored-by: Liam Aharon <[email protected]>

[litep2p](https://github.com/altonen/litep2p) is a libp2p-compatible P2P
networking library. It supports all of the features of `rust-libp2p`
that are currently being utilized by Polkadot SDK.

Compared to `rust-libp2p`, `litep2p` has a quite different architecture
which is why the new `litep2p` network backend is only able to use a
little of the existing code in `sc-network`. The design has been mainly
influenced by how we'd wish to structure our networking-related code in
Polkadot SDK: independent higher-levels protocols directly communicating
with the network over links that support bidirectional backpressure. A
good example would be `NotificationHandle`/`RequestResponseHandle`
abstractions which allow, e.g., `SyncingEngine` to directly communicate
with peers to announce/request blocks.

I've tried running `polkadot --network-backend litep2p` with a few
different peer configurations and there is a noticeable reduction in
networking CPU usage. For high load (`...

This MR contains two major changes and some maintenance cleanup.  

## 1. Free Standing Pallet Benchmark Runner

Closes https://github.com/paritytech/polkadot-sdk/issues/3045, depends
on your runtime exposing the `GenesisBuilderApi` (like
https://github.com/paritytech/polkadot-sdk/pull/1492).

Introduces a new binary crate: `frame-omni-bencher`.  
It allows to directly benchmark a WASM blob - without needing a node or
chain spec.

This makes it much easier to generate pallet weights and should allow us
to remove bloaty code from the node.
It should work for all FRAME runtimes that dont use 3rd party host calls
or non `BlakeTwo256` block hashing (basically all polkadot parachains
should work).

It is 100% backwards compatible with the old CLI args, when the `v1`
compatibility command is used. This is done to allow for forwards
compatible addition of new commands.

### Example (full example in the Rust docs)

Installing the CLI:
```sh
cargo install --locked --path subs...

Improves `adder-collator` to also compute the parachain velocity. The
velocity is defined as number of parachain blocks progressing per relay
chain block.

In this test we're asserting that the elastic parachain always
progresses by 3 blocks per RCB, while the non-elastic parachain
progresses normally - 1 block per RCB.

---------

Signed-off-by: Andrei Sandu <[email protected]>
Co-authored-by: ordian <[email protected]>

With Coretime enabled we can no longer assume there is a static 1:1
mapping between core index and para id. This mapping should be obtained
from the scheduler/claimqueue on block by block basis.

This PR modifies `para_id()` (from `CoreState`) to return the scheduled
`ParaId` for occupied cores and removes its usages in the code.

Closes https://github.com/paritytech/polkadot-sdk/issues/3948

---------

Co-authored-by: Andrei Sandu <[email protected]>

Signed-off-by: hongkuang <[email protected]>

Signed-off-by: divdeploy <[email protected]>

Signed-off-by: Andrei Sandu <[email protected]>

On top of https://github.com/paritytech/polkadot-sdk/pull/3879

I've also moved the previous test where we ensure multiple cores per
para doesn't break non elastic parachains.

---------

Signed-off-by: Andrei Sandu <[email protected]>
Co-authored-by: Javier Viola <[email protected]>

The runtime now can provide a number of predefined presets of
`RuntimeGenesisConfig` struct. This presets are intended to be used in
different deployments, e.g.: `local`, `staging`, etc, and should be
included into the corresponding chain-specs.

Having `GenesisConfig` presets in runtime allows to fully decouple node
from runtime types (the problem is described in #1984).

**Summary of changes:**
- The `GenesisBuilder` API was adjusted to enable this functionality
(and provide better naming - #150):
   ```rust
    fn preset_names() -> Vec<PresetId>;
fn get_preset(id: Option<PresetId>) -> Option<serde_json::Value>;
//`None` means default
    fn build_state(value: serde_json::Value);
    pub struct PresetId(Vec<u8>);
   ```

- **Breaking change**: Old `create_default_config` method was removed,
`build_config` was renamed to `build_state`. As a consequence a node
won't be able to interact with genesis config for older runtimes. The
cleanup was made for sake of API simplicity. Also IMO maintaining
compatibility with old API is not so crucial.
- Reference implementation was provided for `substrate-test-runtime` and
`rococo` runtimes. For rococo new
[`genesis_configs_presets`](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b/polkadot/runtime/rococo/src/genesis_config_presets.rs#L530)
module was added and is used in `GenesisBuilder`
[_presets-related_](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b/polkadot/runtime/rococo/src/lib.rs#L2462-L2485)
methods.

- The `chain-spec-builder` util was also improved and allows to
([_doc_](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b/substrate/bin/utils/chain-spec-builder/src/lib.rs#L19)):
   - list presets provided by given runtime (`list-presets`),
- display preset or default config provided by the runtime
(`display-preset`),
   - build chain-spec using named preset (`create ... named-preset`),


- The `ChainSpecBuilder` is extended with
[`with_genesis_config_preset_name`](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b/substrate/client/chain-spec/src/chain_spec.rs#L447)
method which allows to build chain-spec using named preset provided by
the runtime. Sample usage on the node side
[here](https://github.com/paritytech/polkadot-sdk/blob/2caffaae

/polkadot/node/service/src/chain_spec.rs#L404).

Implementation of #1984.
fixes: #150
part of: #25

---------

Co-authored-by: Sebastian Kunert <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>