Makes https://github.com/paritytech/polkadot-sdk/pull/4035 easier to
review

Currently, PVFs are re-prepared if any execution environment parameter
changes. As we've recently seen on Kusama and Polkadot, that may lead to
a severe finality lag because every validator has to re-prepare every
PVF. That cannot be avoided altogether; however, we could cease
re-preparing PVFs when a change in the execution environment can't lead
to a change in the artifact itself. For example, it's clear that
changing the execution timeout cannot affect the artifact.

In this PR, I'm introducing a separate hash for the subset of execution
environment parameters that changes only if a preparation-related
parameter changes. It introduces some minor code duplication, but
without that, the scope of changes would be much bigger.

TODO:
- [x] Add a test to ensure the artifact is not re-prepared if
non-preparation-related parameter is changed
- [x] Add a test to ensure the artifact is re-prepared if a
preparation-related parameter is changed
- [x] Add comments, warnings, and, possibly, a test to ensure a new
parameter ever added to the executor environment parameters will be
evaluated by the author of changes with respect to its artifact
preparation impact and added to the new hash preimage if needed.

Closes #4132

... a few sessions too late :(, this already happened on polkadot, so as
of now there are no known relay-chains without async backing enabled in
runtime, but let's fix it in case someone else wants to repeat our
steps.

Fixes: https://github.com/paritytech/polkadot-sdk/issues/4226

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>

Changes:
- Add new error variant `ProcessMessageError::StackLimitReached` and
treat XCM error `ExceedsStackLimit` as such.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Branislav Kontur <[email protected]>

Revert "pallet-xcm: Deprecate `execute` and `send` in favor of
`execute_blob` and `send_blob` (#3749)"

This reverts commit feee773d

.

---------

Co-authored-by: Adrian Catangiu <[email protected]>
Co-authored-by: Javier Bullrich <[email protected]>

This PR:
- introduces `AllowHrmpNotificationsFromRelayChain` barrier for allowing
HRMP notifications just from the relay chain (to fulfill safety
assumptions -
[see](https://github.com/paritytech/polkadot-sdk/blob/master/polkadot/xcm/src/v4/mod.rs#L532))
- sets it up for all testnet SP parachains

Continuation of: https://github.com/paritytech/polkadot-sdk/pull/3696

Change `transfer_assets_using_type()` to not assume `DepositAssets` as
the intended use of the assets on the destination.

Instead provides the caller with the ability to specify custom XCM that
be executed on `dest` chain as the last step of the transfer, thus
allowing custom usecases for the transferred assets. E.g. some are
used/swapped/etc there, while some are sent further to yet another
chain.

Note: this is a follow-up on
https://github.com/paritytech/polkadot-sdk/pull/3695, bringing in an API
change for `transfer_assets_using_type()`. This is ok as the previous
version has not been yet released. Thus, its first release will include
the new API proposed by this PR.

This allows usecases such as:
https://forum.polkadot.network/t/managing-sas-on-multiple-reserve-chains-for-same-asset/7538/4

BTW: all this pallet-xcm asset transfers code will be massively reduced
once we have https://github.com/paritytech/xcm-format/pull/54

---------

Signed-off-by: Adrian Catangiu <[email protected]>

Part of https://github.com/paritytech/polkadot-sdk/issues/4126 we want
to safely increase the execute_workers_max_num gradually from chain to
chain and assess if there are any negative impacts.

This PR performs the necessary plumbing to be able to increase it based
on the chain id, it increase the number of execution workers from 2 to 4
on test network but lives kusama and polkadot unchanged until we gather
more data.

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>

Ensure outbound XCMs are decodable with limits + add `EnsureDecodableXcm` router (for testing purposes) (#4186)

This PR:
- adds `EnsureDecodableXcm` (testing) router that attempts to *encode*
and *decode* passed XCM `message` to ensure that the receiving side will
be able to decode, at least with the same XCM version.
- fixes `pallet_xcm` / `pallet_xcm_benchmarks` assets data generation

Relates to investigation of
https://substrate.stackexchange.com/questions/11288 and missing fix
https://github.com/paritytech/polkadot-sdk/pull/2129 which did not get
into the fellows 1.1.X release.

## Questions/TODOs

- [x] fix XCM benchmarks, which produces undecodable data - new router
catched at least two cases
  - `BoundedVec exceeds its limit`
  - `Fungible asset of zero amount is not allowed`  
- [x] do we need to add `sort` to the `prepend_with` as we did for
reanchor [here](https://github.com/paritytech/polkadot-sdk/pull/2129)?
@serban300 (**created separate/follow-up PR**:
https://github.com/paritytech/polkadot-sdk/pull/4235)
- [x] We added decoding check to `XcmpQueue` -> `validate_xcm_nesting`,
why not to added to the `ParentAsUmp` or `ChildParachainRouter`?
@franciscoaguirre

 (**created separate/follow-up PR**:
https://github.com/paritytech/polkadot-sdk/pull/4236)
- [ ] `SendController::send_blob` replace `VersionedXcm::<()>::decode(`
with `VersionedXcm::<()>::decode_with_depth_limit(MAX_XCM_DECODE_DEPTH,
data)` ?

---------

Co-authored-by: Adrian Catangiu <[email protected]>

This is for adding onfinality polkadot bootnode. Please correct me if
this is not the right place for adding a new bootnode

This should be configured on the runtime level and not somewhere inside
the pallet.

---------

Co-authored-by: Adrian Catangiu <[email protected]>
Co-authored-by: Branislav Kontur <[email protected]>

This PR:
- moves `validate_xcm_nesting` from `XcmpQueue` into the `VersionedXcm`
- adds `validate_xcm_nesting` to the `ParentAsUmp`
- adds `validate_xcm_nesting` to the `ChildParachainRouter`


Based on discussion
[here](https://github.com/paritytech/polkadot-sdk/pull/4186#discussion_r1571344270)
and/or
[here](https://github.com/paritytech/polkadot-sdk/pull/4186#discussion_r1572076666)
and/or [here]()

## Question/TODO

- [x] To the
[comment](https://github.com/paritytech/polkadot-sdk/pull/4186#discussion_r1572072295)
- Why was `validate_xcm_nesting` added just to the `XcmpQueue` router
and nowhere else? What kind of problem `MAX_XCM_DECODE_DEPTH` is
solving? (see
[comment](https://github.com/paritytech/polkadot-sdk/pull/4236#discussion_r1574605191))

Add a metric to be able to understand the time jobs are waiting in the
execution queue waiting for an available worker.
https://github.com/paritytech/polkadot-sdk/issues/4126

Signed-off-by: Alexandru Gheorghe <[email protected]>

Basically combines all the recommended calls into one
`build_using_defaults()` call or `init_with_defaults()` when there are
some custom changes required.

Co-authored-by: alvicsam <[email protected]>

This PR:
- sanitizes all `UniversalLocation`s with `GlobalConsensus` (when
possible) - addressing
[comment](https://github.com/paritytech/polkadot-sdk/pull/4025#discussion_r1557361473)
- adds `DefaultConfig` for `pallet-xcm-benchmarks` for `system`

Adds sorting to the XCM Assets' `prepend_with`, which could modify the
order of `AssetId` locations.

Relates to: https://github.com/paritytech/polkadot-sdk/pull/4186 (the
same fix for `reanchored`)
Part of: https://github.com/paritytech/polkadot-sdk/pull/2129

Co-authored-by: Bastian Köcher <[email protected]>

follow up of https://github.com/paritytech/polkadot-sdk/pull/2604
closes https://github.com/paritytech/polkadot-sdk/pull/2604

- [x] take relevant changes from Marcin's PR 
- [x] extract common duplicate code for workers (low-hanging fruits)

~Some unpassed ci problems are more general and should be fixed in
master (see https://github.com/paritytech/polkadot-sdk/pull/4074)~

Proposed labels: **T0-node**, **R0-silent**, **I4-refactor**

-----

kusama address: FZXVQLqLbFV2otNXs6BMnNch54CFJ1idpWwjMb3Z8fTLQC6

---------

Co-authored-by: s0me0ne-unkn0wn <[email protected]>

Related to https://github.com/paritytech/polkadot-sdk/issues/4126
discussion

Currently all preparations have same priority and this is not ideal in
all cases. This change should improve the finality time in the context
of on-demand parachains and when `ExecutorParams` are updated on-chain
and a rebuild of all artifacts is required. The desired effect is to
speed up approval and dispute PVF executions which require preparation
and delay backing executions which require preparation.

---------

Signed-off-by: Andrei Sandu <[email protected]>

- Returned latency (with it, results are more stable)
- The threshold is weakened
- Increased number of runs

approval-voting: Make sure we always mark approved candidates approved in a different relay chain context (#4153)

... see for more detail why this is needed

https://github.com/paritytech/polkadot-sdk/issues/4149#issuecomment-2058472444

## TODO:
- [x] Unittests
- [x] Replicate scenario from
https://github.com/paritytech/polkadot-sdk/issues/4149 and confirm this
fixes it: https://github.com/paritytech/polkadot-sdk/issues/4149 [
Replicated on a zombienet with some hacked nodes, that we can end up in
this state where no-wake is schedule and the nodes are pending new
assignments]

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>

The `next_retry_time` gets populated when a request receives an error
timeout or any other error, after thatn next_retry would check all
requests in the queue returns the smallest one, which then gets used to
move the main loop by creating a Delay
```
futures_timer::Delay::new(instant.saturating_duration_since(Instant::now())).await,
```

However when we retry a task for the first time we still keep it in the
queue an mark it as in flight so its next_retry_time would be the oldest
and it would be small than `now`, so the Delay will always triggers, so
that would make the main loop essentially busy wait untill we received a
response for the retry request.

Fix this by excluding the tasks that are already in-flight.

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>

Block reversion of the current block is technically possible as can be
seen from

https://github.com/paritytech/polkadot-sdk/blob/39b1f50f/polkadot/runtime/parachains/src/disputes.rs#L1215-L1223

- [x] Fix the test

cc https://github.com/paritytech/ci_cd/issues/974

---------

Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <[email protected]>

# Description
- Link to issue: https://github.com/paritytech/polkadot-sdk/issues/3651

polkadot address: 19nSqFQorfF2HxD3oBzWM3oCh4SaCRKWt1yvmgaPYGCo71J

Preparation for https://github.com/paritytech/polkadot-sdk/pull/3935

Changes:
- Add some `default-features = false` for the case that a crate and that
dependency both support nostd builds.
- Shuffle files around of some benchmarking-only crates. These
conditionally disabled the `cfg_attr` for nostd and pulled in libstd.
Example [here](https://github.com/ggwpez/zepter/pull/95). The actual
logic is moved into a `inner.rs` to preserve nostd capability of the
crate in case the benchmarking feature is disabled.
- Add some `use sp_std::vec` where needed.
- Remove some `optional = true` in cases where it was not optional.
- Removed one superfluous `cfg_attr(not(feature = "std"), no_std..`.

All in all this should be logical no-op.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

This PR introduces changes enabling the transfer of coretime regions via
XCM.

TL;DR: There are two primary issues that are resolved in this PR:

1. The `mint` and `burn` functions were not implemented for coretime
regions. These operations are essential for moving assets to and from
the XCM holding register.
2. The transfer of non-fungible assets through XCM was previously
disallowed. This was due to incorrectly benchmarking non-fungible asset
transfers via XCM, which led to assigning it a weight of `Weight::Max`,
effectively preventing its execution.

### `mint_into` and `burn` implementation

This PR addresses the issue with cross-chain transferring regions back
to the Coretime chain. Remote reserve transfers are performed by
withdrawing and depositing the asset to and from the holding registry.
This requires the asset to support burning and minting functionality.

This PR adds burning and minting; however, they work a bit differently
than usual so that the associated region record is not lost when
burning. Instead of removing all the data, burning will set the owner of
the region to `None`, and when minting it back, it will set it to an
actual value. So, when cross-chain transferring, withdrawing into the
registry will remove the region from its original owner, and when
depositing it from the registry, it will set its owner to another
account

This was originally implemented in this PR: #3455, however we decided to
move all of it to this single PR
(https://github.com/paritytech/polkadot-sdk/pull/3455#discussion_r1547324892)

### Fixes made in this PR

- Update the `XcmReserveTransferFilter` on coretime chain since it is
meant as a reserve chain for coretime regions.
- Update the XCM benchmark to use `AssetTransactor` instead of assuming
`pallet-balances` for fungible transfers.
- Update the XCM benchmark to properly measure weight consumption for
nonfungible reserve asset transfers. ATM reserve transfers via the
extrinsic do not work since the weight for it is set to `Weight::max()`.

Closes: https://github.com/paritytech/polkadot-sdk/issues/865

---------

Co-authored-by: Branislav Kontur <[email protected]>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Dónal Murray <[email protected]>

Introduce types to define 1:1 balance conversion for different relative
asset ids/locations of native asset.

Examples:
native asset on Asset Hub presented as `VersionedLocatableAsset` type in
the context of Relay Chain is
```
{
  `location`: (0, Parachain(1000)),
  `asset_id`: (1, Here),
}
```
and it's balance should be converted 1:1 by implementations of
`ConversionToAssetBalance` trait.

---------

Co-authored-by: Branislav Kontur <[email protected]>

Will make https://github.com/paritytech/polkadot-sdk/pull/4035 easier to
review (the mentioned PR already does this move so the diff will be
clearer).

Also called out as part of:
https://github.com/paritytech/polkadot-sdk/pull/3233#discussion_r1490867383

As discovered during investigation of
https://github.com/paritytech/polkadot-sdk/issues/3314 and
https://github.com/paritytech/polkadot-sdk/issues/3673 there are active
validators which accidentally might change their network key during
restart, that's not a safe operation when you are in the active set
because of distributed nature of DHT, so the old records would still
exist in the network until they expire 36h, so unless they have a good
reason validators should avoid changing their key when they restart
their nodes.

There is an effort in parallel to improve this situation
https://github.com/paritytech/polkadot-sdk/pull/3786, but those changes
are way more intrusive and will need more rigorous testing, additionally
they will reduce the time to less than 36h, but the propagation won't be
instant anyway, so not changing your network during restart should be
the safest way to run your node, unless you have a really good reason to
change it.

## Proposal
1. Do not auto-generate the network if the network file does not exist
in the provided path. Nodes where the key file does not exist will get
the following error:
```
Error: 
   0: Starting an authorithy without network key in /home/alexggh/.local/share/polkadot/chains/ksmcc3/network/secret_ed25519.
      
       This is not a safe operation because the old identity still lives in the dht for 36 hours.
      
       Because of it your node might suffer from not being properly connected to other nodes for validation purposes.
      
       If it is the first time running your node you could use one of the following methods.
      
       1. Pass --unsafe-force-node-key-generation and make sure you remove it for subsequent node restarts
      
       2. Separetly generate the key with: polkadot key generate-node-key --file <YOUR_PATH_TO_NODE_KEY>
```

2. Add an explicit parameters for nodes that do want to change their
network despite the warnings or if they run the node for the first time.
`--unsafe-force-node-key-generation`

3. For `polkadot key generate-node-key` add two new mutually exclusive
parameters `base_path` and `default_base_path` to help with the key
generation in the same path the polkadot main command would expect it.
 
4. Modify the installation scripts to auto-generate a key in default
path if one was not present already there, this should help with making
the executable work out of the box after an instalation.

## Notes

Nodes that do not have already the key persisted will fail to start
after this change, however I do consider that better than the current
situation where they start but they silently hide that they might not be
properly connected to their peers.

## TODO
- [x] Make sure only nodes that are authorities on producation chains
will be affected by this restrictions.
- [x] Proper PRDOC, to make sure node operators are aware this is
coming.

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>
Co-authored-by: Dmitry Markin <[email protected]>
Co-authored-by: s0me0ne-unkn0wn <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Verified by running a node using `--reserved-only` and
`--reserved-nodes`.

Changes:
- Add pallet-parameters to Rococo to configure the NIS and preimage
pallet.
- Fix names of expanded dynamic params. Apparently, `to_class_case`
removes suffix `s`, and `Nis` becomes `Ni` 😑

. Now using
`to_pascal_case`.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Alessandro Siniscalchi <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: command-bot <>

This PR introduces `BlockHashProvider` into `pallet_mmr::Config`
This type is used to get `block_hash` for a given `block_number` rather
than directly using `frame_system::Pallet::block_hash`

The `DefaultBlockHashProvider` uses `frame_system::Pallet::block_hash`
to get the `block_hash`

Closes: #4062

This PR mainly removes `xcm::v3` stuff from `assets-common` to make it
more generic and facilitate the transition to newer XCM versions. Some
of the implementations here used hard-coded `xcm::v3::Location`, but now
it's up to the runtime to configure according to its needs.

Additional/consequent changes:
- `penpal` runtime uses now `xcm::latest::Location` for `pallet_assets`
as `AssetId`, because we don't care about migrations here
- it pretty much simplify xcm-emulator integration tests, where we don't
need now a lots of boilerplate conversions:
      ```
      v3::Location::try_from(...).expect("conversion works")`
      ```
- xcm-emulator tests
- split macro `impl_assets_helpers_for_parachain` to the
`impl_assets_helpers_for_parachain` and
`impl_foreign_assets_helpers_for_parachain` (avoids using hard-coded
`xcm::v3::Location`)

# Description

Add `transfer_assets_using()` for transferring assets from local chain
to destination chain using explicit XCM transfer types such as:
- `TransferType::LocalReserve`: transfer assets to sovereign account of
destination chain and forward a notification XCM to `dest` to mint and
deposit reserve-based assets to `beneficiary`.
- `TransferType::DestinationReserve`: burn local assets and forward a
notification to `dest` chain to withdraw the reserve assets from this
chain's sovereign account and deposit them to `beneficiary`.
- `TransferType::RemoteReserve(reserve)`: burn local assets, forward XCM
to `reserve` chain to move reserves from this chain's SA to `dest`
chain's SA, and forward another XCM to `dest` to mint and deposit
reserve-based assets to `beneficiary`. Typically the remote `reserve` is
Asset Hub.
- `TransferType::Teleport`: burn local assets and forward XCM to `dest`
chain to mint/teleport assets and deposit them to `beneficiary`.

By default, an asset's reserve is its origin chain. But sometimes we may
want to explicitly use another chain as reserve (as long as allowed by
runtime `IsReserve` filter).

This is very helpful for transferring assets with multiple configured
reserves (such as Asset Hub ForeignAssets), when the transfer strictly
depends on the used reserve.

E.g. For transferring Foreign Assets over a bridge, Asset Hub must be
used as the reserve location.

# Example usage scenarios

## Transfer bridged ethereum ERC20-tokenX between ecosystem parachains.

ERC20-tokenX is registered on AssetHub as a ForeignAsset by the
Polkadot<>Ethereum bridge (Snowbridge). Its asset_id is something like
`(parents:2, (GlobalConsensus(Ethereum), Address(tokenX_contract)))`.
Its _original_ reserve is Ethereum (only we can't use Ethereum as a
reserve in local transfers); but, since tokenX is also registered on
AssetHub as a ForeignAsset, we can use AssetHub as a reserve.

With this PR we can transfer tokenX from ParaA to ParaB while using
AssetHub as a reserve.

## Transfer AssetHub ForeignAssets between parachains

AssetA created on ParaA but also registered as foreign asset on Asset
Hub. Can use AssetHub as a reserve.

And all of the above can be done while still controlling transfer type
for `fees` so mixing assets in same transfer is supported.

# Tests

Added integration tests for showcasing:
- transferring local (not bridged) assets from parachain over bridge
using local Asset Hub reserve,
- transferring foreign assets from parachain to Asset Hub,
- transferring foreign assets from Asset Hub to parachain,
- transferring foreign assets from parachain to parachain using local
Asset Hub reserve.

---------

Co-authored-by: Branislav Kontur <[email protected]>
Co-authored-by: command-bot <>

Fixes https://github.com/paritytech/polkadot-sdk/issues/3576

Required by elastic scaling collators.
Deprecates old API: `candidate_pending_availability`.

TODO:
- [x] PRDoc

---------

Signed-off-by: Andrei Sandu <[email protected]>