This workflow will automatically add issues related to async backing to
the Parachain team board, updating a custom "meta" field.

Requested by @the-right-joyce

Closes: https://github.com/paritytech/polkadot-sdk/issues/4100

Fixes https://github.com/paritytech/polkadot-sdk/issues/3999



---------

Co-authored-by: Branislav Kontur <[email protected]>

This PR introduces `BlockHashProvider` into `pallet_mmr::Config`
This type is used to get `block_hash` for a given `block_number` rather
than directly using `frame_system::Pallet::block_hash`

The `DefaultBlockHashProvider` uses `frame_system::Pallet::block_hash`
to get the `block_hash`

Closes: #4062

This PR mainly removes `xcm::v3` stuff from `assets-common` to make it
more generic and facilitate the transition to newer XCM versions. Some
of the implementations here used hard-coded `xcm::v3::Location`, but now
it's up to the runtime to configure according to its needs.

Additional/consequent changes:
- `penpal` runtime uses now `xcm::latest::Location` for `pallet_assets`
as `AssetId`, because we don't care about migrations here
- it pretty much simplify xcm-emulator integration tests, where we don't
need now a lots of boilerplate conversions:
      ```
      v3::Location::try_from(...).expect("conversion works")`
      ```
- xcm-emulator tests
- split macro `impl_assets_helpers_for_parachain` to the
`impl_assets_helpers_for_parachain` and
`impl_foreign_assets_helpers_for_parachain` (avoids using hard-coded
`xcm::v3::Location`)

1. The `CustomFmtContext::ContextWithFormatFields` enum arm isn't
actually used and thus we don't need the enum anymore.

2. We don't do anything with most of the normalized metadata that's
created by calling `event.normalized_metadata();` - the `target` we can
get from `event.metadata.target()` and level we can get from
`event.metadata.level()` - let's just call them direct to simplify
things. (`event.metadata()` is just a field call under the hood)

Changelog: No functional changes, might run a tad faster with lots of
logging turned on.

---------

Co-authored-by: Bastian Köcher <[email protected]>

- Progresses https://github.com/paritytech/polkadot-sdk/issues/3155

### What's inside

A job, that will take each of the three
[templates](https://github.com/paritytech/polkadot-sdk/tree/master/templates),
yank them out of the monorepo workspace, and push to individual
repositories
([1](https://github.com/paritytech/polkadot-sdk-minimal-template),
[2](https://github.com/paritytech/polkadot-sdk-parachain-template),
[3](https://github.com/paritytech/polkadot-sdk-solochain-template)).

In case the build/test does not succeed, a PR such as [this
one](https://github.com/paritytech-stg/polkadot-sdk-solochain-template/pull/2)
gets created instead.

I'm proposing a manual dispatch trigger for now - so we can test and
iterate faster - and change it to fully automatic triggered by releases
later.

The manual trigger looks like this:

<img width="340px"
src="https://github.com/paritytech/polkadot-sdk/assets/12039224/e87e0fda-23a3-4735-9035-af801e8417fc"/>

### How it works

The job replaces dependencies [referenced by
git](https://github.com/paritytech/polkadot-sdk/blob/d733c77ee2d2e8e2d5205c552a5efb2e5b5242c8/templates/minimal/pallets/template/Cargo.toml#L25)
with a reference to released crates using
[psvm](https://github.com/paritytech/psvm).

It creates a new workspace for the template, and adapts what's needed
from the `polkadot-sdk` workspace.

### See the results

The action has been tried out in staging, and the results can be
observed here:

- [minimal
stg](https://github.com/paritytech-stg/polkadot-sdk-minimal-template/)
- [parachain
stg](https://github.com/paritytech-stg/polkadot-sdk-parachain-template/)
- [solochain
stg](https://github.com/paritytech-stg/polkadot-sdk-solochain-template/)

These are based on the `1.9.0` release (using `release-crates-io-v1.9.0`
branch).

# Description

Add `transfer_assets_using()` for transferring assets from local chain
to destination chain using explicit XCM transfer types such as:
- `TransferType::LocalReserve`: transfer assets to sovereign account of
destination chain and forward a notification XCM to `dest` to mint and
deposit reserve-based assets to `beneficiary`.
- `TransferType::DestinationReserve`: burn local assets and forward a
notification to `dest` chain to withdraw the reserve assets from this
chain's sovereign account and deposit them to `beneficiary`.
- `TransferType::RemoteReserve(reserve)`: burn local assets, forward XCM
to `reserve` chain to move reserves from this chain's SA to `dest`
chain's SA, and forward another XCM to `dest` to mint and deposit
reserve-based assets to `beneficiary`. Typically the remote `reserve` is
Asset Hub.
- `TransferType::Teleport`: burn local assets and forward XCM to `dest`
chain to mint/teleport assets and deposit them to `beneficiary`.

By default, an asset's reserve is its origin chain. But sometimes we may
want to explicitly use another chain as reserve (as long as allowed by
runtime `IsReserve` filter).

This is very helpful for transferring assets with multiple configured
reserves (such as Asset Hub ForeignAssets), when the transfer strictly
depends on the used reserve.

E.g. For transferring Foreign Assets over a bridge, Asset Hub must be
used as the reserve location.

# Example usage scenarios

## Transfer bridged ethereum ERC20-tokenX between ecosystem parachains.

ERC20-tokenX is registered on AssetHub as a ForeignAsset by the
Polkadot<>Ethereum bridge (Snowbridge). Its asset_id is something like
`(parents:2, (GlobalConsensus(Ethereum), Address(tokenX_contract)))`.
Its _original_ reserve is Ethereum (only we can't use Ethereum as a
reserve in local transfers); but, since tokenX is also registered on
AssetHub as a ForeignAsset, we can use AssetHub as a reserve.

With this PR we can transfer tokenX from ParaA to ParaB while using
AssetHub as a reserve.

## Transfer AssetHub ForeignAssets between parachains

AssetA created on ParaA but also registered as foreign asset on Asset
Hub. Can use AssetHub as a reserve.

And all of the above can be done while still controlling transfer type
for `fees` so mixing assets in same transfer is supported.

# Tests

Added integration tests for showcasing:
- transferring local (not bridged) assets from parachain over bridge
using local Asset Hub reserve,
- transferring foreign assets from parachain to Asset Hub,
- transferring foreign assets from Asset Hub to parachain,
- transferring foreign assets from parachain to parachain using local
Asset Hub reserve.

---------

Co-authored-by: Branislav Kontur <[email protected]>
Co-authored-by: command-bot <>

Fixes https://github.com/paritytech/polkadot-sdk/issues/3576



Required by elastic scaling collators.
Deprecates old API: `candidate_pending_availability`.

TODO:
- [x] PRDoc

---------

Signed-off-by: Andrei Sandu <[email protected]>

As requested in
https://github.com/paritytech/parity-bridges-common/pull/2873#discussion_r1558974215

Small adjustments which should make understanding what is going on much
easier for future readers.

Initialization is a bit messy, the very least we should do is adding
documentation to make it harder to use wrongly.

I was thinking about calling `request_core_count` right from
`start_sales`, but as explained in the docs, this is not necessarily
what you want.

---------

Co-authored-by: eskimor <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Dónal Murray <[email protected]>

Implements https://github.com/polkadot-fellows/RFCs/issues/82


Allow any parachain to have bidirectional channel with any system
parachains

Looking for initial feedbacks before continue finish it

TODOs:
- [x] docs
- [x] benchmarks
- [x] tests

---------

Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: Adrian Catangiu <[email protected]>

One more try to make this test robust from a resource perspective.

---------

Signed-off-by: Andrei Sandu <[email protected]>
Co-authored-by: Javier Viola <[email protected]>

Completes the removal of `try-runtime-cli` logic from `polkadot-sdk`.

Implements the idea from
https://github.com/paritytech/polkadot-sdk/pull/3899
- Removed latencies
- Number of runs reduced from 50 to 5, according to local runs it's
quite enough
- Network message is always sent in a spawned task, even if latency is
zero. Without it, CPU time sometimes spikes.
- Removed the `testnet` profile because we probably don't need that
debug additions.

After the local tests I can't say that it brings a significant
improvement in the stability of the results. However, I belive it is
worth trying and looking at the results over time.

Currently `subsystem-regression-tests` job fails if the first benchmarks
fail and there is no result for the second benchmark. Also dividing the
job makes the pipeline faster (currently it's a longest job)

cc https://github.com/paritytech/ci_cd/issues/969
cc @AndreiEres



---------

Co-authored-by: Andrei Eres <[email protected]>

This is phase 2 of async backing enablement for the system parachains on
the production networks. ~~It should be merged after
polkadot-fellows/runtimes#228 is enacted. After it is released,~~ all
the system parachain collators should be upgraded, and then we can
proceed with phase 3, which will enable async backing in the runtimes.

UPDATE: Indeed, we don't need to wait for the runtime upgrade enactions.
The lookahead collator handles the transition by itself, so we can
upgrade ASAP.

## Scope of changes

Here, we eliminate the dichotomy of having "generic Aura collators" for
the production system parachains and "lookahead Aura collators" for the
testnet system parachains. Now, all the collators are started as
lookahead ones, preserving the logic of transferring from the shell node
to Aura-enabled collators for the asset hubs. So, indeed, it simplifies
the parachain service logic, which cannot but rejoice.

Closes #4041

Changes:
- Increase cache size and reduce retries.
- Ignore Substrate SE links :(
- Fix broken link.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

fixes https://github.com/paritytech/polkadot-sdk/issues/4067



Also add an early bail out for look ahead collator such that we don't
waste time if a CollatorFn is not set.

TODO:
- [x] add test.
- [x] Polkadot System Parachain burn-in.

---------

Signed-off-by: Andrei Sandu <[email protected]>

Same issue but about av-cores was fixed in
https://github.com/paritytech/polkadot-sdk/pull/1403



Signed-off-by: Andrei Sandu <[email protected]>

- Update internal logic so that the storage_base_deposit does not
include ED
- add v16 migration to update ContractInfo struct with this change

Before:
<img width="820" alt="Screenshot 2024-03-21 at 11 23 29"
src="https://github.com/paritytech/polkadot-sdk/assets/521091/a0a8df0d-e743-42c5-9e16-cf2ec1aa949c">

After:
![Screenshot 2024-03-21 at 11 23
42](https://github.com/paritytech/polkadot-sdk/assets/521091/593235b0-b866-4915-b653-2071d793228b

)

---------

Co-authored-by: Cyrill Leutwiler <[email protected]>
Co-authored-by: command-bot <>

Dear team, dear @NachoPal @joepetrowski @bkchr @ggwpez

,

This is a retry of #3957, after merging master as advised!,

Many thanks!

**_Milos_**

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>

Reuse wasmi Module when validating.
Prepare the code for 0.32 and the addition of Module::new_unchecked

This tiny PR extends the `on_validated_block_announce` log with the bad
PeerID.
Used to identify if the peerID is malicious by correlating with other
logs (ie peer-set).

While at it, have removed the `\n` from a multiline log, which did not
play well with
[sub-triage-logs](https://github.com/lexnv/sub-triage-logs/tree/master

).

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

[Weights
compare](https://weights.tasty.limo/compare?unit=weight&ignore_errors=true&threshold=10&method=asymptotic&repo=polkadot-sdk&old=master&new=pg%2Fbench_tweaks&path_pattern=substrate%2Fframe%2F**%2Fsrc%2Fweights.rs%2Cpolkadot%2Fruntime%2F*%2Fsrc%2Fweights%2F**%2F*.rs%2Cpolkadot%2Fbridges%2Fmodules%2F*%2Fsrc%2Fweights.rs%2Ccumulus%2F**%2Fweights%2F*.rs%2Ccumulus%2F**%2Fweights%2Fxcm%2F*.rs%2Ccumulus%2F**%2Fsrc%2Fweights.rs)

Note: Raw weights change does not mean much here, as this PR reduce the
scope of what is benchmarked, they are therefore decreased by a good
margin. One should instead print the Schedule using

cargo test --features runtime-benchmarks bench_print_schedule --
--nocapture
or following the instructions from the
[README](https://github.com/paritytech/polkadot-sdk/tree/pg/bench_tweaks/substrate/frame/contracts#schedule)
for looking at the Schedule of a specific runtime

---------

Co-authored-by: command-bot <>

Part of #3326 

@Kianenigma @ggwpez

 

polkadot address: 12poSUQPtcF1HUPQGY3zZu2P8emuW9YnsPduA4XG3oCEfJVp

---------

Signed-off-by: Matteo Muraca <[email protected]>
Co-authored-by: ordian <[email protected]>

The plain `people-kusama` and `coretime-kusama` chainspecs were uploaded
at https://github.com/paritytech/polkadot-sdk/pull/3961. Only binaries
with compatible runtime versions can run with plain chainspec. For
example:

One of the latest master builds fails:

```
docker run paritypr/polkadot-parachain-debug:master-216509db --chain coretime-kusama
...
Error: Service(Client(Storage("wasm call error Other: Exported method GenesisBuilder_get_preset is not found"))
```

Master build from 5 days ago:

```
docker run paritypr/polkadot-parachain-debug:master-68cdb126 --chain coretime-kusama
...
2024-04-08 16:51:02 [Parachain] 🔨 Initializing Genesis block/state (state: 0xc418…889c, header-hash: 0x638c…d050)
2024-04-08 16:51:03 [Relaychain] 🔨 Initializing Genesis block/state (state: 0xb000…ef6b, header-hash: 0xb0a8…dafe)
2024-04-08 16:51:03 [Relaychain] 👴 Loading GRANDPA authority set from genesis on what appears to be first startup.
2024-04-08 16:51:03 [Relaychain] 👶 Creating empty BABE epoch changes on what appears to be first startup.
2024-04-08 16:51:03 [Relaychain] 🏷 Local node identity is: 12D3KooWQEp2uPow3FnngGmy9dYQ3qxY1GkmumS5MqBWEQscwTyy
2024-04-08 16:51:03 [Relaychain] 📦 Highest known block at #0
...
```

## Changes:

1. Rename:
```
coretime-kusama.json -> coretime-kusama-genesis.json
people-kusama.json -> people-kusama-genesis.json
```

2. Generate raw chainspec:

```
docker run --rm -v $(pwd):/dir paritypr/polkadot-parachain-debug:master-68cdb126 build-spec --raw --chain /dir/people-kusama-genesis.json > people-kusama.json 
docker run --rm -v $(pwd):/dir paritypr/polkadot-parachain-debug:master-68cdb126 build-spec --raw --chain /dir/coretime-kusama-genesis.json > coretime-kusama.json
```

## Tests:

New chainspec can run on the latest master build:

```
docker run -it --rm -v $(pwd):/dir paritypr/polkadot-parachain-debug:master-216509db --chain /dir/coretime-kusama.json
...
2024-04-09 16:44:39 [Relaychain] ⚙️ Syncing, target=#22665488 (8 peers), best: #2371 (0x19f8…5f3a), finalized #2048 (0xede6…f879), ⬇ 388.6kiB/s ⬆ 87.0kiB/s
2024-04-09 16:44:39 [Parachain] 💤 Idle (6 peers), best: #0 (0x638c…d050), finalized #0 (0x638c…d050), ⬇ 6.3kiB/s ⬆ 2.8kiB/s
```

Plain chainspec will fail:

```
docker run -it --rm -v $(pwd):/dir paritypr/polkadot-parachain-debug:master-216509db --chain /dir/coretime-kusama-genesis.json
... 
Error: Service(Client(Storage("wasm call error Other: Exported method GenesisBuilder_get_preset is not found")))
```

Serban Iorga authored Apr 09, 2024 and Bastian Köcher committed Apr 10, 2024

* Migrate fee payment from `Currency` to `fungible` (#2292)

Part of https://github.com/paritytech/polkadot-sdk/issues/226
Related https://github.com/paritytech/polkadot-sdk/issues/1833

- Deprecate `CurrencyAdapter` and introduce `FungibleAdapter`
- Deprecate `ToStakingPot` and replace usage with `ResolveTo`
- Required creating a new `StakingPotAccountId` struct that implements
`TypedGet` for the staking pot account ID
- Update parachain common utils `DealWithFees`, `ToAuthor` and
`AssetsToBlockAuthor` implementations to use `fungible`
- Update runtime XCM Weight Traders to use `ResolveTo` instead of
`ToStakingPot`
- Update runtime Transaction Payment pallets to use `FungibleAdapter`
instead of `CurrencyAdapter`
- [x] Blocked by https://github.com/paritytech/polkadot-sdk/pull/1296,
needs the `Unbalanced::decrease_balance` fix

(cherry picked from commit bda4e75a

)

* Upgrade `trie-db` from `0.28.0` to `0.29.0` (#3982)

- What does this PR do?
1. Upgrades `trie-db`'s version to the latest release. This release
includes, among others, an implementation of `DoubleEndedIterator` for
the `TrieDB` struct, allowing to iterate both backwards and forwards
within the leaves of a trie.
2. Upgrades `trie-bench` to `0.39.0` for compatibility.
3. Upgrades `criterion` to `0.5.1` for compatibility.
- Why are these changes needed?
Besides keeping up with the upgrade of `trie-db`, this specifically adds
the functionality of iterating back on the leafs of a trie, with
`sp-trie`. In a project we're currently working on, this comes very
handy to verify a Merkle proof that is the response to a challenge. The
challenge is a random hash that (most likely) will not be an existing
leaf in the trie. So the challenged user, has to provide a Merkle proof
of the previous and next existing leafs in the trie, that surround the
random challenged hash.

Without having DoubleEnded iterators, we're forced to iterate until we
find the first existing leaf, like so:
```rust
        // ************* VERIFIER (RUNTIME) *************
        // Verify proof. This generates a partial trie based on the proof and
        // checks that the root hash matches the `expected_root`.
        let (memdb, root) = proof.to_memory_db(Some(&root)).unwrap();
        let trie = TrieDBBuilder::<LayoutV1<RefHasher>>::new(&memdb, &root).build();

        // Print all leaf node keys and values.
        println!("\nPrinting leaf nodes of partial tree...");
        for key in trie.key_iter().unwrap() {
            if key.is_ok() {
                println!("Leaf node key: {:?}", key.clone().unwrap());

                let val = trie.get(&key.unwrap());

                if val.is_ok() {
                    println!("Leaf node value: {:?}", val.unwrap());
                } else {
                    println!("Leaf node value: None");
                }
            }
        }

        println!("RECONSTRUCTED TRIE {:#?}", trie);

        // Create an iterator over the leaf nodes.
        let mut iter = trie.iter().unwrap();

        // First element with a value should be the previous existing leaf to the challenged hash.
        let mut prev_key = None;
        for element in &mut iter {
            if element.is_ok() {
                let (key, _) = element.unwrap();
                prev_key = Some(key);
                break;
            }
        }
        assert!(prev_key.is_some());

        // Since hashes are `Vec<u8>` ordered in big-endian, we can compare them directly.
        assert!(prev_key.unwrap() <= challenge_hash.to_vec());

        // The next element should exist (meaning there is no other existing leaf between the
        // previous and next leaf) and it should be greater than the challenged hash.
        let next_key = iter.next().unwrap().unwrap().0;
        assert!(next_key >= challenge_hash.to_vec());
```

With DoubleEnded iterators, we can avoid that, like this:
```rust
        // ************* VERIFIER (RUNTIME) *************
        // Verify proof. This generates a partial trie based on the proof and
        // checks that the root hash matches the `expected_root`.
        let (memdb, root) = proof.to_memory_db(Some(&root)).unwrap();
        let trie = TrieDBBuilder::<LayoutV1<RefHasher>>::new(&memdb, &root).build();

        // Print all leaf node keys and values.
        println!("\nPrinting leaf nodes of partial tree...");
        for key in trie.key_iter().unwrap() {
            if key.is_ok() {
                println!("Leaf node key: {:?}", key.clone().unwrap());

                let val = trie.get(&key.unwrap());

                if val.is_ok() {
                    println!("Leaf node value: {:?}", val.unwrap());
                } else {
                    println!("Leaf node value: None");
                }
            }
        }

        // println!("RECONSTRUCTED TRIE {:#?}", trie);
        println!("\nChallenged key: {:?}", challenge_hash);

        // Create an iterator over the leaf nodes.
        let mut double_ended_iter = trie.into_double_ended_iter().unwrap();

        // First element with a value should be the previous existing leaf to the challenged hash.
        double_ended_iter.seek(&challenge_hash.to_vec()).unwrap();
        let next_key = double_ended_iter.next_back().unwrap().unwrap().0;
        let prev_key = double_ended_iter.next_back().unwrap().unwrap().0;

        // Since hashes are `Vec<u8>` ordered in big-endian, we can compare them directly.
        println!("Prev key: {:?}", prev_key);
        assert!(prev_key <= challenge_hash.to_vec());

        println!("Next key: {:?}", next_key);
        assert!(next_key >= challenge_hash.to_vec());
```
- How were these changes implemented and what do they affect?
All that is needed for this functionality to be exposed is changing the
version number of `trie-db` in all the `Cargo.toml`s applicable, and
re-exporting some additional structs from `trie-db` in `sp-trie`.

---------

Co-authored-by: Bastian Köcher <[email protected]>
(cherry picked from commit 4e73c0fc

)

* Update polkadot-sdk refs

* Fix Cargo.lock

---------

Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Facundo Farall <[email protected]>

Serban Iorga authored Apr 02, 2024 and Bastian Köcher committed Apr 10, 2024

* Use workspace.[authors|edition]

* Add repository.workspace = true

* Upgrade dependencies to the polkadot-sdk versions

* Upgrade async-std version

* Update jsonrpsee version

* cargo update

* use ci-unified image

Svyatoslav Nikolsky authored Apr 01, 2024 and Bastian Köcher committed Apr 10, 2024

* added CLI arguments: full WS URI + separate for WS path URI component + additional log

* URI -> URL?

* added TODO

* fmt

dependabot[bot] authored Apr 01, 2024 and Bastian Köcher committed Apr 10, 2024

Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.36.0 to 1.37.0.
- [Release notes](https://github.com/tokio-rs/tokio/releases)
- [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.36.0...tokio-1.37.0

)

---
updated-dependencies:
- dependency-name: tokio
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

dependabot[bot] authored Apr 01, 2024 and Bastian Köcher committed Apr 10, 2024

Bumps [scale-info](https://github.com/paritytech/scale-info) from 2.11.0 to 2.11.1.
- [Release notes](https://github.com/paritytech/scale-info/releases)
- [Changelog](https://github.com/paritytech/scale-info/blob/master/CHANGELOG.md)
- [Commits](https://github.com/paritytech/scale-info/compare/v2.11.0...v2.11.1

)

---
updated-dependencies:
- dependency-name: scale-info
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>