Based on https://github.com/paritytech/polkadot-sdk/pull/3331
Related to https://github.com/paritytech/polkadot-sdk/issues/3268

Implements migrations with customizable block number to relay height
number translation function.

Adds block to relay height migration code for rococo and westend.

---------

Co-authored-by: DavidK <davidk@parity.io>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

Closes #2082

change send xcm to use `xcm::executor::FeeManager` to determine if the
sender should be charged.

I had to change the `FeeManager` of the penpal config to ensure the same
test behaviour as before. For the other tests, I'm using the
`FeeManager` from the `xcm::executor::FeeManager` as this one is used to
check if the fee can be waived on the charge fees method.

---------

Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: GitHub Action <action@github.com>

There is no need to verify in the `pallet-membership` benchmark that the
`MemembershipChanged` implementation works as the pallet thinks it
should work. If you for example set it to `()`, `get_prime()` will
always return `None`.

TLDR: Remove the checks of `MembershipChanged` in the benchmarks to
support any kind of implementation.

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>

This PR adds size-limits to the fork-aware transaction pool.

**Review Notes**
- Existing
[`TrackedMap`](https://github.com/paritytech/polkadot-sdk/blob/58fd5ae4/substrate/client/transaction-pool/src/graph/tracked_map.rs#L33-L41)
is used in internal mempool to track the size of extrinsics:

https://github.com/paritytech/polkadot-sdk/blob/58fd5ae4

/substrate/client/transaction-pool/src/graph/tracked_map.rs#L33-L41

- In this PR, I also removed the logic that kept transactions in the
`tx_mem_pool` if they were immediately dropped by the views. Initially,
I implemented this as an improvement: if there was available space in
the _mempool_ and all views dropped the transaction upon submission, the
transaction would still be retained in the _mempool_.

However, upon further consideration, I decided to remove this
functionality to reduce unnecessary complexity. Now, when all views drop
a transaction during submission, it is automatically rejected, with the
`submit/submit_and_watch` call returning `ImmediatelyDropped`.


Closes: #5476

---------

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>

# Description

This seems to be an old artifact of the long closed
https://github.com/paritytech/substrate/issues/6827 that I noticed when
working on related code earlier.

## Integration

`NetworkStarter` was removed, simply remove its usage:
```diff
-let (network, system_rpc_tx, tx_handler_controller, start_network, sync_service) =
+let (network, system_rpc_tx, tx_handler_controller, sync_service) =
    build_network(BuildNetworkParams {
...
-start_network.start_network();
```

## Review Notes

Changes are trivial, the only reason for this to not be accepted is if
it is desired to not start network automatically for whatever reason, in
which case the description of network starter needs to change.

# Checklist

* [x] My PR includes a detailed description as outlined in the
"Description" and its two subsections above.
* [ ] My PR follows the [labeling requirements](

https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTIN...

Kudos to @EclesioMeloJunior for noticing it 

Also added a regression test for it. The existing unit test was
exercising only the case where the full chain is reverted

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>

Part of:
#6202

---------

Co-authored-by: Giuseppe Re <giuseppe.re@parity.io>
Co-authored-by: GitHub Action <action@github.com>

This PR introduces usage of `build_struct_json_patch` macro in all
runtimes (also guides) within the code base.

It also fixes macro to support _field init shorthand_, and _Struct
Update_ syntax which were missing in original implementation.

Follow up of #5700 and #5813

This PR exposes additional syncing types to facilitate the development
of a custom syncing strategy based on the existing Polkadot syncing
strategy. Specifically, my goal is to isolate the state sync and chain
sync components, allowing the state to be downloaded from the P2P
network without running a full regular Substrate node. I also need to
intercept the state responses during the state sync process.

The newly exposed types are necessary to implement this custom syncing
strategy.

Part of:

- #6202.

---------

Co-authored-by: Giuseppe Re <giuseppe.re@parity.io>
Co-authored-by: GitHub Action <action@github.com>

Fixes https://github.com/paritytech/polkadot-sdk/issues/6172

# Description

Implemented benchmarks for Notifications and RequestResponse protocols
with libp2p implementation. These benchmarks allow us to monitor
regressions and implement fixes before they are observed in real chain.
In the future, they can be used for targeted optimizations of litep2p
compared to libp2p.

Part of https://github.com/paritytech/polkadot-sdk/issues/5220

Next steps:
- Add benchmarks for litep2p implementation
- Optimize load to get better results
- Add benchmarks to CI to catch regressions



## Integration

Benchmarks don't affect downstream projects.

---------

Co-authored-by: alvicsam <alvicsam@gmail.com>
Co-authored-by: GitHub Action <action@github.com>

# Description

E2E tests to Polkadot/Kusama's people chains (in
https://github.com/open-web3-stack/polkadot-ecosystem-tests/pull/63)
revealed that 2 of the identity pallet's extrinsics did not emit events
in case of success:
* `pallet_identity::rename_sub`, and
* `pallet_identity::set_subs`

This PR fixes that.

## Integration

Other than 2 extrinsics emiting an event when previously they did not,
no other behavior in pallets/extrinsics was modified, so no integration
is needed.

## Review Notes

N/A

This PR changes the behavior of `json_patch::merge` function which no
longer removes any keys from the base JSON object.

fixes: #6306

---------

Co-authored-by: GitHub Action <action@github.com>

This PR has small addition to the db package version. As `cargodeb`
takes the version from the `*.toml` file, this PR adds an extra flag to
the `cargodeb` command so that the version of the deb package matches
the `polkadot` node version.

Fixes https://github.com/paritytech/polkadot-sdk/issues/5530

This PR introduces the removal of backing jobs that have been back
pressured for longer than `allowedAncestryLen`, as these candidates are
no longer viable.

It is reasonable to expect a result for a backing job execution within
`allowedAncestryLen` blocks. Therefore, we set the job TTL as a relay
block number and synchronize the validation host by sending activated
leaves.

---------

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>

This PR ensures that external addresses with different PeerIDs are not
propagated to the higher layer of the network code.

While at it, this ensures that libp2p only adds the `/p2p/peerid` part
to the discovered address if it does not contain it already.

This is a followup from:
- https://github.com/paritytech/polkadot-sdk/pull/6298

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>

fixes https://github.com/paritytech/polkadot-sdk/issues/5966

---------

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

# Description

This is a continuation of
https://github.com/paritytech/polkadot-sdk/pull/5666 that finally fixes
https://github.com/paritytech/polkadot-sdk/issues/5333.

This should allow developers to create custom syncing strategies or even
the whole syncing engine if they so desire. It also moved syncing engine
creation and addition of corresponding protocol outside
`build_network_advanced` method, which is something Bastian expressed as
desired in
https://github.com/paritytech/polkadot-sdk/issues/5#issuecomment-1700816458

Here I replaced strategy-specific types and methods in `SyncingStrategy`
trait with generic ones. Specifically `SyncingAction` is now used by all
strategies instead of strategy-specific types with conversions.
`StrategyKey` was an enum with a fixed set of options and now replaced
with an opaque type that strategies create privately and send to upper
layers as an opaque type. Requests and responses are now handled in a
generic way regardless of the strategy, which reduced and simplified
strategy API.

`PolkadotSyncingStrategy` now lives in its dedicated module (had to edit
.gitignore for this) like other strategies.

`build_network_advanced` takes generic `SyncingService` as an argument
alongside with a few other low-level types (that can probably be
extracted in the future as well) without any notion of specifics of the
way syncing is actually done. All the protocol and tasks are created
outside and not a part of the network anymore. It still adds a bunch of
protocols like for light client and some others that should eventually
be restructured making `build_network_advanced` just building generic
network and not application-specific protocols handling.

## Integration

Just like https://github.com/paritytech/polkadot-sdk/pull/5666
introduced `build_polkadot_syncing_strategy`, this PR introduces
`build_default_block_downloader`, but for convenience and to avoid
typical boilerplate a simpler high-level function
`build_default_syncing_engine` is added that will take care of creating
typical block downloader, syncing strategy and syncing engine, which is
what most users will be using going forward. `build_network` towards the
end of the PR was renamed to `build_network_advanced` and
`build_network`'s API was reverted to
pre-https://github.com/paritytech/polkadot-sdk/pull/5666, so most users
will not see much of a difference during upgrade unless they opt-in to
use new API.

## Review Notes

For `StrategyKey` I was thinking about using something like private type
and then storing `TypeId` inside instead of a static string in it, let
me know if that would preferred.

The biggest change happened to requests that different strategies make
and how their responses are handled. The most annoying thing here is
that block response decoding, in contrast to all other responses, is
dependent on request. This meant request had to be sent throughout the
system. While originally `Response` was `Vec<u8>`, I didn't want to
re-encode/decode request and response just to fit into that API, so I
ended up with `Box<dyn Any + Send>`. This allows responses to be truly
generic and each strategy will know how to downcast it back to the
concrete type when handling the response.

Import queue refactoring was needed to move `SyncingEngine` construction
out of `build_network` that awkwardly implemented for `SyncingService`,
but due to `&mut self` wasn't usable on `Arc<SyncingService>` for no
good reason. `Arc<SyncingService>` itself is of course useless, but
refactoring to replace it with just `SyncingService` was unfortunately
rejected in https://github.com/paritytech/polkadot-sdk/pull/5454

As usual I recommend to review this PR as a series of commits instead of
as the final diff, it'll make more sense that way.

# Checklist

* [x] My PR includes a detailed description as outlined in the
"Description" and its two subsections above.
* [x] My PR follows the [labeling requirements](

https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process
) of this project (at minimum one label for `T` required)
* External contributors: ask maintainers to put the right label on your
PR.
* [x] I have made corresponding changes to the documentation (if
applicable)

This PR fixes the `0002-validators-warp-sync` zombienet test by matching
for a specific error case.

The following PR introduced a new error for validators that don't have
public addresses available for publishing in the DHT:
- https://github.com/paritytech/polkadot-sdk/pull/6298
The log line was moved from `eprintln!("Warning: ...` in sc-cli to the
authority discovery where it is printed properly as an error.


cc @paritytech/sdk-node

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

https://github.com/paritytech/polkadot-sdk/issues/6343

Part of:

- #6202.

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Guillaume Thiolliere <guillaume.thiolliere@parity.io>
Co-authored-by: Giuseppe Re <giuseppe.re@parity.io>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.30 to
0.3.31.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/rust-lang/futures-rs/releases">futures's
releases</a>.</em></p>
<blockquote>
<h2>0.3.31</h2>
<ul>
<li>Fix use after free of task in <code>FuturesUnordered</code> when
dropped future panics (<a
href="https://redirect.github.com/rust-lang/futures-rs/issues/2886">#2886</a>)</li>
<li>Fix soundness bug in <code>task::waker_ref</code> (<a
href="https://redirect.github.com/rust-lang/futures-rs/issues/2830">#2830</a>)
This is a breaking change but allowed because it is soundness bug
fix.</li>
<li>Fix bugs in <code>AsyncBufRead::read_line</code> and
<code>AsyncBufReadExt::lines</code> (<a
href="https://redirect.github.com/rust-lang/futures-rs/issues/2884">#2884</a>)</li>
<li>Fix parsing issue in
<code>select!</code>/<code>select_biased!</code> (<a
href="https://redirect.github.com/rust-lang/futures-rs/is...

closes https://github.com/paritytech/polkadot-sdk/issues/6358

Adds the following runtime api to pallet-nomination-pools.
`pool_accounts(pool_id)`: Returns `(bonded_account, reward_account)`
associated with the `pool_id`.

cc: @rossbulat



---------

Co-authored-by: command-bot <>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>

# Context

This PR aims to introduce XCMv5, for now it's in progress and will be
updated over time.
This branch will serve as a milestone branch for merging in all features
we want to add to XCM, roughly outlined
[here](https://github.com/polkadot-fellows/xcm-format/issues/60).
More features could be added.

## TODO
- [x] Migrate foreign assets from v3 to v4
- [x] Setup v5 skeleton
- [x] Remove XCMv2
- [x] https://github.com/paritytech/polkadot-sdk/pull/5390
- [x] https://github.com/paritytech/polkadot-sdk/pull/5585
- [x] https://github.com/paritytech/polkadot-sdk/pull/5420
- [x] https://github.com/paritytech/polkadot-sdk/pull/5876
- [x] https://github.com/paritytech/polkadot-sdk/pull/5971
- [x] https://github.com/paritytech/polkadot-sdk/pull/6148
- [x] https://github.com/paritytech/polkadot-sdk/pull/6228

Fixes #3434 
Fixes https://github.com/paritytech/polkadot-sdk/issues/4190
Fixes https://github.com/paritytech/polkadot-sdk/issues/5209
Fixes https://github.com/paritytech/polkadot-sdk/issues/5241
Fixes https://github.com/paritytech/polkadot-sdk/issues/4284

---------

Signed-off-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Andrii <ndk@parity.io>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Joseph Zhao <65984904+programskillforverification@users.noreply.github.com>
Co-authored-by: Nazar Mokrynskyi <nazar@mokrynskyi.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: command-bot <>
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Serban Iorga <serban@parity.io>

Relax requirements for `assign_core` so that it accepts updates for the
last scheduled entry.

Fixes #6102

---------

Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: GitHub Action <action@github.com>

closes #1890 

### Overview 

Introduces similar checker struct to `CheckTraitDecls` in
`decl_runtime_apis!` - `CheckTraitImpls`. Overrides
`visit::visit_type_path` to detect usage of `Self` as a type argument
within the scope of `impl_runtime_apis!`.

**Note**: only prevents the usage of `Self` as a type argument in an
angle bracket `<>`, as it is the only use case that fails to compile.
For example, the code
[below](https://github.com/paritytech/polkadot-sdk/blob/master/cumulus/parachains/runtimes/bridge-hubs/bridge-hub-westend/src/lib.rs#L1002)
compiles fine:

```rs
impl BridgeMessagesConfig<WithBridgeHubRococoMessagesInstance> for Runtime {
  fn is_relayer_rewarded(relayer: &Self::AccountId) -> bool {
	  let bench_lane_id = <Self as BridgeMessagesConfig<WithBridgeHubRococoMessagesInstance>>::bench_lane_id();
	  // ...
```

### Result

Given a block of code like this:

```rs
impl_runtime_apis! {
	impl apis::Core<Block> for Runtime {
		fn initialize_block(header: &HeaderFor<Self>) -> ExtrinsicInclusionMode {
			let _: HeaderFor<Self> = header.clone();
			RuntimeExecutive::initialize_block(header)
		}
		// ...
         }
// ...
```

<details open>

<summary>Output:</summary>

```bash
$ cargo build --release -p minimal-template-node
  error: `Self` can not be used as type argument in the scope of `impl_runtime_apis!`. Use `Runtime` instead.
     --> /polkadot-sdk/templates/minimal/runtime/src/lib.rs:133:11
      |
  133 |             let _: HeaderFor<Self> = header.clone();
      |                    ^^^^^^^^^^^^^^^

  error: `Self` can not be used as type argument in the scope of `impl_runtime_apis!`. Use `Runtime` instead.
     --> /polkadot-sdk/templates/minimal/runtime/src/lib.rs:132:32
      |
  132 |         fn initialize_block(header: &HeaderFor<Self>) -> ExtrinsicInclusionMode {
```

</details>

---------

Co-authored-by: Pavlo Khrystenko <p.khrystenko@gmail.com>
Co-authored-by: Pavlo Khrystenko <45178695+pkhry@users.noreply.github.com>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>

Part of https://github.com/paritytech/polkadot-sdk/issues/5047

TODO:

- [x] prdoc
- [x] fix/add tests

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

A quick fix to the step that generates a temporary token used in the
pipeline

# Description

Prior to this PR, the metadata for runtime APIs was entirely based on
that generated by `decl_runtime_apis`. It therefore didn't take into
account that `impl_runtime_apis` might implement older versions of APIs
than what has been declared.

This PR filters the returned runtime API metadata to only include
methods actually implemented, and also avoids including methods labelled
with `changed_in` (which the previous code was atempting to do already
but not successfully, owing to the attr being removed prior to the
check).

We also change all version related things to be `u32`s (rather than
VERSION being `u32` and `api_version`s being `u64`) for consistency /
ease of comparison.

A test is added which works with both the `enable-staging-api` feature
in api/tests enabled or disabled, to check all of this.

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: GitHub Action <action@github.com>

The tests used the same paths. When run on CI, each test is run in its
own process and thus, this "serial_test" crate wasn't used. The tests
are now using their own thread local tempdir, which ensures that the
tests are working when running in parallel in the same program or when
being run individually.

Previously, when receiving the `SetReservedPeers { reserved }` all peers
not in the `reserved` set were removed.

This is incorrect, the intention of `SetReservedPeers` is to change the
active set of reserved peers and disconnect previously reserved peers
not in the new set.

While at it, have added a few other improvements to make the peerset
more robust:
- `SetReservedPeers`: does not disconnect all peers
- `SetReservedPeers`: if a reserved peer is no longer reserved, the
peerset tries to move the peers to the regular set if the slots allow
this move. This ensures the (now regular) peer counts towards slot
allocation.
- every 1 seconds: If we don't have enough connect peers, add the
reserved peers to the list that the peerstore ignores. Reserved peers
are already connected and the peerstore might return otherwise a
reserved peer


### Next Steps

- [x] More testing

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

Resolves https://github.com/paritytech/polkadot-sdk/issues/5929

Migrates `ChildBountyDescriptions` to be indexed instead of unique child
bounty id unique per all child bounties in the pallet to be unique per
every parent bounty.

Migrates `(ParentBounty, ChildBounty)` keys inside `ChildBounties`
storage item to use new `ChildBounty` ids starting from `0`.

@paritytech/frame-coders

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: DavidK <davidk@parity.io>
Co-authored-by: muharem <ismailov.m.h@gmail.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

There was no good example of what is patch and full genesis config file.
Some explanation and example were added to the `chain-spec-builder` doc.

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>

Remove `sp_runtime::RuntimeString` and replace with `Cow<'static, str>` or `String` depending on use case (#5693)

# Description

As described in https://github.com/paritytech/polkadot-sdk/issues/4001
`RuntimeVersion` was not encoded consistently using serde. Turned out it
was a remnant of old times and no longer actually needed. As such I
removed it completely in this PR and replaced with `Cow<'static, str>`
for spec/impl names and `String` for error cases.

Fixes https://github.com/paritytech/polkadot-sdk/issues/4001.

## Integration

For downstream projects the upgrade will primarily consist of following
two changes:
```diff
#[sp_version::runtime_version]
pub const VERSION: RuntimeVersion = RuntimeVersion {
-	spec_name: create_runtime_str!("statemine"),
-	impl_name: create_runtime_str!("statemine"),
+	spec_name: alloc::borrow::Cow::Borrowed("statemine"),
+	impl_name: alloc::borrow::Cow::Borrowed("statemine"),
```
```diff
		fn dispatch_benchmark(
			config: frame_benchmarking::BenchmarkConfig
-		) -> Result<Vec<frame_benchmarking::BenchmarkBatch>, sp_runtime::RuntimeString> {
+		) -> Result<Vec<frame_benchmarking::BenchmarkBatch>, alloc::string::String> {
```

SCALE encoding/decoding remains the same as before, but serde encoding
in runtime has changed from bytes to string (it was like this in `std`
environment already), which most projects shouldn't have issues with. I
consider the impact of serde encoding here low due to the type only
being used in runtime version struct and mostly limited to runtime
internals, where serde encoding/decoding of this data structure is quite
unlikely (though we did hit exactly this edge-case ourselves


).

## Review Notes

Most of the changes are trivial and mechanical, the only non-trivial
change is in
`substrate/primitives/version/proc-macro/src/decl_runtime_version.rs`
where macro call expectation in `sp_version::runtime_version`
implementation was replaced with function call expectation.

# Checklist

* [x] My PR includes a detailed description as outlined in the
"Description" and its two subsections above.
* [ ] My PR follows the [labeling requirements](

https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process
) of this project (at minimum one label for `T` required)
* External contributors: ask maintainers to put the right label on your
PR.
* [ ] I have made corresponding changes to the documentation (if
applicable)

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Guillaume Thiolliere <guillaume.thiolliere@parity.io>
Co-authored-by: Bastian Köcher <info@kchr.de>

I've broken this test with
https://github.com/paritytech/polkadot-sdk/pull/5883 and this is the
fix.

The benchmark is now updated to use proper core index and session index
for the generated candidates.

TODO:
- [ ] <del> PRDoc </del>

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Part of:

- #6202.

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Giuseppe Re <giuseppe.re@parity.io>

We were not passing through the `RUSTUP_TOOLCHAIN` variable to the
`build.rs` script of our fixtures. This means that setting the toolchain
like `cargo +1.81 build` had no effect on the fixture build. It would
always fall back to the default toolchain.

---------

Co-authored-by: GitHub Action <action@github.com>

make the eth-rpc proxy /health and /health/readiness from the proxied
substrate chain
see #4802

---------

Co-authored-by: GitHub Action <action@github.com>