Backport #6684 into `stable2412` from bkontur.

See the
[documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md)
on how to use this bot.

<!--
  # To be used by other automation, do not modify:
  original-pr-number: #${pull_number}
-->

Co-authored-by: Branislav Kontur <bkontur@gmail.com>

Backport #6677 into `stable2412` from lexnv.

See the
[documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md)
on how to use this bot.

<!--
  # To be used by other automation, do not modify:
  original-pr-number: #${pull_number}
-->

Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>

This PR is a backport for:
- https://github.com/paritytech/polkadot-sdk/pull/5732

cc @paritytech/subxt-team

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Egor_P <egor@parity.io>

Backport #6603 into `stable2412` from lexnv.

See the
[documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md)
on how to use this bot.

<!--
  # To be used by other automation, do not modify:
  original-pr-number: #${pull_number}
-->

Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Egor_P <egor@parity.io>

Backport #6640 into the `stable2412` from master

This PR contains backports of:
- https://github.com/paritytech/polkadot-sdk/pull/6465
- https://github.com/paritytech/polkadot-sdk/pull/6536
- https://github.com/paritytech/polkadot-sdk/pull/6579
- https://github.com/paritytech/polkadot-sdk/pull/6645

(when those were merged, bot didn't create backport PRs to stable2412)

---------

Co-authored-by: GitHub Action <action@github.com>

# Description

This PR adds the required changes to release `polkadot`,
`polkadot-parachain` and `polkadot-omni-node` binaries built on Apple
Sillicon macos.

## Integration

This addresses requests from the community for such binaries: #802, and
they should be part of the Github release page.

## Review Notes

Test on paritytech-stg solely focused on macos binaries:
https://github.com/paritytech-stg/polkadot-sdk/actions/runs/11824692766/job/32946793308,
except the steps related to `pgpkms` (which need AWS credentials,
missing from paritytech-stg). The binary names don't have a `darwin-arm`
identifier, and conflict with the existing x86_64-linux binaries. I
haven't tested building everything on `paritytech-stg` because the
x86_64-linux builds run on `unbutu-latest-m` which isn't enabled on
`pairtytech-stg` (and I haven't asked CI team to enable one), so testing
how to go around naming conflicts should be covered next.

### TODO

- [x] Test the workflow start to end (especially the last bits related
to uploading the binaries on S3 and ensuring the previous binaries and
the new ones coexist harmoniously on S3/action artifacts storage without
naming conflicts) @EgorPopelyaev
- [x] Publish the arm binaries on the Github release page - to clarify
what's needed @iulianbarbu

 . Current practice is to manually publish the
binaries built via `release-build-rc.yml` workflow, taken from S3. Would
be great to have the binaries there in the first place before working on
automating this, but I would also do it in a follow up PR.

### Follow ups

- [ ] unify the binaries building under
`release-30_publish_release_draft.yml` maybe?
- [ ] automate binary artifacts upload to S3 in
`release-30_publish_release_draft.yml`

Signed-off-by: Iulian Barbu <iulian.barbu@parity.io>
Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>

This PR is a backport of #6418.

For context, `TransactionExtension`, introduced in #3685, is part of the
`stable2412` release, and this PR brings important fixes and quality of
life improvements. Doing the backport now allows us to not break the
interface later.

Opened against #6473 as the changes in the original PR were made on top
of the changes in this backport.

---------

Signed-off-by: georgepisaltu <george.pisaltu@parity.io>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: GitHub Action <action@github.com>

This PR updates the litep2p version on stable2412.
The litep2p requires a user opt-in / non-default backend, and changes
should not affect testing / stable libp2p versions.

For more details see PR:
- https://github.com/paritytech/polkadot-sdk/pull/6484

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

Co-authored-by: GitHub Action <action@github.com>

add transaction source in the transaction extension API.

Transaction extension is introduced in stable2412 so better not to break
again later.

Backporting the new V5 `ExecuteWithOrigin` instruction to stable2412

---------

Signed-off-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Andrii <ndk@parity.io>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Joseph Zhao <65984904+programskillforverification@users.noreply.github.com>
Co-authored-by: Nazar Mokrynskyi <nazar@mokrynskyi.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

Part of:

- #6202.

---------

Co-authored-by: Giuseppe Re <giuseppe.re@parity.io>
Co-authored-by: GitHub Action <action@github.com>

Fixes https://github.com/paritytech/polkadot-sdk/issues/6172

# Description

Implemented benchmarks for Notifications and RequestResponse protocols
with libp2p implementation. These benchmarks allow us to monitor
regressions and implement fixes before they are observed in real chain.
In the future, they can be used for targeted optimizations of litep2p
compared to libp2p.

Part of https://github.com/paritytech/polkadot-sdk/issues/5220

Next steps:
- Add benchmarks for litep2p implementation
- Optimize load to get better results
- Add benchmarks to CI to catch regressions



## Integration

Benchmarks don't affect downstream projects.

---------

Co-authored-by: alvicsam <alvicsam@gmail.com>
Co-authored-by: GitHub Action <action@github.com>

# Description

E2E tests to Polkadot/Kusama's people chains (in
https://github.com/open-web3-stack/polkadot-ecosystem-tests/pull/63)
revealed that 2 of the identity pallet's extrinsics did not emit events
in case of success:
* `pallet_identity::rename_sub`, and
* `pallet_identity::set_subs`

This PR fixes that.

## Integration

Other than 2 extrinsics emiting an event when previously they did not,
no other behavior in pallets/extrinsics was modified, so no integration
is needed.

## Review Notes

N/A

This PR changes the behavior of `json_patch::merge` function which no
longer removes any keys from the base JSON object.

fixes: #6306

---------

Co-authored-by: GitHub Action <action@github.com>

This PR has small addition to the db package version. As `cargodeb`
takes the version from the `*.toml` file, this PR adds an extra flag to
the `cargodeb` command so that the version of the deb package matches
the `polkadot` node version.

Fixes https://github.com/paritytech/polkadot-sdk/issues/5530

This PR introduces the removal of backing jobs that have been back
pressured for longer than `allowedAncestryLen`, as these candidates are
no longer viable.

It is reasonable to expect a result for a backing job execution within
`allowedAncestryLen` blocks. Therefore, we set the job TTL as a relay
block number and synchronize the validation host by sending activated
leaves.

---------

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>

This PR ensures that external addresses with different PeerIDs are not
propagated to the higher layer of the network code.

While at it, this ensures that libp2p only adds the `/p2p/peerid` part
to the discovered address if it does not contain it already.

This is a followup from:
- https://github.com/paritytech/polkadot-sdk/pull/6298

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>

fixes https://github.com/paritytech/polkadot-sdk/issues/5966

---------

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

# Description

This is a continuation of
https://github.com/paritytech/polkadot-sdk/pull/5666 that finally fixes
https://github.com/paritytech/polkadot-sdk/issues/5333.

This should allow developers to create custom syncing strategies or even
the whole syncing engine if they so desire. It also moved syncing engine
creation and addition of corresponding protocol outside
`build_network_advanced` method, which is something Bastian expressed as
desired in
https://github.com/paritytech/polkadot-sdk/issues/5#issuecomment-1700816458

Here I replaced strategy-specific types and methods in `SyncingStrategy`
trait with generic ones. Specifically `SyncingAction` is now used by all
strategies instead of strategy-specific types with conversions.
`StrategyKey` was an enum with a fixed set of options and now replaced
with an opaque type that strategies create privately and send to upper
layers as an opaque type. Requests and responses are now handled in a
generic way regardl...

This PR fixes the `0002-validators-warp-sync` zombienet test by matching
for a specific error case.

The following PR introduced a new error for validators that don't have
public addresses available for publishing in the DHT:
- https://github.com/paritytech/polkadot-sdk/pull/6298
The log line was moved from `eprintln!("Warning: ...` in sc-cli to the
authority discovery where it is printed properly as an error.


cc @paritytech/sdk-node

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

https://github.com/paritytech/polkadot-sdk/issues/6343

Part of:

- #6202.

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Guillaume Thiolliere <guillaume.thiolliere@parity.io>
Co-authored-by: Giuseppe Re <giuseppe.re@parity.io>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.30 to
0.3.31.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/rust-lang/futures-rs/releases">futures's
releases</a>.</em></p>
<blockquote>
<h2>0.3.31</h2>
<ul>
<li>Fix use after free of task in <code>FuturesUnordered</code> when
dropped future panics (<a
href="https://redirect.github.com/rust-lang/futures-rs/issues/2886">#2886</a>)</li>
<li>Fix soundness bug in <code>task::waker_ref</code> (<a
href="https://redirect.github.com/rust-lang/futures-rs/issues/2830">#2830</a>)
This is a breaking change but allowed because it is soundness bug
fix.</li>
<li>Fix bugs in <code>AsyncBufRead::read_line</code> and
<code>AsyncBufReadExt::lines</code> (<a
href="https://redirect.github.com/rust-lang/futures-rs/issues/2884">#2884</a>)</li>
<li>Fix parsing issue in
<code>select!</code>/<code>select_biased!</code> (<a
href="https://redirect.github.com/rust-lang/futures-rs/is...

closes https://github.com/paritytech/polkadot-sdk/issues/6358

Adds the following runtime api to pallet-nomination-pools.
`pool_accounts(pool_id)`: Returns `(bonded_account, reward_account)`
associated with the `pool_id`.

cc: @rossbulat



---------

Co-authored-by: command-bot <>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>

# Context

This PR aims to introduce XCMv5, for now it's in progress and will be
updated over time.
This branch will serve as a milestone branch for merging in all features
we want to add to XCM, roughly outlined
[here](https://github.com/polkadot-fellows/xcm-format/issues/60).
More features could be added.

## TODO
- [x] Migrate foreign assets from v3 to v4
- [x] Setup v5 skeleton
- [x] Remove XCMv2
- [x] https://github.com/paritytech/polkadot-sdk/pull/5390
- [x] https://github.com/paritytech/polkadot-sdk/pull/5585
- [x] https://github.com/paritytech/polkadot-sdk/pull/5420
- [x] https://github.com/paritytech/polkadot-sdk/pull/5876
- [x] https://github.com/paritytech/polkadot-sdk/pull/5971
- [x] https://github.com/paritytech/polkadot-sdk/pull/6148
- [x] https://github.com/paritytech/polkadot-sdk/pull/6228

Fixes #3434 
Fixes https://github.com/paritytech/polkadot-sdk/issues/4190
Fixes https://github.com/paritytech/polkadot-sdk/issues/5209
Fixes https://github.com/paritytech/polkadot-sdk/issues/5241
Fixes https://github.com/paritytech/polkadot-sdk/issues/4284

---------

Signed-off-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Andrii <ndk@parity.io>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Joseph Zhao <65984904+programskillforverification@users.noreply.github.com>
Co-authored-by: Nazar Mokrynskyi <nazar@mokrynskyi.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: command-bot <>
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Serban Iorga <serban@parity.io>

Relax requirements for `assign_core` so that it accepts updates for the
last scheduled entry.

Fixes #6102

---------

Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: GitHub Action <action@github.com>

closes #1890 

### Overview 

Introduces similar checker struct to `CheckTraitDecls` in
`decl_runtime_apis!` - `CheckTraitImpls`. Overrides
`visit::visit_type_path` to detect usage of `Self` as a type argument
within the scope of `impl_runtime_apis!`.

**Note**: only prevents the usage of `Self` as a type argument in an
angle bracket `<>`, as it is the only use case that fails to compile.
For example, the code
[below](https://github.com/paritytech/polkadot-sdk/blob/master/cumulus/parachains/runtimes/bridge-hubs/bridge-hub-westend/src/lib.rs#L1002)
compiles fine:

```rs
impl BridgeMessagesConfig<WithBridgeHubRococoMessagesInstance> for Runtime {
  fn is_relayer_rewarded(relayer: &Self::AccountId) -> bool {
	  let bench_lane_id = <Self as BridgeMessagesConfig<WithBridgeHubRococoMessagesInstance>>::bench_lane_id();
	  // ...
```

### Result

Given a block of code like this:

```rs
impl_runtime_apis! {
	impl apis::Core<Block> for Runtime {
		fn initialize_block(header: &HeaderFor<Self>) -> ExtrinsicInclusionMode {
			let _: HeaderFor<Self> = header.clone();
			RuntimeExecutive::initialize_block(header)
		}
		// ...
         }
// ...
```

<details open>

<summary>Output:</summary>

```bash
$ cargo build --release -p minimal-template-node
  error: `Self` can not be used as type argument in the scope of `impl_runtime_apis!`. Use `Runtime` instead.
     --> /polkadot-sdk/templates/minimal/runtime/src/lib.rs:133:11
      |
  133 |             let _: HeaderFor<Self> = header.clone();
      |                    ^^^^^^^^^^^^^^^

  error: `Self` can not be used as type argument in the scope of `impl_runtime_apis!`. Use `Runtime` instead.
     --> /polkadot-sdk/templates/minimal/runtime/src/lib.rs:132:32
      |
  132 |         fn initialize_block(header: &HeaderFor<Self>) -> ExtrinsicInclusionMode {
```

</details>

---------

Co-authored-by: Pavlo Khrystenko <p.khrystenko@gmail.com>
Co-authored-by: Pavlo Khrystenko <45178695+pkhry@users.noreply.github.com>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>

Part of https://github.com/paritytech/polkadot-sdk/issues/5047

TODO:

- [x] prdoc
- [x] fix/add tests

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

A quick fix to the step that generates a temporary token used in the
pipeline

# Description

Prior to this PR, the metadata for runtime APIs was entirely based on
that generated by `decl_runtime_apis`. It therefore didn't take into
account that `impl_runtime_apis` might implement older versions of APIs
than what has been declared.

This PR filters the returned runtime API metadata to only include
methods actually implemented, and also avoids including methods labelled
with `changed_in` (which the previous code was atempting to do already
but not successfully, owing to the attr being removed prior to the
check).

We also change all version related things to be `u32`s (rather than
VERSION being `u32` and `api_version`s being `u64`) for consistency /
ease of comparison.

A test is added which works with both the `enable-staging-api` feature
in api/tests enabled or disabled, to check all of this.

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: GitHub Action <action@github.com>