- Breaking down the integration-test into multiple tests
- Fix tx hash to use expected keccak-256
- Add option to ethers.js example to connect to westend and use a
private key

---------

Co-authored-by: GitHub Action <action@github.com>

This PR fixes an issue that I discovered using connecting to the RPC via
localhost using cURL, where cURL tries to connect to via ipv6 before
ipv4 when querying `localhost` which messed up the http host filter
whereas it would connect to the address `[::1]::9944 host_header:
localhost:9944` but the ipv6 interface only whitelisted `[::1]:9944`
which this fixes.

So let's whitelist all localhost interfaces to avoid such weird
edge-cases.

### Behavior before this PR

```bash
$ polkadot --chain westend-dev &
$ curl -v \
     -H 'Content-Type: application/json' \
     -d '{"jsonrpc":"2.0","id":"id","method":"system_name"}' \
     http://localhost:9944
* Host localhost:9944 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:9944...
* Connected to localhost (::1) port 9944
> POST / HTTP/1.1
> Host: localhost:9944
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 50
>
< HTTP/1.1 403 Forbidden
< content-type: text/plain
< content-length: 41
< date: Tue, 12 Nov 2024 13:03:49 GMT
<
Provided Host header is not whitelisted.
* Connection #0 to host localhost left intact
```

### Behavior after this PR
```bash
$ polkadot --chain westend-dev &
➜ wasm-tests (update-artifacts-1731284930) ✗ curl -v \
     -H 'Content-Type: application/json' \
     -d '{"jsonrpc":"2.0","id":"id","method":"system_name"}' \
     http://localhost:9944
* Host localhost:9944 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:9944...
* Connected to localhost (::1) port 9944
> POST / HTTP/1.1
> Host: localhost:9944
> User-Agent: curl/8.5.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 50
>
< HTTP/1.1 200 OK
< content-type: application/json; charset=utf-8
< vary: origin, access-control-request-method, access-control-request-headers
< content-length: 54
< date: Tue, 12 Nov 2024 13:02:57 GMT
<
* Connection #0 to host localhost left intact
{"jsonrpc":"2.0","id":"id","result":"Parity Polkadot"}%
```

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: command-bot <>

Added `ExecuteWithOrigin` instruction according to the old XCM RFC 38:
https://github.com/polkadot-fellows/xcm-format/pull/38.

This instruction allows you to descend or clear while going back again.

## TODO
- [x] Implementation
- [x] Unit tests
- [x] Integration tests
- [x] Benchmarks
- [x] PRDoc

## Future work

Modify `WithComputedOrigin` barrier to allow, for example, fees to be
paid with a descendant origin using this instruction.

---------

Signed-off-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Andrii <ndk@parity.io>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Joseph Zhao <65984904+programskillforverification@users.noreply.github.com>
Co-authored-by: Nazar Mokrynskyi <nazar@mokrynskyi.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: command-bot <>

This PR addresses an issue mentioned
[here](https://github.com/paritytech/polkadot-sdk/issues/6424#issuecomment-2467042441).
The problem was that when the prdoc file has two audiences, but only one
description like in
[prdoc_5660](https://github.com/paritytech/polkadot-sdk/blob/master/prdoc/1.16.0/pr_5660.prdoc)
it was ignored by the template.

Very tiny change that helps with debugging of transactions propagation
by referring to the same type alias not only at receiving side, but also
on the sending size for symmetry

# Description

Resolves #6193

This PR introduces `ConstUint` as a replacement for existing constant
getter types like `ConstU8`, `ConstU16`, etc., providing a more flexible
and unified approach.

## Integration

This update is backward compatible, so developers can choose to adopt
`ConstUint` in new implementations or continue using the existing types
as needed.

## Review Notes

`ConstUint` is a convenient alternative to `ConstU8`, `ConstU16`, and
similar types, particularly useful for configuring `DefaultConfig` in
pallets. It enables configuring the underlying integer for a specific
type without the need to update all dependent types, offering enhanced
flexibility in type management.

# Checklist

* [x] My PR includes a detailed description as outlined in the
"Description" and its two subsections above.
* [ ] My PR follows the [labeling requirements](

https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process
) of this project (at minimum one label for `T` required)
* External contributors: ask maintainers to put the right label on your
PR.
* [ ] I have made corresponding changes to the documentation (if
applicable)
* [ ] I have added tests that prove my fix is effective or that my
feature works (if applicable)

Part of:

- #6202.

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Giuseppe Re <giuseppe.re@parity.io>

Adds NoOp implementation for the `Polling` trait and updates benchmarks
in `pallet-ranked-collective`.

---------

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

- Fix bare_eth_transact so that it estimate more precisely the
transaction fee
- Add some context to the build.rs to make it easier to troubleshoot
errors
- Add TransactionBuilder for the RPC tests.
- Improve error message, proxy rpc error from the node and handle
reverted error message
- Add logs in ReceiptInfo

---------

Co-authored-by: GitHub Action <action@github.com>

Migrates pallet-nft-fractionalization to benchmarking v2 syntax.

Part of:
* #6202

---------

Co-authored-by: Giuseppe Re <giuseppe.re@parity.io>
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>

Based on https://github.com/paritytech/polkadot-sdk/pull/3331
Related to https://github.com/paritytech/polkadot-sdk/issues/3268

Implements migrations with customizable block number to relay height
number translation function.

Adds block to relay height migration code for rococo and westend.

---------

Co-authored-by: DavidK <davidk@parity.io>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

Closes #2082

change send xcm to use `xcm::executor::FeeManager` to determine if the
sender should be charged.

I had to change the `FeeManager` of the penpal config to ensure the same
test behaviour as before. For the other tests, I'm using the
`FeeManager` from the `xcm::executor::FeeManager` as this one is used to
check if the fee can be waived on the charge fees method.

---------

Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: GitHub Action <action@github.com>

There is no need to verify in the `pallet-membership` benchmark that the
`MemembershipChanged` implementation works as the pallet thinks it
should work. If you for example set it to `()`, `get_prime()` will
always return `None`.

TLDR: Remove the checks of `MembershipChanged` in the benchmarks to
support any kind of implementation.

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>

This PR adds size-limits to the fork-aware transaction pool.

**Review Notes**
- Existing
[`TrackedMap`](https://github.com/paritytech/polkadot-sdk/blob/58fd5ae4/substrate/client/transaction-pool/src/graph/tracked_map.rs#L33-L41)
is used in internal mempool to track the size of extrinsics:

https://github.com/paritytech/polkadot-sdk/blob/58fd5ae4

/substrate/client/transaction-pool/src/graph/tracked_map.rs#L33-L41

- In this PR, I also removed the logic that kept transactions in the
`tx_mem_pool` if they were immediately dropped by the views. Initially,
I implemented this as an improvement: if there was available space in
the _mempool_ and all views dropped the transaction upon submission, the
transaction would still be retained in the _mempool_.

However, upon further consideration, I decided to remove this
functionality to reduce unnecessary complexity. Now, when all views drop
a transaction during submission, it is automatically rejected, with the
`submit/submit_and_watch` call returning `ImmediatelyDropped`.


Closes: #5476

---------

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>

# Description

This seems to be an old artifact of the long closed
https://github.com/paritytech/substrate/issues/6827 that I noticed when
working on related code earlier.

## Integration

`NetworkStarter` was removed, simply remove its usage:
```diff
-let (network, system_rpc_tx, tx_handler_controller, start_network, sync_service) =
+let (network, system_rpc_tx, tx_handler_controller, sync_service) =
    build_network(BuildNetworkParams {
...
-start_network.start_network();
```

## Review Notes

Changes are trivial, the only reason for this to not be accepted is if
it is desired to not start network automatically for whatever reason, in
which case the description of network starter needs to change.

# Checklist

* [x] My PR includes a detailed description as outlined in the
"Description" and its two subsections above.
* [ ] My PR follows the [labeling requirements](

https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process
) of this project (at minimum one label for `T` required)
* External contributors: ask maintainers to put the right label on your
PR.

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>

Kudos to @EclesioMeloJunior for noticing it 

Also added a regression test for it. The existing unit test was
exercising only the case where the full chain is reverted

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>

Part of:
#6202

---------

Co-authored-by: Giuseppe Re <giuseppe.re@parity.io>
Co-authored-by: GitHub Action <action@github.com>

This PR introduces usage of `build_struct_json_patch` macro in all
runtimes (also guides) within the code base.

It also fixes macro to support _field init shorthand_, and _Struct
Update_ syntax which were missing in original implementation.

Follow up of #5700 and #5813

This PR exposes additional syncing types to facilitate the development
of a custom syncing strategy based on the existing Polkadot syncing
strategy. Specifically, my goal is to isolate the state sync and chain
sync components, allowing the state to be downloaded from the P2P
network without running a full regular Substrate node. I also need to
intercept the state responses during the state sync process.

The newly exposed types are necessary to implement this custom syncing
strategy.

Part of:

- #6202.

---------

Co-authored-by: Giuseppe Re <giuseppe.re@parity.io>
Co-authored-by: GitHub Action <action@github.com>

Fixes https://github.com/paritytech/polkadot-sdk/issues/6172

# Description

Implemented benchmarks for Notifications and RequestResponse protocols
with libp2p implementation. These benchmarks allow us to monitor
regressions and implement fixes before they are observed in real chain.
In the future, they can be used for targeted optimizations of litep2p
compared to libp2p.

Part of https://github.com/paritytech/polkadot-sdk/issues/5220

Next steps:
- Add benchmarks for litep2p implementation
- Optimize load to get better results
- Add benchmarks to CI to catch regressions



## Integration

Benchmarks don't affect downstream projects.

---------

Co-authored-by: alvicsam <alvicsam@gmail.com>
Co-authored-by: GitHub Action <action@github.com>

# Description

E2E tests to Polkadot/Kusama's people chains (in
https://github.com/open-web3-stack/polkadot-ecosystem-tests/pull/63)
revealed that 2 of the identity pallet's extrinsics did not emit events
in case of success:
* `pallet_identity::rename_sub`, and
* `pallet_identity::set_subs`

This PR fixes that.

## Integration

Other than 2 extrinsics emiting an event when previously they did not,
no other behavior in pallets/extrinsics was modified, so no integration
is needed.

## Review Notes

N/A

This PR changes the behavior of `json_patch::merge` function which no
longer removes any keys from the base JSON object.

fixes: #6306

---------

Co-authored-by: GitHub Action <action@github.com>

This PR has small addition to the db package version. As `cargodeb`
takes the version from the `*.toml` file, this PR adds an extra flag to
the `cargodeb` command so that the version of the deb package matches
the `polkadot` node version.

Fixes https://github.com/paritytech/polkadot-sdk/issues/5530

This PR introduces the removal of backing jobs that have been back
pressured for longer than `allowedAncestryLen`, as these candidates are
no longer viable.

It is reasonable to expect a result for a backing job execution within
`allowedAncestryLen` blocks. Therefore, we set the job TTL as a relay
block number and synchronize the validation host by sending activated
leaves.

---------

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>

This PR ensures that external addresses with different PeerIDs are not
propagated to the higher layer of the network code.

While at it, this ensures that libp2p only adds the `/p2p/peerid` part
to the discovered address if it does not contain it already.

This is a followup from:
- https://github.com/paritytech/polkadot-sdk/pull/6298

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>

fixes https://github.com/paritytech/polkadot-sdk/issues/5966

---------

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

# Description

This is a continuation of
https://github.com/paritytech/polkadot-sdk/pull/5666 that finally fixes
https://github.com/paritytech/polkadot-sdk/issues/5333.

This should allow developers to create custom syncing strategies or even
the whole syncing engine if they so desire. It also moved syncing engine
creation and addition of corresponding protocol outside
`build_network_advanced` method, which is something Bastian expressed as
desired in
https://github.com/paritytech/polkadot-sdk/issues/5#issuecomment-1700816458

Here I replaced strategy-specific types and methods in `SyncingStrategy`
trait with generic ones. Specifically `SyncingAction` is now used by all
strategies instead of strategy-specific types with conversions.
`StrategyKey` was an enum with a fixed set of options and now replaced
with an opaque type that strategies create privately and send to upper
layers as an opaque type. Requests and responses are now handled in a
generic way regardless of the strategy, which reduced and simplified
strategy API.

`PolkadotSyncingStrategy` now lives in its dedicated module (had to edit
.gitignore for this) like other strategies.

`build_network_advanced` takes generic `SyncingService` as an argument
alongside with a few other low-level types (that can probably be
extracted in the future as well) without any notion of specifics of the
way syncing is actually done. All the protocol and tasks are created
outside and not a part of the network anymore. It still adds a bunch of
protocols like for light client and some others that should eventually
be restructured making `build_network_advanced` just building generic
network and not application-specific protocols handling.

## Integration

Just like https://github.com/paritytech/polkadot-sdk/pull/5666
introduced `build_polkadot_syncing_strategy`, this PR introduces
`build_default_block_downloader`, but for convenience and to avoid
typical boilerplate a simpler high-level function
`build_default_syncing_engine` is added that will take care of creating
typical block downloader, syncing strategy and syncing engine, which is
what most users will be using going forward. `build_network` towards the
end of the PR was renamed to `build_network_advanced` and
`build_network`'s API was reverted to
pre-https://github.com/paritytech/polkadot-sdk/pull/5666, so most users
will not see much of a difference during upgrade unless they opt-in to
use new API.

## Review Notes

For `StrategyKey` I was thinking about using something like private type
and then storing `TypeId` inside instead of a static string in it, let
me know if that would preferred.

The biggest change happened to requests that different strategies make
and how their responses are handled. The most annoying thing here is
that block response decoding, in contrast to all other responses, is
dependent on request. This meant request had to be sent throughout the
system. While originally `Response` was `Vec<u8>`, I didn't want to
re-encode/decode request and response just to fit into that API, so I
ended up with `Box<dyn Any + Send>`. This allows responses to be truly
generic and each strategy will know how to downcast it back to the
concrete type when handling the response.

Import queue refactoring was needed to move `SyncingEngine` construction
out of `build_network` that awkwardly implemented for `SyncingService`,
but due to `&mut self` wasn't usable on `Arc<SyncingService>` for no
good reason. `Arc<SyncingService>` itself is of course useless, but
refactoring to replace it with just `SyncingService` was unfortunately
rejected in https://github.com/paritytech/polkadot-sdk/pull/5454

As usual I recommend to review this PR as a series of commits instead of
as the final diff, it'll make more sense that way.

# Checklist

* [x] My PR includes a detailed description as outlined in the
"Description" and its two subsections above.
* [x] My PR follows the [labeling requirements](

https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process
) of this project (at minimum one label for `T` required)
* External contributors: ask maintainers to put the right label on your
PR.
* [x] I have made corresponding changes to the documentation (if
applicable)

This PR fixes the `0002-validators-warp-sync` zombienet test by matching
for a specific error case.

The following PR introduced a new error for validators that don't have
public addresses available for publishing in the DHT:
- https://github.com/paritytech/polkadot-sdk/pull/6298
The log line was moved from `eprintln!("Warning: ...` in sc-cli to the
authority discovery where it is printed properly as an error.


cc @paritytech/sdk-node

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

https://github.com/paritytech/polkadot-sdk/issues/6343

Part of:

- #6202.

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Guillaume Thiolliere <guillaume.thiolliere@parity.io>
Co-authored-by: Giuseppe Re <giuseppe.re@parity.io>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.30 to
0.3.31.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/rust-lang/futures-rs/releases">futures's
releases</a>.</em></p>
<blockquote>
<h2>0.3.31</h2>
<ul>
<li>Fix use after free of task in <code>FuturesUnordered</code> when
dropped future panics (<a
href="https://redirect.github.com/rust-lang/futures-rs/issues/2886">#2886</a>)</li>
<li>Fix soundness bug in <code>task::waker_ref</code> (<a
href="https://redirect.github.com/rust-lang/futures-rs/issues/2830">#2830</a>)
This is a breaking change but allowed because it is soundness bug
fix.</li>
<li>Fix bugs in <code>AsyncBufRead::read_line</code> and
<code>AsyncBufReadExt::lines</code> (<a
href="https://redirect.github.com/rust-lang/futures-rs/issues/2884">#2884</a>)</li>
<li>Fix parsing issue in
<code>select!</code>/<code>select_biased!</code> (<a
href="https://redirect.github.com/rust-lang/futures-rs/is...

closes https://github.com/paritytech/polkadot-sdk/issues/6358

Adds the following runtime api to pallet-nomination-pools.
`pool_accounts(pool_id)`: Returns `(bonded_account, reward_account)`
associated with the `pool_id`.

cc: @rossbulat



---------

Co-authored-by: command-bot <>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>

# Context

This PR aims to introduce XCMv5, for now it's in progress and will be
updated over time.
This branch will serve as a milestone branch for merging in all features
we want to add to XCM, roughly outlined
[here](https://github.com/polkadot-fellows/xcm-format/issues/60).
More features could be added.

## TODO
- [x] Migrate foreign assets from v3 to v4
- [x] Setup v5 skeleton
- [x] Remove XCMv2
- [x] https://github.com/paritytech/polkadot-sdk/pull/5390
- [x] https://github.com/paritytech/polkadot-sdk/pull/5585
- [x] https://github.com/paritytech/polkadot-sdk/pull/5420
- [x] https://github.com/paritytech/polkadot-sdk/pull/5876
- [x] https://github.com/paritytech/polkadot-sdk/pull/5971
- [x] https://github.com/paritytech/polkadot-sdk/pull/6148
- [x] https://github.com/paritytech/polkadot-sdk/pull/6228

Fixes #3434 
Fixes https://github.com/paritytech/polkadot-sdk/issues/4190
Fixes https://github.com/paritytech/polkadot-sdk/issues/5209
Fixes https://github.com/paritytech/polkadot-sdk/issues/5241
Fixes https://github.com/paritytech/polkadot-sdk/issues/4284

---------

Signed-off-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Andrii <ndk@parity.io>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Joseph Zhao <65984904+programskillforverification@users.noreply.github.com>
Co-authored-by: Nazar Mokrynskyi <nazar@mokrynskyi.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: command-bot <>
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Serban Iorga <serban@parity.io>

Relax requirements for `assign_core` so that it accepts updates for the
last scheduled entry.

Fixes #6102

---------

Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: GitHub Action <action@github.com>

closes #1890 

### Overview 

Introduces similar checker struct to `CheckTraitDecls` in
`decl_runtime_apis!` - `CheckTraitImpls`. Overrides
`visit::visit_type_path` to detect usage of `Self` as a type argument
within the scope of `impl_runtime_apis!`.

**Note**: only prevents the usage of `Self` as a type argument in an
angle bracket `<>`, as it is the only use case that fails to compile.
For example, the code
[below](https://github.com/paritytech/polkadot-sdk/blob/master/cumulus/parachains/runtimes/bridge-hubs/bridge-hub-westend/src/lib.rs#L1002)
compiles fine:

```rs
impl BridgeMessagesConfig<WithBridgeHubRococoMessagesInstance> for Runtime {
  fn is_relayer_rewarded(relayer: &Self::AccountId) -> bool {
	  let bench_lane_id = <Self as BridgeMessagesConfig<WithBridgeHubRococoMessagesInstance>>::bench_lane_id();
	  // ...
```

### Result

Given a block of code like this:

```rs
impl_runtime_apis! {
	impl apis::Core<Block> for Runtime {
		fn initialize_block(header: &HeaderFor<Self>) -> ExtrinsicInclusionMode {
			let _: HeaderFor<Self> = header.clone();
			RuntimeExecutive::initialize_block(header)
		}
		// ...
         }
// ...
```

<details open>

<summary>Output:</summary>

```bash
$ cargo build --release -p minimal-template-node
  error: `Self` can not be used as type argument in the scope of `impl_runtime_apis!`. Use `Runtime` instead.
     --> /polkadot-sdk/templates/minimal/runtime/src/lib.rs:133:11
      |
  133 |             let _: HeaderFor<Self> = header.clone();
      |                    ^^^^^^^^^^^^^^^

  error: `Self` can not be used as type argument in the scope of `impl_runtime_apis!`. Use `Runtime` instead.
     --> /polkadot-sdk/templates/minimal/runtime/src/lib.rs:132:32
      |
  132 |         fn initialize_block(header: &HeaderFor<Self>) -> ExtrinsicInclusionMode {
```

</details>

---------

Co-authored-by: Pavlo Khrystenko <p.khrystenko@gmail.com>
Co-authored-by: Pavlo Khrystenko <45178695+pkhry@users.noreply.github.com>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>

Part of https://github.com/paritytech/polkadot-sdk/issues/5047

TODO:

- [x] prdoc
- [x] fix/add tests

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

A quick fix to the step that generates a temporary token used in the
pipeline