Backport #5649 into `stable2409` from bkontur.

See the
[documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md)
on how to use this bot.

<!--
  # To be used by other automation, do not modify:
  original-pr-number: #${pull_number}
-->

---------

Co-authored-by: Branislav Kontur <bkontur@gmail.com>

Backport #5655 into `stable2409` from ggwpez.

See the
[documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md)
on how to use this bot.

<!--
  # To be used by other automation, do not modify:
  original-pr-number: #${pull_number}
-->

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Backport of #5509

Co-authored-by: Dónal Murray <donal.murray@parity.io>

Backport #5327 into `stable2409` from bkontur.

See the
[documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md)
on how to use this bot.

<!--
  # To be used by other automation, do not modify:
  original-pr-number: #${pull_number}
-->

Co-authored-by: Branislav Kontur <bkontur@gmail.com>

Backport #5671 into `stable2409` from claravanstaden.

See the
[documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md)
on how to use this bot.

<!--
  # To be used by other automation, do not modify:
  original-pr-number: #${pull_number}
-->

Co-authored-by: Clara van Staden <claravanstaden64@gmail.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>

Backport #5563 into `stable2409` from acatangiu.

See the
[documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md)
on how to use this bot.

<!--
  # To be used by other automation, do not modify:
  original-pr-number: #${pull_number}
-->

Co-authored-by: Adrian Catangiu <adrian@parity.io>

Backport #5747 into `stable2409` from yrong.

See the
[documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md)
on how to use this bot.

<!--
  # To be used by other automation, do not modify:
  original-pr-number: #${pull_number}
-->

Co-authored-by: Ron <yrong1997@gmail.com>

Backport #5594 into `stable2409` from Moliholy.

See the
[documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md)
on how to use this bot.

<!--
  # To be used by other automation, do not modify:
  original-pr-number: #${pull_number}
-->

Co-authored-by: José Molina Colmenero <jose@blockdeep.io>

Backport #5712 into `stable2409` from franciscoaguirre.

Now the XCM emulator has a log every time `execute_with` is called, to know
which chain is being used.
Also, the logs for UMP, DMP, HRMP processing were included in the `xcm` log filter
and changed from showing the message as an array of bytes to a hex string.
This means running the tests with `RUST_LOG=xcm` should give you everything you
need, you can always filter by `RUST_LOG=xcm::hrmp` or any other if you need it.

Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>

Backport #5713 into `stable2409` from bkchr.

See the
[documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md)
on how to use this bot.

<!--
  # To be used by other automation, do not modify:
  original-pr-number: #${pull_number}
-->

Co-authored-by: Bastian Köcher <git@kchr.de>

Backport #3049 into `stable2409` from bgallois.

See the
[documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md)
on how to use this bot.

<!--
  # To be used by other automation, do not modify:
  original-pr-number: #${pull_number}
-->

Co-authored-by: Benjamin Gallois <gallois.benjamin08@gmail.com>

Backport #5580 into `stable2409` from gui1117.

See the
[documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md)
on how to use this bot.

<!--
  # To be used by other automation, do not modify:
  original-pr-number: #${pull_number}
-->

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

As requested here:
https://github.com/paritytech/polkadot-sdk/pull/5688#issuecomment-2352939516

I don't think it need to be backported to 2407, as the issue was not
present there yet.

Backport #5695 into `stable2409` from ggwpez.

See the
[documentation](https://github.com/paritytech/polkadot-sdk/blob/master/docs/BACKPORT.md)
on how to use this bot.

<!--
  # To be used by other automation, do not modify:
  original-pr-number: #${pull_number}
-->

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Co-authored-by: Ron <yrong1997@gmail.com>
Co-authored-by: Clara van Staden <claravanstaden64@gmail.com>
Co-authored-by: Alistair Singh <alistair.singh7@gmail.com>
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>

Backport https://github.com/paritytech/polkadot-sdk/pull/5678 into
stable2409

Co-authored-by: ggwpez <ggwpez@users.noreply.github.com>

Backport #5644 into `stable2409` (cc @bkchr

).

<!--
  # To be used by other automation, do not modify:
  original-pr-number: #${pull_number}
-->

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

This backports https://github.com/paritytech/polkadot-sdk/pull/5127, to
the stable branch.

Unfortunately https://polkadot.subsquare.io/referenda/1051 passed after
the cut-off deadline and I missed the window of getting this PR merged.

The change itself is super low-risk it just prints a new message to
validators that starting with January 2025 the required minimum of
hardware cores will be 8, I see value in getting this in front of the
validators as soon as possible.

Since we did not release things yet and it does not invalidate any QA we
already did, it should be painless to include it in the current release.

(cherry picked from commit a947cb83)

Backport #5632 into `stable2409` (cc @clangenb).

<!--
  # To be used by other automation, do not modify:
  original-pr-number: #${pull_number}
-->

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: clangenb <37865735+clangenb@users.noreply.github.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Backport #5581 into `stable2409` (cc @franciscoaguirre

).

The dry-run shows in `forwarded_xcms` all the messages in the queues
at the time of calling the API.
Each time the API is called, the result could be different.
You could get messages even if you dry-run something that doesn't send
a message, like a `System::remark`.

This commit fixes this by clearing the message queues before doing the
dry-run, so the only messages left are the ones the users of the API actually
care about.

Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>

This backports original PR:
https://github.com/paritytech/polkadot-sdk/pull/5527 to the release
branch


```

The https://github.com/paritytech/polkadot-sdk/issues/5512 has surfaced that we reported a `BestBlock` event for a block not previously reported via `NewBlock`.

This is because of a race between:
- the stream of events that announces new blocks
- `self.client.info().best_block`

It is possible that `client.info()` contains newer information than the information polled from the block stream (that may be lagging).

To mitigate this, instead of relying on the client's info use the last finalized block to emit a new event.

There are two cases when a new best block event is emitted:
- The best block is in the pruned list and is reported immediately
- The best block is not a descendant of the last finalized block 

Closes: https://github.com/paritytech/polkadot-sdk/issues/5512 

Thanks @j...

Allow free Snowbridge consensus updates, if the header interval is
larger than the configured value (set to 32, so once a epoch).

This PR also moves the Rococo Snowbridge pallet config into its own
module.

Original PR: https://github.com/Snowfork/polkadot-sdk/pull/159

----------

Original `pr_5201.prdoc` is present but moved to release dir, ergo
`R0-Silent` for this backport PR.

---------

Co-authored-by: Clara van Staden <claravanstaden64@gmail.com>
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>

Relates to:
https://github.com/paritytech/parity-bridges-common/issues/2451
Closes: https://github.com/paritytech/parity-bridges-common/issues/2500

## Summary

Now, the bridging pallet supports only static lanes, which means lanes
that are hard-coded in the runtime files. This PR fixes that and adds
support for dynamic, also known as permissionless, lanes. This means
that allowed origins (relay chain, sibling parachains) can open and
close bridges (through BridgeHubs) with another bridged (substrate-like)
consensus using just `xcm::Transact` and `OriginKind::Xcm`.

_This PR is based on the migrated code from the Bridges V2
[branch](https://github.com/paritytech/polkadot-sdk/pull/4427) from the
old `parity-bridges-common`
[repo](https://github.com/paritytech/parity-bridges-common/tree/bridges-v2)._

## Explanation

Please read
[bridges/modules/xcm-bridge-hub/src/lib.rs](https://github.com/paritytech/polkadot-sdk/blob/149b0ac2

/bridges/modules/xcm-bridge-hub/src/lib.rs#L17-L136)
to understand how managing bridges works. The basic concepts around
`BridgeId` and `LaneId` are also explained there.


## TODO

- [x] search and fix for comment: `// TODO:(bridges-v2) - most of that
stuff was introduced with free header execution:
https://github.com/paritytech/polkadot-sdk/pull/4102` - more info in the
comment
[bellow](https://github.com/paritytech/polkadot-sdk/pull/4427#issuecomment-2126625043)
- [x] TODO: there's only one impl of `EnsureOrigin<Success = Location>`

## TODO - not blocking review

**benchmarking:**
- [x] regenerate all relevant weights for BH/AH runtimes
- [ ] regenerate default weights for bridging pallets e.g.
`modules/messages/src/weights.rs`
- [ ] add benchmarks for `xcm-bridge-hub` pallet
https://github.com/paritytech/polkadot-sdk/issues/5550

**testing:**
- [ ] add xcm-emulator tests for Rococo/Penpal to Westend/Penpal with
full opening channel and sending/receiving `xcm::Transact`

**migrations:**
- [x] add migrations for BridgeHubRococo/Westend
https://github.com/paritytech/parity-bridges-common/issues/2794 (to be
reusable for P/K bridge)
  - [x] check also storage migration, if needed for pallets
  - [ ] migration for XCM type (optional)
  - [x] migration for static lanes to the dynamic (reuse for fellows)

**investigation:**
- [ ] revisit
https://github.com/paritytech/parity-bridges-common/issues/2380
- [ ] check congestion around `LocalXcmChannelManager` and
`OutboundLanesCongestedSignals` impls -
https://github.com/paritytech/polkadot-sdk/issues/5551
  - to be reusable for polkadot-fellows
- return `report_bridge_status` was remove, so we need to `XcmpQueue`
alternative?

---------

Signed-off-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Svyatoslav Nikolsky <svyatonik@gmail.com>
Co-authored-by: command-bot <>
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>

Bumps [toml](https://github.com/toml-rs/toml) from 0.8.8 to 0.8.12.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/toml-rs/toml/commit/3a777b326b8c91f48c79ea6fc21aabc695f7dc3d"><code>3a777b3</code></a>
chore: Release</li>
<li><a
href="https://github.com/toml-rs/toml/commit/79799052128f3ea64316d3acc4c54e63fc6f285c"><code>7979905</code></a>
docs: Update changelog</li>
<li><a
href="https://github.com/toml-rs/toml/commit/487768d8d7c41b12eb7547cab4e964b3dede002b"><code>487768d</code></a>
Merge pull request <a
href="https://redirect.github.com/toml-rs/toml/issues/703">#703</a> from
epage/overflow</li>
<li><a
href="https://github.com/toml-rs/toml/commit/6987f77649aacc7a6cc2b9cff0d1f22c260f9643"><code>6987f77</code></a>
chore(ci): Run with default opt-level</li>
<li><a
href="https://github.com/toml-rs/toml/commit/21f545d05ca57560485f24cbf78aaf8478a52c5d"><code>21f545d</code></a>
fix(parser): Don't stackoverflow on opt-level=0</li>
<li><a
href="http...

Bumps [clap_complete](https://github.com/clap-rs/clap) from 4.4.0 to
4.5.13.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/clap-rs/clap/releases">clap_complete's
releases</a>.</em></p>
<blockquote>
<h2>v4.5.13</h2>
<h2>[4.5.13] - 2024-07-31</h2>
<h3>Fixes</h3>
<ul>
<li><em>(derive)</em> Improve error message when
<code>#[flatten]</code>ing an optional <code>#[group(skip)]</code></li>
<li><em>(help)</em> Properly wrap long subcommand descriptions in
help</li>
</ul>
<h2>v4.5.12</h2>
<h2>[4.5.12] - 2024-07-31</h2>
<h2>v4.5.10</h2>
<h2>[4.5.10] - 2024-07-23</h2>
<h2>v4.5.9</h2>
<h2>[4.5.9] - 2024-07-09</h2>
<h3>Fixes</h3>
<ul>
<li><em>(error)</em> When defining a custom help flag, be sure to
suggest it like we do the built-in one</li>
</ul>
<h2>v4.5.8</h2>
<h2>[4.5.8] - 2024-06-28</h2>
<h3>Fixes</h3>
<ul>
<li>Reduce extra flushes</li>
</ul>
<h2>v4.5.7</h2>
<h2>[4.5.7] - 2024-06-10</h2>
<h3>Fixes</h3>
<ul>
<li>Clean up error message when too few arguments for
<code>num_args</code></li>
</ul>
<h2>v4.5.6</h2>
<h2>[4.5.6] - 2024-06-06</h2>
<h2>v4.5.4</h2>
<h2>[4.5.4] - 2024-03-25</h2>
<h3>Fixes</h3>
<ul>
<li><em>(derive)</em> Allow non-literal <code>#[arg(id)]</code>
attributes again</li>
</ul>
<h2>v4.5.3</h2>
<h2>[4.5.3] - 2024-03-15</h2>
<h3>Internal</h3>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/clap-rs/clap/blob/master/CHANGELOG.md">clap_complete's
changelog</a>.</em></p>
<blockquote>
<h2>[4.5.13] - 2024-07-31</h2>
<h3>Fixes</h3>
<ul>
<li><em>(derive)</em> Improve error message when
<code>#[flatten]</code>ing an optional <code>#[group(skip)]</code></li>
<li><em>(help)</em> Properly wrap long subcommand descriptions in
help</li>
</ul>
<h2>[4.5.12] - 2024-07-31</h2>
<h2>[4.5.11] - 2024-07-25</h2>
<h2>[4.5.10] - 2024-07-23</h2>
<h2>[4.5.9] - 2024-07-09</h2>
<h3>Fixes</h3>
<ul>
<li><em>(error)</em> When defining a custom help flag, be sure to
suggest it like we do the built-in one</li>
</ul>
<h2>[4.5.8] - 2024-06-28</h2>
<h3>Fixes</h3>
<ul>
<li>Reduce extra flushes</li>
</ul>
<h2>[4.5.7] - 2024-06-10</h2>
<h3>Fixes</h3>
<ul>
<li>Clean up error message when too few arguments for
<code>num_args</code></li>
</ul>
<h2>[4.5.6] - 2024-06-06</h2>
<h2>[4.5.5] - 2024-06-06</h2>
<h3>Fixes</h3>
<ul>
<li>Allow <code>exclusive</code> to override
<code>required_unless_present</code>,
<code>required_unless_present_any</code>,
<code>required_unless_present_all</code></li>
</ul>
<h2>[4.5.4] - 2024-03-25</h2>
<h3>Fixes</h3>
<ul>
<li><em>(derive)</em> Allow non-literal <code>#[arg(id)]</code>
attributes again</li>
</ul>
<h2>[4.5.3] - 2024-03-15</h2>
<h3>Internal</h3>
<ul>
<li><em>(derive)</em> Update <code>heck</code></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/clap-rs/clap/commit/d222ae4cb62d08b4d8f635aa80ddb3c880b82e6e"><code>d222ae4</code></a>
chore: Release</li>
<li><a
href="https://github.com/clap-rs/clap/commit/a8abcb40c5f2628bfa671adf61a090a1bbfbcfa2"><code>a8abcb4</code></a>
docs: Update changelog</li>
<li><a
href="https://github.com/clap-rs/clap/commit/2690e1bdb19df3e4dde7a50fc33b14a3bf6f0b8a"><code>2690e1b</code></a>
Merge pull request <a
href="https://redirect.github.com/clap-rs/clap/issues/5621">#5621</a>
from shannmu/dynamic_valuehint</li>
<li><a
href="https://github.com/clap-rs/clap/commit/7fd7b3e40bd835070253432accf4076bb020beda"><code>7fd7b3e</code></a>
feat(clap_complete): Support to complete custom value of argument</li>
<li><a
href="https://github.com/clap-rs/clap/commit/fc6aaca52b42d0e4ae13805e7480cbb05f63a0ca"><code>fc6aaca</code></a>
Merge pull request <a
href="https://redirect.github.com/clap-rs/clap/issues/5638">#5638</a>
from epage/cargo</li>
<li><a
href="https://github.com/clap-rs/clap/commit/631e54bc715ed2fa53d8457dc273a25b92d3b354"><code>631e54b</code></a>
docs(cookbook): Style cargo plugin</li>
<li><a
href="https://github.com/clap-rs/clap/commit/6fb49d08bb2acfbc2f2aa5f717ccd4a4018ca872"><code>6fb49d0</code></a>
Merge pull request <a
href="https://redirect.github.com/clap-rs/clap/issues/5636">#5636</a>
from gibfahn/styles_const</li>
<li><a
href="https://github.com/clap-rs/clap/commit/6f215eee98c4f73099b0ede2ac62ba019ada24ce"><code>6f215ee</code></a>
refactor(styles): make styles example use a const</li>
<li><a
href="https://github.com/clap-rs/clap/commit/bbb2e6fdde1c724e39c2f2616332310252c12ab8"><code>bbb2e6f</code></a>
test: Add test case for completing custom value of argument</li>
<li><a
href="https://github.com/clap-rs/clap/commit/999071c46dca0367d93f66ecd97b2e3507963284"><code>999071c</code></a>
fix: Change <code>visible</code> to <code>hidden</code></li>
<li>Additional commits viewable in <a
href="https://github.com/clap-rs/clap/compare/clap_complete-v4.4.0...clap_complete-v4.5.13">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=clap_complete&package-manager=cargo&previous-version=4.4.0&new-version=4.5.13)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

There is a race condition when a validator sends its heads to the
collator, but the collator doesn't yet know these heads. Before it is
aware of these heads by importing the block(s), any collation registered
on the collator is not announced to the validators. The collations
aren't advertised, because the collator doesn't know yet that these
heads of the validator are descendants of the collations relay parent.

The solution is to store these unknown heads of the validators and to
handle them when the collator updates its own view.

This improves `sc-service` API by not requiring the whole
`&Configuration`, using specific configuration options instead.
`RpcConfiguration` was also extracted from `Configuration` to group all
RPC options together.

We don't use Substrate's CLI and would rather not use `Configuration`
either, but some key public functions require it even though they
ignored most of the fields anyway.

`RpcConfiguration` is very helpful not just for consolidation of the
fields, but also to finally make RPC optional for our use case, while
Substrate still runs RPC server on localhost even if listening address
is explicitly set to `None`, which is annoying (and I suspect there is a
reason for it, so didn't want to change the default just yet).

While this is a breaking change, most developers will not notice it if
they use higher-level APIs.

Fixes https://github.com/paritytech/polkadot-sdk/issues/2897

---------

Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>

# Context

Fees can already be paid in other assets locally thanks to the Trader
implementations we have.
This doesn't work when sending messages because delivery fees go through
a different mechanism altogether.
The idea is to fix this leveraging the `AssetExchanger` config item
that's able to turn the asset the user wants to pay fees in into the
asset the router expects for delivery fees.

# Main addition

An adapter was needed to use `pallet-asset-conversion` for exchanging
assets in XCM.
This was created in
https://github.com/paritytech/polkadot-sdk/pull/5130.

The XCM executor was modified to use `AssetExchanger` (when available)
to swap assets to pay for delivery fees.

## Limitations

We can only pay for delivery fees in different assets in intermediate
hops. We can't pay in different assets locally. The first hop will
always need the native token of the chain (or whatever is specified in
the `XcmRouter`).
This is a byproduct of using the `BuyExecution` instruction to know
which asset should be used for delivery fee payment.
Since this instruction is not present when executing an XCM locally, we
are left with this limitation.
To illustrate this limitation, I'll show two scenarios. All chains
involved have pools.

### Scenario 1

Parachain A --> Parachain B

Here, parachain A can use any asset in a pool with its native asset to
pay for local execution fees.
However, as of now we can't use those for local delivery fees.
This means transfers from A to B need some amount of A's native token to
pay for delivery fees.

### Scenario 2

Parachain A --> Parachain C --> Parachain B

Here, Parachain C's remote delivery fees can be paid with any asset in a
pool with its native asset.
This allows a reserve asset transfer between A and B with C as the
reserve to only need A's native token at the starting hop.
After that, it could all be pool assets.

## Future work

The fact that delivery fees go through a totally different mechanism
results in a lot of bugs and pain points.
Unfortunately, this is not so easy to solve in a backwards compatible
manner.
Delivery fees will be integrated into the language in future XCM
versions, following
https://github.com/polkadot-fellows/xcm-format/pull/53.

Old PR: https://github.com/paritytech/polkadot-sdk/pull/4375.