This PR introduces a feature that allows to optionally enable using the
full PoV size.

Technically, we're ready to enable it by default, but as corresponding
runtime changes have not been propagated to the system parachain
runtimes yet, doing so could put them at risk. On the other hand, there
are teams that could benefit from it right now, and it makes no sense
for them to wait for the fellowship release and everything.

---------

Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>

`validity_vote` function is also called from `import_statement` not only
from `import_candidate`. The proof still holds if we assume seconded
statements always come before valid statements, which should be the case
but is not actually enforced by this module at all. On top of this the
proof is not local.

Co-authored-by: eskimor <eskimor@no-such-url.com>

There are numerous reasons for invalid imports, most of them would
likely be caused by bugs. On the other side, dispute distribution
handles all connections fairly, thus there is little harm in keeping a
problematic connection open.

---------

Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: ordian <write@reusable.software>

https://github.com/paritytech/polkadot-sdk/commit/03c45b91



---------

Co-authored-by: Gonçalo Pestana <g6pestana@gmail.com>

Bumps the known_good_semver group with 1 update:
[serde_json](https://github.com/serde-rs/json).

Updates `serde_json` from 1.0.120 to 1.0.121
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/serde-rs/json/releases">serde_json's
releases</a>.</em></p>
<blockquote>
<h2>v1.0.121</h2>
<ul>
<li>Optimize position search in error path (<a
href="https://redirect.github.com/serde-rs/json/issues/1160">#1160</a>,
thanks <a

href="https://github.com/purplesyringa"><code>@​purplesyringa</code></a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a

href="https://github.com/serde-rs/json/commit/eca2658a22cb39952783cb6914eb18242659f66a"><code>eca2658</code></a>
Release 1.0.121</li>
<li><a

href="https://github.com/serde-rs/json/commit/b0d678cfb473386830d559b6ab255d9e21ba39c5"><code>b0d678c</code></a>
Merge pull request <a
href="https://redirect.github.com/serde-rs/json/issues/1160">#1160</a>
from iex-rs/efficient-position</li>
<li><a

href="https://github.com/serde-rs/json/commit/b1edc7d13f72880fd0ac569403a409e5f7961d5f"><code>b1edc7d</code></a>
Optimize position search in error path</li>
<li><a

href="https://github.com/serde-rs/json/commit/40dd7f5e862436f02471fe076f3486c55e472bc2"><code>40dd7f5</code></a>
Merge pull request <a
href="https://redirect.github.com/serde-rs/json/issues/1159">#1159</a>
from iex-rs/fix-recursion</li>
<li><a

href="https://github.com/serde-rs/json/commit/6a306e6ee9f47f3b37088217ffe3ebe9bbb54e5a"><code>6a306e6</code></a>
Move call to tri! out of check_recursion!</li>
<li><a

href="https://github.com/serde-rs/json/commit/3f1c6de4af28b1f6c5100da323f2bffaf7c2083f"><code>3f1c6de</code></a>
Ignore byte_char_slices clippy lint in test</li>
<li><a

href="https://github.com/serde-rs/json/commit/3fd6f5f49dc1c732d9b1d7dfece4f02c0d440d39"><code>3fd6f5f</code></a>
Merge pull request <a
href="https://redirect.github.com/serde-rs/json/issues/1153">#1153</a>
from dpathakj/master</li>
<li><a

href="https://github.com/serde-rs/json/commit/fcb5e83e44abe0f9c27c755a240a6ad56312c090"><code>fcb5e83</code></a>
Correct documentation URL for Value's Index impl.</li> <li>See full diff
in <a

href="https://github.com/serde-rs/json/compare/v1.0.120...v1.0.121">compare
view</a></li>
</ul>
</details>
<br />

[![Dependabot compatibility

score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=serde_json&package-manager=cargo&previous-version=1.0.120&new-version=1.0.121)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary> <br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions

</details>

---------

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Bastian Köcher <info@kchr.de>

Backports #5273 & #5281

---------

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>

Backport https://github.com/paritytech/polkadot-sdk/pull/4937 on the
stable release

Backports https://github.com/paritytech/polkadot-sdk/pull/5321 on the
stable release

Fix compilation issue for Rust stable 1.80 and nightly 1.82

`cargo update -p time`

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

This PR adds the possibility to set the docker stable release tag as an
input parameter to the produced docker images, so that it matches with
the release version

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Jun Jiang <jasl9187@hotmail.com>

Backport #4812 and #5189 to fix the CI.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Morgan Adamiec <morgan@parity.io>

Fix https://github.com/paritytech/polkadot-sdk/issues/3487.

---------

Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com>
Co-authored-by: Dmitry Lavrenov <39522748+dmitrylavrenov@users.noreply.github.com>

[stable2407 backport] runtime: make the candidate relay parent progression check more stric… (#5157)

Backports https://github.com/paritytech/polkadot-sdk/pull/5113 on top of
stable2407

This PR backports hotfix from #5103

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>

This release includes: https://github.com/libp2p/rust-libp2p/pull/5482

Which fixes substrate node crashing with libp2p trace:

```
 0: sp_panic_handler::set::{{closure}}
   1: std::panicking::rust_panic_with_hook
   2: std::panicking::begin_panic::{{closure}}
   3: std::sys_common::backtrace::__rust_end_short_backtrace
   4: std::panicking::begin_panic
   5: <quicksink::SinkImpl<S,F,T,A,E> as futures_sink::Sink<A>>::poll_ready
   6: <rw_stream_sink::RwStreamSink<S> as futures_io::if_std::AsyncWrite>::poll_write
   7: <libp2p_noise::io::framed::NoiseFramed<T,S> as futures_sink::Sink<&alloc::vec::Vec<u8>>>::poll_ready
   8: <libp2p_noise::io::Output<T> as futures_io::if_std::AsyncWrite>::poll_write
   9: <yamux::frame::io::Io<T> as futures_sink::Sink<yamux::frame::Frame<()>>>::poll_ready
  10: yamux::connection::Connection<T>::poll_next_inbound
  11: <libp2p_yamux::Muxer<C> as libp2p_core::muxing::StreamMuxer>::poll
  12: <libp2p_core::muxing::boxed::Wrap<T> as libp2p_core::muxing::StreamMuxer>::poll
  13: <libp2p_core::muxing::boxed::Wrap<T> as libp2p_core::muxing::StreamMuxer>::poll
  14: libp2p_swarm::connection::pool::task::new_for_established_connection::{{closure}}
  15: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
  16: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  17: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
  18: std::panicking::try
  19: tokio::runtime::task::harness::Harness<T,S>::poll
  20: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  21: tokio::runtime::scheduler::multi_thread::worker::Context::run
  22: tokio::runtime::context::set_scheduler
  23: tokio::runtime::context::runtime::enter_runtime
  24: tokio::runtime::scheduler::multi_thread::worker::run
  25: tokio::runtime::task::core::Core<T,S>::poll
  26: tokio::runtime::task::harness::Harness<T,S>::poll
  27: std::sys_common::backtrace::__rust_begin_short_backtrace
  28: core::ops::function::FnOnce::call_once{{vtable.shim}}
  29: std::sys::pal::unix::thread::Thread::new::thread_start
  30: <unknown>
  31: <unknown>


Thread 'tokio-runtime-worker' panicked at 'SinkImpl::poll_ready called after error.', /home/ubuntu/.cargo/registry/src/index.crates.io-6f17d22bba15001f/quicksink-0.1.2/src/lib.rs:158
```

Closes: https://github.com/paritytech/polkadot-sdk/issues/4934

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

After the merge of #4922 we saw failing zombienet tests with the
following error:
```
2024-07-09 10:30:09 Error applying finality to block (0xb9e1d3d9cb2047fe61667e28a0963e0634a7b29781895bc9ca40c898027b4c09, 56685): UnknownBlock: Header was not found in the database: 0x0000000000000000000000000000000000000000000000000000000000000000    
2024-07-09 10:30:09 GRANDPA voter error: could not complete a round on disk: UnknownBlock: Header was not found in the database: 0x0000000000000000000000000000000000000000000000000000000000000000    
```

[Example](https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6662262)

The crashing situation is warp-sync related. After warp syncing, it can
happen that there are gaps in block ancestry where we don't have the
header. At the same time, the genesis hash is in the set of leaves. In
`displaced_leaves_after_finalizing` we then iterate from the finalized
block backwards until we hit an unknown block, crashing the node.

This PR makes the detection of displaced branches resilient against
unknown block in the finalized block chain.

cc @nazar-pc (github won't let me request a review from you)

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: command-bot <>

This PR contains adjustments of the node release pipelines so that it
will be possible to use those to trigger release actions based on the
`stable` branch.

Previously the whole pipeline of the flows from [creation of the
`rc-tag`](https://github.com/paritytech/polkadot-sdk/blob/master/.github/workflows/release-10_rc-automation.yml)
(v1.15.0-rc1, v1.15.0-rc2, etc) till [the release draft
creation](https://github.com/paritytech/polkadot-sdk/blob/master/.github/workflows/release-30_publish_release_draft.yml)
was triggered on push to the node release branch. As we had the node
release branch and the crates release branch separately, it worked fine.

From now on, as we are switching to the one branch approach, for the
first iteration I would like to keep things simple to see how the new
release process will work with both parts (crates and node) made from
one branch.

Changes made: 

- The first step in the pipeline (rc-tag creation) will be triggered
manually instead of the push to the branch
- The tag version will be set manually from the input instead of to be
taken from the branch name
- Docker image will be additionally tagged as `stable`



Closes: https://github.com/paritytech/release-engineering/issues/214

Resolves https://github.com/paritytech/polkadot-sdk/issues/4468

Gives instructions on how to enable elastic scaling MVP to parachain
teams.

Still a draft because it depends on further changes we make to the
slot-based collator:
https://github.com/paritytech/polkadot-sdk/pull/4097

Parachains cannot use this yet because the collator was not released and
no relay chain network has been configured for elastic scaling yet

Closes https://github.com/paritytech/polkadot-sdk/issues/577

### Changed
- `orchestra` updated to 0.4.0
- `PeerViewChange` sent with high priority and should be processed first
in a queue.
- To count them in tests added tracker to TestSender and TestOverseer.
It acts more like a smoke test though.

### Testing on Versi

The changes were tested on Versi with two objectives:
1. Make sure the node functionality does not change.
2. See how the changes affect performance.

Test setup:
- 2.5 hours for each case
- 100 validators
- 50 parachains
- validatorsPerCore = 2
- neededApprovals = 100
- nDelayTranches = 89
- relayVrfModuloSamples = 50

During the test period, all nodes ran without any crashes, which
satisfies the first objective.

To estimate the change in performance we used ToF charts. The graphs
show that there are no spikes in the top as before. This proves that our
hypothesis is correct.

### Normalized charts with ToF

![image](https://github.com/user-attachments/assets/0d49d0db-8302-4a8c-a557-501856805ff5)
[Before](https://grafana.teleport.parity.io/goto/ZoR53ClSg?orgId=1)


![image](https://github.com/user-attachments/assets/9cc73784-7e45-49d9-8212-152373c05880)
[After](https://grafana.teleport.parity.io/goto/6ux5qC_IR?orgId=1)

### Conclusion

The prioritization of subsystem messages reduces the ToF of the
networking subsystem, which helps faster propagation of gossip messages.

cc https://github.com/paritytech/ci_cd/issues/939