Add cargo +nightly fmt

We want to add linter to the entire node based on a template. Just like
`cargo clippy -- --deny missing_docs`. And we have the error (pasted at
the end).
The dummy crate is used to test whether the WASM toolchain is installed
and working as expected. And for some reason this dummy crate included
as a target for the linter. I added an attribute to pass the check.
```
note: To improve backtraces for build dependencies, set the CARGO_PROFILE_DEV_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation.

Caused by:
  process didn't exit successfully: `/Users/hrls/src/atleta/target/debug/build/atleta-runtime-b15153eff20cbe96/build-script-build` (exit status: 1)
  --- stderr
  Rust WASM target for toolchain stable-aarch64-apple-darwin is not properly installed; please install it!

  Further error information:
  ------------------------------------------------------------
     Compiling dummy-crate v1.0.0 (/var/folders/h1/_5gdnk8901n959lc28fwx8400000gn/T/.tmpUQCLaV)
  error: missing documentation for the crate
   --> src/main.rs:1:1
    |
  1 | fn main() {}
    | ^^^^^^^^^^^^
    |
    = note: requested on the command line with `-D missing-docs`

  error: could not compile `dummy-crate` (bin "dummy-crate") due to 1 previous error
  ------------------------------------------------------------


```

Co-authored-by: Bastian Köcher <git@kchr.de>

Closes #4951 (hopefully)

@alvicsam can you please check if it passes in the new environment?

This release includes: https://github.com/libp2p/rust-libp2p/pull/5482

Which fixes substrate node crashing with libp2p trace:

```
 0: sp_panic_handler::set::{{closure}}
   1: std::panicking::rust_panic_with_hook
   2: std::panicking::begin_panic::{{closure}}
   3: std::sys_common::backtrace::__rust_end_short_backtrace
   4: std::panicking::begin_panic
   5: <quicksink::SinkImpl<S,F,T,A,E> as futures_sink::Sink<A>>::poll_ready
   6: <rw_stream_sink::RwStreamSink<S> as futures_io::if_std::AsyncWrite>::poll_write
   7: <libp2p_noise::io::framed::NoiseFramed<T,S> as futures_sink::Sink<&alloc::vec::Vec<u8>>>::poll_ready
   8: <libp2p_noise::io::Output<T> as futures_io::if_std::AsyncWrite>::poll_write
   9: <yamux::frame::io::Io<T> as futures_sink::Sink<yamux::frame::Frame<()>>>::poll_ready
  10: yamux::connection::Connection<T>::poll_next_inbound
  11: <libp2p_yamux::Muxer<C> as libp2p_core::muxing::StreamMuxer>::poll
  12: <libp2p_core::muxing::boxed::Wrap<T> as libp2p_core::muxing::StreamMuxer>::poll
  13: <libp2p_core::muxing::boxed::Wrap<T> as libp2p_core::muxing::StreamMuxer>::poll
  14: libp2p_swarm::connection::pool::task::new_for_established_connection::{{closure}}
  15: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
  16: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  17: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
  18: std::panicking::try
  19: tokio::runtime::task::harness::Harness<T,S>::poll
  20: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  21: tokio::runtime::scheduler::multi_thread::worker::Context::run
  22: tokio::runtime::context::set_scheduler
  23: tokio::runtime::context::runtime::enter_runtime
  24: tokio::runtime::scheduler::multi_thread::worker::run
  25: tokio::runtime::task::core::Core<T,S>::poll
  26: tokio::runtime::task::harness::Harness<T,S>::poll
  27: std::sys_common::backtrace::__rust_begin_short_backtrace
  28: core::ops::function::FnOnce::call_once{{vtable.shim}}
  29: std::sys::pal::unix::thread::Thread::new::thread_start
  30: <unknown>
  31: <unknown>


Thread 'tokio-runtime-worker' panicked at 'SinkImpl::poll_ready called after error.', /home/ubuntu/.cargo/registry/src/index.crates.io-6f17d22bba15001f/quicksink-0.1.2/src/lib.rs:158
```

Closes: https://github.com/paritytech/polkadot-sdk/issues/4934

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

After the merge of #4922 we saw failing zombienet tests with the
following error:
```
2024-07-09 10:30:09 Error applying finality to block (0xb9e1d3d9cb2047fe61667e28a0963e0634a7b29781895bc9ca40c898027b4c09, 56685): UnknownBlock: Header was not found in the database: 0x0000000000000000000000000000000000000000000000000000000000000000    
2024-07-09 10:30:09 GRANDPA voter error: could not complete a round on disk: UnknownBlock: Header was not found in the database: 0x0000000000000000000000000000000000000000000000000000000000000000    
```

[Example](https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6662262

)

The crashing situation is warp-sync related. After warp syncing, it can
happen that there are gaps in block ancestry where we don't have the
header. At the same time, the genesis hash is in the set of leaves. In
`displaced_leaves_after_finalizing` we then iterate from the finalized
block backwards until we hit an unknown block, crashing the node.

This PR makes the detection of displaced branches resilient against
unknown block in the finalized block chain.

cc @nazar-pc (github won't let me request a review from you)

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: command-bot <>

This PR contains adjustments of the node release pipelines so that it
will be possible to use those to trigger release actions based on the
`stable` branch.

Previously the whole pipeline of the flows from [creation of the
`rc-tag`](https://github.com/paritytech/polkadot-sdk/blob/master/.github/workflows/release-10_rc-automation.yml)
(v1.15.0-rc1, v1.15.0-rc2, etc) till [the release draft
creation](https://github.com/paritytech/polkadot-sdk/blob/master/.github/workflows/release-30_publish_release_draft.yml)
was triggered on push to the node release branch. As we had the node
release branch and the crates release branch separately, it worked fine.

From now on, as we are switching to the one branch approach, for the
first iteration I would like to keep things simple to see how the new
release process will work with both parts (crates and node) made from
one branch.

Changes made: 

- The first step in the pipeline (rc-tag creation) will be triggered
manually instead of the push to the branch
- The tag version will be set manually from the input instead of to be
taken from the branch name
- Docker image will be additionally tagged as `stable`



Closes: https://github.com/paritytech/release-engineering/issues/214

Resolves https://github.com/paritytech/polkadot-sdk/issues/4468

Gives instructions on how to enable elastic scaling MVP to parachain
teams.

Still a draft because it depends on further changes we make to the
slot-based collator:
https://github.com/paritytech/polkadot-sdk/pull/4097

Parachains cannot use this yet because the collator was not released and
no relay chain network has been configured for elastic scaling yet

Closes https://github.com/paritytech/polkadot-sdk/issues/577

### Changed
- `orchestra` updated to 0.4.0
- `PeerViewChange` sent with high priority and should be processed first
in a queue.
- To count them in tests added tracker to TestSender and TestOverseer.
It acts more like a smoke test though.

### Testing on Versi

The changes were tested on Versi with two objectives:
1. Make sure the node functionality does not change.
2. See how the changes affect performance.

Test setup:
- 2.5 hours for each case
- 100 validators
- 50 parachains
- validatorsPerCore = 2
- neededApprovals = 100
- nDelayTranches = 89
- relayVrfModuloSamples = 50

During the test period, all nodes ran without any crashes, which
satisfies the first objective.

To estimate the change in performance we used ToF charts. The graphs
show that there are no spikes in the top as before. This proves that our
hypothesis is correct.

### Normalized charts with ToF

![image](https://github.com/user-attachments/assets/0d49d0db-8302-4a8c-a557-501856805ff5)
[Before](https://grafana.teleport.parity.io/goto/ZoR53ClSg?orgId=1)


![image](https://github.com/user-attachments/assets/9cc73784-7e45-49d9-8212-152373c05880)
[After](https://grafana.teleport.parity.io/goto/6ux5qC_IR?orgId=1)

### Conclusion

The prioritization of subsystem messages reduces the ToF of the
networking subsystem, which helps faster propagation of gossip messages.

cc https://github.com/paritytech/ci_cd/issues/939

A baseline for the statement-distribution regression test was set only
in the beginning and now we see that the actual values a bit lower.

<img width="1001" alt="image"
src="https://github.com/user-attachments/assets/40b06eec-e38f-43ad-b437-89eca502aa66">


[Source](https://paritytech.github.io/polkadot-sdk/bench/statement-distribution-regression-bench)

Introduce transient storage, which behaves identically to regular
storage but is kept only in memory and discarded after every
transaction. This functionality is similar to the `TSTORE` and `TLOAD`
operations used in Ethereum.

The following new host functions have been introduced:
`get_transient_storage`
`set_transient_storage`
`take_transient_storage`
`clear_transient_storage`
`contains_transient_storage`
Note: These functions are declared as `unstable` and thus are not
activated.

---------

Co-authored-by: command-bot <>
Co-authored-by: PG Herveou <pgherveou@gmail.com>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>

This is part of the work to further optimize the approval subsystems, if
you want to understand the full context start with reading
https://github.com/paritytech/polkadot-sdk/pull/4849#issue-2364261568

,
however that's not necessary, as this change is self-contained and nodes
would benefit from it regardless of subsequent changes landing or not.

While testing with 1000 validators I found out that the logic for
determining the validators an assignment should be gossiped to is taking
a lot of time, because it always iterated through all the peers, to
determine which are X and Y neighbours and to which we should randomly
gossip(4 samples).

This could be actually optimised, so we don't have to iterate through
all peers for each new assignment, by fetching the list of X and Y peer
ids from the topology first and then stopping the loop once we took the
4 random samples.

With this improvements we reduce the total CPU time spent in
approval-distribution with 15% on networks with 500 validators and 20%
on networks with 1000 validators.

## Test coverage:

`propagates_assignments_along_unshared_dimension` and
`propagates_locally_generated_assignment_to_both_dimensions` cover
already logic and they passed, confirm that there is no breaking change.

Additionally, the approval voting benchmark measure the traffic sent to
other peers, so I confirmed that for various network size there is no
difference in the size of the traffic sent to other peers.

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>

As per #3326, removes usage of the `pallet::getter` macro from the
`session` pallet. The syntax `StorageItem::<T, I>::get()` should be used
instead.

Also, adds public functions for compatibility.

NOTE: The `./historical` directory has not been modified.

cc @muraca

polkadot address: 5GsLutpKjbzsbTphebs9Uy4YK6gTN47MAaz6njPktidjR5cp

---------

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>

This will ensure that malicious code can not access other parts of the
project.

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>

The `DhtEvent::ValuePut` was not propagated back to the higher levels.

This PR ensures we'll send the ValuePut event similarly to
`DhtEvent::ValuePutFailed`

### Next Steps
- [ ] A bit more testing

Thanks @alexggh for catching this 

 

cc @paritytech/networking

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

Co-authored-by: Bastian Köcher <git@kchr.de>

## Testing

Both Bridges zombienet tests passed, e.g.:
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6698640
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6698641
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6700072
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6700073

This should remove nearly all usage of `sp-std` except:
- bridge and bridge-hubs
- a few of frames re-export `sp-std`, keep them for now
- there is a usage of `sp_std::Writer`, I don't have an idea how to move
it

Please review proc-macro carefully. I'm not sure I'm doing it the right
way.

Note: need `/bot fmt`

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: command-bot <>

It says `Will be removed after July 2023` but that's not true 



---------

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>

A malicious peer can submit random bytes on transaction protocol.
In this case, the peer is not disconnected or reported back to the
peerstore.

This PR ensures the peer's reputation is properly reported.

Discovered during testing:
- https://github.com/paritytech/polkadot-sdk/pull/4977




cc @paritytech/networking

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

Part of: https://github.com/paritytech/polkadot-sdk/issues/239

Polkadot address: 12GyGD3QhT4i2JJpNzvMf96sxxBLWymz4RdGCxRH5Rj5agKW

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>

Part of https://github.com/paritytech/polkadot-sdk/issues/4334

Added the command sync. Follow up on #4701 

This PR is blocked until a new machine for the sync gets deployed.

## Summary

This PR contains migrated code from the Bridges V2
[branch](https://github.com/paritytech/polkadot-sdk/pull/4427) from the
old `parity-bridges-common`
[repo](https://github.com/paritytech/parity-bridges-common/tree/bridges-v2).
Even though the PR looks large, it does not (or should not) contain any
significant changes (also not relevant for audit).
This PR is a requirement for permissionless lanes, as they were
implemented on top of these changes.

## TODO

- [x] generate fresh weights for BridgeHubs
- [x] run `polkadot-fellows` bridges zombienet tests with actual runtime
1.2.5. or 1.2.6 to check compatibility
-  working, checked with 1.2.8 fellows BridgeHubs
- [x] run `polkadot-sdk` bridges zombienet tests
  -  with old relayer in CI (1.6.5) 
- [x] run `polkadot-sdk` bridges zombienet tests (locally) - with the
relayer based on this branch -
https://github.com/paritytech/parity-bridges-common/pull/3022
- [x] check/fix relayer companion in bridges repo -
https://github.com/paritytech/parity-bridges-common/pull/3022
- [x] extract pruning stuff to separate PR
https://github.com/paritytech/polkadot-sdk/pull/4944

Relates to:
https://github.com/paritytech/parity-bridges-common/issues/2976
Relates to:
https://github.com/paritytech/parity-bridges-common/issues/2451



---------

Signed-off-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Serban Iorga <serban@parity.io>
Co-authored-by: Svyatoslav Nikolsky <svyatonik@gmail.com>
Co-authored-by: command-bot <>

This improves logging in the xcm-executor to have better debugability
when executing a XCM message.

Following PR for https://github.com/paritytech/polkadot-sdk/pull/4941
that removes usage of `sp-std` on templates

`sp-std` crate was proposed to deprecate on
https://github.com/paritytech/polkadot-sdk/issues/2101



@Kianenigma

---------

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

Fixed the mentioned issue:
https://github.com/paritytech/command-bot/issues/113#issuecomment-2222277552

Now it will properly comment when the old bot gets triggered.

Explains one of the annoying parts of FRAME storage that we have seen
multiple times in PBA everyone gets stuck on.

I have not updated the other two templates for now, and only reflected
it in the parachain template. That can happen in a follow-up.

- [x] Update possible answers in SE about the same topic.

---------

Co-authored-by: Serban Iorga <serban@parity.io>
Co-authored-by: command-bot <>

Enabling this feature when building the `polkadot ` crate will lead it
to being enabled for the builtin westend and rococo runtimes. The result
of that is that a merkleized metadata hash will be computed (at some
time cost) in those runtimes, which will allow transactions which
include a hash via the `CheckMetadataHash` extension to work.

The idea is that this is useful for being able to test/experiment with
the `CheckMetadataHash` extension against local nodes.

---------

Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <git@kchr.de>

It was added to v4 and v3 but was missing from v2

The update is tracked by:
https://github.com/paritytech/polkadot-sdk/issues/3699

However, this is not worth doing at this point since it will change in
the future for phase 2 of the implementation.

Still, it's useful to let people know that the information is not the
most up to date.

`polkadot-parachain` simplifications and deduplications

Details in the commit messages. Just copy-pasting the last commit
description since it introduces the biggest changes:

```
    Implement a more structured way to define a node spec
    
    - use traits instead of bounds for `rpc_ext_builder()`,
      `build_import_queue()`, `start_consensus()`
    - add a `NodeSpec` trait for defining the specifications of a node
    - deduplicate the code related to building a node's components /
      starting a node
```

The other changes are much smaller, most of them trivial and are
isolated in separate commits.

Some of the commands needed update for seamless "newbie" Polkadot SDK
templates experience

Fixes #3770 

Added `clear_origin` as an allowed command after commands that load the
holdings register, in the safe xcm builder.

Checklist
- [x] My PR includes a detailed description as outlined in the
"Description" section above
- [x] My PR follows the [labeling
requirements](https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process)
of this project (at minimum one label for T required)
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works (if applicable)

---------

Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
Co-authored-by: gupnik <mail.guptanikhil@gmail.com>

This PR bumps the last time of a reputation update of a peer.
Doing so ensures the peer remains in the peerstore for longer than 1
hour.

Libp2p updates the `last_updated` field as well.

Small summary for the peerstore:
- A: when peers are reported the `last_updated` time is set to current
time (not done before this PR)
- B: peers that were not updated for 1 hour are removed from the
peerstore
- the reputation of the peers is decaying to zero over time
- peers are reported with a reputation change (positive or negative
depending on the behavior)

Because, (A) was not updating the `last_updated` time, we might lose the
reputation of peers that are constantly updated after 1hour because of
(B).

cc @paritytech/networking

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

Co-authored-by: command-bot <>

Now: 

![image](https://github.com/paritytech/polkadot-sdk/assets/5718007/01545872-e21b-48b9-9913-4b940ff3f361)

Before:

![image](https://github.com/paritytech/polkadot-sdk/assets/5718007/09ae9fa4-eca8-43f7-a083-a4555e392610)


We sadly can not remove the `_macro_name`, but this way we at least show
the correct macro plus docs.

This PR backports regular version bumps and prdocs reordering from the
1.14.0 release branch to master

Functions `can_decrease` and `can_increase` do not return successful
consequence results for assets undergoing destruction; instead, they
return the `UnknownAsset` consequence variant.

This update aligns their behavior with similar functions, such as
`reducible_balance`, `increase_balance`, `decrease_balance`, and `burn`,
which return an `AssetNotLive` error for assets in the process of being
destroyed.