Shouldn't matter much, but this is run on every produced block so free
performance

This PR punishes behaviors that deviate from the notification spec. When
a peer misbehaves by writing data on an unidirectional read stream, the
peer is banned and disconnected immediately.

In this PR:
- The `NotificationOutError` is enriched with termination reason and
made publically available for higher levels
- The protocol misbehavior is propagated through the `CloseDesired`
events
- The network behavior of the protocol is responsible for banning the
peer.
- The peer is banned immediately and, as a result, the reputation system
disconnects the malicious / misbehaving peer
- Logs are enriched with protocol names

Closes: https://github.com/paritytech/polkadot-sdk/issues/7722


cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>

This PR makes the litep2p backend the default network backend in Kusama.

We performed a gradual rollout in Kusama by asking validators to
manually switch to litep2p. The rollout went smoothly, with 250
validators running litep2p without issues. This PR represents the next
step in testing the backend at scale.

Thanks to everyone who contributed to making this happen! A special
shoutout to the validators for their prompt support and cooperation 



While at it, the litep2p release is bumped to the latest 0.9.2, which
downgrades a spamming log to debug.


### CLI Testing Done


```
### Kusama without network backend specified
RUST_LOG=info ./target/release/polkadot --chain kusama --pruning=1000 --in-peers 50 --out-peers 50 --sync=warp --detailed-log-output
2025-03-10 14:24:18.503  INFO main sub-libp2p: Running litep2p network backend

### Kusama with libp2p
RUST_LOG=info ./target/release/polkadot --chain kusama --pruning=1000 --in-peers 50 --out-peers 50 --sync=warp --detailed-log-output --network-backend libp2p
INFO main sub-libp2p: Running libp2p network backend

### Kusama with litep2p
RUST_LOG=info ./target/release/polkadot --chain kusama --pruning=1000 --in-peers 50 --out-peers 50 --sync=warp --detailed-log-output --network-backend litep2p
INFO main sub-libp2p: Running litep2p network backend

### Polkadot without network backend specified
RUST_LOG=info ./target/release/polkadot --chain polkadot --pruning=1000 --in-peers 50 --out-peers 50 --sync=warp --detailed-log-output
2025-03-10 14:27:03.762  INFO main sub-libp2p: Running libp2p network backend 

 ```

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>

#### PR Description

This pull request introduces measures to handle finality stalls by :
- notifying outdated transactions with a
[`FinalityTimeout`](https://github.com/paritytech/polkadot-sdk/blob/d821c84d/substrate/client/transaction-pool/api/src/lib.rs#L145-L147)
event.
- removing outdated views from the `view_store`

An item is considered _outdated_ when the difference between its
associated block and the current block exceeds a pre-defined threshold.

#### Note for Reviewers
The core logic is provided in the following small commits:
- `ViewStore`: new method
[`finality_stall_view_cleanup`](https://github.com/paritytech/polkadot-sdk/blob/d821c84d/substrate/client/transaction-pool/src/fork_aware_txpool/view_store.rs#L869-L903)
for removing stale views was added: 64267000
- `ForkAwareTransactionPool`: core logic for tracking finality stalls
added here: 7b37ea6f. Entry point in
[`finality_stall_cleanup`](https://github.com/paritytech/polkadot-sdk/blob/d821c84d/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L1096-L1136)
- Some related renaming was made to better reflect purpose/shorten the
names: 1a3a1284, a511601f. Also new method
[`transactions_finality_timeout`](https://github.com/paritytech/polkadot-sdk/blob/a511601f/substrate/client/transaction-pool/src/fork_aware_txpool/multi_view_listener.rs#L771-L790)
for triggering external events was added for `MultiViewListener`.
- `included_transactions` which basically is mapping `block hash ->
included transactions hashes`, is also used to find to included
transactions.

I also sneaked in some minor improvements:
- fixed per-transaction logging: 1572f721
- `handle_pre_finalized` method was removed, it was some old leftover
which is no longer needed: a6f84ad0

,

closes: #5482

---------

Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>

# Description

Builds up towards addressing #5497 by creating some zombienet-sdk code
infra that can be used to spin regular networks, as described in the
fork aware transaction pool testing setup added here #7100. It will be
used for developing tests against such networks, and to also spawn them
on demand locally through tooling that will be developed in follow ups.

## Integration

Node/runtime developers can run tests based on the zombienet-sdk infra
that spins frequently used networks which can be used for analyzing
behavior of various node related components, like fork aware transaction
pool.

## Review Notes

- Uses ttxt API implemented here:
https://github.com/michalkucharczyk/tx-test-tool/pull/22/files
- currently, only two test scenarios are considered: 10k future & 10k
ready txs are sent to two separate networks - one parachain and one
relaychain, asserting at the end on the finalization of all 20k txs on
both networks.

---------

Signed-off-by: Iulian Barbu <iulian.barbu@parity.io>
Co-authored-by: Javier Viola <363911+pepoviola@users.noreply.github.com>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

The tokio instant timer produced an overflow when the `poll_tick` was
called because the timer period was set to `u64::max`. The period is
reduced to accommodate for the following tokio time addition:


Source code
[tokio/time/interval.rs](https://github.com/tokio-rs/tokio/blob/a2b12bd5799f06e912b32ac05a5ffb5cf1fe31cd/tokio/src/time/interval.rs#L478-L485):
```rust
        let next = if now > timeout + Duration::from_millis(5) {
            self.missed_tick_behavior
                .next_timeout(timeout, now, self.period)
        } else {
            timeout
                .checked_add(self.period)
                .unwrap_or_else(Instant::far_future)
        };
```


Detected by:
https://github.com/paritytech/polkadot-sdk/actions/runs/13648141251/job/38150825582?pr=7790


```
──── TRY 1 STDERR:       sc-network protocol::notifications::tests::conformance::litep2p_disconnects_libp2p_substream
thread 'protocol::notifications::tests::conformance::litep2p_disconnects_libp2p_substream' panicked at std/src/time.rs:417:33:
overflow when adding duration to instant
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::option::expect_failed
   3: <std::time::Instant as core::ops::arith::Add<core::time::Duration>>::add
   4: tokio::time::interval::Interval::poll_tick
   5: sc_network::protocol::notifications::tests::conformance::litep2p_disconnects_libp2p_substream::{{closure}}
   6: tokio::runtime::scheduler::current_thread::Context::enter
   7: tokio::runtime::context::scoped::Scoped<T>::set
   8: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
   9: tokio::runtime::runtime::Runtime::block_on
  10: sc_network::protocol::notifications::tests::conformance::litep2p_disconnects_libp2p_substream
  11: core::ops::function::FnOnce::call_once
```

cc @paritytech/networking

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

This PR handles a case where we called the `poll_next` on an outbound
substream notification to check if the stream is closed. It is entirely
possible that the `poll_next` would return an `io::error`, for example
end of file.

This PR ensures that we make the distinction between unexpected incoming
data, and error originated from `poll_next`.

While at it, the bulk of the PR change propagates the PeerID from the
network behavior, through the notification handler, to the notification
outbound stream for logging purposes.

cc @paritytech/networking 

Part of: https://github.com/paritytech/polkadot-sdk/issues/7722

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

This PR ensures compatibility in terms of expectations between the
libp2p and litep2p network backends at the notification protocol level.

The libp2p node is tested with the `Notification` behavior that contains
the protocol controller, while litep2p is tested at the lowest level API
(without substrate shim layers).

## Notification Behavior

(I) Libp2p protocol controller will eagerly reopen a closed substream,
even if it is the one that closed it:
- When a node (libp2p or litep2p) closes the substream with **libp2p**,
the **libp2p** controller will reopen the substream
- When **libp2p** closes the substream with a node (either litep2p with
no controller or libp2p), the **libp2p** controller will reopen the
substream
- However in this case, libp2p was the one closing the substream
signaling it is no longer interested in communicating with the other
side

(II) Notifications are lost and not reported to the higher level in the
following scenario:
- T0: Node A opens a substream with Node B
- T1: Node A closes the substream or the connection with Node B
- T2: Node B sends a notification to Node A => *notification is lost*
and never reported
- T3: Node B detects the closed substream or connection


## Testing

This PR effectively checks:
- connectivity at the notification level
- litep2p rejecting libp2p substream and keep-alive mechanism
functionality
- libp2p disconnecting libp2p and connection re-establishment (and all
the other permutations)
- idling of connections with active substreams and keep-alive mechanism
is not enforced


Prior work:
- https://github.com/paritytech/polkadot-sdk/pull/7361

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>

`sc-cli` brings rocksdb dependency into frame-benchmarking-cli when used
with `default-features = false`.
This PR makes rocksdb deps optional that sc-cli brings in some of the
crates. I think I covered all the crates that depend on sc-cli but
please let me know if I missed any.

Fixes: https://github.com/paritytech/polkadot-sdk/issues/3793

This PR updates litep2p to version 0.9.1. The yamux config is entirely
removed to mirror the libp2p yamux upstream version.
While at it, I had to bump indexmap and URL as well. 


## [0.9.1] - 2025-01-19

This release enhances compatibility between litep2p and libp2p by using
the latest Yamux upstream version. Additionally, it includes various
improvements and fixes to boost the stability and performance of the
WebSocket stream and the multistream-select protocol.

### Changed

- yamux: Switch to upstream implementation while keeping the controller
API ([#320](https://github.com/paritytech/litep2p/pull/320))
- req-resp: Replace SubstreamSet with FuturesStream
([#321](https://github.com/paritytech/litep2p/pull/321))
- cargo: Bring up to date multiple dependencies
([#324](https://github.com/paritytech/litep2p/pull/324))
- build(deps): bump hickory-proto from 0.24.1 to 0.24.3
([#323](https://github.com/paritytech/litep2p/pull/323))
- build(deps): bump openssl from 0.10.66 to 0.10.70
([#322](https://github.com/paritytech/litep2p/pull/322))

### Fixed

- websocket/stream: Fix unexpected EOF on `Poll::Pending` state
poisoning ([#327](https://github.com/paritytech/litep2p/pull/327))
- websocket/stream: Avoid memory allocations on flushing
([#325](https://github.com/paritytech/litep2p/pull/325))
- multistream-select: Enforce `io::error` instead of empty protocols
([#318](https://github.com/paritytech/litep2p/pull/318))
- multistream: Do not wait for negotiation in poll_close
([#319](https://github.com/paritytech/litep2p/pull/319))

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

Ref https://github.com/paritytech/ci_cd/issues/1107

We mainly need that so that we can finally compile the `pallet_revive`
fixtures on stable. I did my best to keep the commits focused on one
thing to make review easier.

All the changes are needed because rustc introduced more warnings or is
more strict about existing ones. Most of the stuff could just be fixed
and the commits should be pretty self explanatory. However, there are a
few this that are notable:

## `non_local_definitions `

A lot of runtimes to write `impl` blocks inside functions. This makes
sense to reduce the amount of conditional compilation. I guess I could
have moved them into a module instead. But I think allowing it here
makes sense to avoid the code churn.

## `unexpected_cfgs`

The FRAME macros emit code that references various features like `std`,
`runtime-benchmarks` or `try-runtime`. If a create that uses those
macros does not have those features we get this warning. Those were
mostly when ...

#### Overview

This pull request refactors the transaction pool `graph` module by
renaming components for better clarity. The `EventHandler` trait was
introduced to enhance flexibility in handling transaction lifecycle
events. Changes include renaming `graph::Listener` to
`graph::EventDispatcher` and moving certain functionalities from `graph`
to `view` module in order to decouple `graph` from `view`-related
specifics.

This PR does not introduce changes in the logic.

#### Notes for Reviewers
All the changes looks dense at first, but in fact following was done:
- The `graph::Listener` was renamed to
[`graph::EventDispatcher`](https://github.com/paritytech/polkadot-sdk/blob/515cb404/substrate/client/transaction-pool/src/graph/listener.rs#L74C12-L74C27),
to better reflect its role in dispatching transaction-related events
from `ValidatedPool`. The `EventDispatcher` now utilizes the `L:
EventHandler` generic type to handle transaction status events.
- The new
[`EventHandler`](https://github.com/paritytech/polkadot-sdk/blob/515cb404/substrate/client/transaction-pool/src/graph/listener.rs#L34)
trait was introduced to handle transaction lifecycle events, improving
implementation flexibility and providing clearer role descriptions
within the system. Introduction of this trait allowed the removal of
`View` related entities (e.g. streams) from the `ValidatedPool`'s event
dispatcher (previously _listener_).
- The _dropped monitoring_ and _aggregated events_ stream
[functionalities](https://github.com/paritytech/polkadot-sdk/blob/515cb404/substrate/client/transaction-pool/src/fork_aware_txpool/view.rs#L157-L188)
and [related
types](https://github.com/paritytech/polkadot-sdk/blob/515cb404/substrate/client/transaction-pool/src/fork_aware_txpool/view.rs#L112-L121)
were moved from `graph::listener` to the `view` module. The
[`ViewPoolObserver`](https://github.com/paritytech/polkadot-sdk/blob/515cb404

/substrate/client/transaction-pool/src/fork_aware_txpool/view.rs#L128C19-L128C35),
which implements `EventHandler`, now provides the implementation of
streams feeding.
- Fields, arguments, and variables previously named `listener` were
renamed to `event_dispatcher` to align with their purpose and type
naming.
- Various structs such as `Pool` and `ValidatedPool` were updated to
include a generic `L: EventHandler` across the codebase.

---------

Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>

# Description

resolve #6468



# Checklist

* [x] My PR includes a detailed description as outlined in the
"Description" and its two subsections above.
* [x] My PR follows the [labeling requirements](

https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process
) of this project (at minimum one label for `T` required)
* External contributors: ask maintainers to put the right label on your
PR.
* [x] I have made corresponding changes to the documentation (if
applicable)
* [x] I have added tests that prove my fix is effective or that my
feature works (if applicable)

---------

Co-authored-by: command-bot <>

This PR modifies the libp2p networking-specific log targets for granular
control (e.g., just enabling trace for req-resp).

Previously, all logs were outputted to `sub-libp2p` target, flooding the
log messages on busy validators.

### Changes
- Discover: `sub-libp2p::discovery`
- Notification/behaviour: `sub-libp2p::notification::behaviour`
- Notification/handler: `sub-libp2p::notification::handler`
- Notification/service: `sub-libp2p::notification::service`
- Notification/upgrade: `sub-libp2p::notification::upgrade`
- Request response: `sub-libp2p::request-response`

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>

#### Description 
Currently the transaction which is reported as invalid by a block
builder (or `removed_invalid` by other components) is silently skipped.

This PR improves this behavior. The transaction pool `report_invalid`
function now accepts optional error associated with every reported
transaction, and also the optional block hash which provides hints how
reported transaction shall be handled. The following API change is
proposed:

https://github.com/paritytech/polkadot-sdk/blob/8be5ef3e/substrate/client/transaction-pool/api/src/lib.rs#L297-L318
Depending on error, the transaction pool can decide if transaction shall
be removed from the view only or entirely from the pool. Invalid event
will be dispatched if required.


#### Notes for reviewers

- Actual logic of removing invalid txs is implented in
[`ViewStore::report_invalid`](https://github.com/paritytech/polkadot-sdk/blob/0fad26c4/substrate/client/transaction-pool/src/fork_aware_txpool/view_store.rs#L657-L680).
Method's doc explains the flow.
- This PR changes `HashMap` to `IndexMap` in revalidation logic. This is
to preserve the original order of transactions (mainly for purposes of
unit tests).
- This PR solves the problem mentioned in:
https://github.com/paritytech/polkadot-sdk/issues/5477#issuecomment-2598809344
(which can now be resolved). The invalid transactions found during
mempool revalidation are now also removed from the `view_store`. No
dangling invalid transaction shall be left in the pool.
(https://github.com/paritytech/polkadot-sdk/pull/6661/commits/bfec2625)
- The support for dropping invalid transactions reported from the views
was also added. This should never happen, but if for any case all views
will report invalid transcation (which previously was valid) the
transaction will be dropped from the pool
(https://github.com/paritytech/polkadot-sdk/pull/6661/commits/48214a38

).



fixes: #6008, #5477

---------

Co-authored-by: command-bot <>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

This PR implements conformance tests between our network backends
(litep2p and libp2p).

The PR creates a setup for extending testing in the future, while
implementing the following tests:
- connectivity check: Connect litep2p -> libp2p and libp2p -> litep2p
- request response check: Send 32 requests from one backend to the other
- notification check: Send 128 ping pong notifications and 128 from one
backend to the other

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

When debugging stuff, it is useful to see the full hashes and not only
the "short form". This makes it easier to read logs and follow blocks.

---------

Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>

#### Overview

This PR introduces a new mechanism to capture and report metrics related
to timings of transaction lifecycle events, which are currently not
available. By exposing these timings, we aim to augment transaction-pool
reliability dashboards and extend existing Grafana boards.

A new `unknown_from_block_import_txs` metric is also introduced. It
provides the number of transactions in imported block which are not
known to the node's transaction pool. It allows to monitor alignment of
transaction pools across the nodes in the network.

#### Notes for reviewers
- **[Per-event
Metrics](https://github.com/paritytech/polkadot-sdk/blob/8a53992e/substrate/client/transaction-pool/src/fork_aware_txpool/metrics.rs#L84-L105)
Collection**: implemented by[
`EventsMetricsCollector`](https://github.com/paritytech/polkadot-sdk/blob/8a53992e/substrate/client/transaction-pool/src/fork_aware_txpool/metrics.rs#L353-L358)
which allows to capture both submission timestamps and transaction
status updates. An asynchronous
[`EventsMetricsCollectorTask`](https://github.com/paritytech/polkadot-sdk/blob/8a53992e/substrate/client/transaction-pool/src/fork_aware_txpool/metrics.rs#L503-L526)
processes the metrics-related messages sent by the
`EventsMetricsCollector` and reports the timings of transaction statuses
updates to Prometheus. This task implements event[
de-duplication](https://github.com/paritytech/polkadot-sdk/blob/8a53992e/substrate/client/transaction-pool/src/fork_aware_txpool/metrics.rs#L458)
using a `HashMap` of
[`TransactionEventMetricsData`](https://github.com/paritytech/polkadot-sdk/blob/8a53992e/substrate/client/transaction-pool/src/fork_aware_txpool/metrics.rs#L424-L435)
entries which also holds transaction submission timestamps used to
[compute
timings](https://github.com/paritytech/polkadot-sdk/blob/8a53992e/substrate/client/transaction-pool/src/fork_aware_txpool/metrics.rs#L489-L495).
Transaction-related items are removed when transaction's final status is
[reported](https://github.com/paritytech/polkadot-sdk/blob/8a53992e/substrate/client/transaction-pool/src/fork_aware_txpool/metrics.rs#L496).
- Transaction submission timestamp is reusing the timestamp of
`TimedTransactionSource` kept in mempool. It is reported to
`EventsMetricsCollector` in
[`submit_at`](https://github.com/paritytech/polkadot-sdk/blob/8a53992e/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L735)
and
[`submit_and_watch`](https://github.com/paritytech/polkadot-sdk/blob/8a53992e/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L836)
methods of `ForkAwareTxPool`.
- Transaction updates are reported to `EventsMetricsCollector` from
`MultiViewListener`
[task](https://github.com/paritytech/polkadot-sdk/blob/8a53992e/substrate/client/transaction-pool/src/fork_aware_txpool/multi_view_listener.rs#L494).
This allows to gather metrics for _watched_ and _non-watched_
transactions (what enables metrics on non-rpc-enabled collators).
- New metric
([`unknown_from_block_import_txs`](https://github.com/paritytech/polkadot-sdk/blob/8a53992e/substrate/client/transaction-pool/src/fork_aware_txpool/metrics.rs#L59-L60))
allowing checking alignment of pools across the network is
[reported](https://github.com/paritytech/polkadot-sdk/blob/8a53992e/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L1288-L1292)
using new `TxMemPool`
[method](https://github.com/paritytech/polkadot-sdk/blob/8a53992e

/substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs#L605-L611).

fixes: #7355, #7448

---------

Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>

- Closes https://github.com/paritytech/license-scanner/issues/44
- Silent because only comments are changed in the crates.

## What's inside

First, we change the file traversal mechanism from shell globbing to
walking through files which happens inside the `license-scanner` - so it
has knowledge about directory structure and can correlate files with
corresponding Cargo manifest.

Next, added `MIT-0` and `Unlicense` to the allowed list of licenses.
`Unlicense` appears only once across {Substrate,Cumulus,Polkadot} - in
`penpal-runtime`.

Finally, updated headers in files that do not match the corresponding
manifest license.

---------

Co-authored-by: cornholio <0@mcornholio.ru>

This PR ensures that litep2p will keep an idle connection alive for 10
seconds.

The bump from 5 seconds is done to mirror the libp2p behavior and
potentially improve connection stability:

https://github.com/paritytech/polkadot-sdk/blob/a07fb323

/substrate/client/network/src/service.rs#L542-L549

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

#### Description
During 2s block investigation it turned out that
[ForkAwareTxPool::register_listeners](https://github.com/paritytech/polkadot-sdk/blob/master/substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs#L1036)
call takes significant amount of time.
```
register_listeners: at HashAndNumber { number: 12, hash: 0xe9a1...0b1d2 } took 200.041933ms
register_listeners: at HashAndNumber { number: 13, hash: 0x5eb8...a87c6 } took 264.487414ms
register_listeners: at HashAndNumber { number: 14, hash: 0x30cb...2e6ec } took 340.525566ms
register_listeners: at HashAndNumber { number: 15, hash: 0x0450...4f05c } took 405.686659ms
register_listeners: at HashAndNumber { number: 16, hash: 0xfa6f...16c20 } took 477.977836ms
register_listeners: at HashAndNumber { number: 17, hash: 0x5474...5d0c1 } took 483.046029ms
register_listeners: at HashAndNumber { number: 18, hash: 0x3ca5...37b78 } took 482.715468ms
register_listeners: at HashAndNumber { number: 19, hash: 0xbfcc...df254 } took 484.206999ms
register_listeners: at HashAndNumber { number: 20, hash: 0xd748...7f027 } took 414.635236ms
register_listeners: at HashAndNumber { number: 21, hash: 0x2baa...f66b5 } took 418.015897ms
register_listeners: at HashAndNumber { number: 22, hash: 0x5f1d...282b5 } took 423.342397ms
register_listeners: at HashAndNumber { number: 23, hash: 0x7a18...f2d03 } took 472.742939ms
register_listeners: at HashAndNumber { number: 24, hash: 0xc381...3fd07 } took 489.625557ms
```

This PR implements the idea outlined in #7071. Instead of having a
separate listener for every transaction in each view, we now use a
single stream of aggregated events per view, with each stream providing
events for all transactions in that view. Each event is represented as a
tuple: (transaction-hash, transaction-status). This significantly reduce
the time required for `maintain`.

#### Review Notes
- single aggregated stream, provided by the individual view delivers
events in form of `(transaction-hash, transaction-status)`,
- `MultiViewListener` now has a task. This task is responsible for:
- polling the stream map (which consists of individual view's aggregated
streams) and the `controller_receiver` which provides side-channel
[commands](https://github.com/paritytech/polkadot-sdk/blob/2b18e080

/substrate/client/transaction-pool/src/fork_aware_txpool/multi_view_listener.rs#L68-L95)
(like `AddView` or `FinalizeTransaction`) sent from the _transaction
pool_.
- dispatching individual transaction statuses and control commands into
the external (created via API, e.g. over RPC) listeners of individual
transactions,
- external listener is responsible for status handling _logic_ (e.g.
deduplication of events, or ignoring some of them) and triggering
statuses to external world (_this was not changed_).
- level of debug messages was adjusted (per-tx messages shall be
_trace_),

Closes #7071

---------

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

Fixes https://github.com/paritytech/polkadot-sdk/issues/7400

This PR modifies the fatxpool to use tracing instead of log for logging.

closes #5490

Polkadot address: 12GyGD3QhT4i2JJpNzvMf96sxxBLWymz4RdGCxRH5Rj5agKW

---------

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

Instead of using libp2p-provided external address candidates,
susceptible to address translation issues, use litep2p-backend approach
based on confirming addresses observed by multiple peers as external.

Fixes https://github.com/paritytech/polkadot-sdk/issues/7207.

---------

Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>

closes #5978

---------

Co-authored-by: command-bot <>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

Revert https://github.com/paritytech/polkadot-sdk/pull/7011 and replace
it with a backward-compatible solution suitable for backporting to a
release branch.

### Review notes
It's easier to review this PR per commit: the first commit is just a
revert, so it's enough to review only the second one, which is almost a
one-liner.

---------

Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Part of: https://github.com/paritytech/polkadot-sdk/issues/6906

# Description

Closes https://github.com/paritytech/polkadot-sdk/issues/7265.

## Integration

Requires changes in
`https://github.com/polkadot-js/api/packages/{rpc-augment,types-support,types}`
to be visible in Polkadot\Substrate Portal and in other libraries where
we should explicitly state RPC methods.

Accompany PR to `polkadot-js/api`:
https://github.com/polkadot-js/api/pull/6070.

## Review Notes

Please put the right label on my PR.

---------

Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <git@kchr.de>

# Derived Dev Accounts

Resolves https://github.com/paritytech/polkadot-sdk/issues/6040

## Description
This update introduces support for creating an arbitrary number of
developer accounts at the genesis block based on a specified derivation
path. This functionality is gated by the runtime-benchmarks feature,
ensuring it is only enabled during benchmarking scenarios.

### Key Features
- Arbitrary Dev Accounts at Genesis: Developers can now specify any
number of accounts to be generated at genesis using a hard derivation
path.

- Default Derivation Path: If no derivation path is provided (i.e., when
`Option<dev_accounts: (..., None)>` is set to `Some` at genesis), the
system will default to the path `//Sender//{}`.

- No Impact on Total Token Issuance: Developer accounts are excluded
from the total issuance of the token supply at genesis, ensuring they do
not affect the overall balance or token distribution.

polkadot address: 14SRqZTC1d8rfxL8W1tBTnfUBPU23ACFVPzp61FyGf4ftUFg

---------

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

This PR enforces that outbound requests are finished within the
specified protocol timeout.

The stable2412 version running libp2p 0.52.4 contains a bug which does
not track request timeouts properly:
- https://github.com/libp2p/rust-libp2p/pull/5429

The issue has been detected while submitting libp2p -> litep2p requests
in kusama. This aims to check that pending outbound requests have not
timedout. Although the issue has been fixed in libp2p, there might be
other cases where this may happen. For example:
- https://github.com/libp2p/rust-libp2p/pull/5417

For more context see:
https://github.com/paritytech/polkadot-sdk/issues/7076#issuecomment-2596085096


1. Ideally, the force-timeout mechanism in this PR should never be
triggered in production. However, origin/stable2412 occasionally
encounters this issue. When this happens, 2 warnings may be generated:
- one warning introduced by this PR wrt force timeout terminating the
request
- possible one warning when the libp2p decides (if at all) to provide
the response back to substrate (as mentioned by @alexggh
[here](https://github.com/paritytech/polkadot-sdk/pull/7222/files#diff-052aeaf79fef3d9a18c2cfd67006aa306b8d52e848509d9077a6a0f2eb856af7L769)
and
[here](https://github.com/paritytech/polkadot-sdk/pull/7222/files#diff-052aeaf79fef3d9a18c2cfd67006aa306b8d52e848509d9077a6a0f2eb856af7L842)

2. This implementation does not propagate to the substrate service the
`RequestFinished { error: .. }`. That event is only used internally by
substrate to increment metrics. However, we don't have the peer
information available to propagate the event properly when we
force-timeout the request. Considering this should most likely not
happen in production (origin/master) and that we'll be able to extract
information by warnings, I would say this is a good tradeoff for code
simplicity:


https://github.com/paritytech/polkadot-sdk/blob/06e3b5c6

/substrate/client/network/src/service.rs#L1543


### Testing

Added a new test to ensure the timeout is reached properly, even if
libp2p does not produce a response in due time.

I've also transitioned the tests to using `tokio::test` due to a
limitation of
[CI](https://github.com/paritytech/polkadot-sdk/actions/runs/12832055737/job/35784043867)

```
--- TRY 1 STDERR:        sc-network request_responses::tests::max_response_size_exceeded ---
thread 'request_responses::tests::max_response_size_exceeded' panicked at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/time/interval.rs:139:26:
there is no reactor running, must be called from the context of a Tokio 1.x runtime
```



cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>

This PR provides the partial results of the `GetRecord` kademlia query.

This significantly improves the authority discovery records, from ~37
minutes to ~2/3 minutes.
In contrast, libp2p discovers authority records in around ~10 minutes. 

The authority discovery was slow because litep2p provided the records
only after the Kademlia query was completed. A normal Kademlia query
completes in around 40 seconds to a few minutes.
In this PR, partial records are provided as soon as they are discovered
from the network.

### Testing Done

Started a node in Kusama with `--validator` and litep2p backend.
The node discovered 996/1000 authority records in ~ 1 minute 45 seconds.

![Screenshot 2025-01-09 at 12 26
08](https://github.com/user-attachments/assets/b618bf7c-2bba-43a0-a021-4047e854c075)


### Before & After

In this image, on the left side is libp2p, in the middle litep2p without
this PR, on the right litep2p with this PR

![Screenshot 2025-01-07 at 17 57
56](https://github.com/user-attachments/assets/a8d467f7-8dc7-461c-bcff-163b94d01ae8)



Closes: https://github.com/paritytech/polkadot-sdk/issues/7077

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

This PR rejects inbound requests from banned peers (reputation is below
the banned threshold).

This mirrors the request-response implementation from the libp2p side.
I won't expect this to get triggered too often, but we'll monitor this
metric.

While at it, have registered a new inbound failure metric to have
visibility into this.

Discovered during the investigation of:
https://github.com/paritytech/polkadot-sdk/issues/7076#issuecomment-2589613046

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

This PR adds the `(litep2p)` suffix to the agent version (user agent) of
the identify protocol.

The change is needed to gain visibility into network backends and
determine exactly the number of validators that are running litep2p.
Using tools like subp2p-explorer, we can determine if the validators are
running litep2p nodes.

This reflects on the identify protocol:

```
info=Identify {
  protocol_version: Some("/substrate/1.0"),
  agent_version: Some("polkadot-parachain/v1.17.0-967989c5

 (kusama-node-name-01) (litep2p)")
  ...
}
```

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

Higher-priority transactions can now replace lower-priority transactions
even when the internal _tx_mem_pool_ is full.

**Notes for reviewers:**
- The _tx_mem_pool_ now maintains information about transaction
priority. Although _tx_mem_pool_ itself is stateless, transaction
priority is updated after submission to the view. An alternative
approach could involve validating transactions at the `at` block, but
this is computationally expensive. To avoid additional validation
overhead, I opted to use the priority obtained from runtime during
submission to the view. This is the rationale behind introducing the
`SubmitOutcome` struct, which synchronously communicates transaction
priority from the view to the pool. This results in a very brief window
during which the transaction priority remains unknown - those
transaction are not taken into consideration while dropping takes place.
In the future, if needed, we could update transaction priority using
view revalidation results to keep this information fully up-to-date (as
priority of transaction may change with chain-state evolution).
- When _tx_mem_pool_ becomes full (an event anticipated to be rare),
transaction priority must be known to perform priority-based removal. In
such cases, the most recent block known is utilized for validation. I
think that speculative submission to the view and re-using the priority
from this submission would be an unnecessary complication.
- Once the priority is determined, lower-priority transactions whose
cumulative size meets or exceeds the size of the new transaction are
collected to ensure the pool size limit is not exceeded.
- Transaction removed from _tx_mem_pool_ , also needs to be removed from
all the views with appropriate event (which is done by
`remove_transaction_subtree`). To ensure complete removal, the
`PendingTxReplacement` struct was re-factored to more generic
`PendingPreInsertTask` (introduced in #6405) which covers removal and
submssion of transaction in the view which may be potentially created in
the background. This is to ensure that removed transaction will not
re-enter to the newly created view.
- `submit_local` implementation was also improved to properly handle
priorities in case when mempool is full. Some missing tests for this
method were also added.

Closes: #5809

---------

Co-authored-by: command-bot <>
Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>

# Description

This PR modifies the hard-coded size of extrinsics cache within
[`PoolRotator`](https://github.com/paritytech/polkadot-sdk/blob/cdf107de/substrate/client/transaction-pool/src/graph/rotator.rs#L36-L45)
to be inline with pool limits.

The problem was, that due to small size (comparing to number of txs in
single block) of hard coded size:

https://github.com/paritytech/polkadot-sdk/blob/cdf107de/substrate/client/transaction-pool/src/graph/rotator.rs#L34
excessive number of unnecessary verification were performed in
`prune_tags`:

https://github.com/paritytech/polkadot-sdk/blob/cdf107de

/substrate/client/transaction-pool/src/graph/pool.rs#L369-L370

This was resulting in quite long durations of `prune_tags` execution
time (which was ok for 6s, but becomes noticable for 2s blocks):
```
Pruning at HashAndNumber { number: 83, ... }. Resubmitting transactions: 6142, reverification took: 237.818955ms    
Pruning at HashAndNumber { number: 84, ... }. Resubmitting transactions: 5985, reverification took: 222.118218ms    
Pruning at HashAndNumber { number: 85, ... }. Resubmitting transactions: 5981, reverification took: 215.546847ms
```

The fix reduces the overhead:
```
Pruning at HashAndNumber { number: 92, ... }. Resubmitting transactions: 6325, reverification took: 14.728354ms    
Pruning at HashAndNumber { number: 93, ... }. Resubmitting transactions: 7030, reverification took: 23.973607ms    
Pruning at HashAndNumber { number: 94, ... }. Resubmitting transactions: 4465, reverification took: 9.532472ms    
```

## Review Notes
I decided to leave the hardocded `EXPECTED_SIZE` for the legacy
transaction pool. Removing verification of transactions during
re-submission may negatively impact the behavior of the legacy
(single-state) pool. As in long-term we probably want to deprecate old
pool, I did not invest time to assess the impact of rotator change in
behavior of the legacy pool.

---------

Co-authored-by: command-bot <>
Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>

# Description

- Used 10 notifications and requests within the benchmarks. After moving
the network workers' initialization out of the benchmarks, it is
acceptable to use this small number without losing precision.
- Removed the 128MB payload that consumed most of the execution time.

Co-authored-by: Dónal Murray <donalm@seadanda.dev>

# Description

Implements NetworkRequest::request for litep2p that we need for
networking benchmarks


## Review Notes

Duplicates implementation for NetworkService

https://github.com/paritytech/polkadot-sdk/blob/5bf9dd2a/substrate/client/network/src/service.rs#L1186-L1205

---------

Co-authored-by: command-bot <>

# Description

This PR removes usage of deprecated `sp-std` from Substrate. (following
PR of #5010)

## Integration

This PR doesn't remove re-exported `sp_std` from any crates yet, so
downstream projects using re-exported `sp_std` will not be affected.

## Review Notes

The existing code using `sp-std` is refactored to use `alloc` and `core`
directly. The key-value maps are instantiated from a vector of tuples
directly instead of using `sp_std::map!` macro.

`sp_std::Writer` is a helper type to use `Vec<u8>` with
`core::fmt::Write` trait. This PR copied it into `sp-runtime`, because
all crates using `sp_std::Writer` (including `sp-runtime` itself,
`frame-support`, etc.) depend on `sp-runtime`.

If this PR is merged, I would write following PRs to remove remaining
usage of `sp-std` from `bridges` and `cumulus`.

---------

Co-authored-by: command-bot <>
Co-authored-by: Guillaume Thiolliere <guillaume.thiolliere@parity.io>
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Bastian Köcher <git@kchr.de>

Co-authored-by: Dónal Murray <donal.murray@parity.io>