# Adding instruction to migrate benchmarking from v1 to v2

Even if the documentation for benchmarking v1 and v2 is clear and
detailed, I feel that adding a migration guide from v1 to v2 would help
doing it quicker.

## Integration
This change only affects documentation, so it does not cause integration
issues.

## Review Notes
I followed the migration procedure I applied in PR
https://github.com/paritytech/polkadot-sdk/pull/6018 . I added
everything from there, but I may be missing some extra steps that are
needed in specific case, so in case you notice something please let me
know.

---------

Co-authored-by: Dónal Murray <donal.murray@parity.io>

# Description

closes #5871

> The chainHead_v1_follow is using unbounded channels to send out
messages on the JSON-RPC connection which may use lots of memory if the
client is slow and can't really keep up with server i.e, substrate may
keep lots of message in memory

This PR changes the outgoing stream to abort and send a `Stop` event
downstream in the case that client doesn't keep up with the producer.

## Integration

*In depth notes about how this PR should be integrated by downstream
projects. This part is mandatory, and should be
reviewed by reviewers, if the PR does NOT have the `R0-Silent` label. In
case of a `R0-Silent`, it can be ignored.*

## Review Notes

- `rpc::Subscription::pipe_from_stream` - now takes `Self` param by
reference, change was made to allow sending events to the `Subscription`
after calls to `pipe_from_stream`.
- `chainhead_follow::submit_events` - now uses `Abortable` stream to end
it early in case
     - connection was closed by the client
     - signal received that subscription should stop 
     - error has occured when processing the events 
     - client can't keep up with the events produced
- TODO: 
  - make the abort logic less hacky

---------

Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>

# Description
Update `ext_code_hash` to match
[EIP-1052](https://eips.ethereum.org/EIPS/eip-1052) specs. Since all
possible results are written into output pointer then there's no need
for a return value.

https://github.com/paritytech/revive/pull/77

Closes #5705

---------

Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

[Issue #4762 ](https://github.com/paritytech/polkadot-sdk/issues/4762)

- Creates an enum for passing context of `service_queues` being called
from within `on_initialize` and `on_idle` hooks. Uses a match statement
inside of `service_queues` to continue using the same code, but NOT
throw a `defensive` if being called within `on_idle` hook.
- The issue requested to not throw the `defensive` if being called from
within `on_idle` hook.
- Created the `ServiceQueuesContext` enum to pass as an argument of
`service_queues` when called within the `on_initialize` and `on_idle`
hooks. A match statement was added inside of `service_queues` to
continue to throw the defensive if called from `on_initialize` but NOT
throw the defensive if called from `on_idle`.

---------

Co-authored-by: gotnoshoeson <milesbrentpatterson@proton.me>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Bastian Köcher <info@kchr.de>

Failed call frames do not produce new return data but still reset it.

---------

Signed-off-by: xermicus <cyrill@parity.io>
Co-authored-by: GitHub Action <action@github.com>

# Description
This PR migrates the staking pallet's benchmarks to the `v2` of pallet
benchmarking tooling provided by
[`frame_benchmarking`](https://github.com/paritytech/polkadot-sdk/tree/master/substrate/frame/benchmarking).

## Integration

N/A

## Review Notes

The PR is straightforward, as were #1676 , #1838 and #1868.

---------

Co-authored-by: Dónal Murray <donal.murray@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

This feature is part of the upcoming metadata V16. The associated types
of the `Config` trait that require the `TypeInfo` or `Parameter` bounds
are included in the metadata of the pallet. The metadata is not yet
exposed to the end-user, however the metadata intermediate
representation (IR) contains these types.

Developers can opt out of metadata collection of the associated types by
specifying `without_metadata` optional attribute to the
`#[pallet::config]`.

Furthermore, the `without_metadata` argument can be used in combination
with the newly added `#[pallet::include_metadata]` attribute to
selectively include only certain associated types in the metadata
collection.

### API Design

- There is nothing to collect from the associated types:

```rust
#[pallet::config]
pub trait Config: frame_system::Config {
		// Runtime events already propagated to the metadata.
		type RuntimeEvent: From<Event<Self>> + IsType<<Self as frame_system::Config>::RuntimeEvent>;

		// Constants are already propagated.
		#[pallet::constant]
		type MyGetParam2: Get<u32>;
	}
```

- Default automatic collection of associated types that require TypeInfo
or Parameter bounds:

```rust
	#[pallet::config]
	pub trait Config: frame_system::Config {
		// Runtime events already propagated to the metadata.
		type RuntimeEvent: From<Event<Self>> + IsType<<Self as frame_system::Config>::RuntimeEvent>;

		// Constants are already propagated.
		#[pallet::constant]
		type MyGetParam2: Get<u32>;

		// Associated type included by default, because it requires TypeInfo bound.
		/// Nonce doc.
		type Nonce: TypeInfo;

		// Associated type included by default, because it requires
		// Parameter bound (indirect TypeInfo).
		type AccountData: Parameter;

		// Associated type without metadata bounds, not included.
		type NotIncluded: From<u8>;
	}
```

- Disable automatic collection

```rust
// Associated types are not collected by default.
	#[pallet::config(without_metadata)]
	pub trait Config: frame_system::Config {
		// Runtime events already propagated to the metadata.
		type RuntimeEvent: From<Event<Self>> + IsType<<Self as frame_system::Config>::RuntimeEvent>;

		// Constants are already propagated.
		#[pallet::constant]
		type MyGetParam2: Get<u32>;

		// Explicitly include associated types.
		#[pallet::include_metadata]
		type Nonce: TypeInfo;

		type AccountData: Parameter;

		type NotIncluded: From<u8>;
	}
```

Builds on top of the PoC:
https://github.com/paritytech/polkadot-sdk/pull/4358
Closes: https://github.com/paritytech/polkadot-sdk/issues/4519

cc @paritytech/subxt-team

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

This PR ensures that the `litep2p.public_addresses()` never grows
indefinitely.
- effectively fixes subtle memory leaks
- fixes authority DHT records being dropped due to size limits being
exceeded
- provides a healthier subset of public addresses to the `/identify`
protocol

This PR adds a new `ExternalAddressExpired` event to the
litep2p/discovery process.

Substrate uses an LRU `address_confirmations` bounded to 32 address
entries.
The oldest entry is propagated via the `ExternalAddressExpired` event
when a new address is added to the list (if capacity is exceeded).

The expired address is then removed from the
`litep2p.public_addresses()`, effectively limiting its size to 32
entries (the size of `address_confirmations` LRU).

cc @paritytech/networking @alexggh



---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>

# Description
Part of https://github.com/paritytech/polkadot-sdk/issues/3326
Removes pallet::getter from pallet-offences from type `Reports`.
Adds a test to verify that retrieval of `Reports` still works with
storage::getter.

polkadot address: 155YyFVoMnv9BY6qxy2i5wxtveUw7WE1n5H81fL8PZKTA1Sd

---------

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Dónal Murray <donal.murray@parity.io>

### Fork-Aware Transaction Pool Implementation

This PR introduces a fork-aware transaction pool (fatxpool) enhancing
transaction management by maintaining the valid state of txpool for
different forks.

### High-level overview
The high level overview was added to
[`sc_transaction_pool::fork_aware_txpool`](https://github.com/paritytech/polkadot-sdk/blob/3ad0a1b7/substrate/client/transaction-pool/src/fork_aware_txpool/mod.rs#L21)
module. Use:
```
cargo  doc --document-private-items -p sc-transaction-pool --open
```
to build the doc. It should give a good overview and nice entry point
into the new pool's mechanics.

<details>
  <summary>Quick overview (documentation excerpt)</summary>

#### View
For every fork, a view is created. The view is a persisted state of the
transaction pool computed and updated at the tip of the fork. The view
is built around the existing `ValidatedPool` structure.

A view is created on every new best block notification. To create a
view, one of the existing views is chosen and cloned.

When the chain progresses, the view is kept in the cache
(`retracted_views`) to allow building blocks upon intermediary blocks in
the fork.

The views are deleted on finalization: views lower than the finalized
block are removed.

The views are updated with the transactions from the mempool—all
transactions are sent to the newly created views.
A maintain process is also executed for the newly created
views—basically resubmitting and pruning transactions from the
appropriate tree route.

##### View store
View store is the helper structure that acts as a container for all the
views. It provides some convenient methods.

##### Submitting transactions
Every transaction is submitted to every view at the tips of the forks.
Retracted views are not updated.
Every transaction also goes into the mempool.

##### Internal mempool
Shortly, the main purpose of an internal mempool is to prevent a
transaction from being lost. That could happen when a transaction is
invalid on one fork and could be valid on another. It also allows the
txpool to accept transactions when no blocks have been reported yet.

The mempool removes its transactions when they get finalized.
Transactions are also periodically verified on every finalized event and
removed from the mempool if no longer valid.

#### Events
Transaction events from multiple views are merged and filtered to avoid
duplicated events.
`Ready` / `Future` / `Inblock` events are originated in the Views and
are de-duplicated and forwarded to external listeners.
`Finalized` events are originated in fork-aware-txpool logic.
`Invalid` events requires special care and can be originated in both
view and fork-aware-txpool logic.

#### Light maintain
Sometime transaction pool does not have enough time to prepare fully
maintained view with all retracted transactions being revalidated. To
avoid providing empty ready transaction set to block builder (what would
result in empty block) the light maintain was implemented. It simply
removes the imported transactions from ready iterator.

#### Revalidation
Revalidation is performed for every view. The revalidation process is
started after a trigger is executed. The revalidation work is terminated
just after a new best block / finalized event is notified to the
transaction pool.
The revalidation result is applied to the newly created view which is
built upon the revalidated view.

Additionally, parts of the mempool are also revalidated to make sure
that no transactions are stuck in the mempool.


#### Logs
The most important log allowing to understand the state of the txpool
is:
```
              maintain: txs:(0, 92) views:[2;[(327, 76, 0), (326, 68, 0)]] event:Finalized { hash: 0x8...f, tree_route: [] }  took:3.463522ms
                             ^   ^         ^     ^   ^  ^      ^   ^  ^        ^                                                   ^
unwatched txs in mempool ────┘   │         │     │   │  │      │   │  │        │                                                   │
   watched txs in mempool ───────┘         │     │   │  │      │   │  │        │                                                   │
                     views  ───────────────┘     │   │  │      │   │  │        │                                                   │
                      1st view block # ──────────┘   │  │      │   │  │        │                                                   │
                           number of ready tx ───────┘  │      │   │  │        │                                                   │
                                numer of future tx ─────┘      │   │  │        │                                                   │
                                        2nd view block # ──────┘   │  │        │                                                   │
                                      number of ready tx ──────────┘  │        │                                                   │
                                           number of future tx ───────┘        │                                                   │
                                                                 event ────────┘                                                   │
                                                                       duration  ──────────────────────────────────────────────────┘
```
It is logged after the maintenance is done.

The `debug` level enables per-transaction logging, allowing to keep
track of all transaction-related actions that happened in txpool.
</details>


### Integration notes

For teams having a custom node, the new txpool needs to be instantiated,
typically in `service.rs` file, here is an example:

https://github.com/paritytech/polkadot-sdk/blob/9c547ff3

/cumulus/polkadot-omni-node/lib/src/common/spec.rs#L152-L161

To enable new transaction pool the following cli arg shall be specified:
`--pool-type=fork-aware`. If it works, there shall be information
printed in the log:
```
2024-09-20 21:28:17.528  INFO main txpool: [Parachain]  creating ForkAware txpool.
````

For debugging the following debugs shall be enabled:
```
      "-lbasic-authorship=debug",
      "-ltxpool=debug",
```
*note:* trace for txpool enables per-transaction logging.

### Future work
The current implementation seems to be stable, however further
improvements are required.
Here is the umbrella issue for future work:
- https://github.com/paritytech/polkadot-sdk/issues/5472


Partially fixes: #1202

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>

fixes #5981 

Could confirm the issue with the added tests:

```
test tests/split_ui/pass/split_call.rs [should pass] ... error
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
error[E0423]: expected value, found attribute macro `origin`
  --> tests/split_ui/pass/split_call.rs:23:1
   |
23 | #[frame_support::pallet(dev_mode)]
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ not a value
   |
   = note: this error originates in the attribute macro `frame_support::pallet` (in Nightly builds, run with -Z macro-backtrace for more info)
```

# Description

`origin` unexpectedly resolved to a macro, which is available at the
span of invocation. The solution here is to use the expansion as a
function instead of a call and pass in the desired values to avoid
ambiguities.

Use the umbrella crate for the parachain template

This covers almost all the dependencies. There are just a few exceptions
for which I created a separate issue:
https://github.com/paritytech/polkadot-sdk/issues/5993

Also related to: https://github.com/paritytech/polkadot-sdk/issues/4782

fixes #5320 @sam0x17 @gupnik 

# Description

The issue could be confirmed with the added example. The cause is for
macro hygiene, `entries` in the `#( #entries_builder )*` expansion won't
be able to reference the `entries` defined outside. The solution here is
to allow the reference to be passed into the expansion with closure.

Or we could just switch to the unhygienic span with `quote::quote!`
instead such that `entries` will resolve to the "outer" definition.

### Improved devx of running many pallets simultaneously

Changes to /cmd:
- Replace (flip) `--continue-on-fail` with `--fail-fast`, but only for
`bench`. This makes all pallets/runtimes run non-stop by default, as it
was primary use-case during tests
- The list of successful/failed pallets was hard to find within tons of
logs, so decided to write only needed logs in a file, and output as a
summary in the workflow and in the comment
- Side fix: updated `tasks_example` to `pallet_example_tasks` to make
compliant with standard naming

<img width="1006" alt="image"
src="https://github.com/user-attachments/assets/14896041-7018-4a0d-92b7-4508e81913c2">

+ added command results to workflow summary:
<img width="1275" alt="image"
src="https://github.com/user-attachments/assets/b4a8afdb-dc9f-4ff9-9720-28a88956035f">

---------

Co-authored-by: GitHub Action <action@github.com>

Some of the features enabled for `polkadot-sdk` umbrella crate were not
necessary for substrate node (e.g. all `cumulus-*` or `polkadot-*`
features) resulting in much longer compilation time. This PR fixes that.

https://github.com/paritytech/polkadot-sdk/pull/5886#discussion_r1793423401

---------

Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Bastian Köcher <git@kchr.de>

This bumps `ethbloom`, `ethereum-types`, `primitive-types` and `rlp` to
their latest version.

Fixes: https://github.com/paritytech/polkadot-sdk/issues/5870

---------

Co-authored-by: command-bot <>
Co-authored-by: ggwpez <ggwpez@users.noreply.github.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Dónal Murray <donal.murray@parity.io>

Changes:
- Add `--exclude-pallets` to exclude some pallets from runtimes where we
dont have genesis presets yet
- Make `--genesis-builder-policy=none` work with `--runtime`
- CI: Run the frame-omni-bencher for all runtimes

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: ggwpez <ggwpez@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: alvicsam <alvicsam@gmail.com>
Co-authored-by: Maksym H <mordamax@gmail.com>
Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
Co-authored-by: Maksym H <1177472+mordamax@users.noreply.github.com>

`AccountData` should be set to `()` and not to `AccountInfo`.

Closes: https://github.com/paritytech/polkadot-sdk/issues/5922

---------

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

Resolves https://github.com/polkadot-fellows/runtimes/issues/459

Tested with rococo/westend/kusama/polkadot runtimes

---------

Co-authored-by: DavidK <davidk@parity.io>
Co-authored-by: Muharem <ismailov.m.h@gmail.com>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Dónal Murray <donal.murray@parity.io>

This is a no-op refactor of staking pallet to move all `T::Currency` api
calls under one module.

A followup PR (https://github.com/paritytech/polkadot-sdk/pull/5501)
will implement the Currency <> Fungible migration for the pallet.

Introduces the new `asset` module that centralizes all interaction with
`T::Currency`. This is an attempt to try minimising staking logic
changes to minimal parts of the codebase.

## Things of note
- `T::Currency::free_balance` in current implementation includes both
staked (locked) and liquid tokens (kinda sounds wrong to call it free
then). This PR renames it to `stakeable_balance` (any better name
suggestions?). With #5501, this will become `free balance that can be
held/staked` + `already held/staked balance`.

This PR adds **static** validation that prevents upload of code that:

1) Contains basic blocks larger than the specified limit (currently
`200`)
2) Contains invalid instructions
3) Uses the `sbrk` instruction

Doing that statically at upload time (instead of at runtime) allows us
to change the basic block limit or add instructions later without
worrying about breaking old code. This is well worth the linear scan of
the whole blob on deployment in my opinion. Please note that those
checks are not applied when existing code is just run (hot path).

Also some drive by fixes:
- Remove superflous `publish = true`
- Abort fixture build on warning and fix existing warnings
- Re-enable optimizations in fixture builds (should be fixed now in
PolkaVM)
- Disable stripping for fixture builds (maybe we can get some line
information on trap via `RUST_LOG`)

---------

Co-authored-by: command-bot <>
Co-authored-by: PG Herveou <pgherveou@gmail.com>

This PR introduces a `VestedTransfer` Trait, which handles making a
transfer while also applying a vesting schedule to that balance.

This can be used in pallets like the Treasury pallet, where now we can
easily introduce a `vested_spend` extrinsic as an alternative to giving
all funds up front.

We implement `()` for the `VestedTransfer` trait, which just returns an
error, and allows anyone to opt out from needing to use or implement
this trait.

This PR also updates the logic of `do_vested_transfer` to remove the
"pre-check" which was needed before we had a default transactional layer
in FRAME.

Finally, I also fixed up some bad formatting in the test.rs file.

---------

Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>

closes #5942

Couldn't find any emissions of `Event::Issued` without amount check
other than in this PR.

Currently, we have;


https://github.com/paritytech/polkadot-sdk/blob/4bda956d/substrate/frame/balances/src/impl_currency.rs#L212-L220

and


https://github.com/paritytech/polkadot-sdk/blob/4bda956d/substrate/frame/balances/src/impl_currency.rs#L293-L306

There are cases during warp sync or re-orgs, where we receive a
notification with a block parent that was not reported in the past. This
PR extends the tracking state to catch those cases and report a `Stop`
event to the user.

This PR adds a new state to the RPC-v2 chainHead to track which blocks
have been reported.

In the past we relied on the pinning mechanism to provide us details if
a block is pinned or not.
However, the pinning state keeps the minimal information around for
pinning. Therefore, unpinning a block will cause the state to disappear.

Closes: https://github.com/paritytech/polkadot-sdk/issues/5761

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

# Description
## What?
Make it possible for other pallets to implement their own logic when a
slash on a balance occurs.

## Why?
In the [introduction of
holds](https://github.com/paritytech/substrate/pull/12951) @gavofyork
said:
> Since Holds are designed to be infallibly slashed, this means that any
logic using a Freeze must handle the possibility of the frozen amount
being reduced, potentially to zero. A permissionless function should be
provided in order to allow bookkeeping to be updated in this instance.

At Polimec we needed to find a way to reduce the vesting schedules of
our users after a slash was made, and after talking to @Kianenigma

 at
the Web3Summit, we realized there was no easy way to implement this with
the current traits, so we came up with this solution.



## How?
- First we abstract the `done_slash` function of holds::Balanced to it's
own trait that any pallet can implement.
- Then we add a config type in pallet-balances that accepts a callback
tuple of all the pallets that implement this trait.
- Finally implement done_slash for pallet-balances such that it calls
the config type.

## Integration
The default implementation of done_slash is still an empty function, and
the new config type of pallet-balances can be set to an empty tuple, so
nothing changes by default.

## Review Notes
- I suggest to focus on the first commit which contains the main logic
changes.
- I also have a working implementation of done_slash for pallet_vesting,
should I add it to this PR?
- If I run `cargo +nightly fmt --all` then I get changes to a lot of
unrelated crates, so not sure if I should run it to avoid the fmt
failure of the CI
- Should I hunt down references to fungible/fungibles documentation and
update it accordingly?

**Polkadot address:** `15fj1UhQp8Xes7y7LSmDYTy349mXvUwrbNmLaP5tQKBxsQY1`

# Checklist

* [x] My PR includes a detailed description as outlined in the
"Description" and its two subsections above.
* [x] My PR follows the [labeling requirements](

https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process
) of this project (at minimum one label for `T` required)
* External contributors: ask maintainers to put the right label on your
PR.
* [ ] I have made corresponding changes to the documentation (if
applicable)

---------

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: command-bot <>
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>

This PR adds a new beefy metric to monitor the number of live beefy
peers.

Part of investigation of litep2p request failures:
https://github.com/paritytech/polkadot-sdk/issues/4985

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

This PR introduces the concept of immutable storage data, used for
[Solidity immutable
variables](https://docs.soliditylang.org/en/latest/contracts.html#immutable).

This is a minimal implementation. Immutable data is attached to a
contract; to keep `ContractInfo` fixed in size, we only store the length
there, and store the immutable data in a dedicated storage map instead.
Which comes at the cost of requiring an additional storage read (costly)
for contracts using this feature.

We discussed more optimal solutions not requiring any additional storage
accesses internally, but they turned out to be non-trivial to implement.
Another optimization benefiting multiple calls to the same contract in a
single call stack would be to cache the immutable data in `Stack`.
However, this potential creates a DOS vulnerability (the attack vector
is to call into as many contracts in a single stack as possible, where
they all have maximum immutable data to fill the cache as efficiently as
possible). So this either has to be guaranteed to be a non-issue by
limits, or, more likely, to have some logic to bound the cache.
Eventually, we should think about introducing the concept of warm and
cold storage reads (akin to EVM). Since immutable variables are commonly
used in contracts, this change is blocking our initial launch and we
should only optimize it properly in follow-ups.

This PR also disables the `set_code_hash` API (which isn't usable for
Solidity contracts without pre-compiles anyways). With immutable storage
attached to contracts, we now want to run the constructor of the new
code hash to collect the immutable data during `set_code_hash`. This
will be implemented in a follow up PR.

---------

Signed-off-by: Cyrill Leutwiler <bigcyrill@hotmail.com>
Signed-off-by: xermicus <cyrill@parity.io>
Co-authored-by: command-bot <>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Co-authored-by: PG Herveou <pgherveou@gmail.com>

- update baseline for pallet_revive
- update cmd pipeline name
- Fix compilation after renaming some of benchmarks in pallet_revive.
[Runtime Dev]. Changed the "instr" benchmark so that it should no longer
return to little weight. It is still bogus but at least benchmarking
should not work. (by @athei

 )

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
Co-authored-by: command-bot <>

Close https://github.com/paritytech/polkadot-sdk/issues/5589

This PR makes it possible for `rpc_v2::Storage::query_iter_paginated` to
be "backpressured" which is achieved by having a channel where the
result is sent back and when this channel is "full" we pause the
iteration.

The chainHead_follow has an internal channel which doesn't represent the
actual connection and that is set to a very small number (16). Recall
that the JSON-RPC server has a dedicate buffer for each connection by
default of 64.

#### Notes

- Because `archive_storage` also depends on
`rpc_v2::Storage::query_iter_paginated` I had to tweak the method to
support limits as well. The reason is that archive_storage won't get
backpressured properly because it's not an subscription. (it would much
easier if it would be a subscription in rpc v2 spec because nothing
against querying huge amount storage keys)
- `query_iter_paginated` doesn't necessarily return the storage "in
order" such as
- `query_iter_paginated(vec![("key1", hash), ("key2", value)], ...)`
could return them in arbitrary order because it's wrapped in
FuturesUnordered but I could change that if we want to process it
inorder (it's slower)
- there is technically no limit on the number of storage queries in each
`chainHead_v1_storage call` rather than the rpc max message limit which
10MB and only allowed to max 16 calls `chainHead_v1_x` concurrently
(this should be fine)

#### Benchmarks using subxt on localhost

- Iterate over 10 accounts on westend-dev -> ~2-3x faster 
- Fetch 1024 storage values (i.e, not descedant values) -> ~50x faster
- Fetch 1024 descendant values -> ~500x faster

The reason for this is because as Josep explained in the issue is that
one is only allowed query five storage items per call and clients has
make lots of calls to drive it forward..

---------

Co-authored-by: command-bot <>
Co-authored-by: James Wilson <james@jsdw.me>

Normally, the BEEFY protocol only accepts a single MMR Root entry in a
commitment's payload. But to be extra careful, when validating
equivocation reports, let's check all the MMR roots, if there are more.

---------

Co-authored-by: Adrian Catangiu <adrian@parity.io>

# Description

close #5641

---------

Co-authored-by: Bastian Köcher <git@kchr.de>

This is a refactor and improvement from:
https://github.com/paritytech/polkadot-sdk/pull/3881

- `sp_runtime::proving_trie` now exposes a `BasicProvingTrie` for both
`base2` and `base16`.
- APIs for `base16` are more focused on single value proofs, also
aligning their APIs with the `base2` trie
- A `ProvingTrie` trait is included which wraps both the `base2` and
`base16` trie, and exposes all APIs needed for an end to end scenario.
- A `ProofToHashes` trait is exposed which can allow us to write proper
benchmarks for the merkle trie.

---------

Co-authored-by: Ankan <10196091+Ank4n@users.noreply.github.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>

# Description

Updates the doc comment on the `import_notification_stream` to make its
behaviour clearer.

Closes [Unexpected behaviour of block
`import_notification_stream`](https://github.com/paritytech/polkadot-sdk/issues/5596).

## Integration

Doesn't apply.

## Review Notes

The old comment docs caused some confusion to myself and some members of
my team, on when this notification stream is triggered. This is
reflected in the linked
[issue](https://github.com/paritytech/polkadot-sdk/issues/5596), and as
discussed there, this PR aims to prevent this confusion in future devs
looking to make use of this functionality.

# Checklist

* [x] My PR includes a detailed description as outlined in the
"Description" and its two subsections above.
* [ ] My PR follows the [labeling requirements](

https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process
) of this project (at minimum one label for `T` required)
* External contributors: ask maintainers to put the right label on your
PR.
* [x] I have made corresponding changes to the documentation (if
applicable)
* [x] I have added tests that prove my fix is effective or that my
feature works (if applicable)

You can remove the "Checklist" section once all have been checked. Thank
you for your contribution!

---------

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>

# Description

`substrate-wasm-builder` can be a build dependency for crates which
develop FRAME runtimes. I had a tough time seeing errors happening in
such crates (e.g. runtimes from the `templates` directory) in my IDE. I
use a combination of rust-analyzer + nvim-lsp + nvim-lspconfig +
rustacean.vim and all of this stack is not able to correctly parse
errors emitted during the `build` phase.

As a matter of fact there is also a cargo issue tracking specifically
this issue where cargo doesn't propagate the `--message-format` type to
the build phase: [here](https://github.com/rust-lang/cargo/issues/14246)
initially and then
[here](https://github.com/rust-lang/cargo/issues/8283). It feels like a
solution for this use case isn't very close, so if it comes to runtimes
development (both as an SDK user and developer), enabling wasm builder
to emit diagnostics messages friendly to IDEs would be useful for
regular workflows where IDEs are used for finding errors instead of
manually running `cargo` commands.

## Integration

It can be an issue if Substrate/FRAME SDKs users and developers rely on
the runtimes' crates build phase output in certain ways. Emitting
compilation messages as json will pollute the regular compilation output
so people that would manually run `cargo build` or `cargo check` on
their crates will have a tougher time extracting the non JSON output.

## Review Notes

Rust IDEs based on rust-analyzer rely on cargo check/clippy to extract
diagnostic information. The information is generated by passing flags
like `--messages-format=json` to the `cargo` commands. The messages are
extracted by rust-analyzer and published to LSP clients that will
populate UIs accordingly.

We need to build against the wasm target by using `message-format=json`
too so that IDEs can show the errors for crates that have a build
dependency on `substrate-wasm-builder`.

---------

Signed-off-by: Iulian Barbu <iulian.barbu@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>

This PR removes a workaround which had a reference comment to a rust
compiler issue. The issue has been fixed and we should be able to remove
that trait bound.

cc https://github.com/paritytech/ci_cd/issues/1035

cc https://github.com/paritytech/ci_cd/issues/1023

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>
Co-authored-by: Maksym H <1177472+mordamax@users.noreply.github.com>
Co-authored-by: gui <gui.thiolliere@gmail.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: ggwpez <ggwpez@users.noreply.github.com>

This PR introduces 2 new syscalls: `return_data_size` and
`return_data_copy`, resembling the semantics of the EVM `RETURNDATASIZE`
and `RETURNDATACOPY` opcodes.

The ownership of `ExecReturnValue` (the return data) has moved to the
`Frame`. This allows implementing the new contract API functionality in
ext with no additional copies. Returned data is passed via contract
memory, memory is (will be) metered, hence the amount of returned data
can not be statically known, so we should avoid storing copies of the
returned data if we can. By moving the ownership of the exectuables
return value into the `Frame` struct we achieve this.

A zero-copy implementation of those APIs would be technically possible
without that internal change by making the callsite in the runtime
responsible for moving the returned data into the frame after any call.
However, resetting the stored output needs to be handled in ext, since
plain transfers will _not_ affect the stored return data (and we don't
want to handle this special call case inside the `runtime` API). This
has drawbacks:
- It can not be tested easily in the mock.
- It introduces an inconsistency where resetting the stored output is
handled in ext, but the runtime API is responsible to store it back
correctly after any calls made. Instead, with ownership of the data in
`Frame`, both can be handled in a single place. Handling both in `fn
run()` is more natural and leaves less room for runtime API bugs.

The returned output is reset each time _before_ running any executable
in a nested stack. This change should not incur any overhead to the
overall memory usage as _only_ the returned data from the last executed
frame will be kept around at any time.

---------

Signed-off-by: Cyrill Leutwiler <bigcyrill@hotmail.com>
Signed-off-by: xermicus <cyrill@parity.io>
Co-authored-by: command-bot <>
Co-authored-by: PG Herveou <pgherveou@gmail.com>

# MBM try-runtime support

This MR adds support to the try-runtime trait such that the
try-runtime-CLI will be able to support MBM testing
[here](https://github.com/paritytech/try-runtime-cli/pull/90). It mainly
adds two feature-gated hooks to the `SteppedMigration` hook to
facilitate testing. These hooks are named `pre_upgrade` and
`post_upgrade` and have the same signature and implications as for
single-block migrations.

## Integration

To make use of this in your Multi-Block-Migration, just implement the
two new hooks and test pre- and post-conditions in them:

```rust
#[cfg(feature = "try-runtime")]
fn pre_upgrade() -> Result<Vec<u8>, frame_support::sp_runtime::TryRuntimeError> {
	// ...
}

#[cfg(feature = "try-runtime")]
fn post_upgrade(prev: Vec<u8>) -> Result<(), frame_support::sp_runtime::TryRuntimeError> {
    // ...
}
```

You may return an error or panic in these functions to indicate failure.
This will then show up in the try-runtime-CLI and can be used in CI for
testing.

Changes:
- Adds `try-runtime` gated methods `pre_upgrade` and `post_upgrade` on
`SteppedMigration`
- Adds `try-runtime` gated methods `nth_pre_upgrade` and
`nth_post_upgrade` on `SteppedMigrations`
- Modifies `pallet_migrations` implementation to run pre_upgrade and
post_upgrade steps at the appropriate times, and panic in the event of
migration failure.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: georgepisaltu <george.pisaltu@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: claravanstaden <claravanstaden64@gmail.com>
Co-authored-by: ggwpez <ggwpez@users.noreply.github.com>
Co-authored-by: georgepisaltu <george.pisaltu@parity.io>