Changes:
- Add pallet-parameters to Rococo to configure the NIS and preimage
pallet.
- Fix names of expanded dynamic params. Apparently, `to_class_case`
removes suffix `s`, and `Nis` becomes `Ni` 😑

. Now using
`to_pascal_case`.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Alessandro Siniscalchi <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: command-bot <>

Closes: https://github.com/paritytech/polkadot-sdk/issues/4100

This PR introduces `BlockHashProvider` into `pallet_mmr::Config`
This type is used to get `block_hash` for a given `block_number` rather
than directly using `frame_system::Pallet::block_hash`

The `DefaultBlockHashProvider` uses `frame_system::Pallet::block_hash`
to get the `block_hash`

Closes: #4062

1. The `CustomFmtContext::ContextWithFormatFields` enum arm isn't
actually used and thus we don't need the enum anymore.

2. We don't do anything with most of the normalized metadata that's
created by calling `event.normalized_metadata();` - the `target` we can
get from `event.metadata.target()` and level we can get from
`event.metadata.level()` - let's just call them direct to simplify
things. (`event.metadata()` is just a field call under the hood)

Changelog: No functional changes, might run a tad faster with lots of
logging turned on.

---------

Co-authored-by: Bastian Köcher <[email protected]>

Small adjustments which should make understanding what is going on much
easier for future readers.

Initialization is a bit messy, the very least we should do is adding
documentation to make it harder to use wrongly.

I was thinking about calling `request_core_count` right from
`start_sales`, but as explained in the docs, this is not necessarily
what you want.

---------

Co-authored-by: eskimor <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Dónal Murray <[email protected]>

Completes the removal of `try-runtime-cli` logic from `polkadot-sdk`.

Closes #4041

Changes:
- Increase cache size and reduce retries.
- Ignore Substrate SE links :(
- Fix broken link.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

- Update internal logic so that the storage_base_deposit does not
include ED
- add v16 migration to update ContractInfo struct with this change

Before:
<img width="820" alt="Screenshot 2024-03-21 at 11 23 29"
src="https://github.com/paritytech/polkadot-sdk/assets/521091/a0a8df0d-e743-42c5-9e16-cf2ec1aa949c">

After:
![Screenshot 2024-03-21 at 11 23
42](https://github.com/paritytech/polkadot-sdk/assets/521091/593235b0-b866-4915-b653-2071d793228b)

---------

Co-authored-by: Cyrill Leutwiler <[email protected]>
Co-authored-by: command-bot <>

Reuse wasmi Module when validating.
Prepare the code for 0.32 and the addition of Module::new_unchecked

This tiny PR extends the `on_validated_block_announce` log with the bad
PeerID.
Used to identify if the peerID is malicious by correlating with other
logs (ie peer-set).

While at it, have removed the `\n` from a multiline log, which did not
play well with
[sub-triage-logs](https://github.com/lexnv/sub-triage-logs/tree/master).

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

[Weights
compare](https://weights.tasty.limo/compare?unit=weight&ignore_errors=true&threshold=10&method=asymptotic&repo=polkadot-sdk&old=master&new=pg%2Fbench_tweaks&path_pattern=substrate%2Fframe%2F**%2Fsrc%2Fweights.rs%2Cpolkadot%2Fruntime%2F*%2Fsrc%2Fweights%2F**%2F*.rs%2Cpolkadot%2Fbridges%2Fmodules%2F*%2Fsrc%2Fweights.rs%2Ccumulus%2F**%2Fweights%2F*.rs%2Ccumulus%2F**%2Fweights%2Fxcm%2F*.rs%2Ccumulus%2F**%2Fsrc%2Fweights.rs)

Note: Raw weights change does not mean much here, as this PR reduce the
scope of what is benchmarked, they are therefore decreased by a good
margin. One should instead print the Schedule using

cargo test --features runtime-benchmarks bench_print_schedule --
--nocapture
or following the instructions from the
[README](https://github.com/paritytech/polkadot-sdk/tree/pg/bench_tweaks/substrate/frame/contracts#schedule)
for looking at the Schedule of a specific runtime

---------

Co-authored-by: command-bot <>

Current benchmarking macro returns a closure with the captured
benchmarked code.
This can cause issues when the benchmarked code has complex lifetime
requirements.

This PR updates the existing macro by injecting the recording parameter
and invoking the start / stop method around the benchmarked block
instead of returning a closure

One other added benefit is that you can write this kind of code now as
well:

```rust
let v;
#[block]
{ v = func.call(); }
dbg!(v); // or assert something on v
```


[Weights compare
link](https://weights.tasty.limo/compare?unit=weight&ignore_errors=true&threshold=10&method=asymptotic&repo=polkadot-sdk&old=pg/fix-weights&new=pg/bench_update&path_pattern=substrate/frame/**/src/weights.rs,polkadot/runtime/*/src/weights/**/*.rs,polkadot/bridges/modules/*/src/weights.rs,cumulus/**/weights/*.rs,cumulus/**/weights/xcm/*.rs,cumulus/**/src/weights.rs)

---------

Co-authored-by: command-bot <>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@pari...>

Fix some broken legacy definitions of pallet_contracts_uapi storage host
functions

Cumulus test-parachain node and test runtime were still using relay
chain consensus and 12s blocktimes. With async backing around the corner
on the major chains we should switch our tests too.

Also needed to nicely test the changes coming to collators in #3168.

### Changes Overview
- Followed the [migration
guide](https://wiki.polkadot.network/docs/maintain-guides-async-backing)
for async backing for the cumulus-test-runtime
- Adjusted the cumulus-test-service to use the correct import-queue,
lookahead collator etc.
- The block validation function now uses the Aura Ext Executor so that
the seal of the block is validated
- Previous point requires that we seal block before calling into
`validate_block`, I introduced a helper function for that
- Test client adjusted to provide a slot to the relay chain proof and
the aura pre-digest

This PR ensures that the transaction API generates an `Invalid` events
for transaction bytes that fail to decode.

The spec mentioned the `Invalid` event at the jsonrpc error section,
however this spec PR makes things clearer:
- https://github.com/paritytech/json-rpc-interface-spec/pull/146

While at it have discovered an inconsistency with the generated events.
The drop event from the transaction pool was incorrectly mapped to the
`invalid` event.

Added tests for the API stabilize the API soon:
- https://github.com/paritytech/json-rpc-interface-spec/pull/144


Closes: https://github.com/paritytech/polkadot-sdk/issues/3083


cc @paritytech/subxt-team

---------

Signed-off-by: Alexandru Vasile <[email protected]>

This PR brings the fix
https://github.com/paritytech/substrate/pull/13396 to polkadot-sdk.

In the past, due to insufficient inbound slot count on polkadot &
kusama, this fix led to low peer count. The situation has improved since
then after changing the default ratio between `--in-peers` &
`--out-peers`.

Nevertheless, it's expected that the reported total peer count with this
fix is going to be lower than without it. This should be seen as the
correct number of working connections reported, as opposed to also
reporting already closed connections, and not as lower count of working
connections with peers.

This PR also removes the peer eviction mechanism, as closed substream
detection is a more granular way of detecting peers that stopped syncing
with us.

The burn-in has been already performed as part of testing these changes
in https://github.com/paritytech/polkadot-sdk/pull/3426.

---------

Co-authored-by: Aaro Altonen <[email protected]>

This is a tiny PR to increase the time a peer remains banned.

A peer is banned when the reputation drops below a threshold.
With every second, the peer reputation is exponentially decayed towards
zero.

For the previous setup:
- decaying to zero from (i32::MAX or i32::MIN) would take 948 seconds
(15mins 48seconds)
- from i32::MIN to escaping the banned threshold would take 10 seconds
This means we are decaying reputation a bit too aggressive and
misbehaving peers can misbehave again in 10 seconds.
Another side effect of this is that we have encountered multiple
warnings caused by a few misbehaving peers.

In the new setup:
- decaying to zero from (i32::MAX or i32::MIN) would take 3544 seconds
(59 minutes)
- from i32::MIN to escaping the banned threshold would take ~69 seconds

This is a followup of:
- https://github.com/paritytech/polkadot-sdk/pull/4000.

### Testing Done
- Created a misbehaving client with
[subp2p-explorer](https://github.com/lexnv/subp2p-explorer), the client
is banned for approx 69seconds until it is allowed to connect again.

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <[email protected]>

# Description
- What does this PR do?
1. Upgrades `trie-db`'s version to the latest release. This release
includes, among others, an implementation of `DoubleEndedIterator` for
the `TrieDB` struct, allowing to iterate both backwards and forwards
within the leaves of a trie.
2. Upgrades `trie-bench` to `0.39.0` for compatibility.
3. Upgrades `criterion` to `0.5.1` for compatibility.
- Why are these changes needed?
Besides keeping up with the upgrade of `trie-db`, this specifically adds
the functionality of iterating back on the leafs of a trie, with
`sp-trie`. In a project we're currently working on, this comes very
handy to verify a Merkle proof that is the response to a challenge. The
challenge is a random hash that (most likely) will not be an existing
leaf in the trie. So the challenged user, has to provide a Merkle proof
of the previous and next existing leafs in the trie, that surround the
random challenged hash.

Without having DoubleEnded iterators, we're forced to iterate until we
find the first existing leaf, like so:
```rust
        // ************* VERIFIER (RUNTIME) *************
        // Verify proof. This generates a partial trie based on the proof and
        // checks that the root hash matches the `expected_root`.
        let (memdb, root) = proof.to_memory_db(Some(&root)).unwrap();
        let trie = TrieDBBuilder::<LayoutV1<RefHasher>>::new(&memdb, &root).build();

        // Print all leaf node keys and values.
        println!("\nPrinting leaf nodes of partial tree...");
        for key in trie.key_iter().unwrap() {
            if key.is_ok() {
                println!("Leaf node key: {:?}", key.clone().unwrap());

                let val = trie.get(&key.unwrap());

                if val.is_ok() {
                    println!("Leaf node value: {:?}", val.unwrap());
                } else {
                    println!("Leaf node value: None");
                }
            }
        }

        println!("RECONSTRUCTED TRIE {:#?}", trie);

        // Create an iterator over the leaf nodes.
        let mut iter = trie.iter().unwrap();

        // First element with a value should be the previous existing leaf to the challenged hash.
        let mut prev_key = None;
        for element in &mut iter {
            if element.is_ok() {
                let (key, _) = element.unwrap();
                prev_key = Some(key);
                break;
            }
        }
        assert!(prev_key.is_some());

        // Since hashes are `Vec<u8>` ordered in big-endian, we can compare them directly.
        assert!(prev_key.unwrap() <= challenge_hash.to_vec());

        // The next element should exist (meaning there is no other existing leaf between the
        // previous and next leaf) and it should be greater than the challenged hash.
        let next_key = iter.next().unwrap().unwrap().0;
        assert!(next_key >= challenge_hash.to_vec());
```

With DoubleEnded iterators, we can avoid that, like this:
```rust
        // ************* VERIFIER (RUNTIME) *************
        // Verify proof. This generates a partial trie based on the proof and
        // checks that the root hash matches the `expected_root`.
        let (memdb, root) = proof.to_memory_db(Some(&root)).unwrap();
        let trie = TrieDBBuilder::<LayoutV1<RefHasher>>::new(&memdb, &root).build();

        // Print all leaf node keys and values.
        println!("\nPrinting leaf nodes of partial tree...");
        for key in trie.key_iter().unwrap() {
            if key.is_ok() {
                println!("Leaf node key: {:?}", key.clone().unwrap());

                let val = trie.get(&key.unwrap());

                if val.is_ok() {
                    println!("Leaf node value: {:?}", val.unwrap());
                } else {
                    println!("Leaf node value: None");
                }
            }
        }

        // println!("RECONSTRUCTED TRIE {:#?}", trie);
        println!("\nChallenged key: {:?}", challenge_hash);

        // Create an iterator over the leaf nodes.
        let mut double_ended_iter = trie.into_double_ended_iter().unwrap();

        // First element with a value should be the previous existing leaf to the challenged hash.
        double_ended_iter.seek(&challenge_hash.to_vec()).unwrap();
        let next_key = double_ended_iter.next_back().unwrap().unwrap().0;
        let prev_key = double_ended_iter.next_back().unwrap().unwrap().0;

        // Since hashes are `Vec<u8>` ordered in big-endian, we can compare them directly.
        println!("Prev key: {:?}", prev_key);
        assert!(prev_key <= challenge_hash.to_vec());

        println!("Next key: {:?}", next_key);
        assert!(next_key >= challenge_hash.to_vec());
```
- How were these changes implemented and what do they affect?
All that is needed for this functionality to be exposed is changing the
version number of `trie-db` in all the `Cargo.toml`s applicable, and
re-exporting some additional structs from `trie-db` in `sp-trie`.

---------

Co-authored-by: Bastian Köcher <[email protected]>

Update the Contracts API to use `WeightMeter`, as it simplifies the code
and makes it easier to reason about, rather than taking a mutable weight
or returning a tuple with the weight consumed

---------

Co-authored-by: Alexander Theißen <[email protected]>

closes https://github.com/paritytech/polkadot-sdk/issues/3894

Allows Nomination Pool configuration to be set by a custom origin
instead of root.

In runtimes, we would set this to be `StakingAdmin`, same as for
pallet-staking.

---------

Co-authored-by: Liam Aharon <[email protected]>

During validation, `SkipCheckIfFeeless` should check if the call is
`feeless` and delegate to the wrapped extension if not.

Closes  https://github.com/paritytech/polkadot-sdk/issues/3737

---------

Co-authored-by: command-bot <>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

[litep2p](https://github.com/altonen/litep2p) is a libp2p-compatible P2P
networking library. It supports all of the features of `rust-libp2p`
that are currently being utilized by Polkadot SDK.

Compared to `rust-libp2p`, `litep2p` has a quite different architecture
which is why the new `litep2p` network backend is only able to use a
little of the existing code in `sc-network`. The design has been mainly
influenced by how we'd wish to structure our networking-related code in
Polkadot SDK: independent higher-levels protocols directly communicating
with the network over links that support bidirectional backpressure. A
good example would be `NotificationHandle`/`RequestResponseHandle`
abstractions which allow, e.g., `SyncingEngine` to directly communicate
with peers to announce/request blocks.

I've tried running `polkadot --network-backend litep2p` with a few
different peer configurations and there is a noticeable reduction in
networking CPU usage. For high load (`...

This MR contains two major changes and some maintenance cleanup.  

## 1. Free Standing Pallet Benchmark Runner

Closes https://github.com/paritytech/polkadot-sdk/issues/3045, depends
on your runtime exposing the `GenesisBuilderApi` (like
https://github.com/paritytech/polkadot-sdk/pull/1492).

Introduces a new binary crate: `frame-omni-bencher`.  
It allows to directly benchmark a WASM blob - without needing a node or
chain spec.

This makes it much easier to generate pallet weights and should allow us
to remove bloaty code from the node.
It should work for all FRAME runtimes that dont use 3rd party host calls
or non `BlakeTwo256` block hashing (basically all polkadot parachains
should work).

It is 100% backwards compatible with the old CLI args, when the `v1`
compatibility command is used. This is done to allow for forwards
compatible addition of new commands.

### Example (full example in the Rust docs)

Installing the CLI:
```sh
cargo install --locked --path subs...

Sometimes you need to debug some issues just by the logs and reconstruct
what happened.
In these scenarios it would be nice to know if a block was imported as
best block, and what it parent was.
So here I propose to change the output of the informant to this:

```
2024-04-05 20:38:22.004  INFO ⋮substrate: [Parachain] ✨ Imported #18 (0xe7b3…4555 -> 0xbd6f…ced7)    
2024-04-05 20:38:24.005  INFO ⋮substrate: [Parachain] ✨ Imported #19 (0xbd6f…ced7 -> 0x4dd0…d81f)    
2024-04-05 20:38:24.011  INFO ⋮substrate: [jobless-children-5352] 🌟 Imported #42 (0xed2e…27fc -> 0x718f…f30e)    
2024-04-05 20:38:26.005  INFO ⋮substrate: [Parachain] ✨ Imported #20 (0x4dd0…d81f -> 0x6e85…e2b8)    
2024-04-05 20:38:28.004  INFO ⋮substrate: [Parachain] 🌟 Imported #21 (0x6e85…e2b8 -> 0xad53…2a97)    
2024-04-05 20:38:30.004  INFO ⋮substrate: [Parachain] 🌟

 Imported #22 (0xad53…2a97 -> 0xa874…890f)    
```

---------

Co-authored-by: Bastian Köcher <[email protected]>

The stream was just used to communicate from the validator the peer
reports back to the gossip engine. Internally the gossip engine just
forwards these reports to the networking engine. So, we can just do this
directly.

The reporting stream was also pumped [in the worker behind the
engine](https://github.com/paritytech/polkadot-sdk/blob/9d626189/substrate/client/consensus/beefy/src/worker.rs#L939).
This means if there was a lot of data incoming over the engine, the
reporting stream was almost never processed and thus, it could have
started to grow and we have seen issues around this.

Partly Closes: https://github.com/paritytech/polkadot-sdk/issues/3945

Signed-off-by: hongkuang <[email protected]>

I don't think there are any more releases to the 0.2.x versions, so best
we're on the 0.3.x release.

No change on the benchmarks, fast local time is still just as fast as
before:

new version bench:
```
fast_local_time         time:   [30.551 ns 30.595 ns 30.668 ns]
```

old version bench:
```
fast_local_time         time:   [30.598 ns 30.646 ns 30.723 ns]
```

---------

Co-authored-by: Bastian Köcher <[email protected]>

1. Add `#[doc(no_inline)]` to frame umbrella crate re-exports that
eventually resolve to `frame_support_procedural` so docs don't look like
the screenshot below and instead link to the proper `frame-support`
docs.
<img width="1512" alt="Screenshot 2024-04-05 at 20 05 01"
src="https://github.com/paritytech/polkadot-sdk/assets/16665596/a41daa4c-ebca-44a4-9fea-f9f336314e13">


2. Remove `"Rust-Analyzer Users: "` prefix from
`frame_support_procedural` doc comments, since these doc comments are
visible in the web documentation and possible to stumble upon especially
when navigating from the frame umbrella crate.

Co-authored-by: Kian Paimani <[email protected]>

Decrements `total_deposit` when collection metadata is cleared in
`pallet-nfts` and `pallet-uniques`.

Defines a runtime api for `pallet-broker` for getting the current price
of a core if there is an ongoing sale.

Closes: #3413

---------

Co-authored-by: Bastian Köcher <[email protected]>

Signed-off-by: divdeploy <[email protected]>

This PR adds a check that `max_timeslices > 0` and errors if not. It
also adds a test for this behaviour and cleans up some misleading docs.

The runtime now can provide a number of predefined presets of
`RuntimeGenesisConfig` struct. This presets are intended to be used in
different deployments, e.g.: `local`, `staging`, etc, and should be
included into the corresponding chain-specs.

Having `GenesisConfig` presets in runtime allows to fully decouple node
from runtime types (the problem is described in #1984).

**Summary of changes:**
- The `GenesisBuilder` API was adjusted to enable this functionality
(and provide better naming - #150):
   ```rust
    fn preset_names() -> Vec<PresetId>;
fn get_preset(id: Option<PresetId>) -> Option<serde_json::Value>;
//`None` means default
    fn build_state(value: serde_json::Value);
    pub struct PresetId(Vec<u8>);
   ```

- **Breaking change**: Old `create_default_config` method was removed,
`build_config` was renamed to `build_state`. As a consequence a node
won't be able to interact with genesis config for older runtimes. The
cleanup was made for sake of API simplicity. Also IMO maintaining
compatibility with old API is not so crucial.
- Reference implementation was provided for `substrate-test-runtime` and
`rococo` runtimes. For rococo new
[`genesis_configs_presets`](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b/polkadot/runtime/rococo/src/genesis_config_presets.rs#L530)
module was added and is used in `GenesisBuilder`
[_presets-related_](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b/polkadot/runtime/rococo/src/lib.rs#L2462-L2485)
methods.

- The `chain-spec-builder` util was also improved and allows to
([_doc_](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b/substrate/bin/utils/chain-spec-builder/src/lib.rs#L19)):
   - list presets provided by given runtime (`list-presets`),
- display preset or default config provided by the runtime
(`display-preset`),
   - build chain-spec using named preset (`create ... named-preset`),


- The `ChainSpecBuilder` is extended with
[`with_genesis_config_preset_name`](https://github.com/paritytech/polkadot-sdk/blob/3b41d66b/substrate/client/chain-spec/src/chain_spec.rs#L447)
method which allows to build chain-spec using named preset provided by
the runtime. Sample usage on the node side
[here](https://github.com/paritytech/polkadot-sdk/blob/2caffaae

/polkadot/node/service/src/chain_spec.rs#L404).

Implementation of #1984.
fixes: #150
part of: #25

---------

Co-authored-by: Sebastian Kunert <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>

This function is currently only exposed to runtime-benchmarks, where it
should be available in all tests.

Also made it available in `std` because this is how `set_block_number`
works in FRAME System, however, not sure if it is needed in the latest
std/no_std paradigm.

---------

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>

Part of https://github.com/paritytech/polkadot-sdk/issues/226 
Related https://github.com/paritytech/polkadot-sdk/issues/1833

- Deprecate `CurrencyAdapter` and introduce `FungibleAdapter`
- Deprecate `ToStakingPot` and replace usage with `ResolveTo`
- Required creating a new `StakingPotAccountId` struct that implements
`TypedGet` for the staking pot account ID
- Update parachain common utils `DealWithFees`, `ToAuthor` and
`AssetsToBlockAuthor` implementations to use `fungible`
- Update runtime XCM Weight Traders to use `ResolveTo` instead of
`ToStakingPot`
- Update runtime Transaction Payment pallets to use `FungibleAdapter`
instead of `CurrencyAdapter`
- [x] Blocked by https://github.com/paritytech/polkadot-sdk/pull/1296,
needs the `Unbalanced::decrease_balance` fix

## Basic example showcasing a migration using the MBM framework

This PR has been built on top of
https://github.com/paritytech/polkadot-sdk/pull/1781 and adds two new
example crates to the `examples` pallet

### Changes Made:

Added the `pallet-example-mbm` crate: This crate provides a minimal
example of a pallet that uses MBM. It showcases a storage migration
where values are migrated from a `u32` to a `u64`.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>

Closes https://github.com/paritytech/polkadot-sdk/issues/2977

The issue appears to stem from the `aquamarine` crate failing to render
diagrams in re-exported crates.

e.g. as raised
[here](https://github.com/paritytech/polkadot-sdk/issues/2977), diagrams
would render at `frame_support::traits::Hooks` but not the re-exported
doc `frame::traits::Hooks`, even if I added `aquamarine` as a `frame`
crate dependency.

To resolve this, I followed advice in
https://github.com/mersinvald/aquamarine/issues/20 to instead render
mermaid diagrams directly using JS by adding an `after-content.js`.

---

Also fixes compile warnings, enables `--all-features` and disallows
future warnings in CI.

---------

Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>