Just to have some information what is the release number that was used
to push a particular commit or PR in the templates repositories.

chore: fix typos

related to
https://github.com/paritytech/parity-bridges-common/issues/2962
on top of #4383

Example:
```sh
RUST_LOG=runtime=trace,rpc=trace,bridge=trace \
        ./target/release/substrate-relay relay-messages-delivery-confirmation bridge-hub-rococo-to-bridge-hub-westend \
        --source-host localhost \
        --source-port 8943 \
        --source-version-mode Auto \
        --source-signer //Eve \
        --source-transactions-mortality 4 \
        --target-host localhost \
        --target-port 8945 \
        --target-version-mode Auto \
        --lane 00000002 \
        --at-target-block 49
```

closes https://github.com/paritytech/parity-bridges-common/issues/2982
closes https://github.com/paritytech/parity-bridges-common/issues/2730

The main change is in the
bridges/relays/lib-substrate-relay/src/finality/target.rs, changes in
other files are just moving the code

~I haven't been able to run zn tests locally - don't know why, but it
keeps failing for me locally with: `
Error running script:
/home/svyatonik/dev/polkadot-sdk/bridges/testing/framework/js-helpers/wait-hrmp-channel-opened.js
Error: Timeout(300), "custom-js
/home/svyatonik/dev/polkadot-sdk/bridges/testing/framework/js-helpers/wait-hrmp-channel-opened.js
within 300 secs" didn't complete on time.`~ The issue was an obsolete
`polkadot-js-api` binary - did `yarn global upgrade` and it is ok now

Marked XCMv2 as deprecated now that we have XCMv4.
It will be removed sometime around June 2024.

---------

Co-authored-by: Branislav Kontur <[email protected]>

availability-recovery: bump chunk fetch threshold to 1MB for Polkadot and 4MB for Kusama + testnets (#4399)

Doing this change ensures that we minimize the CPU usage we spend in
reed-solomon by only doing the re-encoding into chunks if PoV size is
less than 4MB (which means all PoVs right now)
 
Based on susbystem benchmark results we concluded that it is safe to
bump this number higher. At worst case scenario the network pressure for
a backing group of 5 is around 25% of the network bandwidth in hw specs.

Assuming 6s block times (max_candidate_depth 3) and needed_approvals 30
the amount of bandwidth usage of a backing group used would hover above
`30 * 4 * 3 = 360MB` per relay chain block. Given a backing group of 5
that gives 72MB per block per validator -> 12 MB/s.

<details>
<summary>Reality check on Kusama PoV sizes (click for chart)</summary>
<br>
<img width="697" alt="Screenshot 2024-05-07 at 14 30 38"
src="https://github.com/paritytech/polkadot-sdk/assets/54316454/bfed32d4-8623-48b0-9ec0-8b95dd2a9d8c">
</details>

---------

Signed-off-by: Andrei Sandu <[email protected]>

# Umbrella Crate

The Polkadot-SDK "umbrella" is a crate that re-exports all other
published crates. This makes it
possible to have a very small `Cargo.toml` file that only has one
dependency, the umbrella
crate. This helps with selecting the right combination of crate
versions, since otherwise 3rd
party tools are needed to select a compatible set of versions.

## Features

The umbrella crate supports no-std builds and can therefore be used in
the runtime and node.
There are two main features: `runtime` and `node`. The `runtime` feature
enables all `no-std`
crates, while the `node` feature enables all `std` crates. It should be
used like any other
crate in the repo, with `default-features = false`.

For more fine-grained control, additionally, each crate can be enabled
selectively. The umbrella
exposes one feature per dependency. For example, if you only want to use
the `frame-support`
crate, you can enable the `frame-support` feature.

The umbrella exposes a few more general features:
- `tuples-96`: Needs to be enabled for runtimes that have more than 64
pallets.
- `serde`: Specifically enable `serde` en/decoding support.
- `experimental`: Experimental enable experimental features - should not
yet used in production.
- `with-tracing`: Enable tracing support.
- `try-runtime`, `runtime-benchmarks` and `std`: These follow the
standard conventions.
- `runtime`: As described above, enable all `no-std` crates.
- `node`: As described above, enable all `std` crates.
- There does *not* exist a dedicated docs feature. To generate docs,
enable the `runtime` and
`node` feature. For docs.rs the manifest contains specific configuration
to make it show up
  all re-exports.

There is a specific `zepter` check in place to ensure that the features
of the umbrella are
correctly configured. This check is run in CI and locally when running
`zepter`.

## Generation

The umbrella crate needs to be updated every time when a new crate is
added or removed from the
workspace. It is checked in CI by calling its generation script. The
generation script is
located in `./scripts/generate-umbrella.py` and needs dependency
`cargo_workspace`.

Example: `python3 scripts/generate-umbrella.py --sdk . --version 1.9.0`

## Usage

> Note: You can see a live example in the `staging-node-cli` and
`kitchensink-runtime` crates.

The umbrella crate can be added to your runtime crate like this:

`polkadot-sdk = { path = "../../../../umbrella", features = ["runtime"],
default-features =
false}`

or for a node:

`polkadot-sdk = { path = "../../../../umbrella", features = ["node"],
default-features = false
}`

In the code, it is then possible to bring all dependencies into scope
via:

`use polkadot_sdk::*;`

### Known Issues

The only known issue so far is the fact that the `use` statement brings
the dependencies only
into the outer module scope - not the global crate scope. For example,
the following code would
need to be adjusted:

```rust
use polkadot_sdk::*;

mod foo {
   // This does sadly not compile:
   frame_support::parameter_types! { }

   // Instead, we need to do this (or add an equivalent `use` statement):
   polkadot_sdk::frame_support::parameter_types! { }
}
```

Apart from this, no issues are known. There could be some bugs with how
macros locate their own
re-exports. Please compile issues that arise from using this crate.

## Dependencies

The umbrella crate re-exports all published crates, with a few
exceptions:
- Runtime crates like `rococo-runtime` etc are not exported. This
otherwise leads to very weird
  compile errors and should not be needed anyway.
- Example and fuzzing crates are not exported. This is currently
detected by checking the name
of the crate for these magic words. In the future, it will utilize
custom metadata, as it is
  done in the `rococo-runtime` crate.
- The umbrella crate itself. Should be obvious :)

## Follow Ups
- [ ] Re-writing the generator in Rust - the python script is at its
limit.
- [ ] Using custom metadata to exclude some crates instead of filtering
by names.
- [ ] Finding a way to setting the version properly. Currently its
locked in the CI script.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

@serban300

 could you please do the same for the MMR crate? Am not sure
what commit was released since there are no release tags in the repo.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Attempt to avoid specifying `BlockHashCount` for different `mocking::{MockBlock, MockBlockU32, MockBlockU128}` (#4543)

While doing some migration/rebase I came in to the situation, where I
needed to change `mocking::MockBlock` to `mocking::MockBlockU32`:
```
#[derive_impl(frame_system::config_preludes::TestDefaultConfig)]
impl frame_system::Config for TestRuntime {
	type Block = frame_system::mocking::MockBlockU32<TestRuntime>;
	type AccountData = pallet_balances::AccountData<ThisChainBalance>;
}
```
But actual `TestDefaultConfig` for `frame_system` is using `ConstU64`
for `type BlockHashCount = frame_support::traits::ConstU64<10>;`
[here](https://github.com/paritytech/polkadot-sdk/blob/master/substrate/frame/system/src/lib.rs#L303).
Because of this, it force me to specify and add override for `type
BlockHashCount = ConstU32<10>`.

This PR tries to fix this with `TestBlockHashCount` implementation for
`TestDefaultConfig` which supports `u32`, `u64` and `u128` as a
`BlockNumber`.

### How to simulate error
Just by removing `type BlockHashCount = ConstU32<250>;`
[here](https://github.com/paritytech/polkadot-sdk/blob/master/substrate/frame/multisig/src/tests.rs#L44)
```
:~/parity/olkadot-sdk$ cargo test -p pallet-multisig
   Compiling pallet-multisig v28.0.0 (/home/bparity/parity/aaa/polkadot-sdk/substrate/frame/multisig)
error[E0277]: the trait bound `ConstU64<10>: frame_support::traits::Get<u32>` is not satisfied
   --> substrate/frame/multisig/src/tests.rs:41:1
    |
41  | #[derive_impl(frame_system::config_preludes::TestDefaultConfig)]
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `frame_support::traits::Get<u32>` is not implemented for `ConstU64<10>`
    |
    = help: the following other types implement trait `frame_support::traits::Get<T>`:
              <ConstU64<T> as frame_support::traits::Get<u64>>
              <ConstU64<T> as frame_support::traits::Get<std::option::Option<u64>>>
note: required by a bound in `frame_system::Config::BlockHashCount`
   --> /home/bparity/parity/aaa/polkadot-sdk/substrate/frame/system/src/lib.rs:535:24
    |
535 |         type BlockHashCount: Get<BlockNumberFor<Self>>;
    |                              ^^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `Config::BlockHashCount`
    = note: this error originates in the attribute macro `derive_impl` which comes from the expansion of the macro `frame_support::macro_magic::forward_tokens_verbatim` (in Nightly builds, run with -Z macro-backtrace for more info)

For more information about this error, try `rustc --explain E0277`.
error: could not compile `pallet-multisig` (lib test) due to 1 previous error 
```




## For reviewers:

(If there is a better solution, please let me know!)

The first commit contains actual attempt to fix the problem:
https://github.com/paritytech/polkadot-sdk/commit/3c5499e5

.
The second commit is just removal of `BlockHashCount` from all other
places where not needed by default.

Closes: https://github.com/paritytech/polkadot-sdk/issues/1657

---------

Co-authored-by: Bastian Köcher <[email protected]>

We need to use the `polkadot-ckb-merkle-mountain-range` dependency
published on `crates.io` in order to unblock the release of the
`sp-mmr-primitives` crate

The XCM docs were pretty much moved to the new rust docs format in
https://github.com/paritytech/polkadot-sdk/pull/2633, with the addition
of the XCM cookbook, which I plan to add more examples to shortly.

These docs were not mentioned in the polkadot-sdk rust docs, this PR
just mentions them there, so people can actually find them.

Define `OpaqueValue` and use it instead of
`grandpa::OpaqueKeyOwnershipProof` and `beefy:OpaqueKeyOwnershipProof`

Related to
https://github.com/paritytech/polkadot-sdk/pull/4522#discussion_r1608278279

We'll need to introduce a runtime API method that calls the
`report_fork_voting_unsigned()` extrinsic. This method will need to
receive the ancestry proof as a paramater. I'm still not sure, but there
is a chance that we'll send the ancestry proof as an opaque type.

So let's introduce this `OpaqueValue`. We can already use it to replace
`grandpa::OpaqueKeyOwnershipProof` and `beefy:OpaqueKeyOwnershipProof`
and maybe we'll need it for the ancestry proof as well.

---------

Co-authored-by: Bastian Köcher <[email protected]>

fix https://github.com/paritytech/polkadot-sdk/issues/4163

This PR does the following:
Update to pallet-contracts-proc-macro: 
- Parse #[cfg] so we can add a dummy noop host function for benchmark.
- Generate BenchEnv::<host_fn> so we can call host functions directly in
the benchmark.
- Add the weight of the noop host function before calling the host
function itself

Update benchmarks:
- Update all host function benchmark, a host function benchmark now
simply call the host function, instead of invoking the function n times
from within a contract.
- Refactor RuntimeCosts & Schedule, for most host functions, we can now
use the generated weight function directly instead of computing the diff
with the cost! macro

```rust
// Before
#[benchmark(pov_mode = Measured)]
fn seal_input(r: Linear<0, API_BENCHMARK_RUNS>) {
    let code = WasmModule::<T>::from(ModuleDefinition {
        memory: Some(ImportedMemory::max::<T>()),
        imported_functions: vec![ImportedFunction {
            module: "seal0",
            name: "seal_input",
            params: vec![ValueType::I32, ValueType::I32],
            return_type: None,
        }],
        data_segments: vec![DataSegment { offset: 0, value: 0u32.to_le_bytes().to_vec() }],
        call_body: Some(body::repeated(
            r,
            &[
                Instruction::I32Const(4), // ptr where to store output
                Instruction::I32Const(0), // ptr to length
                Instruction::Call(0),
            ],
        )),
        ..Default::default()
    });

    call_builder!(func, code);

    let res;
    #[block]
    {
        res = func.call();
    }
    assert_eq!(res.did_revert(), false);
}
```

```rust
// After
fn seal_input(n: Linear<0, { code::max_pages::<T>() * 64 * 1024 - 4 }>) {
    let mut setup = CallSetup::<T>::default();
    let (mut ext, _) = setup.ext();
    let mut runtime = crate::wasm::Runtime::new(&mut ext, vec![42u8; n as usize]);
    let mut memory = memory!(n.to_le_bytes(), vec![0u8; n as usize],);
    let result;
    #[block]
    {
        result = BenchEnv::seal0_input(&mut runtime, &mut memory, 4, 0)
    }
    assert_ok!(result);
    assert_eq!(&memory[4..], &vec![42u8; n as usize]);
}
``` 

[Weights
compare](https://weights.tasty.limo/compare?unit=weight&ignore_errors=true&threshold=10&method=asymptotic&repo=polkadot-sdk&old=master&new=pg%2Frework-host-benchs&path_pattern=substrate%2Fframe%2Fcontracts%2Fsrc%2Fweights.rs%2Cpolkadot%2Fruntime%2F*%2Fsrc%2Fweights%2F**%2F*.rs%2Cpolkadot%2Fbridges%2Fmodules%2F*%2Fsrc%2Fweights.rs%2Ccumulus%2F**%2Fweights%2F*.rs%2Ccumulus%2F**%2Fweights%2Fxcm%2F*.rs%2Ccumulus%2F**%2Fsrc%2Fweights.rs)

---------

Co-authored-by: command-bot <>
Co-authored-by: Alexander Theißen <[email protected]>

Cherry-picked fix from upcoming
https://github.com/paritytech/polkadot-sdk/pull/4494

---------

Co-authored-by: Svyatoslav Nikolsky <[email protected]>
Co-authored-by: command-bot <>

This one also works and it is easier.

Third and final PR in the set, closes
https://github.com/paritytech/polkadot-sdk/issues/454.

Original PR: https://github.com/paritytech/polkadot-sdk/pull/2680

## Precursors:
- https://github.com/paritytech/polkadot-sdk/pull/3889.
- https://github.com/paritytech/polkadot-sdk/pull/3904.

## Follow up issues/improvements
- https://github.com/paritytech/polkadot-sdk/issues/4404

Overall changes are documented here (lot more visual 😍

):
https://hackmd.io/@ak0n/454-np-governance

## Summary of various roles 🤯
### Pallet Staking
**Nominator**: An account that directly stakes on `pallet-staking` and
nominates a set of validators.
**Stakers**: Common term for nominators and validators.
Virtual Stakers: Same as stakers, but they are keyless accounts and
their locks are managed by a pallet external to `pallet-staking`.

### Pallet Delegated Staking
**Agent**: An account that receives delegation from other accounts
(delegators) and stakes on their behalf. They are also Virtual Stakers
in `pallet-staking` where `pallet-delegated-staking` manages its locks.
**Delegator**: An account that delegates some funds to an agent.

### Pallet Nomination Pools
**Pool account**: Keyless account of a pool where funds are pooled.
Members pledge their funds towards the pools. These are going to become
`Agent` accounts in `pallet-delegated-staking`.
**Pool Members**: They are individual members of the pool who
contributed funds to it. They are also `Delegator` in
`pallet-delegated-staking`.

## Changes
### Multiple Stake strategies

**TransferStake**: The current nomination pool logic can be considered a
staking strategy where delegators transfer funds to pool and stake. In
this scenario, funds are locked in pool account, and users lose the
control of their funds.

**DelegateStake**: With this PR, we introduce a new staking strategy
where individual delegators delegate fund to pool. `Delegate` implies
funds are locked in delegator account itself. Important thing to note
is, pool does not have funds of its own, but it has authorization from
its members to use these funds for staking.

We extract out all the interaction of pool with staking interface into a
new trait `StakeStrategy`. This is the logic that varies between the
above two staking strategies. We use the trait `StakeStrategy` to
implement above two strategies: `TransferStake` and `DelegateStake`.

### NominationPool
Consumes an implementation of `StakeStrategy` instead of
`StakingInterface`. I have renamed it from `Staking` to `StakeAdapter`
to clarify the difference from the earlier used trait.

To enable delegation based staking in pool, Nomination pool can be
configured as:
```
type StakeAdapter = pallet_nomination_pools::adapter::DelegateStake<Self, DelegatedStaking>;
```

Note that with the following configuration, the changes in the PR are
no-op.
```
type StakeAdapter = pallet_nomination_pools::adapter::TransferStake<Self, Staking>;
```

## Deployment roadmap
Plan to enable this only in Westend. In production runtimes, we can keep
pool to use `TransferStake` which will be no functional change.

Once we have a full audit, we can enable this in Kusama followed by
Polkadot.

## TODO
- [x] Runtime level (Westend) migration for existing nomination pools.
- [x] Permissionless call/ pallet::tasks for claiming delegator funds.
- [x] Add/update benches.
- [x] Migration tests.
- [x] Storage flag to mark `DelegateStake` migration and integrity
checks to not allow `TransferStake` for migrated runtimes.

---------

Signed-off-by: Matteo Muraca <[email protected]>
Signed-off-by: Alexandru Gheorghe <[email protected]>
Signed-off-by: Andrei Sandu <[email protected]>
Signed-off-by: Adrian Catangiu <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Signed-off-by: divdeploy <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: hongkuang <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: gemini132 <[email protected]>
Co-authored-by: Matteo Muraca <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Alexandru Gheorghe <[email protected]>
Co-authored-by: Alessandro Siniscalchi <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>
Co-authored-by: Ross Bulat <[email protected]>
Co-authored-by: Serban Iorga <[email protected]>
Co-authored-by: s0me0ne-unkn0wn <[email protected]>
Co-authored-by: Sam Johnson <[email protected]>
Co-authored-by: Adrian Catangiu <[email protected]>
Co-authored-by: Javier Viola <[email protected]>
Co-authored-by: Alexandru Vasile <[email protected]>
Co-authored-by: Niklas Adolfsson <[email protected]>
Co-authored-by: Dastan <[email protected]>
Co-authored-by: Clara van Staden <[email protected]>
Co-authored-by: Ron <[email protected]>
Co-authored-by: Vincent Geddes <[email protected]>
Co-authored-by: Svyatoslav Nikolsky <[email protected]>
Co-authored-by: Michal Kucharczyk <[email protected]>
Co-authored-by: Dino Pačandi <[email protected]>
Co-authored-by: Andrei Eres <[email protected]>
Co-authored-by: Alin Dima <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Branislav Kontur <[email protected]>
Co-authored-by: Sebastian Kunert <[email protected]>
Co-authored-by: gupnik <[email protected]>
Co-authored-by: Vladimir Istyufeev <[email protected]>
Co-authored-by: Lulu <[email protected]>
Co-authored-by: Juan Girini <[email protected]>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Dónal Murray <[email protected]>
Co-authored-by: Shawn Tabrizi <[email protected]>
Co-authored-by: Kutsal Kaan Bilgin <[email protected]>
Co-authored-by: Ermal Kaleci <[email protected]>
Co-authored-by: ordian <[email protected]>
Co-authored-by: divdeploy <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Sergej Sakac <[email protected]>
Co-authored-by: Squirrel <[email protected]>
Co-authored-by: HongKuang <[email protected]>
Co-authored-by: Tsvetomir Dimitrov <[email protected]>
Co-authored-by: Egor_P <[email protected]>
Co-authored-by: Aaro Altonen <[email protected]>
Co-authored-by: Dmitry Markin <[email protected]>
Co-authored-by: Alexandru Vasile <[email protected]>
Co-authored-by: Léa Narzis <[email protected]>
Co-authored-by: Gonçalo Pestana <[email protected]>
Co-authored-by: georgepisaltu <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: PG Herveou <[email protected]>
Co-authored-by: jimwfs <[email protected]>
Co-authored-by: jimwfs <[email protected]>
Co-authored-by: polka.dom <[email protected]>

..and other high level docs.


# Polling 

Please vote in the reactions of this PR 

- 👍 I agree to replace the website of this repo to
https://paritytech.github.io/polkadot-sdk/master/polkadot_sdk_docs/index.html
- 👎 Keep it as polkadot.network
- 🤷

 Different opinion

---------

Co-authored-by: Bastian Köcher <[email protected]>

This implements the `CheckMetadataHash` extension as described in
[RFC78](https://polkadot-fellows.github.io/RFCs/approved/0078-merkleized-metadata.html).

Besides the signed extension, the `substrate-wasm-builder` is extended
to support generating the metadata-hash.

Closes: https://github.com/paritytech/polkadot-sdk/issues/291

---------

Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>

We are actually using an older version, where the log line is in a
different file.

This PR backports version bumps and reorganisation of the prdoc files
from the `1.12.0` release branch back to `master`

Closes #4366

This version includes the latest release of pjs/api
(https://github.com/polkadot-js/api/releases/tag/v11.1.1).
Thx!

This PR introduces custom types / substrate wrappers for `Multiaddr`,
`multiaddr::Protocol`, `Multihash`, `ed25519::*` and supplementary types
like errors and iterators.

This is needed to unblock `libp2p` upgrade PR
https://github.com/paritytech/polkadot-sdk/pull/1631 after
https://github.com/paritytech/polkadot-sdk/pull/2944 was merged.
`libp2p` and `litep2p` currently depend on different versions of
`multiaddr` crate, and introduction of this "common ground" types is
needed to support independent version upgrades of `multiaddr` and
dependent crates in `libp2p` & `litep2p`.

While being just convenient to not tie versions of `libp2p` & `litep2p`
dependencies together, it's currently not even possible to keep `libp2p`
& `litep2p` dependencies updated to the same versions as `multiaddr` in
`libp2p` depends on `libp2p-identity` that we can't include as a
dependency of `litep2p`, which has it's own `PeerId` type. In the
future, to keep things updated on `litep2p` side, we will likely need to
fork `multiaddr` and make it use `litep2p` `PeerId` as a payload of
`/p2p/...` protocol.

With these changes, common code in substrate uses these custom types,
and `litep2p` & `libp2p` backends use corresponding libraries types.

closes https://github.com/paritytech/parity-bridges-common/issues/2963

See issue above for rationale
I've been thinking about adding similar calls to other pallets, but:
- for parachains pallet I haven't been able to think of a case when we
will need that given how long referendum takes. I.e. if storage proof
format changes and we want to unstuck the bridge, it'll take a large a
few weeks to sync a single parachain header, then another weeks for
another and etc.
- for messages pallet I've made the similar call initially, but it just
changes a storage key (`OutboundLanes` and/or `InboundLanes`), so
there's no any logic here and it may be simply done using
`system.set_storage`.

---------

Co-authored-by: command-bot <>

closes https://github.com/paritytech/parity-bridges-common/issues/3000

Recently we've changed our bridge configuration for Rococo <> Westend
and our new relayer has started to submit transactions every ~ `30`
seconds. Eventually, it switches itself into limbo state, where it can't
submit more transactions - all `author_submitAndWatchExtrinsic` calls
are failing with the following error: `ERROR bridge Failed to send
transaction to BridgeHubRococo node: Call(ErrorObject { code:
ServerError(-32006), message: "Too many subscriptions on the
connection", data: Some(RawValue("Exceeded max limit of 1024")) })`.

Some links for those who want to explore:
- server side (node) has a strict limit on a number of active
subscriptions. It fails to open a new subscription if this limit is hit:
https://github.com/paritytech/jsonrpsee/blob/a4533966b997e83632509ad97eea010fc7c3efc0/server/src/middleware/rpc/layer/rpc_service.rs#L122-L132.
The limit is set to `1024` by default;
- internally this limit is a semaphore with `limit` permits:
https://github.com/paritytech/jsonrpsee/blob/a4533966b997e83632509ad97eea010fc7c3efc0/core/src/server/subscription.rs#L461-L485;
- semaphore permit is acquired in the first link;
- the permit is "returned" when the `SubscriptionSink` is dropped:
https://github.com/paritytech/jsonrpsee/blob/a4533966b997e83632509ad97eea010fc7c3efc0/core/src/server/subscription.rs#L310-L325;
- the `SubscriptionSink` is dropped when [this `polkadot-sdk`
function](https://github.com/paritytech/polkadot-sdk/blob/278486f9/substrate/client/rpc/src/utils.rs#L58-L94)
returns. In other words - when the connection is closed, the stream is
finished or internal subscription buffer limit is hit;
- the subscription has the internal buffer, so sending an item contains
of two steps: [reading an item from the underlying
stream](https://github.com/paritytech/polkadot-sdk/blob/278486f9/substrate/client/rpc/src/utils.rs#L125-L141)
and [sending it over the
connection](https://github.com/paritytech/polkadot-sdk/blob/278486f9/substrate/client/rpc/src/utils.rs#L111-L116);
- when the underlying stream is finished, the `inner_pipe_from_stream`
wants to ensure that all items are sent to the subscriber. So it: [waits
until the current send operation
completes](https://github.com/paritytech/polkadot-sdk/blob/278486f9/substrate/client/rpc/src/utils.rs#L146-L148)
and then [send all remaining items from the internal
buffer](https://github.com/paritytech/polkadot-sdk/blob/278486f9/substrate/client/rpc/src/utils.rs#L150-L155).
Once it is done, the function returns, the `SubscriptionSink` is
dropped, semaphore permit is dropped and we are ready to accept new
subscriptions;
- unfortunately, the code just calls the `pending_fut.await.is_err()` to
ensure that [the current send operation
completes](https://github.com/paritytech/polkadot-sdk/blob/278486f9/substrate/client/rpc/src/utils.rs#L146-L148).
But if there are no current send operation (which is normal), then the
`pending_fut` is set to terminated future and the `await` never
completes. Hence, no return from the function, no drop of
`SubscriptionSink`, no drop of semaphore permit, no new subscriptions
allowed (once number of susbcriptions hits the limit.

I've illustrated the issue with small test - you may ensure that if e.g.
the stream is initially empty, the
`subscription_is_dropped_when_stream_is_empty` will hang because
`pipe_from_stream` never exits.

Co-authored-by: Svyatoslav Nikolsky <[email protected]>

Implements https://github.com/paritytech/polkadot-sdk/issues/4429

Collators only need to maintain the implicit view for the paraid they
are collating on.
In this case, bypass prospective-parachains entirely. It's still useful
to use the GetMinimumRelayParents message from prospective-parachains
for validators, because the data is already present there.

This enables us to entirely remove the subsystem from collators, which
consumed resources needlessly

Aims to resolve https://github.com/paritytech/polkadot-sdk/issues/4167 

TODO:
- [x] fix unit tests

As per #3326, removes pallet::getter macro usage from
pallet-fast-unstake. The syntax `StorageItem::<T, I>::get()` should be
used instead.

cc @muraca

---------

Co-authored-by: Liam Aharon <[email protected]>

Replace usage of deprecated
`substrate_rpc_client::ChildStateApi::storage_keys` with
`substrate_rpc_client::ChildStateApi::storage_keys_paged`.

Required for successful scraping of Aleph Zero state.

Co-authored-by: jimwfs <[email protected]>

addresses https://github.com/paritytech/polkadot-sdk/issues/4440 (will
close once we have this in prod runtimes).
related: https://github.com/paritytech/polkadot-sdk/issues/2037.

An extra consumer reference is preventing pools to be destroyed. When a
pool is ready to be destroyed, we
can safely clear the consumer references if any. Notably, I only check
for one extra consumer reference since that is a known bug. Anything
more indicates possibly another issue and we probably don't want to
silently absorb those errors as well.

After this change, pools with extra consumer reference should be able to
destroy normally.

Changes the Ethereum client storage scope to public, so it can be set in
a migration.

When merged, we should backport to the all other release branches:

- [ ] release-crates-io-v1.7.0 - patch release the fellows BridgeHubs
runtimes https://github.com/paritytech/polkadot-sdk/pull/4504
- [ ] release-crates-io-v1.8.0 -
https://github.com/paritytech/polkadot-sdk/pull/4505
- [ ] release-crates-io-v1.9.0 -
https://github.com/paritytech/polkadot-sdk/pull/4506
- [ ] release-crates-io-v1.10.0 -
https://github.com/paritytech/polkadot-sdk/pull/4507
- [ ] release-crates-io-v1.11.0 -
https://github.com/paritytech/polkadot-sdk/pull/4508
- [ ] release-crates-io-v1.12.0 (commit soon)

Co-authored-by: command-bot <>

Before relayer crates have been moved + merged, the `MetricsParams` type
has been created from a `substrate-relay` crate (binary) and hence it
has been setting the `substrate_relay_build_info` metic value properly -
to the binary version. Now it is created from the
`substrate-relay-helper` crate, which has the fixed (it isn't published)
version `0.1.0`, so our relay provides incorrect metric value. This
'breaks' our monitoring tools - we see that all relayers have that
incorrect version, which is not cool.

The idea is to have a global static variable (shame on me) that is
initialized by the binary during initialization like we do with the
logger initialization already. Was considering some alternative options:
- adding a separate argument to every relayer subcommand and propagating
it to the `MetricsParams::new()` causes a lot of changes and introduces
even more noise to the binary code, which is supposed to be as small as
possible in the new design. But I could do that if team thinks it is
better;
- adding a `structopt(skip) pub relayer_version: RelayerVersion`
argument to all subcommand params won't work, because it will be
initialized by default and `RelayerVersion` needs to reside in some util
crate (not the binary), so it'll have the wrong value again.

Using Dynamic Parameters for contracts seems like a bad idea for now.

Given that we have benchmarks for each host function (in addition to our
extrinsics), parameter storage reads will be counted multiple times. We
will work on updates to the benchmarking framework to mitigate this
issue in future iterations.

---------

Co-authored-by: command-bot <>