fixes https://github.com/paritytech/polkadot-sdk/issues/3775



Additionally moves the claim queue fetch utilities into
`subsystem-util`.

TODO:
- [x] fix tests
- [x] add elastic scaling tests

---------

Signed-off-by: Andrei Sandu <[email protected]>

Fixes
https://github.com/paritytech/polkadot-sdk/issues/3884#issuecomment-2026058687

After moving regression tests to benchmarks
(https://github.com/paritytech/polkadot-sdk/pull/3741

) we don't need to
filter tests anymore.

---------

Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>
Co-authored-by: Alin Dima <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Javier Viola <[email protected]>
Co-authored-by: Serban Iorga <[email protected]>
Co-authored-by: Adrian Catangiu <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Alexandru Vasile <[email protected]>
Co-authored-by: Niklas Adolfsson <[email protected]>
Co-authored-by: Dastan <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Clara van Staden <[email protected]>
Co-authored-by: Ron <[email protected]>
Co-authored-by: Vincent Geddes <[email protected]>
Co-authored-by: Svyatoslav Nikolsky <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Adds implementation for `SortedMembers::add` for _pallet-membership_
benchmarks.

---------

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: command-bot <>

Added `TryFrom<&'a RuntimeString> for &'a str`

This PR includes the following 2 improvements:

## Ethereum Client

Author: @yrong 
### Original Upstream PRs
- https://github.com/Snowfork/polkadot-sdk/pull/123
- https://github.com/Snowfork/polkadot-sdk/pull/125

### Description
The Ethereum client syncs beacon headers as they are finalized, and
imports every execution header. When a message is received, it is
verified against the import execution header. This is unnecessary, since
the execution header can be sent with the message as proof. The recent
Deneb Ethereum upgrade made it easier to locate the relevant beacon
header from an execution header, and so this improvement was made
possible. This resolves a concern @svyatonik had in our initial Rococo
PR:
https://github.com/paritytech/polkadot-sdk/pull/2522#discussion_r1431270691

## Inbound Queue

Author: @yrong 
### Original Upstream PR
- https://github.com/Snowfork/polkadot-sdk/pull/118

### Description
When the AH sovereign account (who pays relayer rewards) is depleted,
the inbound message will not fail. The relayer just will not receive
rewards.

Both these changes were done by @yrong, many thanks. ❤

️

---------

Co-authored-by: claravanstaden <Cats 4 life!>
Co-authored-by: Ron <[email protected]>
Co-authored-by: Vincent Geddes <[email protected]>
Co-authored-by: Svyatoslav Nikolsky <[email protected]>

migrations: prevent accidentally using unversioned migrations instead of `VersionedMigration` (#3835)

closes #1324 

#### Problem
Currently, it is possible to accidentally use inner unversioned
migration instead of `VersionedMigration` since both implement
`OnRuntimeUpgrade`.

#### Solution

With this change, we make it clear that value of `Inner` is not intended
to be used directly. It is achieved by bounding `Inner` to new trait
`UncheckedOnRuntimeUpgrade`, which has the same interface (except
`unchecked_` prefix) as `OnRuntimeUpgrade`.

#### `try-runtime` functions

Since developers can implement `try-runtime` for `Inner` value in
`VersionedMigration` and have custom logic for it, I added the same
`try-runtime` functions to `UncheckedOnRuntimeUpgrade`. I looked for a
ways to not duplicate functions, but couldn't find anything that doesn't
significantly change the codebase. So I would appreciate If you have any
suggestions to improve this

cc @liamaharon

 @xlc 

polkadot address: 16FqwPZ8GRC5U5D4Fu7W33nA55ZXzXGWHwmbnE1eT6pxuqcT

---------

Co-authored-by: Liam Aharon <[email protected]>

Working towards migrating the `parity-bridges-common` repo inside
`polkadot-sdk`. This PR upgrades some dependencies in order to align
them with the versions used in `parity-bridges-common`

Related to
https://github.com/paritytech/parity-bridges-common/issues/2538

chainHead: Allow methods to be called from within a single connection context and limit connections (#3481)

This PR ensures that the chainHead RPC class can be called only from
within the same connection context.

The chainHead methods are now registered as raw methods. 
- https://github.com/paritytech/jsonrpsee/pull/1297
The concept of raw methods is introduced in jsonrpsee, which is an async
method that exposes the connection ID:
The raw method doesn't have the concept of a blocking method. Previously
blocking methods are now spawning a blocking task to handle their
blocking (ie DB) access. We spawn the same number of tasks as before,
however we do that explicitly.

Another approach would be implementing a RPC middleware that captures
and decodes the method parameters:
- https://github.com/paritytech/polkadot-sdk/pull/3343
However, that approach is prone to errors since the methods are
hardcoded by name. Performace is affected by the double deserialization
that needs to happen to extract the subscription ID we'd like to limit.
Once from the middleware, and once from the methods itself.

This PR paves the way to implement the chainHead connection limiter:
- https://github.com/paritytech/polkadot-sdk/issues/1505
Registering tokens (subscription ID / operation ID) on the
`RpcConnections` could be extended to return an error when the maximum
number of operations is reached.

While at it, have added an integration-test to ensure that chainHead
methods can be called from within the same connection context.

Before this is merged, a new JsonRPC release should be made to expose
the `raw-methods`:
- [x] Use jsonrpsee from crates io (blocked by:
https://github.com/paritytech/jsonrpsee/pull/1297)

Closes: https://github.com/paritytech/polkadot-sdk/issues/3207




cc @paritytech/subxt-team

---------

Signed-off-by: Alexandru Vasile <[email protected]>
Co-authored-by: Niklas Adolfsson <[email protected]>

This outputs:
```
2024-04-02 14:36:02.135 ERROR tokio-runtime-worker beefy: 🥩 for session starting at block 21990151
no BEEFY authority key found in store, you must generate valid session keys
(https://wiki.polkadot.network/docs/maintain-guides-how-to-validate-polkadot#generating-the-session-keys

)
```
error log entry, once every session, for nodes running with
`Role::Authority` that have no public BEEFY key in their keystore

---------

Co-authored-by: Bastian Köcher <[email protected]>

Pulling the latest changes from `parity-bridges-common`

This version includes:

- Internal metrics of zombienet (used to benchmark with v2).

When using `schedule_code_upgrade` to change the code of a parachain in
the relay chain runtime, we had already fixed to not set the `GoAhead`
signal. This was done to not brick any parachain after the upgrade,
because they were seeing the signal without having any upgrade prepared.
The remaining problem is that the parachain code is only upgraded after
a parachain header was enacted, aka the parachain made some progress.
However, this is quite complicated if the parachain is bricked (which is
the most common scenario why to manually schedule a code upgrade). Thus,
this pull request replaces `SetGoAhead` with `UpgradeStrategy` to signal
to the logic kind of strategy want to use. The strategies are either
`SetGoAheadSignal` or `ApplyAtExpectedBlock`. `SetGoAheadSignal` sets
the go ahead signal as before and awaits a parachain block.
`ApplyAtExpectedBlock` schedules the upgrade and applies it directly at
the `expected_block` without waiting for the parachain to make any kind
of progress.

Fix "double-weights" for extrinsics, use only the ones benchmarked in
the runtime.

Deprecate extrinsics that don't specify WeightLimit, remove their usage
across the repo.

---------

Signed-off-by: Adrian Catangiu <[email protected]>
Co-authored-by: command-bot <>

derive-syn-parse v0.2.0 came out recently which (finally) adds support
for syn 2x.

Upgrading to this will remove many of the places where syn 1x was still
compiling alongside syn 2x in the polkadot-sdk workspace.

This also upgrades `docify` to 0.2.8 which is the version that upgrades
derive-syn-pasre to 0.2.0.

Additionally, this consolidates the `docify` versions in the repo to all
use the latest, and in one case upgrades to the 0.2x syntax where 0.1.x
was still being used.

---------

Co-authored-by: Liam Aharon <[email protected]>

Rejoice! Rejoice! The story is nearly over.

This PR removes stale migrations, auxiliary structures, and package
dependencies, thus making Rococo and Westend totally free from any
`im-online`-related stuff.

`im-online` still stays a part of the Substrate node and its runtime:
https://github.com/paritytech/polkadot-sdk/blob/0d9324847391e902bb42f84f0e76096b1f764efe/substrate/bin/node/runtime/src/lib.rs#L2276-L2277
I'm not sure if it makes sense to remove it from there considering that
we're not removing `im-online` from FRAME. Please share your opinion.

fix https://github.com/paritytech/polkadot-sdk/issues/3921



---------

Signed-off-by: Andrei Sandu <[email protected]>

Fix links

Related CI failure:
https://github.com/paritytech/polkadot-sdk/actions/runs/8455425042/job/23162858534?pr=3859

Runtime release 1.2 includes bumping of the ParachainHost APIs up to
v10, so let's move all the released APIs out of vstaging folder, this PR
does not include any logic changes only renaming of the modules and some
moving around.

Signed-off-by: Alexandru Gheorghe <[email protected]>

Test started failing after
https://github.com/paritytech/polkadot-sdk/commit/66051adb619d2119771920218e2de75fa037d7e8


which enabled approval coalescing, that was expected to happen because
the test required an polkadot_parachain_approval_checking_finality_lag
of 0, which can't happen with max_approval_coalesce_count greater than 1
because we always delay the approval for no_show_duration_ticks/2 in
case we can coalesce it with other approvals.


So relax a bit the restrictions, since we don't actually care that the
lags are 0, but the fact the finalities are progressing and are not
stuck.

Signed-off-by: Alexandru Gheorghe <[email protected]>

Related Issue https://github.com/paritytech/polkadot-sdk/issues/3398

This PR makes permissionless withdrawing the default option, giving any
network participant access to claim pool rewards on member's behalf. Of
course, members can still opt out of this by setting a `Permissioned`
claim permission.

Permissionless claiming has been a part of the nomination pool pallet
for around 9 months now, with very limited uptake (~4% of total pool
members). 1.6% of pool members are using `PermissionlessAll`, strongly
suggesting it is not wanted - it is too ambiguous and doesn't provide
guidance to claimers.

Stakers expect rewards to be claimed on their behalf by default - I have
expanded upon this in detail within the [accompanying issue's
discussion](https://github.com/paritytech/polkadot-sdk/issues/3398).
Other protocols have this behaviour, whereby staking rewards are
received without the staker having to take any action. From this
perspective, permissionless claiming is not intuitive for pool members.
As evidence of this, over 150,000 DOT is currently unclaimed on
Polkadot, and is growing at a non-linear rate.

fixes https://github.com/paritytech/polkadot-sdk/issues/3886



---------

Signed-off-by: Andrei Sandu <[email protected]>

This PR introduces a refactor of the parachain runtime configuration by
consolidating all pallet configurations into a new module named
`configs`. This change aims to improve the readability and
maintainability of the runtime configuration by centralizing all
configuration parameters.

## Changes
- **Creation of `configs.rs`**: A new file `configs.rs` has been added
under `templates/parachain/runtime/src/`, containing all the runtime
configurations previously scattered across `lib.rs`.
- **Refactoring of `lib.rs`**: The `lib.rs` file has been significantly
slimmed down by removing the inline pallet configurations and importing
them from `configs.rs` instead.
- **Optimization of Import Statements**: Reorganized import statements
to clarify the runtime's dependency structure.

### Benefits
- **Improved Readability**: With configurations being centralized,
developers can now easily locate and review runtime parameters without
navigating through the `lib.rs` file.

This refactor does not introduce any changes to the runtime logic but
improves the project structure for better development experience.

---------

Co-authored-by: Kian Paimani <[email protected]>

The metric records the current protocol_version of the validator that
just connected with the peer_map.len(), which contains all peers that
connected, that has the effect the metric will be wrong since it won't
tell us how many peers we have connected per version because it will
always record the total number of peers

Fix this by counting by version inside peer_map, additionally because
that might be a bit heavier than len(), publish it only on-active
leaves.

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>

Part of #3326 

cc @Kianenigma @ggwpez @liamaharon

 

polkadot address: 12poSUQPtcF1HUPQGY3zZu2P8emuW9YnsPduA4XG3oCEfJVp

---------

Signed-off-by: Matteo Muraca <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>

Closes: https://github.com/paritytech/polkadot-sdk/issues/3906

Closes https://github.com/paritytech/polkadot-sdk-docs/issues/70

WIP PR for an overview of how to develop tokens in FRAME. 

- [x] Tokens in Substrate Ref Doc
  - High-level overview of the token-related logic in FRAME
- Improve docs with better explanation of how holds, freezes, ed, free
balance, etc, all work
- [x] Update `pallet_balances` docs
  - Clearly mark what is deprecated (currency)
- [x] Write fungible trait docs
- [x] Evaluate and if required update `pallet_assets`, `pallet_uniques`,
`pallet_nfts` docs
- [x] Absorb https://github.com/paritytech/polkadot-sdk/pull/2683/


- [x] Audit individual trait method docs, and improve if possible

Feel free to suggest additional TODOs for this PR in the comments

---------

Co-authored-by: Bill Laboon <[email protected]>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Sebastian Kunert <[email protected]>

Fixes https://github.com/paritytech/polkadot-sdk/issues/3887.

Currently, all protocols use the same metric name for
`mpsc-notification-to-protocol` this is bad because we can't actually
tell which protocol might cause problems.

This patch proposes we derive the name of the metric from the protocol
name, so that we have separate metrics for each protocol and properly
detect which one is having problem processing its messages.

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>

Removes transient code introduced to clean up offchain database after
`im-online` pallet removal.

Should be merged after #2290 has been enacted.

Somehow https://github.com/paritytech/polkadot-sdk/pull/3795

 was merged
but tests are failing now on master. I suspect that CI is not even
running these tests anymore which is a big issue.

---------

Signed-off-by: Andrei Sandu <[email protected]>

PR adds variables validation and app credentials for pushing into
gh-pages

cc https://github.com/paritytech/ci_cd/issues/934

# Description

Since the binary split additional syscalls are getting blocked in
relation to the workers.

With the hardened systemd file it shows the following warning:

```
Cannot fully enable landlock, a Linux kernel security feature. Running validation of malicious PVF code has a higher risk of compromising this machine. Consider upgrading the kernel version for maximum security. status=Ok(NotEnforced) abi=1
```

For it to work we need to allow additionally:
- mount
- umount2
- pivot_root

and set `RestrictNamespaces=false`

Added new line `SystemCallFilter=pivot_root` because otherwise it would
get blocked by ~\@\privileged

Co-authored-by: s0me0ne-unkn0wn <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Part of: https://github.com/paritytech/polkadot-sdk/issues/239

Polkadot address: 12GyGD3QhT4i2JJpNzvMf96sxxBLWymz4RdGCxRH5Rj5agKW

This PR exports unified hostfunctions needed for parachains. Basicaly
`SubstrateHostFunctions` + `storage_proof_size::HostFunctions`.

Also removes the native executor from the parachain template.

---------

Co-authored-by: Michal Kucharczyk <[email protected]>

Fix reference links

Co-authored-by: Bastian Köcher <[email protected]>

Cleanup tests (-2.7k lines !) using some builder patterns to build
pallet_contracts api calls

This PR significantly refactors the runtime API implementations to
improve project structure, maintainability, and readability. Key changes
include:

1. **Enhancing Visibility**: Adjusts the visibility of
`RUNTIME_API_VERSIONS` in `impl_runtime_apis.rs` to `pub`, making it
accessible throughout the runtime module.
2. **Centralizing API Implementations**: Introduces a new file,
`apis.rs`, within the parachain template's runtime directory.
3. **Streamlining `lib.rs`**: Updates the main runtime library file to
reflect these structural changes. It removes redundant API
implementations and points `VERSION` to the newly exposed
`RUNTIME_API_VERSIONS` from `apis.rs`, simplifying the overall runtime
configuration.

### Motivations Behind the Refactoring:
- **Improved Project Structure**: Centralizing API implementations in
`apis.rs` offers a clearer, more navigable project structure.
- **Better Readability**: Streamlining `lib.rs` and reducing clutter
enhance readability, making it easier for new contributors to understand
the project layout and logic.

### Summary of Changes:
- Made `RUNTIME_API_VERSIONS` public in `impl_runtime_apis.rs`.
- Added `apis.rs` to centralize runtime API implementations.
- Streamlined `lib.rs` to adjust to the refactored project structure.

A prerequisite for adding a stable branch and respecting SemVer on new
stable releases is including SemVer bump levels in our PRDocs.

Next release is scheduled for April 3rd, so it would be great to get
this merged before then.

Also added "None" as a valid bump option, to support test/benchmark
changes and CI to ensure changed crates have an entry.