Rejoice! Rejoice! The story is nearly over.

This PR removes stale migrations, auxiliary structures, and package
dependencies, thus making Rococo and Westend totally free from any
`im-online`-related stuff.

`im-online` still stays a part of the Substrate node and its runtime:
https://github.com/paritytech/polkadot-sdk/blob/0d932484/substrate/bin/node/runtime/src/lib.rs#L2276-L2277
I'm not sure if it makes sense to remove it from there considering that
we're not removing `im-online` from FRAME. Please share your opinion.

fix https://github.com/paritytech/polkadot-sdk/issues/3921

---------

Signed-off-by: Andrei Sandu <[email protected]>

Fix links

Related CI failure:
https://github.com/paritytech/polkadot-sdk/actions/runs/8455425042/job/23162858534?pr=3859

Runtime release 1.2 includes bumping of the ParachainHost APIs up to
v10, so let's move all the released APIs out of vstaging folder, this PR
does not include any logic changes only renaming of the modules and some
moving around.

Signed-off-by: Alexandru Gheorghe <[email protected]>

Test started failing after
https://github.com/paritytech/polkadot-sdk/commit/66051adb


which enabled approval coalescing, that was expected to happen because
the test required an polkadot_parachain_approval_checking_finality_lag
of 0, which can't happen with max_approval_coalesce_count greater than 1
because we always delay the approval for no_show_duration_ticks/2 in
case we can coalesce it with other approvals.


So relax a bit the restrictions, since we don't actually care that the
lags are 0, but the fact the finalities are progressing and are not
stuck.

Signed-off-by: Alexandru Gheorghe <[email protected]>

Related Issue https://github.com/paritytech/polkadot-sdk/issues/3398

This PR makes permissionless withdrawing the default option, giving any
network participant access to claim pool rewards on member's behalf. Of
course, members can still opt out of this by setting a `Permissioned`
claim permission.

Permissionless claiming has been a part of the nomination pool pallet
for around 9 months now, with very limited uptake (~4% of total pool
members). 1.6% of pool members are using `PermissionlessAll`, strongly
suggesting it is not wanted - it is too ambiguous and doesn't provide
guidance to claimers.

Stakers expect rewards to be claimed on their behalf by default - I have
expanded upon this in detail within the [accompanying issue's
discussion](https://github.com/paritytech/polkadot-sdk/issues/3398).
Other protocols have this behaviour, whereby staking rewards are
received without the staker having to take any action. From this
perspective, permissionless claiming is not intuitive for pool members.
As evidence of this, over 150,000 DOT is currently unclaimed on
Polkadot, and is growing at a non-linear rate.

fixes https://github.com/paritytech/polkadot-sdk/issues/3886

---------

Signed-off-by: Andrei Sandu <[email protected]>

This PR introduces a refactor of the parachain runtime configuration by
consolidating all pallet configurations into a new module named
`configs`. This change aims to improve the readability and
maintainability of the runtime configuration by centralizing all
configuration parameters.

## Changes
- **Creation of `configs.rs`**: A new file `configs.rs` has been added
under `templates/parachain/runtime/src/`, containing all the runtime
configurations previously scattered across `lib.rs`.
- **Refactoring of `lib.rs`**: The `lib.rs` file has been significantly
slimmed down by removing the inline pallet configurations and importing
them from `configs.rs` instead.
- **Optimization of Import Statements**: Reorganized import statements
to clarify the runtime's dependency structure.

### Benefits
- **Improved Readability**: With configurations being centralized,
developers can now easily locate and review runtime parameters without
navigating through the `lib.rs` file.

This refactor does not introduce any changes to the runtime logic but
improves the project structure for better development experience.

---------

Co-authored-by: Kian Paimani <[email protected]>

The metric records the current protocol_version of the validator that
just connected with the peer_map.len(), which contains all peers that
connected, that has the effect the metric will be wrong since it won't
tell us how many peers we have connected per version because it will
always record the total number of peers

Fix this by counting by version inside peer_map, additionally because
that might be a bit heavier than len(), publish it only on-active
leaves.

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>

Part of #3326 

cc @Kianenigma @ggwpez @liamaharon

 

polkadot address: 12poSUQPtcF1HUPQGY3zZu2P8emuW9YnsPduA4XG3oCEfJVp

---------

Signed-off-by: Matteo Muraca <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>

Closes: https://github.com/paritytech/polkadot-sdk/issues/3906

Closes https://github.com/paritytech/polkadot-sdk-docs/issues/70

WIP PR for an overview of how to develop tokens in FRAME. 

- [x] Tokens in Substrate Ref Doc
  - High-level overview of the token-related logic in FRAME
- Improve docs with better explanation of how holds, freezes, ed, free
balance, etc, all work
- [x] Update `pallet_balances` docs
  - Clearly mark what is deprecated (currency)
- [x] Write fungible trait docs
- [x] Evaluate and if required update `pallet_assets`, `pallet_uniques`,
`pallet_nfts` docs
- [x] Absorb https://github.com/paritytech/polkadot-sdk/pull/2683/
- [x] Audit individual trait method docs, and improve if possible

Feel free to suggest additional TODOs for this PR in the comments

---------

Co-authored-by: Bill Laboon <[email protected]>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Sebastian Kunert <[email protected]>

Fixes https://github.com/paritytech/polkadot-sdk/issues/3887.

Currently, all protocols use the same metric name for
`mpsc-notification-to-protocol` this is bad because we can't actually
tell which protocol might cause problems.

This patch proposes we derive the name of the metric from the protocol
name, so that we have separate metrics for each protocol and properly
detect which one is having problem processing its messages.

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>

Removes transient code introduced to clean up offchain database after
`im-online` pallet removal.

Should be merged after #2290 has been enacted.

Somehow https://github.com/paritytech/polkadot-sdk/pull/3795 was merged
but tests are failing now on master. I suspect that CI is not even
running these tests anymore which is a big issue.

---------

Signed-off-by: Andrei Sandu <[email protected]>

PR adds variables validation and app credentials for pushing into
gh-pages

cc https://github.com/paritytech/ci_cd/issues/934

# Description

Since the binary split additional syscalls are getting blocked in
relation to the workers.

With the hardened systemd file it shows the following warning:

```
Cannot fully enable landlock, a Linux kernel security feature. Running validation of malicious PVF code has a higher risk of compromising this machine. Consider upgrading the kernel version for maximum security. status=Ok(NotEnforced) abi=1
```

For it to work we need to allow additionally:
- mount
- umount2
- pivot_root

and set `RestrictNamespaces=false`

Added new line `SystemCallFilter=pivot_root` because otherwise it would
get blocked by ~\@\privileged

Co-authored-by: s0me0ne-unkn0wn <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Part of: https://github.com/paritytech/polkadot-sdk/issues/239

Polkadot address: 12GyGD3QhT4i2JJpNzvMf96sxxBLWymz4RdGCxRH5Rj5agKW

This PR exports unified hostfunctions needed for parachains. Basicaly
`SubstrateHostFunctions` + `storage_proof_size::HostFunctions`.

Also removes the native executor from the parachain template.

---------

Co-authored-by: Michal Kucharczyk <[email protected]>

Fix reference links

Co-authored-by: Bastian Köcher <[email protected]>

Cleanup tests (-2.7k lines !) using some builder patterns to build
pallet_contracts api calls

This PR significantly refactors the runtime API implementations to
improve project structure, maintainability, and readability. Key changes
include:

1. **Enhancing Visibility**: Adjusts the visibility of
`RUNTIME_API_VERSIONS` in `impl_runtime_apis.rs` to `pub`, making it
accessible throughout the runtime module.
2. **Centralizing API Implementations**: Introduces a new file,
`apis.rs`, within the parachain template's runtime directory.
3. **Streamlining `lib.rs`**: Updates the main runtime library file to
reflect these structural changes. It removes redundant API
implementations and points `VERSION` to the newly exposed
`RUNTIME_API_VERSIONS` from `apis.rs`, simplifying the overall runtime
configuration.

### Motivations Behind the Refactoring:
- **Improved Project Structure**: Centralizing API implementations in
`apis.rs` offers a clearer, more navigable project structure.
- **Better Readability**: Streamlining `lib.rs` and reducing clutter
enhance readability, making it easier for new contributors to understand
the project layout and logic.

### Summary of Changes:
- Made `RUNTIME_API_VERSIONS` public in `impl_runtime_apis.rs`.
- Added `apis.rs` to centralize runtime API implementations.
- Streamlined `lib.rs` to adjust to the refactored project structure.

A prerequisite for adding a stable branch and respecting SemVer on new
stable releases is including SemVer bump levels in our PRDocs.

Next release is scheduled for April 3rd, so it would be great to get
this merged before then.

Also added "None" as a valid bump option, to support test/benchmark
changes and CI to ensure changed crates have an entry.

This PR add `#[deprecated]` attribute to v1 and v2 of the schedule
trait. Proposed in this issue:
https://github.com/paritytech/polkadot-sdk/issues/3676

```rust
#[allow(deprecated)]
#[deprecated = "traits::schedule::v1 is deprecated. Please use v3 instead."]
pub mod v1 {
...
}

#[allow(deprecated)]
#[deprecated = "traits::schedule::v2 is deprecated. Please use v3 instead."]
pub mod v2 {
...
}
```

polkadot address: 19nSqFQorfF2HxD3oBzWM3oCh4SaCRKWt1yvmgaPYGCo71J

---------

Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>

There has been a case that a referenda failed because the length given
to `submit` was incorrect. The pallet can actually check the length if
the pre-image already exists to ensure that these kind of issues are not
happening again.

When the scheduler fails to lookup a `call`, it should unrequest it,
because it will not be required anymore.

This PR adds a new extrinsic `Call::restore_ledger ` gated by
`StakingAdmin` origin that restores a corrupted staking ledger. This
extrinsic will be used to recover ledgers that were affected by the
issue discussed in
https://github.com/paritytech/polkadot-sdk/issues/3245.

The extrinsic will re-write the storage items associated with a stash
account provided as input parameter. The data used to reset the ledger
can be either i) fetched on-chain or ii) partially/totally set by the
input parameters of the call.

In order to use on-chain data to restore the staking locks, we need a
way to read the current lock in the balances pallet. This PR adds a
`InspectLockableCurrency` trait and implements it in the pallet
balances. An alternative would be to tightly couple staking with the
pallet balances but that's inelegant (an example of how it would look
like in [this
branch](https://github.com/paritytech/polkadot-sdk/tree/gpestana/ledger-badstate-clean_tightly)).

More details on the type of corruptions and corresponding fixes
https://hackmd.io/DLb5jEYWSmmvqXC9ae4yRg?view#/

We verified that the `Call::restore_ledger` does fix all current
corrupted ledgers in Polkadot and Kusama. You can verify it here
https://hackmd.io/v-XNrEoGRpe7APR-EZGhOA.

**Changes introduced**
- Adds `Call::restore_ledger ` extrinsic to recover a corrupted ledger;
- Adds trait `frame_support::traits::currency::InspectLockableCurrency`
to allow external pallets to read current locks given an account and
lock ID;
- Implements the `InspectLockableCurrency` in the pallet-balances.
- Adds staking locks try-runtime checks
(https://github.com/paritytech/polkadot-sdk/issues/3751)

**Todo**
- [x] benchmark `Call::restore_ledger`
- [x] throughout testing of all ledger recovering cases
- [x] consider adding the staking locks try-runtime checks to this PR
(https://github.com/paritytech/polkadot-sdk/issues/3751)
- [x] simulate restoring all ledgers
(https://hackmd.io/Dsa2tvhISNSs7zcqriTaxQ?view) in Polkadot and Kusama
using chopsticks -- https://hackmd.io/v-XNrEoGRpe7APR-EZGhOA

Related to https://github.com/paritytech/polkadot-sdk/issues/3245
Closes https://github.com/paritytech/polkadot-sdk/issues/3751

---------

Co-authored-by: command-bot <>

Few fixes in CI to publish benchmarks jobs.

cc https://github.com/paritytech/ci_cd/issues/934

This will make it possible to use remaining weight on idle for
processing enqueued messages.
More context here https://github.com/paritytech/polkadot-sdk/issues/3709

---------

Co-authored-by: Adrian Catangiu <[email protected]>

This works only for collators that implement the `collator_fn` allowing
`collation-generation` subsystem to pull collations triggered on new
heads.

Also enables
`request_v2::CollationFetchingResponse::CollationWithParentHeadData` for
test adder/undying collators.

TODO:
- [x] fix tests
- [x] new tests
- [x] PR doc

---------

Signed-off-by: Andrei Sandu <[email protected]>

PR adds CI jobs that collect subsystem-benchmarks results and publishes
them to gh-pages.

cc https://github.com/paritytech/ci_cd/issues/934
cc @AndreiEres

The authority-discovery mechanism has implemented a few exponential
timers for:
- publishing the authority records
- goes from 2 seconds (when freshly booted) to 1 hour if the node is
long-running
  - set to 1 hour after successfully publishing the authority record
- discovering other authority records
- goes from 2 seconds (when freshly booted) to 10 minutes if the node is
long-running

This PR resets the exponential publishing and discovery interval to
defaults ensuring that long-running nodes:
- will retry publishing the authority records as aggressively as freshly
booted nodes
- Currently, if a long-running node fails to publish the DHT record when
the keys change (ie DhtEvent::ValuePutFailed), it will only retry after
1 hour
- will rediscover other authorities faster (since there is a chance that
other authority keys changed)

The subp2p-explorer has difficulties discovering the authorities when
the authority set changes in the first few hours. This might be entirely
due to the recursive nature of the DHT and the needed time to propagate
the records. However, there is a small chance that the authority
publishing failed and is only retried in 1h.

Let me know if this makes sense 🙏

 

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <[email protected]>
Co-authored-by: Dmitry Markin <[email protected]>

`execute` and `send` try to decode the xcm in the parameters before
reaching the filter line.
The new extrinsics decode only after the filter line.
These should be used instead of the old ones.

## TODO
- [x] Tests
- [x] Generate weights
- [x] Deprecation issue ->
https://github.com/paritytech/polkadot-sdk/issues/3771
- [x] PRDoc
- [x] Handle error in pallet-contracts

This would make writing XCMs in PJS Apps more difficult, but here's the
fix for that: https://github.com/polkadot-js/apps/pull/10350.
Already deployed! https://polkadot.js.org/apps/#/utilities/xcm

Supersedes https://github.com/paritytech/polkadot-sdk/pull/1798/

---------

Co-authored-by: PG Herveou <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: Adrian Catangiu <[email protected]>

This is a long due chore ...

---------

Signed-off-by: Andrei Sandu <[email protected]>
Co-authored-by: ordian <[email protected]>

bump version to `1.3.97` (follow up from
https://github.com/paritytech/polkadot-sdk/pull/3805)

---------

Co-authored-by: Bastian Köcher <[email protected]>

Fixes #3826.

The docs on the `candidates` field of `BlockEntry` were incorrectly
stating that they are sorted by core index. The (incorrect) optimization
was introduced in #3747 based on this assumption. The actual ordering is
based on `CandidateIncluded` events ordering in the runtime. We revert
this optimization here.

- [x] verify the underlying issue
- [x] add a regression test

---------

Co-authored-by: Bastian Köcher <[email protected]>