Changes:
- `QueueFootprint` gets a new field; `ready_pages` that contains the
non-overweight and not yet processed pages.
- `XCMP` queue pallet is change to use the `ready_pages` instead of
`pages` to calculate the channel suspension thresholds.

This should give the XCMP queue pallet a more correct view of when to
suspend channels.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

# Description

Removed deprecated type `GenesisConfig` from the codebase.

Closes https://github.com/paritytech/polkadot-sdk/issues/175

# Checklist

- [x] My PR includes a detailed description as outlined in the
"Description" section above
- [x] My PR follows the [labeling requirements](CONTRIBUTING.md#Process)
of this project (at minimum one label for `T`
  required)
- [x] I have made corresponding changes to the documentation (if
applicable)

---------

Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Michal Kucharczyk <[email protected]>

The first step towards
https://github.com/paritytech/polkadot-sdk/issues/3155

Brings all templates under the following structure

```
templates
|   parachain
|   |   polkadot-launch
|   |   runtime              --> parachain-template-runtime
|   |   pallets              --> pallet-parachain-template
|   |   node                 --> parachain-template-node
|   minimal
|   |   runtime              --> minimal-template-runtime
|   |   pallets              --> pallet-minimal-template
|   |   node                 --> minimal-template-node
|   solochain
|   |   runtime              --> solochain-template-runtime
|   |   pallets              --> pallet-template (the naming is not consistent here)
|   |   node                 --> solochain-template-node
```

The only note-worthy changes in this PR are: 

- More `Cargo.toml` fields are forwarded to use the one from the
workspace.
- parachain template now has weights and benchmarks
- adds a shell pallet to the mi...

Part of #3326 
This one is easier as all the storage items are public. 

@ggwpez @Kianenigma @shawntabrizi



---------

Signed-off-by: Matteo Muraca <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: Kian Paimani <[email protected]>

Fixes failures like:
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5436619#L3319

After some discussion with @kogeler

 after the we added the rate-limit
middleware it may slow down
the rpc call timings metrics significantly because it works as follows:

1. The rate limit guard is checked when the call comes and if a slot is
available -> process the call
2. If no free spot is available then the call will be sleeping
`jitter_delay + min_time_rate_guard` then woken up and checked at most
ten times
3. If no spot is available after 10 iterations -> the call is rejected
(this may take tens of seconds)

Thus, this PR adds a label "is_rate_limited" to filter those out on the
metrics "substrate_rpc_calls_time" and "substrate_rpc_calls_finished".

I had to merge two middleware layers Metrics and RateLimit to avoid
shared state in a hacky way.

---------

Co-authored-by: James Wilson <[email protected]>

Latest Rust nightly complains with `the item `Result` is imported
redundantly`

Co-authored-by: Sebastian Kunert <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>

Closes #2160

First part of [Extrinsic
Horizon](https://github.com/paritytech/polkadot-sdk/issues/2415)

Introduces a new trait `TransactionExtension` to replace
`SignedExtension`. Introduce the idea of transactions which obey the
runtime's extensions and have according Extension data (né Extra data)
yet do not have hard-coded signatures.

Deprecate the terminology of "Unsigned" when used for
transactions/extrinsics owing to there now being "proper" unsigned
transactions which obey the extension framework and "old-style" unsigned
which do not. Instead we have __*General*__ for the former and
__*Bare*__ for the latter. (Ultimately, the latter will be phased out as
a type of transaction, and Bare will only be used for Inherents.)

Types of extrinsic are now therefore:
- Bare (no hardcoded signature, no Extra data; used to be known as
"Unsigned")
- Bare transactions (deprecated): Gossiped, validated with
`ValidateUnsigned` (deprecated) and the `_bare_compat` bits of
`TransactionExtension` (deprecated).
  - Inherents: Not gossiped, validated with `ProvideInherent`.
- Extended (Extra data): Gossiped, validated via `TransactionExtension`.
  - Signed transactions (with a hardcoded signature).
  - General transactions (without a hardcoded signature).

`TransactionExtension` differs from `SignedExtension` because:
- A signature on the underlying transaction may validly not be present.
- It may alter the origin during validation.
- `pre_dispatch` is renamed to `prepare` and need not contain the checks
present in `validate`.
- `validate` and `prepare` is passed an `Origin` rather than a
`AccountId`.
- `validate` may pass arbitrary information into `prepare` via a new
user-specifiable type `Val`.
- `AdditionalSigned`/`additional_signed` is renamed to
`Implicit`/`implicit`. It is encoded *for the entire transaction* and
passed in to each extension as a new argument to `validate`. This
facilitates the ability of extensions to acts as underlying crypto.

There is a new `DispatchTransaction` trait which contains only default
function impls and is impl'ed for any `TransactionExtension` impler. It
provides several utility functions which reduce some of the tedium from
using `TransactionExtension` (indeed, none of its regular functions
should now need to be called directly).

Three transaction version discriminator ("versions") are now
permissible:
- 0b000000100: Bare (used to be called "Unsigned"): contains Signature
or Extra (extension data). After bare transactions are no longer
supported, this will strictly identify an Inherents only.
- 0b100000100: Old-school "Signed" Transaction: contains Signature and
Extra (extension data).
- 0b010000100: New-school "General" Transaction: contains Extra
(extension data), but no Signature.

For the New-school General Transaction, it becomes trivial for authors
to publish extensions to the mechanism for authorizing an Origin, e.g.
through new kinds of key-signing schemes, ZK proofs, pallet state,
mutations over pre-authenticated origins or any combination of the
above.

## Code Migration

### NOW: Getting it to build

Wrap your `SignedExtension`s in `AsTransactionExtension`. This should be
accompanied by renaming your aggregate type in line with the new
terminology. E.g. Before:

```rust
/// The SignedExtension to the basic transaction logic.
pub type SignedExtra = (
	/* snip */
	MySpecialSignedExtension,
);
/// Unchecked extrinsic type as expected by this runtime.
pub type UncheckedExtrinsic =
	generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, SignedExtra>;
```

After:

```rust
/// The extension to the basic transaction logic.
pub type TxExtension = (
	/* snip */
	AsTransactionExtension<MySpecialSignedExtension>,
);
/// Unchecked extrinsic type as expected by this runtime.
pub type UncheckedExtrinsic =
	generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, TxExtension>;
```

You'll also need to alter any transaction building logic to add a
`.into()` to make the conversion happen. E.g. Before:

```rust
fn construct_extrinsic(
		/* snip */
) -> UncheckedExtrinsic {
	let extra: SignedExtra = (
		/* snip */
		MySpecialSignedExtension::new(/* snip */),
	);
	let payload = SignedPayload::new(call.clone(), extra.clone()).unwrap();
	let signature = payload.using_encoded(|e| sender.sign(e));
	UncheckedExtrinsic::new_signed(
		/* snip */
		Signature::Sr25519(signature),
		extra,
	)
}
```

After:

```rust
fn construct_extrinsic(
		/* snip */
) -> UncheckedExtrinsic {
	let tx_ext: TxExtension = (
		/* snip */
		MySpecialSignedExtension::new(/* snip */).into(),
	);
	let payload = SignedPayload::new(call.clone(), tx_ext.clone()).unwrap();
	let signature = payload.using_encoded(|e| sender.sign(e));
	UncheckedExtrinsic::new_signed(
		/* snip */
		Signature::Sr25519(signature),
		tx_ext,
	)
}
```

### SOON: Migrating to `TransactionExtension`

Most `SignedExtension`s can be trivially converted to become a
`TransactionExtension`. There are a few things to know.

- Instead of a single trait like `SignedExtension`, you should now
implement two traits individually: `TransactionExtensionBase` and
`TransactionExtension`.
- Weights are now a thing and must be provided via the new function `fn
weight`.

#### `TransactionExtensionBase`

This trait takes care of anything which is not dependent on types
specific to your runtime, most notably `Call`.

- `AdditionalSigned`/`additional_signed` is renamed to
`Implicit`/`implicit`.
- Weight must be returned by implementing the `weight` function. If your
extension is associated with a pallet, you'll probably want to do this
via the pallet's existing benchmarking infrastructure.

#### `TransactionExtension`

Generally:
- `pre_dispatch` is now `prepare` and you *should not reexecute the
`validate` functionality in there*!
- You don't get an account ID any more; you get an origin instead. If
you need to presume an account ID, then you can use the trait function
`AsSystemOriginSigner::as_system_origin_signer`.
- You get an additional ticket, similar to `Pre`, called `Val`. This
defines data which is passed from `validate` into `prepare`. This is
important since you should not be duplicating logic from `validate` to
`prepare`, you need a way of passing your working from the former into
the latter. This is it.
- This trait takes two type parameters: `Call` and `Context`. `Call` is
the runtime call type which used to be an associated type; you can just
move it to become a type parameter for your trait impl. `Context` is not
currently used and you can safely implement over it as an unbounded
type.
- There's no `AccountId` associated type any more. Just remove it.

Regarding `validate`:
- You get three new parameters in `validate`; all can be ignored when
migrating from `SignedExtension`.
- `validate` returns a tuple on success; the second item in the tuple is
the new ticket type `Self::Val` which gets passed in to `prepare`. If
you use any information extracted during `validate` (off-chain and
on-chain, non-mutating) in `prepare` (on-chain, mutating) then you can
pass it through with this. For the tuple's last item, just return the
`origin` argument.

Regarding `prepare`:
- This is renamed from `pre_dispatch`, but there is one change:
- FUNCTIONALITY TO VALIDATE THE TRANSACTION NEED NOT BE DUPLICATED FROM
`validate`!!
- (This is different to `SignedExtension` which was required to run the
same checks in `pre_dispatch` as in `validate`.)

Regarding `post_dispatch`:
- Since there are no unsigned transactions handled by
`TransactionExtension`, `Pre` is always defined, so the first parameter
is `Self::Pre` rather than `Option<Self::Pre>`.

If you make use of `SignedExtension::validate_unsigned` or
`SignedExtension::pre_dispatch_unsigned`, then:
- Just use the regular versions of these functions instead.
- Have your logic execute in the case that the `origin` is `None`.
- Ensure your transaction creation logic creates a General Transaction
rather than a Bare Transaction; this means having to include all
`TransactionExtension`s' data.
- `ValidateUnsigned` can still be used (for now) if you need to be able
to construct transactions which contain none of the extension data,
however these will be phased out in stage 2 of the Transactions Horizon,
so you should consider moving to an extension-centric design.

## TODO

- [x] Introduce `CheckSignature` impl of `TransactionExtension` to
ensure it's possible to have crypto be done wholly in a
`TransactionExtension`.
- [x] Deprecate `SignedExtension` and move all uses in codebase to
`TransactionExtension`.
  - [x] `ChargeTransactionPayment`
  - [x] `DummyExtension`
  - [x] `ChargeAssetTxPayment` (asset-tx-payment)
  - [x] `ChargeAssetTxPayment` (asset-conversion-tx-payment)
  - [x] `CheckWeight`
  - [x] `CheckTxVersion`
  - [x] `CheckSpecVersion`
  - [x] `CheckNonce`
  - [x] `CheckNonZeroSender`
  - [x] `CheckMortality`
  - [x] `CheckGenesis`
  - [x] `CheckOnlySudoAccount`
  - [x] `WatchDummy`
  - [x] `PrevalidateAttests`
  - [x] `GenericSignedExtension`
  - [x] `SignedExtension` (chain-polkadot-bulletin)
  - [x] `RefundSignedExtensionAdapter`
- [x] Implement `fn weight` across the board.
- [ ] Go through all pre-existing extensions which assume an account
signer and explicitly handle the possibility of another kind of origin.
- [x] `CheckNonce` should probably succeed in the case of a non-account
origin.
- [x] `CheckNonZeroSender` should succeed in the case of a non-account
origin.
- [x] `ChargeTransactionPayment` and family should fail in the case of a
non-account origin.
  - [ ] 
- [x] Fix any broken tests.

---------

Signed-off-by: georgepisaltu <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Signed-off-by: Alexandru Gheorghe <[email protected]>
Signed-off-by: Andrei Sandu <[email protected]>
Co-authored-by: Nikhil Gupta <[email protected]>
Co-authored-by: georgepisaltu <[email protected]>
Co-authored-by: Chevdor <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Maciej <[email protected]>
Co-authored-by: Javier Viola <[email protected]>
Co-authored-by: Marcin S. <[email protected]>
Co-authored-by: Tsvetomir Dimitrov <[email protected]>
Co-authored-by: Javier Bullrich <[email protected]>
Co-authored-by: Koute <[email protected]>
Co-authored-by: Adrian Catangiu <[email protected]>
Co-authored-by: Vladimir Istyufeev <[email protected]>
Co-authored-by: Ross Bulat <[email protected]>
Co-authored-by: Gonçalo Pestana <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Svyatoslav Nikolsky <[email protected]>
Co-authored-by: André Silva <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: s0me0ne-unkn0wn <[email protected]>
Co-authored-by: ordian <[email protected]>
Co-authored-by: Sebastian Kunert <[email protected]>
Co-authored-by: Aaro Altonen <[email protected]>
Co-authored-by: Dmitry Markin <[email protected]>
Co-authored-by: Alexandru Vasile <[email protected]>
Co-authored-by: Alexander Samusev <[email protected]>
Co-authored-by: Julian Eager <[email protected]>
Co-authored-by: Michal Kucharczyk <[email protected]>
Co-authored-by: Davide Galassi <[email protected]>
Co-authored-by: Dónal Murray <[email protected]>
Co-authored-by: yjh <[email protected]>
Co-authored-by: Tom Mi <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Will | Paradox | ParaNodes.io <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Joshy Orndorff <[email protected]>
Co-authored-by: Joshy Orndorff <[email protected]>
Co-authored-by: PG Herveou <[email protected]>
Co-authored-by: Alexander Theißen <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Juan Girini <[email protected]>
Co-authored-by: bader y <[email protected]>
Co-authored-by: James Wilson <[email protected]>
Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: asynchronous rob <[email protected]>
Co-authored-by: Parth <[email protected]>
Co-authored-by: Andrew Jones <[email protected]>
Co-authored-by: Jonathan Udd <[email protected]>
Co-authored-by: Serban Iorga <[email protected]>
Co-authored-by: Egor_P <[email protected]>
Co-authored-by: Branislav Kontur <[email protected]>
Co-authored-by: Evgeny Snitko <[email protected]>
Co-authored-by: Just van Stam <[email protected]>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: gupnik <[email protected]>
Co-authored-by: dzmitry-lahoda <[email protected]>
Co-authored-by: zhiqiangxu <[email protected]>
Co-authored-by: Nazar Mokrynskyi <[email protected]>
Co-authored-by: Anwesh <[email protected]>
Co-authored-by: cheme <[email protected]>
Co-authored-by: Sam Johnson <[email protected]>
Co-authored-by: kianenigma <[email protected]>
Co-authored-by: Jegor Sidorenko <[email protected]>
Co-authored-by: Muharem <[email protected]>
Co-authored-by: joepetrowski <[email protected]>
Co-authored-by: Alexandru Gheorghe <[email protected]>
Co-authored-by: Gabriel Facco de Arruda <[email protected]>
Co-authored-by: Squirrel <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>
Co-authored-by: georgepisaltu <[email protected]>
Co-authored-by: command-bot <>

Closes https://github.com/paritytech/polkadot-sdk-docs/issues/35

- Moves pallet proc macro docs to `frame_support`
- Adds missing docs
- Revise revise existing docs, adding compiling doctests where
appropriate

---------

Co-authored-by: command-bot <>
Co-authored-by: Kian Paimani <[email protected]>

Step in https://github.com/paritytech/polkadot-sdk/issues/171

This PR removes the need to specify `as [disambiguation_path]` for cases
where the trait definition resides within the same scope as default impl
path.

For example, in the following macro invocation
```rust
#[derive_impl(frame_system::config_preludes::TestDefaultConfig as frame_system::DefaultConfig)]
impl frame_system::Config for Runtime {
   ...
}
```
the trait `DefaultConfig` lies within the `frame_system` scope and
`TestDefaultConfig` impls the `DefaultConfig` trait. Using this
information, we can compute the disambiguation path internally, thus
removing the need of an explicit specification.

In cases where the trait lies outside this scope, we would still need to
specify it explicitly, but this should take care of most (if not all)
uses of `derive_impl` within FRAME's context.

Fixes https://github.com/paritytech/polkadot-sdk/issues/3496.

This fixes an issue introduced in
https://github.com/paritytech/substrate/pull/14101, in which I removed
the `Call` enum's documentation and replaced it with a link to the
`Pallet` struct, but this also removed any docs related to call from the
metadata.

I tried to add a regression test for this, but it seems to me that this
is not possible, given that using `type-info` we only assert in type-ids
for `Call`, `Event` and `Error`. I removed some doc comments from a test
setup in `frame-support-test` to demonstrate the issue there. @jsdw do
you have any comments on this?

I also fixed a small issue in the custom html/css of `polkadot-sdk-doc`
crate, making sure it does not affect the rust-doc page of all other
crates.

- [x] Investigate a regression test
- [x] prdoc

# Description

*Removes `sp_weights::OldWeight` and its usage*

Fixes #144

---------

Co-authored-by: Liam Aharon <[email protected]>

Fixing: 
```
Verification failed for block 0x07bbf1e04121d70a4bdb21cc055132b53ac2390fa95c4d05497fc91b1e8bf7f5 received from (12D3KooWJzLd8skcAgA24EcJey7aJAhYctfUxWGjSP5Usk9wbpPZ): "Header 0x07bbf1e04121d70a4bdb21cc055132b53ac2390fa95c4d05497fc91b1e8bf7f5 rejected: too far in the future"   
```

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>
Co-authored-by: Dmitry Sinyavin <[email protected]>
Co-authored-by: s0me0ne-unkn0wn <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

This MR is the merge of
https://github.com/paritytech/substrate/pull/14414 and
https://github.com/paritytech/substrate/pull/14275. It implements
[RFC#13](https://github.com/polkadot-fellows/RFCs/pull/13), closes
https://github.com/paritytech/polkadot-sdk/issues/198.

----- 

This Merge request introduces three major topicals:

1. Multi-Block-Migrations
1. New pallet `poll` hook for periodic service work
1. Replacement hooks for `on_initialize` and `on_finalize` in cases
where `poll` cannot be used

and some more general changes to FRAME.  
The changes for each topical span over multiple crates. They are listed
in topical order below.

# 1.) Multi-Block-Migrations

Multi-Block-Migrations are facilitated by creating `pallet_migrations`
and configuring `System::Config::MultiBlockMigrator` to point to it.
Executive picks this up and triggers one step of the migrations pallet
per block.
The chain is in lockdown mode for as long as an MBM is ongoing.
Executive does this by polling `MultiBlockMigrator::ongoing` and not
allowing any transaction in a block, if true.

A MBM is defined through trait `SteppedMigration`. A condensed version
looks like this:
```rust
/// A migration that can proceed in multiple steps.
pub trait SteppedMigration {
	type Cursor: FullCodec + MaxEncodedLen;
	type Identifier: FullCodec + MaxEncodedLen;

	fn id() -> Self::Identifier;

	fn max_steps() -> Option<u32>;

	fn step(
		cursor: Option<Self::Cursor>,
		meter: &mut WeightMeter,
	) -> Result<Option<Self::Cursor>, SteppedMigrationError>;
}
```

`pallet_migrations` can be configured with an aggregated tuple of these
migrations. It then starts to migrate them one-by-one on the next
runtime upgrade.
Two things are important here:
- 1. Doing another runtime upgrade while MBMs are ongoing is not a good
idea and can lead to messed up state.
- 2. **Pallet Migrations MUST BE CONFIGURED IN `System::Config`,
otherwise it is not used.**

The pallet supports an `UpgradeStatusHandler` that can be used to notify
external logic of upgrade start/finish (for example to pause XCM
dispatch).

Error recovery is very limited in the case that a migration errors or
times out (exceeds its `max_steps`). Currently the runtime dev can
decide in `FailedMigrationHandler::failed` how to handle this. One
follow-up would be to pair this with the `SafeMode` pallet and enact
safe mode when an upgrade fails, to allow governance to rescue the
chain. This is currently not possible, since governance is not
`Mandatory`.

## Runtime API

- `Core`: `initialize_block` now returns `ExtrinsicInclusionMode` to
inform the Block Author whether they can push transactions.

### Integration

Add it to your runtime implementation of `Core` and `BlockBuilder`:
```patch
diff --git a/runtime/src/lib.rs b/runtime/src/lib.rs
@@ impl_runtime_apis! {
	impl sp_block_builder::Core<Block> for Runtime {
-		fn initialize_block(header: &<Block as BlockT>::Header) {
+		fn initialize_block(header: &<Block as BlockT>::Header) -> RuntimeExecutiveMode {
			Executive::initialize_block(header)
		}

		...
	}
```

# 2.) `poll` hook

A new pallet hook is introduced: `poll`. `Poll` is intended to replace
mostly all usage of `on_initialize`.
The reason for this is that any code that can be called from
`on_initialize` cannot be migrated through an MBM. Currently there is no
way to statically check this; the implication is to use `on_initialize`
as rarely as possible.
Failing to do so can result in broken storage invariants.

The implementation of the poll hook depends on the `Runtime API` changes
that are explained above.

# 3.) Hard-Deadline callbacks

Three new callbacks are introduced and configured on `System::Config`:
`PreInherents`, `PostInherents` and `PostTransactions`.
These hooks are meant as replacement for `on_initialize` and
`on_finalize` in cases where the code that runs cannot be moved to
`poll`.
The reason for this is to make the usage of HD-code (hard deadline) more
explicit - again to prevent broken invariants by MBMs.

# 4.) FRAME (general changes)

## `frame_system` pallet

A new memorize storage item `InherentsApplied` is added. It is used by
executive to track whether inherents have already been applied.
Executive and can then execute the MBMs directly between inherents and
transactions.

The `Config` gets five new items:
- `SingleBlockMigrations` this is the new way of configuring migrations
that run in a single block. Previously they were defined as last generic
argument of `Executive`. This shift is brings all central configuration
about migrations closer into view of the developer (migrations that are
configured in `Executive` will still work for now but is deprecated).
- `MultiBlockMigrator` this can be configured to an engine that drives
MBMs. One example would be the `pallet_migrations`. Note that this is
only the engine; the exact MBMs are injected into the engine.
- `PreInherents` a callback that executes after `on_initialize` but
before inherents.
- `PostInherents` a callback that executes after all inherents ran
(including MBMs and `poll`).
- `PostTransactions` in symmetry to `PreInherents`, this one is called
before `on_finalize` but after all transactions.

A sane default is to set all of these to `()`. Example diff suitable for
any chain:
```patch
@@ impl frame_system::Config for Test {
 	type MaxConsumers = ConstU32<16>;
+	type SingleBlockMigrations = ();
+	type MultiBlockMigrator = ();
+	type PreInherents = ();
+	type PostInherents = ();
+	type PostTransactions = ();
 }
```

An overview of how the block execution now looks like is here. The same
graph is also in the rust doc.

<details><summary>Block Execution Flow</summary>
<p>

![Screenshot 2023-12-04 at 19 11
29](https://github.com/paritytech/polkadot-sdk/assets/10380170/e88a80c4-ef11-4faa-8df5-8b33a724c054)

</p>
</details> 

## Inherent Order

Moved to https://github.com/paritytech/polkadot-sdk/pull/2154

---------------


## TODO

- [ ] Check that `try-runtime` still works
- [ ] Ensure backwards compatibility with old Runtime APIs
- [x] Consume weight correctly
- [x] Cleanup

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Juan Girini <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Gavin Wood <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

- deprecation companion:
https://github.com/substrate-developer-hub/substrate-docs/pull/2136
- inspired by
https://substrate.stackexchange.com/questions/11058/how-can-i-create-ocw-that-wont-activates-every-block-but-will-activates-only-w/11060#11060

---------

Co-authored-by: Sergej Sakac <[email protected]>

This PR adds tests for the `transaction_broadcast` method.


The testing needs to coordinate the following components:
- The `TestApi` marks transactions as invalid and implements
`ChainApi::validate_transaction`
- this is what dictates if a transaction is valid or not and is called
from within the `BasicPool`
- The `BasicPool` which maintains the transactions and implements
`submit_and_watch` needed by the tx broadcast to submit the transaction
- The status of the transaction pool is exposed by mocking the BasicPool
- The `ChainHeadMockClient` which mocks the
`BlockchainEvents::import_notification_stream` needed by the tx
broadcast to know to which blocks the transaction is submitted

The following changes have been added to the substrate testing to
accommodate this:
- `TestApi` gets ` remove_invalid`, counterpart to `add_invalid` to
ensure an invalid transaction can become valid again; as well as a
priority setter for extrinsics
- `BasicPool` test constructor is extended with options for the
`PoolRotator`
- this mechanism is needed because transactions are banned for 30mins
(default) after they are declared invalid
  - testing bypasses this by providing a `Duration::ZERO`

### Testing Scenarios

- Capture the status of the transaction as it is normally broadcasted
- `transaction_stop` is valid while the transaction is in progress
- A future transaction is handled when the dependencies are completed
- Try to resubmit the transaction at a later block (currently invalid)
- An invalid transaction status is propagated; the transaction is marked
as temporarily banned; then the ban expires and transaction is
resubmitted
  
This builds on top of:
https://github.com/paritytech/polkadot-sdk/pull/3079
Part of: https://github.com/paritytech/polkadot-sdk/issues/3084

cc @paritytech/subxt-team

---------

Signed-off-by: Alexandru Vasile <[email protected]>
Co-authored-by: James Wilson <[email protected]>

Closes https://github.com/paritytech/polkadot-sdk-docs/issues/55

- Changes 'current storage version' terminology to less ambiguous
'in-code storage version' (suggestion by @ggwpez

)
- Adds a new example pallet `pallet-example-single-block-migrations`
- Adds a new reference doc to replace
https://docs.substrate.io/maintain/runtime-upgrades/ (temporarily living
in the pallet while we wait for developer hub PR to merge)
- Adds documentation for the `storage_alias` macro
- Improves `trait Hooks` docs 
- Improves `trait GetStorageVersion` docs
- Update the suggested patterns for using `VersionedMigration`, so that
version unchecked migrations are never exported
- Prevents accidental usage of version unchecked migrations in runtimes

https://github.com/paritytech/substrate/pull/14421#discussion_r1255467895
- Unversioned migration code is kept inside `mod version_unchecked`,
versioned code is kept in `pub mod versioned`
- It is necessary to use modules to limit visibility because the inner
migration must be `pub`. See
https://github.com/rust-lang/rust/issues/30905 and

https://internals.rust-lang.org/t/lang-team-minutes-private-in-public-rules/4504/40
for more.

### todo

- [x] move to reference docs to proper place within sdk-docs (now that
https://github.com/paritytech/polkadot-sdk/pull/2102 is merged)
- [x] prdoc

---------

Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Juan <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: gupnik <[email protected]>

resolve https://github.com/paritytech/polkadot-sdk/issues/3116

a follow-up on
https://github.com/paritytech/polkadot-sdk/pull/3061#pullrequestreview-1847530265:

- [x] reuse collator overseer builder for polkadot-node and collator
- [x] run zombienet test (0001-parachains-smoke-test.toml)
- [x] make wasm build errors more user-friendly for an easier problem
detection when using different toolchains in Rust

---------

Co-authored-by: ordian <[email protected]>
Co-authored-by: s0me0ne-unkn0wn <[email protected]>

Introduce storage attr macro `#[disable_try_decode_storage]` and set it on `System::Events` and `ParachainSystem::HostConfiguration` (#3454)

Closes https://github.com/paritytech/polkadot-sdk/issues/2560

Allows marking storage items with `#[disable_try_decode_storage]`, and
uses it with `System::Events`.

Question: what's the recommended way to write a test for this? I
couldn't find a test for similar existing macro `#[whitelist_storage]`.

Does the following: 

- Add a reference doc page named `frame_runtime_types`, which explains
what types like `RuntimeOrigin`, `RuntimeCall` etc are.
- On top of it, it adds a reference doc page called `frame_origin` which
explains a few important patterns that we use around origins
- And finally brushes up `#[frame::origin]` docs. 
- Updates the theme, sidebar and favicon to look like: 

<img width="1728" alt="Screenshot 2024-02-20 at 12 16 00"
src="https://github.com/paritytech/polkadot-sdk/assets/5588131/6d60a16b-2081-411b-8869-43b91920cca9">


All of this was inspired by
https://substrate.stackexchange.com/questions/10992/how-do-you-find-the-public-key-for-the-medium-spender-track-origin/10993

closes https://github.com/paritytech/polkadot-sdk-docs/issues/45
closes https://github.com/paritytech/polkadot-sdk-docs/issues/43
contributes / overlaps with
https://github.com/paritytech/polkadot-sdk/pull/2638 cc @liamaharon


deprecation companion:
https://github.com/substrate-developer-hub/substrate-docs/pull/2131
pba-content companion:
https://github.com/Polkadot-Blockchain-Academy/pba-content/pull/977

---------

Co-authored-by: Radha <[email protected]>
Co-authored-by: Sebastian Kunert <[email protected]>
Co-authored-by: Gonçalo Pestana <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>

# Description

*Refactors `String::from_utf8` usage in the pallet benchmarking

Fixes #389

---------

Co-authored-by: command-bot <>

While investigating some pruning issues I found some room for
improvement in the notification pin handling.

**Problem:** It was not possible to define an upper limit on
notification pins. The block pinning cache has a limit, but only handles
bodies and justifications.

After this PR, bookkeeping for notifications is managed in the pinning
worker. A limit can be defined in the worker. If that limit is crossed,
blocks that were pinned for that notification are unpinned, which now
affects the state as well as bodies and justifications. The pinned
blocks cache still has a limit, but should never be hit.

closes #19

---------

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: André Silva <[email protected]>

Instead of only generating the error, we now generate the actual code
and the error. This generates in total less errors and helps the user to
identify the actual problem and not being confronted with tons of
errors.

Changes the runtime hash algorithm used in
`resolve_state_version_from_wasm` from `DefaultHasher` to a
caller-provided one (usually `HashingFor<Block>`), to match the one used
elsewhere.

This fixes an issue where the runtime wasm is compiled 3 times when
starting the `tanssi-node` with `--dev`. With this fix, the runtime wasm
is only compiled 2 times. The other redundant compilation is caused by
the `GenesisConfigBuilderRuntimeCaller` struct, which ignores the
runtime cache.

---------

Co-authored-by: Bastian Köcher <[email protected]>

# Runtime side for PoV Reclaim

## Implementation Overview
- Hostfunction to fetch the storage proof size has been added to the
PVF. It uses the size tracking recorder that was introduced in my
previous PR.
- Mechanisms to use the reclaim HostFunction have been introduced.
- 1. A SignedExtension that checks the node-reported proof size before
and after application of an extrinsic. Then it reclaims the difference.
- 2. A manual helper to make reclaiming easier when manual interaction
is required, for example in `on_idle` or other hooks.
- In order to utilize the manual reclaiming, I modified `WeightMeter` to
support the reduction of consumed weight, at least for storage proof
size.

## How to use
To enable the general functionality for a parachain:
1. Add the SignedExtension to your parachain runtime. 
2. Provide the HostFunction to the node
3. Enable proof recording during block import

## TODO
- [x] PRDoc

---------

Co-authored-by: Dmitry Markin <[email protected]>
Co-authored-by: Davide Galassi <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

## Summary
* use benchamarked weights instead of hardcoded ones for
`pallet-membership`
* rename benchmark to match extrinsic name
* remove unnecessary dependency from `clear_prime`

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>

Add a `ApiVersion` constant to the pallet-contracts Config to
communicate with developers the current state of the host functions
exposed by the pallet

As part of BEEFY worker/voter initialization the task waits for certain
chain and backend conditions to be fulfilled:
- BEEFY consensus enabled on-chain & GRANDPA best finalized higher than
on-chain BEEFY genesis block,
- backend has synced headers for BEEFY mandatory blocks between best
BEEFY and best GRANDPA.

During this waiting time, any messages gossiped on the BEEFY topic for
current chain get enqueued in the gossip engine, leading to RAM bloating
and output warning/error messages when the wait time is non-negligible
(like during a clean sync).

This PR adds logic to pump the gossip engine while waiting for other
things to make sure gossiped messages get consumed (practically
discarded until worker is fully initialized).

Also raises the warning threshold for enqueued messages from 10k to
100k. This is in line with the other gossip protocols on the node.

Fixes https://github.com/paritytech/polkadot-sdk/issues/3390

---------

Signed-off-by: Adrian Catangiu <[email protected]>

This PR fixes a subtle bug in `wasm-builder` first introduced in
https://github.com/paritytech/polkadot-sdk/pull/1851 (sorry, my bad! I
should have caught this during review) where the status code of the
`cargo` subprocess is not properly checked, which results in builds
silently succeeding when they shouldn't (that is: if we successfully
build a runtime blob, and then modify the code so that it won't compile,
and recompile it again, then the build will succeed and silently use the
*old* blob).

cc @athei This is the bug you were seeing.

[edit]Also fixes a similar PolkaVM-specific bug where I accidentally
used the wrong comparison operator.[/edit]

Closes https://github.com/paritytech/polkadot-sdk/issues/2713

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: André Silva <[email protected]>

part of #3326 

@ggwpez @Kianenigma @shawntabrizi



---------

Signed-off-by: Matteo Muraca <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>

In Tanssi, we need a way to stop the collator code and then start it
again. This is to support rotating the same collator between different
runtimes. Currently, this works very well, except for the proposer
metrics, because they only get registered the first time they are
started. Afterwards, we see this warning log:

> Failed to register proposer prometheus metrics: Duplicate metrics
collector registration attempted


~~So this PR adds a method to set metrics, to allow us to register
metrics manually before creating the `ProposerFactory`, and then clone
the same metrics every time we need to start the collator.~~ Implemented
Clone instead

This PR backports the changes from the rpc-v2 spec:
https://github.com/paritytech/json-rpc-interface-spec/pull/134

The `Broadcasted` event has been removed:
- it is hard to enforce a `Dropped { broadcasted: bool }` event in cases
of a load-balancer being placed in front of an RPC server
- when the server exists, it is impossible to guarantee this field if
the server did not previously send a `Broadcasted` event
- the number of peers reported by this event does not guarantee that
peers are unique
- the same peer can disconnect and reconnect, increasing this metric
number
- the number of peers that receive this transaction offers no guarantee
about the transaction being included in the chain at a later time


cc @paritytech/subxt-team

---------

Signed-off-by: Alexandru Vasile <[email protected]>
Co-authored-by: James Wilson <[email protected]>

closes #2844 

- adds `list-pallets` option which prints all unique available pallets
for benchmarking
```bash
./target/release/node benchmark pallet --list=pallets
```
- adds `all` option which runs benchmarks for all available pallets and
extrinsics (equivalent to `--pallet * --extrinsic *`)
```bash
./target/release/node benchmark pallet --all
```

- use the `list=pallets` syntax in `run_all_benchmarks.sh` script

cc ggwpez

---------

Co-authored-by: Bastian Köcher <[email protected]>

fixes https://github.com/paritytech/polkadot-sdk/issues/3407

This PR is fixing a bug in the sync mechanism between wasmi and
pallet-contracts. This bug leads to essentially double charging all the
gas that was used during the execution of the host function. When the
`call` host function is used for recursion this will lead to a quadratic
amount of gas consumption with regard to the nesting depth.We also took
the chance to refactor the code in question and improve the rust docs.

The bug was caused by not updating `GasMeter::executor_consumed`
(previously `engine_consumed`) when leaving the host function. This lead
to the value being stale (too low) when entering another host function.

---------

Co-authored-by: PG Herveou <[email protected]>