Closes: https://github.com/paritytech/polkadot-sdk/issues/7033

Co-authored-by: Dónal Murray <donalm@seadanda.dev>
Co-authored-by: Dónal Murray <donal.murray@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

Implement cumulus StorageWeightReclaim as wrapping transaction extension + frame system ReclaimWeight (#6140)

(rebasing of https://github.com/paritytech/polkadot-sdk/pull/5234)

## Issues:

* Transaction extensions have weights and refund weight. So the
reclaiming of unused weight must happen last in the transaction
extension pipeline. Currently it is inside `CheckWeight`.
* cumulus storage weight reclaim transaction extension misses the proof
size of logic happening prior to itself.

## Done:

* a new storage `ExtrinsicWeightReclaimed` in frame-system. Any logic
which attempts to do some reclaim must use this storage to avoid double
reclaim.
* a new function `reclaim_weight` in frame-system pallet: info and post
info in arguments, read the already reclaimed weight, calculate the new
unused weight from info and post info. do the more accurate reclaim if
higher.
* `CheckWeight` is unchanged and still reclaim the weight in post
dispatch
* `ReclaimWeight` is a new transaction extension in frame system. For
solo chains it must be used last in the transactino extension pipeline.
It does the final most accurate reclaim
* `StorageWeightReclaim` is moved from cumulus primitives into its own
pallet (in order to define benchmark) and is changed into a wrapping
transaction extension.
It does the recording of proof size and does the reclaim using this
recording and the info and post info. So parachains don't need to use
`ReclaimWeight`. But also if they use it, there is no bug.

    ```rust
  /// The TransactionExtension to the basic transaction logic.
pub type TxExtension =
cumulus_pallet_weight_reclaim::StorageWeightReclaim<
         Runtime,
         (
                 frame_system::CheckNonZeroSender<Runtime>,
                 frame_system::CheckSpecVersion<Runtime>,
                 frame_system::CheckTxVersion<Runtime>,
                 frame_system::CheckGenesis<Runtime>,
                 frame_system::CheckEra<Runtime>,
                 frame_system::CheckNonce<Runtime>,
                 frame_system::CheckWeight<Runtime>,
pallet_transaction_payment::ChargeTransactionPayment<Runtime>,
                 BridgeRejectObsoleteHeadersAndMessages,

(bridge_to_rococo_config::OnBridgeHubWestendRefundBridgeHubRococoMessages,),
frame_metadata_hash_extension::CheckMetadataHash<Runtime>,
         ),
  >;
  ```

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: georgepisaltu <52418509+georgepisaltu@users.noreply.github.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: command-bot <>

- **[docs] Fix release naming**
- **Remove outdated and unmaintained file**

Closes https://github.com/paritytech/polkadot-sdk/issues/6998

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

If you see the doc
https://paritytech.github.io/polkadot-sdk/master/polkadot_sdk_docs/polkadot_sdk/frame_runtime/index.html
The runtime part introduction is missing.

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Changes:
- Make R0-silent not run the semver check again. Originally I thought
this would be good to have a bullet-proof check, but it now often
triggers when CI or unrelated files are changed. In the end, the
developer has to make the right choice here - and always will need to.
So bringing back the R0 label gives more power to the devs and should
increase dev velocity. We still need to ensure that every use of this
label is well understood, and not just used out of lazyness.
- Fix `/cmd prdoc` bump levels
- Update docs

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

It doesn't make sense to only reorder the features array.

For example:

This makes it hard for me to compare the dependencies and features,
especially some crates have a really really long dependencies list.
```toml​
[dependencies]
c = "*"
a = "*"
b = "*"

[features]
std = [
  "a",
  "b",
  "c",
]
```

This makes my life easier.
```toml​
[dependencies]
a = "*"
b = "*"
c = "*"

[features]
std = [
  "a",
  "b",
  "c",
]
```

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: command-bot <>

# Description

Get runtime's metadata, parse it and verify pallets list for a pallet
named `ParachainSystem` (for now), and block number to be the same for
both node and runtime. Ideally we'll add other pallets checks too, at
least a small set of pallets we think right away as mandatory for
parachain compatibility.
Closes: #5565 

## Integration

Runtime devs must be made aware that to be fully compatible with Omni
Node, certain naming conventions should be respected when defining
pallets (e.g we verify parachain-system pallet existence by searching
for a pallet with `name` `ParachainSystem` in runtime's metadata). Not
finding such a pallet will not influence the functionality yet, but by
doing these checks we could provide useful feedback for runtimes that
are clearly not implementing what's required for full parachain
compatibility with Omni Node.

## Review Notes

- [x] parachain system check
- [x] check frame_system's metadata to ensure the block number in there
is the same as the one in the node side
- [x] add tests for the previous checking logic
- [x] update omni node polkadot-sdk docs to make these conventions
visible.
- [ ] add more pallets checks?

---------

Signed-off-by: Iulian Barbu <iulian.barbu@parity.io>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

# Description

**Understood assignment:**
Initial assignment description is in #6194.
In order to Simplify the display of commands and ensure they are tested
for chain spec builder's `polkadot-sdk` reference docs, find every
occurrence of `#[docify::export]` where `process:Command` is used, and
replace the use of `process:Command` by `run_cmd!` from the `cmd_lib
crate`.

---------

Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>

Close: #5858

---------

Co-authored-by: Bastian Köcher <git@kchr.de>

cc https://github.com/paritytech/ci_cd/issues/1088

In `chain-spec-guide-runtime` crate's tests, there was assumption that
release version of wasm blob exists. This PR uses
`chain_spec_guide_runtime::runtime::WASM_BINARY_PATH` const to use
correct path to runtime blob.

---------

Co-authored-by: GitHub Action <action@github.com>

This PR introduces usage of `build_struct_json_patch` macro in all
runtimes (also guides) within the code base.

It also fixes macro to support _field init shorthand_, and _Struct
Update_ syntax which were missing in original implementation.

Follow up of #5700 and #5813

Remove `sp_runtime::RuntimeString` and replace with `Cow<'static, str>` or `String` depending on use case (#5693)

# Description

As described in https://github.com/paritytech/polkadot-sdk/issues/4001
`RuntimeVersion` was not encoded consistently using serde. Turned out it
was a remnant of old times and no longer actually needed. As such I
removed it completely in this PR and replaced with `Cow<'static, str>`
for spec/impl names and `String` for error cases.

Fixes https://github.com/paritytech/polkadot-sdk/issues/4001.

## Integration

For downstream projects the upgrade will primarily consist of following
two changes:
```diff
#[sp_version::runtime_version]
pub const VERSION: RuntimeVersion = RuntimeVersion {
-	spec_name: create_runtime_str!("statemine"),
-	impl_name: create_runtime_str!("statemine"),
+	spec_name: alloc::borrow::Cow::Borrowed("statemine"),
+	impl_name: alloc::borrow::Cow::Borrowed("statemine"),
```
```diff
		fn dispatch_benchmark(
			config: frame_benchmarking::BenchmarkConfig
-		) -> Result<Vec<frame_benchmarking::BenchmarkBatch>, sp_runtime::RuntimeString> {
+		) -> Result<Vec<frame_benchmarking::BenchmarkBatch>, alloc::string::String> {
```

SCALE encoding/decoding remains the same as before, but serde encoding
in runtime has changed from bytes to string (it was like this in `std`
environment already), which most projects shouldn't have issues with. I
consider the impact of serde encoding here low due to the type only
being used in runtime version struct and mostly limited to runtime
internals, where serde encoding/decoding of this data structure is quite
unlikely (though we did hit exactly this edge-case ourselves


).

## Review Notes

Most of the changes are trivial and mechanical, the only non-trivial
change is in
`substrate/primitives/version/proc-macro/src/decl_runtime_version.rs`
where macro call expectation in `sp_version::runtime_version`
implementation was replaced with function call expectation.

# Checklist

* [x] My PR includes a detailed description as outlined in the
"Description" and its two subsections above.
* [ ] My PR follows the [labeling requirements](

https://github.com/paritytech/polkadot-sdk/blob/master/docs/contributor/CONTRIBUTING.md#Process
) of this project (at minimum one label for `T` required)
* External contributors: ask maintainers to put the right label on your
PR.
* [ ] I have made corresponding changes to the documentation (if
applicable)

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Guillaume Thiolliere <guillaume.thiolliere@parity.io>
Co-authored-by: Bastian Köcher <info@kchr.de>

This PR adds `build_struct_json_patch` which helps to generate a JSON
used for preset.

Here is doc and example:

https://github.com/paritytech/polkadot-sdk/blob/d868b858/substrate/frame/support/src/generate_genesis_config.rs#L168-L266

And real-world usage:

https://github.com/paritytech/polkadot-sdk/blob/d868b858

/cumulus/parachains/runtimes/assets/asset-hub-rococo/src/genesis_config_presets.rs#L37-L61

Closes #5700

---------

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

provides low-level documentation on how the omni-node is meant to work.
This is meant to act as reusable material for other teams (e.g.
Papermoon and W3F) to use and integrate into the high level Polkadot
documentation.

Broadly speaking, for omni-node to have great rust-docs, we need to
focus on the following crates, all of which got a bit of love in this
PR:

1. `sp-genesis-builder`
2. `polkadot-omni-node`
3. `polkadot-omni-node-lib`
4. `frame-omni-bencher`

On top of this, we have now: 

* `polkadot_sdk_docs::guides` contains two new steps demonstrating the
most basic version of composing your pallet, putting it into a runtime,
and putting that runtime into omni-node
* `polkadot_sdk_docs::reference_docs::omni_node` to explain in more
detail how omni-node differs from the old-school node.
* `polkadot_sdk_docs::reference_docs::frame_weight_benchmarking` to
finally have a minimal reference about weights and benchmarking.
* It provides tests for some of the steps in
https://github.com/paritytech/polkadot-sdk/issues/5568


closes https://github.com/paritytech/polkadot-sdk/issues/5568
closes https://github.com/paritytech/polkadot-sdk/issues/4781

Next steps

- [x] Ensure the README of the parachain template is up-to-date.
@iulianbarbu


- [ ] Readme for `polkadot-omni-node` and similar is updated. For now,
use `cargo-readme` and copy over the rust-docs.

To build the branch locally and run this:
https://paritytech.github.io/polkadot-sdk/master/polkadot_sdk_docs/meta_contributing/index.html#how-to-develop-locally

---------

Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

Original PR https://github.com/paritytech/polkadot-sdk/pull/2280
reverted in https://github.com/paritytech/polkadot-sdk/pull/3665

This PR reintroduces the reverted functionality with additional changes,
related effort
[here](https://github.com/paritytech/polkadot-sdk/pull/3623).
Description is copied over from the original PR

First part of [Extrinsic
Horizon](https://github.com/paritytech/polkadot-sdk/issues/2415)

Introduces a new trait `TransactionExtension` to replace
`SignedExtension`. Introduce the idea of transactions which obey the
runtime's extensions and have according Extension data (né Extra data)
yet do not have hard-coded signatures.

Deprecate the terminology of "Unsigned" when used for
transactions/extrinsics owing to there now being "proper" unsigned
transactions which obey the extension framework and "old-style" unsigned
which do not. Instead we have __*General*__ for the former and
__*Bare*__ for ...

### Improved devx of running many pallets simultaneously

Changes to /cmd:
- Replace (flip) `--continue-on-fail` with `--fail-fast`, but only for
`bench`. This makes all pallets/runtimes run non-stop by default, as it
was primary use-case during tests
- The list of successful/failed pallets was hard to find within tons of
logs, so decided to write only needed logs in a file, and output as a
summary in the workflow and in the comment
- Side fix: updated `tasks_example` to `pallet_example_tasks` to make
compliant with standard naming

<img width="1006" alt="image"
src="https://github.com/user-attachments/assets/14896041-7018-4a0d-92b7-4508e81913c2">

+ added command results to workflow summary:
<img width="1275" alt="image"
src="https://github.com/user-attachments/assets/b4a8afdb-dc9f-4ff9-9720-28a88956035f">

---------

Co-authored-by: GitHub Action <action@github.com>

- moved the omni-node lib from
  `cumulus/polkadot-parachain/polkadot-parachain-lib` to
  `cumulus/polkadot-omni-node/lib`
- renamed `polkadot-parachain-lib` to `polkadot-omni-node-lib`
- added `polkadot-omni-node` binary

Related to https://github.com/paritytech/polkadot-sdk/issues/5566

Related to
https://github.com/paritytech/polkadot-sdk/pull/5924#issuecomment-2393558697

improve prdoc arguments validation & help:
- convert audiences options to snake_case. Fixes
https://github.com/paritytech/polkadot-sdk/issues/5927
  - support more than one audiences
  - define allowed bump options
- infer --pr from the actual PR (now it's optional, can still be
overwritten)


![image](https://github.com/user-attachments/assets/24e18fe2-2f67-4ce0-90e4-34f6c2f860c9)

Test evidence:
https://github.com/paritytech-stg/polkadot-sdk/pull/52/commits/6dd274e3678d287c163cfa6cb696acee9852767d

cc https://github.com/paritytech/ci_cd/issues/1039

Resolves https://github.com/paritytech/polkadot-sdk/issues/5026

This PR adds support for starting a dev node with manual seal consensus.
This can be done by using the `--dev-block-time` argument . For example:
```
polkadot-parachain --chain asset-hub-rococo-dev --dev-block-time 5000 --tmp
```

Partially implements
https://github.com/paritytech/polkadot-sdk/issues/5048

- adds a core selection runtime API to cumulus and a generic way of
configuring it for a parachain
- modifies the slot based collator to utilise the claim queue and the
generic core selection

What's left to be implemented (in a follow-up PR):
- add the UMP signal for core selection into the parachain-system pallet

View the RFC for more context:
https://github.com/polkadot-fellows/RFCs/pull/103

---------

Co-authored-by: command-bot <>

Fixes https://github.com/paritytech/polkadot-sdk/issues/5647

- [x]  create new command (reusing original py module)
- [x]  add unit-test cases (just the fact of proxy)
- [x]  update docs

Update `RuntimeVerison` type and use `system_version` to derive extrinsics root `StateVersion` instead of `V0` (#4257)

This PR 
- Renames `RuntimeVersion::state_version` to `system_version`
- Uses `Runtime::system_version` to derive extrinsics root
`StateVersion` instead of default `StateVersion::V0`

This PR should not be breaking any existing chains so long as they use
same `RuntimeVersion::state_version` for `Runtime::system_version`

Using `RuntimeVersion::system_version = 2` will make the extrinsics root
to use `StateVersion::V1` instead of `V0`

RFC for this change - https://github.com/polkadot-fellows/RFCs/pull/42

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Nazar Mokrynskyi <nazar@mokrynskyi.com>

Prdoc is now published as a Parity fork under the
[`parity-prdoc`](https://crates.io/crates/parity-prdoc) crate after the
directions diverged from the ideas of the original creator (discussions
[here](https://github.com/paritytech/prdoc/pull/40) and
[here](https://github.com/paritytech/prdoc/issues/36)).
Now updating the install instructions here.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Currently, the logo on the docs site blends into the background on the
light theme:

<img width="320px"
src="https://github.com/user-attachments/assets/4cded445-d3af-4100-bd11-6a33da836cb1">


[Here](https://internals.rust-lang.org/t/feedback-on-new-rust-documentation-logo-background/12787)
is a discussion related to this. There doesn't seem to be a perfect way
to handle this situation.

---

Silent because the docs crate is not part of the regular release
process.

---------

Co-authored-by: ordian <write@reusable.software>

When using with `polkadot-parachain`, you usually need to specify the
`relay_chain` and `para_id` fields in the chain spec.

With this PR it can be achieved by specifying newly added `--para-id`
and `--relay-chain` command line args, e.g:
```
chain-spec-builder create -r _runtime.wasm  --para-id 100 --relay-chain xxx default
```

This was implemented by simple _json_ blobs merging.

Additionally unit tests covering basic functionality were added.

Also adds a fix for not overwriting the chain spec with the default
config each time, swallowing not standard fields is also fixed.

Fixes: #4873

---------

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

Co-authored-by: Javier Viola <363911+pepoviola@users.noreply.github.com>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: command-bot <>

# Description

Trivial doc fixes:

* Replace the word `reminder` with `remainder` so that the English
matches the code intent.
* Explicit instruct the reader to `clone`.

## Review Notes

* Trivial

Co-authored-by: Jan-Jan <111935+Jan-Jan@users.noreply.github.com>

Adds a bot that automatically opens MRs into the `stable2407` branch
when the `A4-needs-backport` label is applied to a merged MR.

TODO:
- [x] ~~Settle on label vs error message trade-off~~ (resolved)

docs:

# Backporting

This document explains how to backport a merged PR from `master` to one
of the `stable*` branches. Backports should only be used to fix bugs or
security issues - never to introduce new features.

## Steps

1. Fix a bug through a PR that targets `master`.
2. Add label `A4-needs-backport` to the PR.
4. Merge the PR into `master`.
5. Wait for the bot to open the backport PR.
6. Ensure the change is audited or does not need audit.
7. Merge the backport PR. 

The label can also be added after the PR is merged.

## Example

For example here where the dev triggered the process by adding the label
after merging:


![backport-ex2](https://github.com/user-attachments/assets/c7b686db-a0fe-41f1-9d6f-959a5a7097b1)

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Closes https://github.com/paritytech/product-engineering/issues/93

- Deprecates old command bot bash scripts. New cmd implementation is
re-written on Python
- Deprecates `sync` command, which was never used
- Benchmarks:
- Uses new
[frame-omni-bencher](https://crates.io/crates/frame-omni-bencher)
- Simplifies usage to only providing a runtime and/or pallet name (even
multiple runtimes or pallets)
- Supports sub-modules (like `/cmd bench --runtime dev --pallet
pallet_asset_conversion_ops`)
- Can regenerate all weights with one command (substrate, polkadot,
cumulus) for provided pallet(s) name
- Adds [subweight](https://crates.io/crates/subweight-core) diff as a
result of bench command

Co-authored-by: kianenigma <kian@parity.io>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

A follow-up to https://github.com/paritytech/polkadot-sdk/pull/5447

Instead of having an option to select a PR template (like is the case
with issues), this change will make the one PR template we have the
default, it will show up for every new PR.

Also updated one link so it works properly in the PR body.

## Context

Currently, many crates have no readme files, even though they are public
and available on crates.io.
Others have just a couple of words that does not look super presentable.

Even though probably nobody starts a journey with `polkadot-sdk` or its
documentation with a readme of a random low-level crate, I think it
would look more mature to have a little better readmes there.

So, in an attempt to improve [the
aesthetics](https://github.com/paritytech/eng-automation/issues/10) of
`polkadot-sdk`, I propose a set of consistent, branded, better-looking
readmes for all published crates.

## What's inside

- ~~New readme files for published crates.~~
- A python script to generate new readmes, for the crates that have
none.
  - It will skip crates that do have a readme, and private crates.
- Added a new image asset to the repo - logo with a background.
- The main readme of the repo uses a [nice
trick](https://github.com/paritytech/polkadot-sdk/blob/ce6938ae/README.md?plain=1#L4-L5)
to accompany both light and dark modes - but that doesn't work on
`crates.io` so a single logo with a background is needed.

## Example

### Current

![Screenshot 2024-08-04 at 16 13
36](https://github.com/user-attachments/assets/3ae0881d-0f40-4614-9d2c-3c95029f0820)

### Changed

![Screenshot 2024-08-04 at 16 12
28](https://github.com/user-attachments/assets/fa7eadc8-aec8-4f77-84d9-54d63ce189cd)

# Description
Updates `template.rs` to reflect the two OZ templates available and a
short description

# Checklist

* [x] My PR includes a detailed description as outlined in the
"Description" and its two subsections above.
* [x] My PR follows the [labeling requirements](CONTRIBUTING.md#Process)
of this project (at minimum one label for `T`
  required)
* External contributors: ask maintainers to put the right label on your
PR.
* [x] I have made corresponding changes to the documentation (if
applicable)

---------

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

Related to https://github.com/paritytech/polkadot-sdk/issues/5210

## Before

<img width="320px"
src="https://github.com/user-attachments/assets/27c4c50b-632b-4a3b-a154-0e828a1ac650"/>

## After

<img width="320px"
src="https://github.com/user-attachments/assets/314954fa-7f49-415d-82dd-a9a99983a283"/>

- Where applicable, use a regular [`reference`] instead of
`../../../reference/index.html`.
- Typos.
- Update a link to `polkadot-evm` which has moved out of the monorepo.
- ~~The link specification for `chain_spec_builder` is invalid~~
(actually it was valid) - it works fine without it.

Part of https://github.com/paritytech/eng-automation/issues/10