## Description

The `frame-benchmarking-cli` crate has not been buildable without the
`rocksdb` feature since version 1.17.0.

**Error:**  
```rust
self.database()?.unwrap_or(Database::RocksDb),
                             ^^^^^^^ variant or associated item not found in `Database`
```

This issue is also related to the `rocksdb` feature bleeding (#3793),
where the `rocksdb` feature was always activated even when compiling
this crate with `--no-default-features`.

**Fix:**  
- Resolved the error by choosing `paritydb` as the default database when
compiled without the `rocksdb` feature.
- Fixed the issue where the `sc-cli` crate's `rocksdb` feature was
always active, even compiling `frame-benchmarking-cli` with
`--no-default-features`.

## Review Notes

Fix the crate to be built without rocksdb, not intended to solve #3793.

---------

Co-authored-by: command-bot <>

Changes:
- Automatically detect if current env is tty. If not disable the spinner
logging.
- Add `Result` type.
- Format log style.

Originally reported from:
-
https://github.com/hack-ink/polkadot-runtime-releaser/blob/4811d2b419649a73edd5bd1f748a858b846eb139/action/try-runtime/action.yml#L75-L91
-
https://github.com/hack-ink/polkadot-runtime-releaser-workshop/pull/3#issuecomment-2563883943

Closes #7010.

---

Polkadot address: 156HGo9setPcU2qhFMVWLkcmtCEGySLwNqa3DaEiYSWtte4Y

---------

Signed-off-by: Xavier Lau <x@acg.box>
Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

# Description

This PR removes the outdated warning message from the
`frame-omni-bencher` CLI that states the tool is "not yet battle
tested". Fixes #7019

## Integration

No integration steps are required.

## Review Notes

The functionality of the tool remains unchanged. Removes the warning
message from the CLI output.

---------

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>

Using `{:#?}` for the error details is a bit annoying, this change makes
a more consistent formatting style for error messages.

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: command-bot <>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

It doesn't make sense to only reorder the features array.

For example:

This makes it hard for me to compare the dependencies and features,
especially some crates have a really really long dependencies list.
```toml​
[dependencies]
c = "*"
a = "*"
b = "*"

[features]
std = [
  "a",
  "b",
  "c",
]
```

This makes my life easier.
```toml​
[dependencies]
a = "*"
b = "*"
c = "*"

[features]
std = [
  "a",
  "b",
  "c",
]
```

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: command-bot <>

# Description

Get runtime's metadata, parse it and verify pallets list for a pallet
named `ParachainSystem` (for now), and block number to be the same for
both node and runtime. Ideally we'll add other pallets checks too, at
least a small set of pallets we think right away as mandatory for
parachain compatibility.
Closes: #5565 

## Integration

Runtime devs must be made aware that to be fully compatible with Omni
Node, certain naming conventions should be respected when defining
pallets (e.g we verify parachain-system pallet existence by searching
for a pallet with `name` `ParachainSystem` in runtime's metadata). Not
finding such a pallet will not influence the functionality yet, but by
doing these checks we could provide useful feedback for runtimes that
are clearly not implementing what's required for full parachain
compatibility with Omni Node.

## Review Notes

- [x] parachain system check
- [x] check frame_system's metadata to ensure the block number in there
is the same as the one in the node side
- [x] add tests for the previous checking logic
- [x] update omni node polkadot-sdk docs to make these conventions
visible.
- [ ] add more pallets checks?

---------

Signed-off-by: Iulian Barbu <iulian.barbu@parity.io>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

Close: #5858

---------

Co-authored-by: Bastian Köcher <git@kchr.de>

# Description

Closes #6335.

## Integration

N/A

## Review Notes

`RuntimeTarget` is converted to return path to the custom target JSON
file

---------

Signed-off-by: Jarkko Sakkinen <jarkko@parity.io>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>
Co-authored-by: Koute <koute@users.noreply.github.com>

This PR exposes the *unstable* metadata V16. The metadata is exposed
under the unstable u32::MAX number.

Developers can start experimenting with the new features of the metadata
v16. *Please note that this metadata is under development and expect
breaking changes until stabilization.*

The `ExtrinsicMetadata` trait receives a breaking change. Its associated
type `VERSION` is rename to `VERSIONS` and now supports a constant
static list of metadata versions.
The versions implemented for `UncheckedExtrinsic` are v4 (legacy
version) and v5 (new version).

For metadata collection, it is assumed that all `TransactionExtensions`
are under version 0.

Builds on top of: https://github.com/paritytech/polkadot-sdk/pull/5274

Closes: https://github.com/paritytech/polkadot-sdk/issues/5980
Closes: https://github.com/paritytech/polkadot-sdk/issues/5347
Closes: https://github.com/paritytech/polkadot-sdk/issues/5285

cc @paritytech/subxt-team

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: James Wilson <james@jsdw.me>
Co-authored-by: GitHub Action <action@github.com>

Fix prometheus metrics not shutting down if there are open connections.
I fixed the same issue in the past but it broke again after a dependecy
upgrade.

See also:

https://github.com/paritytech/polkadot-sdk/pull/1637

When using multiple instances of the same pallet, each instance was
executed with the components of all instances. While actually each
instance should only be executed with the components generated for the
particular instance. The problem here was that in the runtime only the
pallet-name was used to determine if a certain pallet should be
benchmarked. When using instances, the pallet name is the same for both
of these instances. The solution is to also take the instance name into
account.

The fix requires to change the `Benchmark` runtime api to also take the
`instance`. The node side is written in a backwards compatible way to
also support runtimes which do not yet support the `instance` parameter.

---------

Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: clangenb <37865735+clangenb@users.noreply.github.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>

# Description

Closes #5940  

## Integration

Node devs that rely on templates' nodes binaries for minimal or
parachain would need to follow the updated templates' README.mds again
to find how to build the nodes' binaries.

## Review Notes

Conditional compilation of virtual workspaces would compile the
`members` list as if we passed `--workspace` flag to `cargo build` ,
except when adding a `default-members` list which will be used for any
cargo command executed in the virtual workspace root. To build the full
members list needs passing `--workspace` flag.

Other options investigated:
- feature guard the `node` crate by defining a feature in the `node`
crate, but it feels too complex since all code needs to be feature
guarded. I haven't tried it but technically speaking it might work. I
think though it looks awkward and my opinion is that the alternative is
better.
- defining features in the virtual workspace's Cargo.toml doesn't work
(thought that I might create a feature that will have a dependency on
the `node` crate and then not passing the feature to cargo build results
in ignoring the `node` crate)
- skipping compilation by using an environment variable, read in the
build script, that will exit compilation abruptly if not set, but I
couldn't make it work.
- exclude the crate from the members list and build it specifically by
passing `--package minimal-template-node` flag to the `cargo build`
command. This has the disadvantage of not allowing IDEs based on rust
analyzer to index/compile the node crate.

My conclusion is that any option would require two commands to build the
template, one with the node and one without, and both must be included
in the README or templates usage documentation. If it comes which ones
to pick I am in favor of the `default-members` option, which requires
minimal intervention and expresses how cargo commands are executed on
top of the workspace members, and what's left out from regular usage.

### Testing

Testing was conducted as described bellow:

- [x] zombienet with `minimal-template-node` , `parachain-template-node`
and `polkadot-omni-node`. Things work as expected.
- [x] no chopsticks testing was conducted - feels a bit out of scope for
OmniNode related docs and overall testing when promoting it over the
templates' nodes.
- [x] testing the changes for the sync templates workflow (ignore the
added comment from the Cargo.tomls, it was removed here on this branch:
[99bff3e2](https://github.com/paritytech/polkadot-sdk/pull/5954/commits/99bff3e2

)):
[minimal](https://github.com/paritytech-stg/polkadot-sdk-minimal-template/pull/22/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542R9),
[parachain](https://github.com/paritytech-stg/polkadot-sdk-parachain-template/pull/19/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542R9),
[solochain](https://github.com/paritytech-stg/polkadot-sdk-solochain-template/pull/17/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542R9).
The links correspond to PRs opened by a bot after manually starting the
sync-templates workflow on `paritytech-stg` org to test the end result
of the `Cargo.toml` changes.

---------

Signed-off-by: Iulian Barbu <iulian.barbu@parity.io>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

# Benchmark Overhead Command for Parachains

This implements the `benchmark overhead` command for parachains. Full
context is available at:
https://github.com/paritytech/polkadot-sdk/issues/5303. Previous attempt
was this https://github.com/paritytech/polkadot-sdk/pull/5283, but here
we have integration into frame-omni-bencher and improved tooling.

## Changes Overview

Users are now able to use `frame-omni-bencher` to generate
`extrinsic_weight.rs` and `block_weight.rs` files for their runtime. The
core logic for generating these remains untouched; this PR provides
mostly machinery to make it work for parachains at all.

Similar to the pallet benchmarks, we gain the option to benchmark based
on just a runtime:

```
frame-omni-bencher v1 benchmark overhead --runtime {{runtime}}
```

or with a spec:

```
frame-omni-bencher v1 benchmark overhead --chain {{spec}} --genesis-builder spec
```

In this case, the genesis state is generated from the runtime presets.
However, it is also possible to use `--chain` and genesis builder `spec`
to generate the genesis state from the chain spec.

Additionally, we use metadata to perform some checks based on the
pallets the runtime exposes:

- If we see the `ParaInherent` pallet, we assume that we are dealing
with a relay chain. This means that we don't need proof recording during
import (since there is no storage weight).
- If we detect the `ParachainSystem` pallet, we assume that we are
dealing with a parachain and take corresponding actions like patching a
para id into the genesis state.

On the inherent side, I am currently supplying the standard inherents
every parachain needs.

In the current state, `frame-omni-bencher` supports all system chains.
In follow-up PRs, we could add additional inherents to increase
compatibility.

Since we are building a block during the benchmark, we also need to
build an extrinsic. By default, I am leveraging subxt to build the xt
dynamically. If a chain is not compatible with the `SubstrateConfig`
that comes with `subxt`, it can provide a custom extrinsic builder to
benchmarking-cli. This requires either a custom bencher implementation
or an integration into the parachains node.

Also cumulus-test-runtime has been migrated to provide genesis configs.

## Chain Compatibility
The current version here is compatible with the system chains and common
substrate chains. The way to go for others would be to customize the
frame-omni-bencher by providing a custom extrinsicbuilder. I did an
example implementation that works for mythical:
https://github.com/skunert/mythical-bencher

## Follow-Ups
- After #6040 is finished, we should integrate this here to make the
tooling truly useful. In the current form, the state is fairly small and
not representative.

## How to Review
I recommend starting from
[here](https://github.com/paritytech/polkadot-sdk/pull/5891/files#diff-50830ff756b3ac3403b7739d66c9e3a5185dbea550669ca71b28d19c7a2a54ecR264),
this method is the main entry point for omni-bencher and `polkadot`
binary.

TBD:
- [x] PRDoc

---------

Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>

Original PR https://github.com/paritytech/polkadot-sdk/pull/2280
reverted in https://github.com/paritytech/polkadot-sdk/pull/3665

This PR reintroduces the reverted functionality with additional changes,
related effort
[here](https://github.com/paritytech/polkadot-sdk/pull/3623).
Description is copied over from the original PR

First part of [Extrinsic
Horizon](https://github.com/paritytech/polkadot-sdk/issues/2415)

Introduces a new trait `TransactionExtension` to replace
`SignedExtension`. Introduce the idea of transactions which obey the
runtime's extensions and have according Extension data (né Extra data)
yet do not have hard-coded signatures.

Deprecate the terminology of "Unsigned" when used for
transactions/extrinsics owing to there now being "proper" unsigned
transactions which obey the extension framework and "old-style" unsigned
which do not. Instead we have __*General*__ for the former and
__*Bare*__ for ...

### Fork-Aware Transaction Pool Implementation

This PR introduces a fork-aware transaction pool (fatxpool) enhancing
transaction management by maintaining the valid state of txpool for
different forks.

### High-level overview
The high level overview was added to
[`sc_transaction_pool::fork_aware_txpool`](https://github.com/paritytech/polkadot-sdk/blob/3ad0a1b7/substrate/client/transaction-pool/src/fork_aware_txpool/mod.rs#L21)
module. Use:
```
cargo  doc --document-private-items -p sc-transaction-pool --open
```
to build the doc. It should give a good overview and nice entry point
into the new pool's mechanics.

<details>
  <summary>Quick overview (documentation excerpt)</summary>

#### View
For every fork, a view is created. The view is a persisted state of the
transaction pool computed and updated at the tip of the fork. The view
is built around the existing `ValidatedPool` structure.

A view is created on every new best block notification. To create a
view, one of the existing views is chosen and cloned.

When the chain progresses, the view is kept in the cache
(`retracted_views`) to allow building blocks upon intermediary blocks in
the fork.

The views are deleted on finalization: views lower than the finalized
block are removed.

The views are updated with the transactions from the mempool—all
transactions are sent to the newly created views.
A maintain process is also executed for the newly created
views—basically resubmitting and pruning transactions from the
appropriate tree route.

##### View store
View store is the helper structure that acts as a container for all the
views. It provides some convenient methods.

##### Submitting transactions
Every transaction is submitted to every view at the tips of the forks.
Retracted views are not updated.
Every transaction also goes into the mempool.

##### Internal mempool
Shortly, the main purpose of an internal mempool is to prevent a
transaction from being lost. That could happen when a transaction is
invalid on one fork and could be valid on another. It also allows the
txpool to accept transactions when no blocks have been reported yet.

The mempool removes its transactions when they get finalized.
Transactions are also periodically verified on every finalized event and
removed from the mempool if no longer valid.

#### Events
Transaction events from multiple views are merged and filtered to avoid
duplicated events.
`Ready` / `Future` / `Inblock` events are originated in the Views and
are de-duplicated and forwarded to external listeners.
`Finalized` events are originated in fork-aware-txpool logic.
`Invalid` events requires special care and can be originated in both
view and fork-aware-txpool logic.

#### Light maintain
Sometime transaction pool does not have enough time to prepare fully
maintained view with all retracted transactions being revalidated. To
avoid providing empty ready transaction set to block builder (what would
result in empty block) the light maintain was implemented. It simply
removes the imported transactions from ready iterator.

#### Revalidation
Revalidation is performed for every view. The revalidation process is
started after a trigger is executed. The revalidation work is terminated
just after a new best block / finalized event is notified to the
transaction pool.
The revalidation result is applied to the newly created view which is
built upon the revalidated view.

Additionally, parts of the mempool are also revalidated to make sure
that no transactions are stuck in the mempool.


#### Logs
The most important log allowing to understand the state of the txpool
is:
```
              maintain: txs:(0, 92) views:[2;[(327, 76, 0), (326, 68, 0)]] event:Finalized { hash: 0x8...f, tree_route: [] }  took:3.463522ms
                             ^   ^         ^     ^   ^  ^      ^   ^  ^        ^                                                   ^
unwatched txs in mempool ────┘   │         │     │   │  │      │   │  │        │                                                   │
   watched txs in mempool ───────┘         │     │   │  │      │   │  │        │                                                   │
                     views  ───────────────┘     │   │  │      │   │  │        │                                                   │
                      1st view block # ──────────┘   │  │      │   │  │        │                                                   │
                           number of ready tx ───────┘  │      │   │  │        │                                                   │
                                numer of future tx ─────┘      │   │  │        │                                                   │
                                        2nd view block # ──────┘   │  │        │                                                   │
                                      number of ready tx ──────────┘  │        │                                                   │
                                           number of future tx ───────┘        │                                                   │
                                                                 event ────────┘                                                   │
                                                                       duration  ──────────────────────────────────────────────────┘
```
It is logged after the maintenance is done.

The `debug` level enables per-transaction logging, allowing to keep
track of all transaction-related actions that happened in txpool.
</details>


### Integration notes

For teams having a custom node, the new txpool needs to be instantiated,
typically in `service.rs` file, here is an example:

https://github.com/paritytech/polkadot-sdk/blob/9c547ff3

/cumulus/polkadot-omni-node/lib/src/common/spec.rs#L152-L161

To enable new transaction pool the following cli arg shall be specified:
`--pool-type=fork-aware`. If it works, there shall be information
printed in the log:
```
2024-09-20 21:28:17.528  INFO main txpool: [Parachain]  creating ForkAware txpool.
````

For debugging the following debugs shall be enabled:
```
      "-lbasic-authorship=debug",
      "-ltxpool=debug",
```
*note:* trace for txpool enables per-transaction logging.

### Future work
The current implementation seems to be stable, however further
improvements are required.
Here is the umbrella issue for future work:
- https://github.com/paritytech/polkadot-sdk/issues/5472


Partially fixes: #1202

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>

Changes:
- Add `--exclude-pallets` to exclude some pallets from runtimes where we
dont have genesis presets yet
- Make `--genesis-builder-policy=none` work with `--runtime`
- CI: Run the frame-omni-bencher for all runtimes

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: ggwpez <ggwpez@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: alvicsam <alvicsam@gmail.com>
Co-authored-by: Maksym H <mordamax@gmail.com>
Co-authored-by: Alexander Samusev <41779041+alvicsam@users.noreply.github.com>
Co-authored-by: Maksym H <1177472+mordamax@users.noreply.github.com>

# Description

close #5641

---------

Co-authored-by: Bastian Köcher <git@kchr.de>

# Description

`substrate-wasm-builder` can be a build dependency for crates which
develop FRAME runtimes. I had a tough time seeing errors happening in
such crates (e.g. runtimes from the `templates` directory) in my IDE. I
use a combination of rust-analyzer + nvim-lsp + nvim-lspconfig +
rustacean.vim and all of this stack is not able to correctly parse
errors emitted during the `build` phase.

As a matter of fact there is also a cargo issue tracking specifically
this issue where cargo doesn't propagate the `--message-format` type to
the build phase: [here](https://github.com/rust-lang/cargo/issues/14246)
initially and then
[here](https://github.com/rust-lang/cargo/issues/8283). It feels like a
solution for this use case isn't very close, so if it comes to runtimes
development (both as an SDK user and developer), enabling wasm builder
to emit diagnostics messages friendly to IDEs would be useful for
regular workflows where IDEs are used...

cc https://github.com/paritytech/ci_cd/issues/1035

cc https://github.com/paritytech/ci_cd/issues/1023

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: command-bot <>
Co-authored-by: Maksym H <1177472+mordamax@users.noreply.github.com>
Co-authored-by: gui <gui.thiolliere@gmail.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: ggwpez <ggwpez@users.noreply.github.com>

The genesis state is currently partially provided via
`OverlayedChanges`, but these changes are reset by the runtime after the
first repetition, causing the second repitition to use an invalid
genesis state.

Changes:
- Provide the genesis state as a `Storage` without any
`OverlayedChanges` to make it work correctly with repetitions.
- Add `--genesis-builder-preset` option to use different genesis preset
names.
- Improve error messages.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: ggwpez <ggwpez@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>

# Description

When we start a node with connections to external RPC servers (as a
minimal node), we lack metrics around how many individual calls we're
doing to the remote RPC servers and their duration. This PR adds metrics
that measure durations of each RPC call made by the minimal nodes, and
implicitly how many calls there are.

Closes #5409 
Closes #5689

## Integration

Node operators should be able to track minimal node metrics and decide
appropriate actions according to how the metrics are interpreted/felt.
The added metrics can be observed by curl'ing the prometheus metrics
endpoint for the ~relaychain~ parachain (it was changed based on the
review). The metrics are represented by
~`polkadot_parachain_relay_chain_rpc_interface`~
`relay_chain_rpc_interface` namespace (I realized lining up
`parachain_relay_chain` in the same metric might be confusing :).
Excerpt from the curl:

```
relay_chain_rpc_interface_bucket{method="chain_getBlockHash",chain="rococo_local_testnet",le="0.001"} 15
relay_chain_rpc_interface_bucket{method="chain_getBlockHash",chain="rococo_local_testnet",le="0.004"} 23
relay_chain_rpc_interface_bucket{method="chain_getBlockHash",chain="rococo_local_testnet",le="0.016"} 23
relay_chain_rpc_interface_bucket{method="chain_getBlockHash",chain="rococo_local_testnet",le="0.064"} 23
relay_chain_rpc_interface_bucket{method="chain_getBlockHash",chain="rococo_local_testnet",le="0.256"} 24
relay_chain_rpc_interface_bucket{method="chain_getBlockHash",chain="rococo_local_testnet",le="1.024"} 24
relay_chain_rpc_interface_bucket{method="chain_getBlockHash",chain="rococo_local_testnet",le="4.096"} 24
relay_chain_rpc_interface_bucket{method="chain_getBlockHash",chain="rococo_local_testnet",le="16.384"} 24
relay_chain_rpc_interface_bucket{method="chain_getBlockHash",chain="rococo_local_testnet",le="65.536"} 24
relay_chain_rpc_interface_bucket{method="chain_getBlockHash",chain="rococo_local_testnet",le="+Inf"} 24
relay_chain_rpc_interface_sum{method="chain_getBlockHash",chain="rococo_local_testnet"} 0.11719075
relay_chain_rpc_interface_count{method="chain_getBlockHash",chain="rococo_local_testnet"} 24
```

## Review Notes

The way we measure durations/hits is based on `HistogramVec` struct
which allows us to collect timings for each RPC client method called
from the minimal node., It can be extended to measure the RPCs against
other dimensions too (status codes, response sizes, etc). The timing
measuring is done at the level of the `relay-chain-rpc-interface`, in
the `RelayChainRpcClient` struct's method 'request_tracing'. A single
entry point for all RPC requests done through the
relay-chain-rpc-interface. The requests durations will fall under
exponential buckets described by start `0.001`, factor `4` and count
`9`.

---------

Signed-off-by: Iulian Barbu <iulian.barbu@parity.io>

Introduces a trait for verifying existence proofs in the runtime. The
trait is implemented for the 16 patricia merkle tree and the binary
tree.

---------

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: command-bot <>

Fixes: https://github.com/paritytech/polkadot-sdk/issues/5122.

This PR extends the existing single core `benchmark_cpu` to also build a
score of the entire processor by spawning `EXPECTED_NUM_CORES(8)`
threads and averaging their throughput.

This is better than simply checking the number of cores, because also
covers multi-tenant environments where the OS sees a high number of
available CPUs, but because it has to share it with the rest of his
neighbours its total throughput does not satisfy the minimum
requirements.


## TODO
- [x] Obtain reference values on the reference hardware.

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>

rpc server: listen to `ipv6 socket` if available and `--experimental-rpc-endpoint` CLI option (#4792)

Close https://github.com/paritytech/polkadot-sdk/issues/3488,
https://github.com/paritytech/polkadot-sdk/issues/4331

This changes/adds the following:

1. The default setting is that substrate starts a rpc server that
listens to localhost both Ipv4 and Ipv6 on the same port. Ipv6 is
allowed to fail because some platforms may not support it
2. A new RPC CLI option `--experimental-rpc-endpoint` which allow to
configure arbitrary listen addresses including the port, if this is
enabled no other interfaces are enabled.
3. If the local addr is not found for any of the sockets the server is
not started throws an error.
4. Remove the deny_unsafe from the RPC implementations instead this is
an extension to allow different polices for different interfaces/sockets
such one may enable unsafe on local interface and safe on only the
external interface.

So for instance in this PR it's now possible to start up three RPC
endpoints as follows:
```
$ polkadot --experimental-rpc-endpoint "listen-addr=127.0.0.1:9944,rpc-methods=unsafe" --experimental-rpc-endpoint "listen-addr=0.0.0.0:9945,rpc-methods=safe,rate-limit=100" --experimental-rpc-endpoint "listen-addr=[::1]:9944,optional=true"
```

#### Needs to be addressed

~1. Support binding to a random port if it's fails with the default
stuff for backward compatible reasons~
~2. How to sync that the rpc CLI params and that the rpc-listen-addr
align, hard to maintain...~
~3. Add similar warning prints for exposing unsafe methods on external
interfaces..~
~4. Inline todos + the hacky String conversion from rpc params.~

#### Cons with this PR

Manual strings parsing impl more error-prone than relying on clap....

//cc @jsdw @BulatSaif @PierreBesson @bkchr



---------

Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

as part of paritytech/devops/3502, try-runtime nodes were migrated to a
new provider with a new domain address. The PR fixes all the references
to try-runtime nodes on rococo, westend, kusama and polkadot networks.

---------

Co-authored-by: ArshamTeymouri <ArshamTeymouri@users.noreply.github.com>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>

Changes:
- Set default level to `Info` again. Seems like a dependency update set
it to something higher.
- Fix docs to not use `--locked` since we rely on dependency bumps via
cargo.
- Add README with rust docs.
- Fix bug where the node ignored `--heap-pages` argument.

You can test the `--heap-pages` bug by running this command on master
and then on this branch. Note that it should fail because of the very
low heap pages arg:
`cargo run --release --bin polkadot --features=runtime-benchmarks --
benchmark pallet --chain=dev --steps=10 --repeat=30
--wasm-execution=compiled --heap-pages=8 --pallet=frame-system
--extrinsic="*"`

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: ggwpez <ggwpez@users.noreply.github.com>

The CI isn't happy with the amount of output:
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/7035621/raw

---------

Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>

Since `May 2023` after
https://github.com/paritytech/substrate/pull/13548 optimization,
`Blake2256` is faster with about 30%, that means that there is a
difference of ~30% between the benchmark values we ask validators to run
against and the machine we use for generating the weights.So if all
validators, just barely pass the benchmarks our weights are potentially
underestimated with about ~20%, so let's bring this two in sync.

Same thing happened when we merged
https://github.com/paritytech/polkadot-sdk/pull/2524 in `Nov 2023`
SR25519-Verify became faster with about 10-15%

## Results

Generated on machine from here:
https://github.com/paritytech/devops/pull/3210
```
+----------+----------------+--------------+-------------+-------------------+
| Category | Function       | Score        | Minimum     | Result            |
+============================================================================+
| CPU      | BLAKE2-256     | 1.00 GiBs    | 783.27 MiBs |  Pass (130.7 %) |
|----------+----------------+--------------+-------------+-------------------|
| CPU      | SR25519-Verify | 637.62 KiBs  | 560.67 KiBs |  Pass (113.7 %) |
|----------+----------------+--------------+-------------+-------------------|
| Memory   | Copy           | 12.19 GiBs   | 11.49 GiBs  | 

 Pass (106.1 %) |
```

Discovered and discussed here:
https://github.com/paritytech/polkadot-sdk/pull/5127#issuecomment-2258423469

## Downsides

Machines that barely passed the benchmark will suddenly find themselves
bellow the benchmark, but since that is just an warning and everything
else continues as before it shouldn't be too impactful and should give
the validators the necessary information that they need to become
compliant, since they actually aren't when compared with the used
weights.

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>

I propose to have `ProofSizeExt` available during benchmarking so we can
improve the accuracy for extensions using it.

Another thing we could do is to also enable recording for the timing
benchmark here:

https://github.com/paritytech/polkadot-sdk/blob/035211d7/substrate/utils/frame/benchmarking-cli/src/pallet/command.rs#L232

Parachains will need to have recording enabled during import for
reclaim, so we could enable it here and provide a flag
`--disable-proof-recording` for scenarios where one does not want it.
Happy to hear opinions about this.

This PR replaces env_logger with sp_tracing because of an issue with
env_logger and gum #4660

---------

Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Andrei Eres <eresav@me.com>

A bit of a controversial move, but a good preparation for even further
reducing the traffic on outdated content of `substrate.io`. Current
status:

<img width="728" alt="Screenshot 2024-07-15 at 11 32 48"
src="https://github.com/user-attachments/assets/df33b164-0ce7-4ac4-bc97-a64485f12571">

Previously, I was in favor of changing the domain of the rust-docs to
something like `polkadot-sdk.parity.io` or similar, but I think the
current format is pretty standard and has a higher chance of staying put
over the course of time:

`<org-name>.github.io/<repo-name>` ->
`https://paritytech.github.io/polkadot-sdk/`

part of https://github.com/paritytech/eng-automation/issues/10

When building multiple runtimes in parallel, each of them will try to
use the concurrency set by the parent cargo process. For example, in a
system with 8 cpu cores, building 3 runtimes in parallel creates 8 * 3
tasks. This results in the system hanging because of the high cpu and
memory usage.

This PR allows the substrate_wasm_builder to use the same [jobserver][1]
as the parent cargo process, making all invocations of cargo share the
same concurrency pool. So in a system with 8 cores, there will never be
more than 8 tasks running at the same time.

Implementation roughly based on [cargo][2] but with less unsafe.

This can be tested by telling cargo to use half the cpu cores, like
`cargo build -j4` in an 8 core machine. Before this PR it will use 100%
cpu when building 2 runtimes in parallel, after this PR it will always
use 50%.

[1]:
https://doc.rust-lang.org/cargo/reference/build-scripts.html#jobserver
[2]:
https://github.com/rust-lang/cargo/blob/d1b5f0759eedf5f1126c781c64232856956069ad/src/cargo/util/context/mod.rs#L271

---------

Co-authored-by: Bastian Köcher <git@kchr.de>

Closes: https://github.com/paritytech/polkadot-sdk/issues/3192

We want to add linter to the entire node based on a template. Just like
`cargo clippy -- --deny missing_docs`. And we have the error (pasted at
the end).
The dummy crate is used to test whether the WASM toolchain is installed
and working as expected. And for some reason this dummy crate included
as a target for the linter. I added an attribute to pass the check.
```
note: To improve backtraces for build dependencies, set the CARGO_PROFILE_DEV_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation.

Caused by:
  process didn't exit successfully: `/Users/hrls/src/atleta/target/debug/build/atleta-runtime-b15153eff20cbe96/build-script-build` (exit status: 1)
  --- stderr
  Rust WASM target for toolchain stable-aarch64-apple-darwin is not properly installed; please install it!

  Further error information:
  ------------------------------------------------------------
     Compiling dummy-crate v1.0.0 (/var/folders/h1/_5gdnk8901n959lc28fwx8400000gn/T/.tmpUQCLaV)
  error: missing documentation for the crate
   --> src/main.rs:1:1
    |
  1 | fn main() {}
    | ^^^^^^^^^^^^
    |
    = note: requested on the command line with `-D missing-docs`

  error: could not compile `dummy-crate` (bin "dummy-crate") due to 1 previous error
  ------------------------------------------------------------


```

Co-authored-by: Bastian Köcher <git@kchr.de>

Partly fixes
https://github.com/paritytech/polkadot-sdk/pull/4890#discussion_r1655548633

Still the offchain API needs to be updated to hyper v1.0 and I opened an
issue for it, it's using low-level http body features that have been
removed

This is PR updates jsonrpsee v0.23 which mainly changes:
- Add `Extensions` which we now is using to get the connection id (used
by the rpc spec v2 impl)
- Update hyper to v1.0, http v1.0, soketto and related crates
(hyper::service::make_service_fn is removed)
- The subscription API for the client is modified to know why a
subscription was closed.

Full changelog here:
https://github.com/paritytech/jsonrpsee/releases/tag/v0.23.0

---------

Co-authored-by: Bastian Köcher <git@kchr.de>

After preparing in https://github.com/paritytech/polkadot-sdk/pull/4633,
we can lift also all internal dependencies up to the workspace.

This does not actually change anything, but uses `workspace = true` for
all dependencies. You can check it with:
```bash
git checkout -q $(git merge-base oty-lift-all-deps origin/master)
cargo tree -e features > master.out

git checkout -q oty-lift-all-deps
cargo tree -e features > new.out
diff master.out new.out
```

It did not yet lift 100% of dependencies, some inside of `target.*` or
some that had conflicting aliases introduced recently. But i will do
these together in a follow-up with CI checks.

Can be reproduced with [zepter](https://github.com/ggwpez/zepter/):
`zepter transpose d lift-to-workspace "regex:.*" --version-resolver
highest --skip-package "polkadot-sdk" --ignore-errors --fix`.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

## TODO

- [x] test/confirm that the release script is ok

---------

Co-authored-by: Javier Viola <363911+pepoviola@users.noreply.github.com>
Co-authored-by: Egor_P <egor@parity.io>

This branch propose to avoid clones in append by storing offset and size
in previous overlay depth.
That way on rollback we can just truncate and change size of existing
value.
To avoid copy it also means that :

- append on new overlay layer if there is an existing value: create a
new Append entry with previous offsets, and take memory of previous
overlay value.
- rollback on append: restore value by applying offsets and put it back
in previous overlay value
- commit on append: appended value overwrite previous value (is an empty
vec as the memory was taken). offsets of commited layer are dropped, if
there is offset in previous overlay layer they are maintained.
- set value (or remove) when append offsets are present: current
appended value is moved back to previous overlay value with offset
applied and current empty entry is overwrite (no offsets kept).

The modify mechanism is not needed anymore.
This branch lacks testing and break some existing genericity (bit of
duplicated code), but good to have to check direction.

Generally I am not sure if it is worth or we just should favor
differents directions (transients blob storage for instance), as the
current append mechanism is a bit tricky (having a variable length in
first position means we sometime need to insert in front of a vector).

Fix #30.

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: EgorPopelyaev <egor@parity.io>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>