Fixes https://github.com/paritytech/polkadot-sdk/issues/5296

The code do reduce or increase the weight by comparing
`benchmarked_weight` and `consumed_weight`.

But `benchmarked_weight` is the pre dispatch weight. not the post
dispatch weight that is actually written into the block weight by
`CheckWeight`.

So in case the consumed weight was: `pre dispatch weight > consumed
weight > post dispatch weight` then the reclaim code was reducing the
block weight instead of increasing it.

Might explain this issue even better
https://github.com/paritytech/polkadot-sdk/issues/5229

@skunert 
@s0me0ne-unkn0wn

Closes https://github.com/paritytech/ci_cd/issues/1012

In the real world, not all assets are sufficient. This aligns our
emulated networks to that reality. Only DOT and USDT are sufficient "by
default".

Updating the BHR and BHW runtime versions as a result of the changes in
https://github.com/paritytech/polkadot-sdk/pull/5074/

Despite what we had in the [original
request](https://github.com/paritytech/polkadot-sdk/issues/3155#issuecomment-1979037109),
I'm proposing a change to open a PR to the destination template
repositories instead of pushing the code.

This will give it a chance to run through the destination CI before
making changes, and to set stricter branch protection in the destination
repos.

This PR adds the possibility to set the docker stable release tag as an
input parameter to the produced docker images, so that it matches with
the release version

PR adds github-action for jobs test-linux-stable-oldkernel.
PR waits the latest release of forklift.

cc https://github.com/paritytech/ci_cd/issues/939
cc https://github.com/paritytech/ci_cd/issues/1006

---------

Co-authored-by: Maksym H <1177472+mordamax@users.noreply.github.com>

https://github.com/paritytech/polkadot-sdk/pull/4527/files#r1706673828

- Part of https://github.com/paritytech/ci_cd/issues/1006
- Closes: https://github.com/paritytech/ci_cd/issues/1010
- Related: https://github.com/paritytech/polkadot-sdk/pull/4405

- Possibly affecting how frame-omni-bencher works on different runtimes:
https://github.com/paritytech/polkadot-sdk/pull/5083

Currently works in parallel with gitlab short benchmarks. 
Triggered only by adding `GHA-migration` label to assure smooth
transition (kind of feature-flag).
Later when tested on random PRs we'll remove the gitlab and turn on by
default these tests

---------

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Uses custom metadata to exclude chain-specific crates.  
The only concern is that devs who want to use chain-specific crates,
still need to select matching versions numbers. Could possibly be
addresses with chain-specific umbrella crates, but currently it should
be possible to use [psvm](https://github.com/paritytech/psvm).

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Since `May 2023` after
https://github.com/paritytech/substrate/pull/13548 optimization,
`Blake2256` is faster with about 30%, that means that there is a
difference of ~30% between the benchmark values we ask validators to run
against and the machine we use for generating the weights.So if all
validators, just barely pass the benchmarks our weights are potentially
underestimated with about ~20%, so let's bring this two in sync.

Same thing happened when we merged
https://github.com/paritytech/polkadot-sdk/pull/2524 in `Nov 2023`
SR25519-Verify became faster with about 10-15%

## Results

Generated on machine from here:
https://github.com/paritytech/devops/pull/3210
```
+----------+----------------+--------------+-------------+-------------------+
| Category | Function       | Score        | Minimum     | Result            |
+============================================================================+
| CPU      | BLAKE2-256     | 1...

### Context

Since Rococo is now deprecated, we need another testnet to detect
bleeding-edge changes to Substrate, Polkadot, & BEEFY consensus
protocols that could brick the bridge.

It's the mirror PR of https://github.com/Snowfork/polkadot-sdk/pull/157
which has reviewed by Snowbridge team internally.

Synced with @acatangiu

 about that in channel
https://matrix.to/#/!gxqZwOyvhLstCgPJHO:matrix.parity.io/$N0CvTfDSl3cOQLEJeZBh-wlKJUXx7EDHAuNN5HuYHY4?via=matrix.parity.io&via=parity.io&via=matrix.org

---------

Co-authored-by: Clara van Staden <claravanstaden64@gmail.com>

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Follow-up to https://github.com/paritytech/polkadot-sdk/pull/4457, looks
like more things were missing

---------

Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>

Addresses
https://github.com/paritytech/polkadot-sdk/pull/5085#issuecomment-2265725858

Luckily, in the rest of the script, github API allows (or forces?) us to
read the state of PRs the same way as we read the state of issues, so it
works without any more changes.

This PR shows a warning when the `--public-addr` is not provided for
validators.

In the future, we'll transform this warning into a hard failure.
Validators are encouraged to provide this parameter for better
availability over the network.

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

I propose to have `ProofSizeExt` available during benchmarking so we can
improve the accuracy for extensions using it.

Another thing we could do is to also enable recording for the timing
benchmark here:

https://github.com/paritytech/polkadot-sdk/blob/035211d7/substrate/utils/frame/benchmarking-cli/src/pallet/command.rs#L232

Parachains will need to have recording enabled during import for
reclaim, so we could enable it here and provide a flag
`--disable-proof-recording` for scenarios where one does not want it.
Happy to hear opinions about this.

Closes #4517

Polkadot address: 12gMhxHw8QjEwLQvnqsmMVY1z5gFa54vND74aMUbhhwN6mJR

---------

Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>

Test currently failing, therefore improving to include a file from the
same crate to not trip up the caching.

R0 silent since this is only modifying unpublished crates.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Dónal Murray <donal.murray@parity.io>

[pallet_contracts] Increase the weight of the deposit_event host function to limit the memory used by events. (#4973)

This PR updates the weight of the `deposit_event` host function by
adding
a fixed ref_time of 60,000 picoseconds per byte. Given a block time of 2
seconds
and this specified ref_time, the total allocation size is 32MB.

---------

Co-authored-by: Alexander Theißen <alex.theissen@me.com>

This Workflow is not supposed to run in the paritytech/polkadot-sdk
repo. This Workflow is supposed to run only in the forks of the repo, in
`paritytech-release/polkadot-sdk` specifically, to automatically
maintain the critical fork synced with the upstream. This Workflow
should be always disabled in the paritytech/polkadot-sdk repo.

- Progresses https://github.com/paritytech/polkadot-sdk/issues/5226

There is no actual `try-runtime` or `runtime-benchmarks` functionality
in the minimal template at the moment.

The errors on polkadot-parachain are not printed with their full display
context(what is marked with `#[error(`) because main returns plain
Result and the error will be shown in its Debug format, that's not
consistent with how the polkadot binary behave and is not user friendly
since it does not tell them why they got the error.

Fix it by using `color_eyre` as polkadot already does it. 

Fixes: https://github.com/paritytech/polkadot-sdk/issues/5211

## Output before
```
Error: NetworkKeyNotFound("/acala/data/Collator2/chains/mandala-tc9/network/secret_ed25519")
```

## Output after
```
Error: 
   0: Starting an authorithy without network key in /home/alexggh/.local/share/polkadot-parachain/chains/asset-hub-kusama/network/secret_ed25519.
      
       This is not a safe operation because other authorities in the network may depend on your node having a stable identity.
      
       Otherwise these other authorities may not being able to reach you.
      
       If it is the first time running your node you could use one of the following methods:
      
       1. [Preferred] Separately generate the key with: <NODE_BINARY> key generate-node-key --base-path <YOUR_BASE_PATH>
      
       2. [Preferred] Separately generate the key with: <NODE_BINARY> key generate-node-key --file <YOUR_PATH_TO_NODE_KEY>
      
       3. [Preferred] Separately generate the key with: <NODE_BINARY> key generate-node-key --default-base-path
      
       4. [Unsafe] Pass --unsafe-force-node-key-generation and make sure you remove it for subsequent node restarts

```

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>

This PR adds functionality that allows tasks to enable auto-renewal.
Each task eligible for renewal can enable auto-renewal.

A new storage value is added to track all the cores with auto-renewal
enabled and the associated task running on the core. The `BoundedVec` is
sorted by `CoreIndex` to make disabling auto-renewal more efficient.

Cores are renewed at the start of a new bulk sale. If auto-renewal
fails(e.g. due to the sovereign account of the task not holding
sufficient balance), an event will be emitted, and the renewal will
continue for the other cores.

The two added extrinsics are:
- `enable_auto_renew`: Extrinsic for enabling auto renewal.
- `disable_auto_renew`: Extrinsic for disabling auto renewal.

TODOs:
- [x] Write benchmarks for the newly added extrinsics.

Closes: #4351

---------

Co-authored-by: Dónal Murray <donalm@seadanda.dev>

network/strategy: Backoff and ban overloaded peers to avoid submitting the same request multiple times (#5029)

This PR avoids submitting the same block or state request multiple times
to the same slow peer.

Previously, we submitted the same request to the same slow peer, which
resulted in reputation bans on the slow peer side.
Furthermore, the strategy selected the same slow peer multiple times to
submit queries to, although a better candidate may exist.

Instead, in this PR we:
- introduce a `DisconnectedPeers` via LRU with 512 peer capacity to only
track the state of disconnected peers with a request in flight
- when the `DisconnectedPeers` detects a peer disconnected with a
request in flight, the peer is backed off
  - on the first disconnection: 60 seconds
  - on second disconnection: 120 seconds
- on the third disconnection the peer is banned, and the peer remains
banned until the peerstore decays its reputation
  
This PR lifts the pressure from overloaded nodes that cannot process
requests in due time.
And if a peer is detected to be slow after backoffs, the peer is banned.

Theoretically, submitting the same request multiple times can still
happen when:
- (a) we backoff and ban the peer 
- (b) the network does not discover other peers -- this may also be a
test net
- (c) the peer gets reconnected after the reputation decay and is still
slow to respond



Aims to improve:
- https://github.com/paritytech/polkadot-sdk/issues/4924
- https://github.com/paritytech/polkadot-sdk/issues/531

Next Steps:
- Investigate the network after this is deployed, possibly bumping the
keep-alive timeout or seeing if there's something else misbehaving




This PR builds on top of:
- https://github.com/paritytech/polkadot-sdk/pull/4987


### Testing Done
- Added a couple of unit tests where test harness were set in place

- Local testnet

```bash
13:13:25.102 DEBUG tokio-runtime-worker sync::persistent_peer_state: Added first time peer 12D3KooWHdiAxVd8uMQR1hGWXccidmfCwLqcMpGwR6QcTP6QRMuD

13:14:39.102 DEBUG tokio-runtime-worker sync::persistent_peer_state: Remove known peer 12D3KooWHdiAxVd8uMQR1hGWXccidmfCwLqcMpGwR6QcTP6QRMuD state: DisconnectedPeerState { num_disconnects: 2, last_disconnect: Instant { tv_sec: 93355, tv_nsec: 942016062 } }, should ban: false

13:16:49.107 DEBUG tokio-runtime-worker sync::persistent_peer_state: Remove known peer 12D3KooWHdiAxVd8uMQR1hGWXccidmfCwLqcMpGwR6QcTP6QRMuD state: DisconnectedPeerState { num_disconnects: 3, last_disconnect: Instant { tv_sec: 93485, tv_nsec: 947551051 } }, should ban: true

13:16:49.108  WARN tokio-runtime-worker peerset: Report 12D3KooWHdiAxVd8uMQR1hGWXccidmfCwLqcMpGwR6QcTP6QRMuD: -2147483648 to -2147483648. Reason: Slow peer after backoffs. Banned, disconnecting.
```

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

While working on #5129 I noticed that after warp sync, nodes would
print:
```
2024-07-29 17:59:23.898 ERROR ⋮beefy: 🥩 Error: ConsensusReset. Restarting voter.    
```

After some debugging I found that we enter the following loop:
1. Wait for beefy pallet to be available: Pallet is detected available
directly after warp sync since we are at the tip.
2. Wait for headers from tip to beefy genesis to be available: During
this time we don't process finality notifications, since we later want
to inspect all the headers for authority set changes.
3. Gap sync finishes, route to beefy genesis is available.
4. The worker starts acting, tries to fetch beefy genesis block. It
fails, since we are acting on old finality notifications where the state
is already pruned.
5. Whole beefy subsystem is being restarted, loading the state from db
again and iterating a lot of headers.

This already happened before #5129.

This PR enables the `chainSpec_v1` class for the polkadot-parachian. 
The chainSpec is part of the rpc-v2 which is spec-ed at:
https://github.com/paritytech/json-rpc-interface-spec/blob/main/src/api/chainSpec.md.

This also paves the way for enabling a future `chainSpec_unstable_spec`
on all nodes.

Closes: https://github.com/paritytech/polkadot-sdk/issues/5191

cc @paritytech/subxt-team

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

Added a new adapter to xcm-builder, the `SingleAssetExchangeAdapter`.
This adapter makes it easy to use `pallet-asset-conversion` for
configuring the `AssetExchanger` XCM config item.

I also took the liberty of adding a new function to the `AssetExchange`
trait, with the following signature:

```rust
fn quote_exchange_price(give: &Assets, want: &Assets, maximal: bool) -> Option<Assets>;
```

The signature is meant to be fairly symmetric to that of
`exchange_asset`.
The way they interact can be seen in the doc comment for it in the
`AssetExchange` trait.

This is a breaking change but is needed for
https://github.com/paritytech/polkadot-sdk/pull/5131.
Another idea is to create a new trait for this but that would require
setting it in the XCM config which is also breaking.

Old PR: https://github.com/paritytech/polkadot-sdk/pull/4375.

---------

Co-authored-by: Adrian Catangiu <adrian@parity.io>

This PR replaces env_logger with sp_tracing because of an issue with
env_logger and gum #4660

---------

Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Andrei Eres <eresav@me.com>

As the release branch name changed starting from this release, this PR
adds it to the sync templates flow so that checkout step worked
properly.

---------

Co-authored-by: rzadp <roopert7@gmail.com>

This PR reduces the occurrences for identified observed addresses.

Litep2p discovers its external addresses by inspecting the
`IdentifyInfo::ObservedAddress` field reported by other peers.
After we get 5 confirmations of the same external observed address (the
address the peer dialed to reach us), the address is reported through
the network layer.

The PR effectively changes this from 5 to 2.
This has a subtle implication on freshly started nodes for the
authority-discovery discussed below.

The PR also makes the authority discovery a bit more robust by not
publishing records if the node doesn't have addresses yet to report.
This aims to fix a scenario where:
- the litep2p node has started, it has some pending observed addresses
but less than 5
- the authorit-discovery publishes a record, but at this time the node
doesn't have any addresses discovered and the record is published
without addresses -> this means other nodes w...

The test name is `test-frame-ui` I don't know if I can also change it to
`test-ui` without breaking other stuff. So I kept the name unchanged.

---------

Co-authored-by: Bastian Köcher <git@kchr.de>

As per #3326, removes usage of the `pallet::getter` macro from the
balances pallet. The syntax `StorageItem::<T, I>::get()` should be used
instead.

Also, adds public functions for compatibility.

cc @muraca

polkadot address: 5GsLutpKjbzsbTphebs9Uy4YK6gTN47MAaz6njPktidjR5cp

---------

Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

This PR reverts the removal of an [`if`
statement](https://github.com/paritytech/polkadot-sdk/commit/7ecf3f75#diff-8bf31ba8d9ebd6377983fd7ecc7f4e41cb1478a600db1a15a578d1ae0e8ed435L370)
merged recently, which affected test output verbosity of several pallets
(e.g. staking, EPM, and potentially others).

More generally, the `UnixTime::now` implementation of the timestamp
pallet should log an error *only* when called at the genesis block.

Adds a snapshot step to the try-runtime check that tries to download a
cached snapshot.
The cache is valid for the current day and is otherwise re-created.

Check is now only limited by build time and docker startup.

![Screenshot 2024-07-30 at 02 02
58](https://github.com/user-attachments/assets/0773e9b9-4a52-4572-a891-74b9d725ba70)

![Screenshot 2024-07-30 at 02 02
20](https://github.com/user-attachments/assets/4685ef17-a04c-4bdc-9d61-311d0010f71c)

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

This PR backports regular version bumps and reorganisation of the
`prdoc` files from the stable release branch back to master

[subsystem-benchmark] Update availability-distribution-regression-bench baseline after recent subsystem changes (#5180)

ISSUE
Link to the issue:
https://github.com/paritytech/polkadot-sdk/issues/3326

Deliverables

[Deprecation] remove pallet::getter usage from pallet-proxy

---------

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>