The `lazy_static` package does not work well in `no-std`: it requires
`spin_no_std` feature, which also will propagate into `std` if enabled.
This is not what we want.

This PR provides simple address uri parser which allows to get rid of
_regex_ which was used to parse the address uri, what in turns allows to
remove lazy_static.

Three regular expressions
(`SS58_REGEX`,`SECRET_PHRASE_REGEX`,`JUNCTION_REGEX`) were replaced with
the parser which unifies all of them.

The new parser does not support Unicode, it is ASCII only.

Related to: #2044

---------

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Koute <[email protected]>
Co-authored-by: command-bot <>

Related to #2285

- save the state of the BEEFY gadget after processing a finality proof.
We need this in order to avoid skipping blocks.
- avoid reprocessing the old state when not necessary

Addresses a bug caused by
https://github.com/paritytech/polkadot-sdk/pull/1189. The changes are
still not released yet, so would like to push the fix soon so it can go
together with the release of the above PR.

`fast_unstake` checks if a staker is exposed in an era. However, this fn
is still returning whether the staker is exposed based on the old
storage item. This PR fixes that by looking in both old and new exposure
storages.

Also adds some integrity tests for paged exposures.

New release includes logic to move all jobs to `spot instances`.
Thx!

The versioned migrations are already there in pallet nomination-pools:

https://github.com/paritytech/polkadot-sdk/blob/f6ee4781/substrate/frame/nomination-pools/src/migration.rs#L27-L48

Just updating the kitchensink runtime to point to them.

This is also nice because it points the dev to an example of how to use
`VersionedMigration`.

Missed in https://github.com/paritytech/polkadot-sdk/pull/1672

cc @joepetrowski

Westend now successfully updated to `spec: 103000`, we **have to
remove** the session keys migration before the next release as it
doesn't gracefully handle reapplying it.

Since preparation for the merge queues needs more time I'm enabling
zombienet jobs in PRs CI back.

Adds a function for querying the last runtime upgrade spec version. This
can be useful for when writing runtime level migrations to ensure that
they are not executed multiple times. An example would be a session key
migration.

---------

Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>

Rococo<>Wococo bridge is replaced by Rococo<Westend bridge, so this PR
removes unneeded code.

- [x] update bridges subtree after
https://github.com/paritytech/parity-bridges-common/pull/2692

---------

Co-authored-by: command-bot <>
Co-authored-by: Svyatoslav Nikolsky <[email protected]>

Refunding surplus happens anyway on xcm_executor::post_process(),
automatically refunding surplus to original_origin at the end of
execution. Since SovereignPaidRemoteExporter doesn't ClearOrigin, it can
simply rely on the automatic mechanism.

Furthermore, RefundSurplus instruction refunds _surplus_. Surplus exists
only as a result of Transact, SetErrorHandler or SetAppendix
instructions, none of which being part of the
SovereignPaidRemoteExporter XCM program. So surplus is always zero here
anyway.

Add collectives and glutton parachain westend runtimes to prepare for
#1737.

The removal of system parachain native runtimes #1737 is blocked until
chainspecs and runtime APIs can be dealt with cleanly (merge of #1256
and follow up PRs).

In the meantime, these additions are ready to be merged to `master`, so
I have separated them out into this PR.

Also marked `bridge-hub-westend` as unimplemented in line with [this
issue](https://github.com/paritytech/parity-bridges-common/issues/2602).

TODO
- [x] add to `command-bot` benchmarks
- [x] add to `command-bot-scripts` benchmarks
- [x] generate weights

---------

Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: Muharem <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Branislav Kontur <[email protected]>

The goal of this PR is to migrate Identity deposits from the Relay Chain
to a system parachain.

The problem I want to solve is that `IdentityOf` and `SubsOf` both store
an amount that's held in reserve as a storage deposit. When migrating to
a parachain, we can take a snapshot of the actual `IdentityInfo` and
sub-account mappings, but should migrate (off chain) the `deposit`s to
zero, since the chain (and by extension, accounts) won't have any funds
at genesis.

The good news is that we expect parachain deposits to be significantly
lower (possibly 100x) on the parachain. That is, a deposit of 21 DOT on
the Relay Chain would need 0.21 DOT on a parachain. This PR proposes to
migrate the deposits in the following way:

1. Introduces a new pallet with two extrinsics: 
- `reap_identity`: Has a configurable `ReapOrigin`, which would be set
to `EnsureSigned` on the Relay Chain (i.e. callable by anyone) and
`EnsureRoot` on the parachain (we don't want identities reaped from
there).
- `poke_deposit`: Checks what deposit the pallet holds (at genesis,
zero) and attempts to update the amount based on the calculated deposit
for storage data.
2. `reap_identity` clears all storage data for a `target` account and
unreserves their deposit.
3. A `ReapIdentityHandler` teleports the necessary DOT to the parachain
and calls `poke_deposit`. Since the parachain deposit is much lower, and
was just unreserved, we know we have enough.

One awkwardness I ran into was that the XCMv3 instruction set does not
provide a way for the system to teleport assets without a fee being
deducted on reception. Users shouldn't have to pay a fee for the system
to migrate their info to a more efficient location. So I wrote my own
program and did the `InitiateTeleport` accounting on my own to send a
program with `UnpaidExecution`. Have discussed an
`InitiateUnpaidTeleport` instruction with @franciscoaguirre

 . Obviously
any chain executing this would have to pass a `Barrier` for free
execution.

TODO:

- [x] Confirm People Chain ParaId
- [x] Confirm People Chain deposit rates (determined in
https://github.com/paritytech/polkadot-sdk/pull/2281)
- [x] Add pallet to Westend

---------

Co-authored-by: Bastian Köcher <[email protected]>

PR prepares CI to the GitHub Merge Queues. All github actions that were
running in PR adjusted so they can run in the merge queues. Zombienet
jobs will do nothing during PRs but they will run during merge queues.

Jobs that will be skipped during PR:
 - all zombienet jobs
 - all publish docker jobs

Jobs that will be skipped during merge queue:
 - check-labels
 - check-prdoc
 - pr-custom-review
 - review trigger

cc https://github.com/paritytech/ci_cd/issues/862

Get rid of public `ChainSync::..._requests()` functions and return all
requests as actions.

---------

Co-authored-by: Sebastian Kunert <[email protected]>

Disallow reserve transfers that use teleportable fees if `(origin,
fees)` matches `XcmTeleportFilter`.

Add regression tests for filtering based on `XcmTeleportFilter` for both
`(limited_)reserve_transfer_assets()` and `(limited_)teleport_assets`
extrinsics.

This PR introduces:
- XCM  host functions `xcm_send`, `xcm_execute`
- An Xcm trait into the config. that proxy these functions to to
`pallet_xcm`, or disable their usage by using `()`.
- A mock_network and xcm_test files to test the newly added xcm-related
functions.

---------

Co-authored-by: Keith Yeung <[email protected]>
Co-authored-by: Sasha Gryaznov <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Alexander Theißen <[email protected]>

Adds a `NodeFeatures` bitfield value to the runtime `HostConfiguration`,
with the purpose of coordinating the enabling of node-side features,
such as: https://github.com/paritytech/polkadot-sdk/issues/628 and
https://github.com/paritytech/polkadot-sdk/issues/598.
These are features that require all validators enable them at the same
time, assuming all/most nodes have upgraded their node versions.

This PR doesn't add any feature yet. These are coming in future PRs.

Also adds a runtime API for querying the state of the client features
and an extrinsic for setting/unsetting a feature by its index in the bitfield.

Note: originally part of:
https://github.com/paritytech/polkadot-sdk/pull/1644, but posted as
standalone to be reused by other PRs until the initial PR is merged

Fixes #1437

Co-authored-by: Sophia Gold <[email protected]>

Co-authored-by: Marcin S <[email protected]>

Remove the genesis hash from tests:
- Clippy was passing on the PR:
https://github.com/paritytech/polkadot-sdk/pull/2296
- Clippy fails on master:
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4328487

This was a race with merging:
https://github.com/paritytech/polkadot-sdk/pull/2295, which introduced
another test that used the `CHAIN_GENESIS`

Signed-off-by: Alexandru Vasile <[email protected]>

Fixes https://github.com/paritytech/polkadot-sdk/issues/106

Port of cumulus PR https://github.com/paritytech/cumulus/pull/2960

This PR adds the ability to bid for collator slots even after the max
number of collators have already registered. This eliminates the first
come, first served mechanism that was in place before.

Key changes:
- added `update_bond` extrinsic to allow registered candidates to adjust
their bonds in order to dynamically control their bids
- added `take_candidate_slot` extrinsic to try to replace an already
existing candidate by bidding more than them
- candidates are now kept in a sorted list in the pallet storage, where
the top `DesiredCandidates` out of `MaxCandidates` candidates in the
list will be selected by the session pallet as collators
- if the candidacy bond is increased through a `set_candidacy_bond`
call, candidates which don't meet the new bond requirements are kicked


# Checklist

- [ ] My PR includes a detailed description as outlined in the
"Description" section above
- [ ] My PR follows the [labeling
requirements](https://github.com/paritytech/polkadot-sdk/blob/master/docs/CONTRIBUTING.md#process)
of this project (at minimum one label for `T` required)
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [ ] I have added tests that prove my fix is effective or that my
feature works (if applicable)
- [ ] If this PR alters any external APIs or interfaces used by
Polkadot, the corresponding Polkadot PR is ready as well as the
corresponding Cumulus PR (optional)

---------

Signed-off-by: georgepisaltu <[email protected]>

The method has been removed from the spec
(https://github.com/paritytech/json-rpc-interface-spec/tree/main/src),
this PR keeps the `chainHead` in sync with that change.

@paritytech/subxt-team

---------

Signed-off-by: Alexandru Vasile <[email protected]>

This PR adds support for multiple hashes being passed to the
`chainHeda_unpin` parameters.

The `hash` parameter is renamed to `hash_or_hashes` per
https://github.com/paritytech/json-rpc-interface-spec/pull/111.

While at it, a new integration test is added to check the unpinning of
multiple hashes. The API is checked against a hash or a vector of
hashes.

cc @paritytech/subxt-team

---------

Signed-off-by: Alexandru Vasile <[email protected]>

Turns out to access environment secrets the workflow must explicitly opt
in to the environment.

- switch from Rococo<>Wococo to Rococo<>Westend bridge
- add bidirectional simple tests
- remove Wococo chains from xcm-emulator
- added tests for assets transfers over Rococo<>Westend bridge 

fixes https://github.com/paritytech/parity-bridges-common/issues/2405

add details in `--dev` flag to tell that it disables local peer
discovery

### Context

When adding automated end-to-end tests, we replaced `--dev` by 

```
`--chain=dev`, `--force-authoring`, `--rpc-cors=all`, `--alice`, and `--tmp` flags
```

as stated in the command line documentation. But the tests started
failing due to the nodes connecting to each other.

### Fix

This PR includes additional command line documentation to explain more
in detail what `--dev` flag inludes.

Closes https://github.com/paritytech/polkadot-sdk/issues/2241

See issue comments for more details about this storage.

## Problem/Motivation
The benchmark for the `ExpectPallet` XCM instruction uses a hard-coded
version `4.0.0` for the `frame_system` pallet. Unfortunately, this
doesn't work for the `polkadot-fellows/runtimes` repository, where we
use dependencies from `crates.io`, e.g.,
[frame-system::23.0.0.0](https://github.com/polkadot-fellows/runtimes/blob/dd7f86f0d50064481ed0b7c0218494a5cfad997e/relay/kusama/Cargo.toml#L83).

Closes: https://github.com/paritytech/polkadot-sdk/issues/2284 

## Solution
This PR fixes the benchmarks that require pallet information and enables
the runtime to provide the correct/custom pallet information. The
default implementation provides `frame_system::Pallet` with index `0`,
where the version is not hard-coded but read from the runtime.


## Local testing

Added log for `T::valid_pallet` to the benchmarks like:
```
let valid_pallet = T::valid_pallet();
log::info!(
	target: "frame::benchmark::pallet",
	"valid_pallet: {}::{}::{}::{}::{}",
	valid_pallet.index,
	valid_pallet.module_name,
	valid_pallet.crate_version.major,
	valid_pallet.crate_version.minor,
	valid_pallet.crate_version.patch,
);
```

Run benchmarks for `westend`:
```
cargo run --bin=polkadot --features=runtime-benchmarks -- benchmark pallet --steps=2 --repeat=1 --extrinsic=* --heap-pages=4096 --json-file=./bench.json --chain=westend-dev --template=./polkadot/xcm/pallet-xcm-benchmarks/template.hbs --pallet=pallet_xcm_benchmarks::generic --output=./polkadot/runtime/westend/src/weights/xcm
```

---

For actual `frame_system` version:
```
[package]
name = "frame-system"
version = "4.0.0-dev"
```

Log dump:
```
2023-11-13 12:56:45 Starting benchmark: pallet_xcm_benchmarks::generic::query_pallet    
2023-11-13 12:56:45 valid_pallet: 0::frame_system::4::0::0    
2023-11-13 12:56:45 valid_pallet: 0::frame_system::4::0::0    
2023-11-13 12:56:45 valid_pallet: 0::frame_system::4::0::0    
2023-11-13 12:56:45 Starting benchmark: pallet_xcm_benchmarks::generic::expect_pallet    
2023-11-13 12:56:45 valid_pallet: 0::frame_system::4::0::0    
2023-11-13 12:56:45 valid_pallet: 0::frame_system::4::0::0    
2023-11-13 12:56:45 valid_pallet: 0::frame_system::4::0::0 
```


For changed `frame_system` version:
```
[package]
name = "frame-system"
version = "5.1.3-dev"
```

Log dump:
```
2023-11-13 12:51:51 Starting benchmark: pallet_xcm_benchmarks::generic::query_pallet    
2023-11-13 12:51:51 valid_pallet: 0::frame_system::5::1::3    
2023-11-13 12:51:51 valid_pallet: 0::frame_system::5::1::3    
2023-11-13 12:51:51 valid_pallet: 0::frame_system::5::1::3    
2023-11-13 12:51:51 Starting benchmark: pallet_xcm_benchmarks::generic::expect_pallet    
2023-11-13 12:51:51 valid_pallet: 0::frame_system::5::1::3    
2023-11-13 12:51:51 valid_pallet: 0::frame_system::5::1::3    
2023-11-13 12:51:51 valid_pallet: 0::frame_system::5::1::3
```

## References

Closes: https://github.com/paritytech/polkadot-sdk/issues/2284

BEEFY ECDSA signatures are on keccak has of the messages. As such we can
not simply call

`EcdsaBlsPair::verify(signature.as_inner_ref(), msg,
self.as_inner_ref())`

because that invokes ecdsa default verification which perfoms blake2
hash which we don't want.

This bring up the second issue makes: This makes `sign` and `verify`
function in `pair_crypto` useless, at least for BEEFY use case.
Moreover, there is no obvious clean way to generate the signature given
that pair_crypto does not exposes `sign_prehashed`. You could in theory
query the keystore for the pair (could you?), invoke `to_raw` and
re-generate each sub-pair and sign using each. But that sounds extremely
anticlimactic and will be frow upon by auditors . So I appreciate any
alternative suggestion.

---------

Co-authored-by: Davide Galassi <[email protected]>
Co-authored-by: Robert Hambrock <[email protected]>

Collators were previously reencoding the available data and checking the
erasure root.
Replace that with just checking the PoV hash, which consumes much less
CPU and takes less time.

We also don't need to check the `PersistedValidationData` hash, as
collators don't use it.

Reason:
https://github.com/paritytech/polkadot-sdk/issues/575#issuecomment-1806572230

After systematic chunks recovery is merged, collators will no longer do
any reed-solomon encoding/decoding, which has proven to be a great CPU
consumer.

Signed-off-by: alindima <[email protected]>

Moved the review event of review-bot to only be triggered in approvals.

Because we only update the required reviews when someone approves, this
will stop the bot from immediately requesting a new review when someone
comments or request changes as they should have been already notified in
the first batch.

## Motivation

`pallet-xcm` is the main user-facing interface for XCM functionality,
including assets manipulation functions like `teleportAssets()` and
`reserve_transfer_assets()` calls.

While `teleportAsset()` works both ways, `reserve_transfer_assets()`
works only for sending reserve-based assets to a remote destination and
beneficiary when the reserve is the _local chain_.

## Solution

This PR enhances `pallet_xcm::(limited_)reserve_withdraw_assets` to
support transfers when reserves are other chains.
This will allow complete, **bi-directional** reserve-based asset
transfers user stories using `pallet-xcm`.

Enables following scenarios:
- transferring assets with local reserve (was previously supported iff
asset used as fee also had local reserve - now it works in all cases),
- transferring assets with reserve on destination,
- transferring assets with reserve on remote/third-party chain (iff
assets and fees have same remote reserve),
- transferring assets with reserve different than the reserve of the
asset to be used as fees - meaning can be used to transfer random asset
with local/dest reserve while using DOT for fees on all involved chains,
even if DOT local/dest reserve doesn't match asset reserve,
- transferring assets with any type of local/dest reserve while using
fees which can be teleported between involved chains.

All of the above is done by pallet inner logic without the user having
to specify which scenario/reserves/teleports/etc. The correct scenario
and corresponding XCM programs are identified, and respectively, built
automatically based on runtime configuration of trusted teleporters and
trusted reserves.

#### Current limitations:
- while `fees` and "non-fee" `assets` CAN have different reserves (or
fees CAN be teleported), the remaining "non-fee" `assets` CANNOT, among
themselves, have different reserve locations (this is also implicitly
enforced by `MAX_ASSETS_FOR_TRANSFER=2`, but this can be safely
increased in the future).
- `fees` and "non-fee" `assets` CANNOT have **different remote**
reserves (this could also be supported in the future, but adds even more
complexity while possibly not being worth it - we'll see what the future
holds).

Fixes https://github.com/paritytech/polkadot-sdk/issues/1584
Fixes https://github.com/paritytech/polkadot-sdk/issues/2055

---------

Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Branislav Kontur <[email protected]>