- unit tests for pov-recovery
- elastic scaling support (recovering multiple candidates in a single
relay chain block)
- also some small cleanups
- also switches to candidates_pending_availability in
`handle_empty_block_announce_data`

Fixes https://github.com/paritytech/polkadot-sdk/issues/3577

After https://github.com/paritytech/polkadot-sdk/pull/4097 is merged, we
should also add a zombienet test, similar to the existing
`0002-pov_recovery.toml` but which has a single collator using elastic
scaling on multiple cores.

take over #2941 
[Weights
compare](https://weights.tasty.limo/compare?unit=weight&ignore_errors=true&threshold=10&method=asymptotic&repo=polkadot-sdk&old=master&new=pg%2Fwasmi-to-v0.32.0-beta.7&path_pattern=substrate%2Fframe%2F**%2Fsrc%2Fweights.rs%2Cpolkadot%2Fruntime%2F*%2Fsrc%2Fweights%2F**%2F*.rs%2Cpolkadot%2Fbridges%2Fmodules%2F*%2Fsrc%2Fweights.rs%2Ccumulus%2F**%2Fweights%2F*.rs%2Ccumulus%2F**%2Fweights%2Fxcm%2F*.rs%2Ccumulus%2F**%2Fsrc%2Fweights.rs)

---------

Co-authored-by: command-bot <>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>

Thanks for @xlc for the original seed info, I've just fixed it up a bit
and added example links.

I've moved the comparison between eth-rpc-api and frontier outside, as
it is opinionation. I think the content there was good but should live
in the README of the corresponding repos. No strong opinion, happy
either way.

---------

Co-authored-by: Bryan Chen <xlchen1291@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Gonçalo Pestana <g6pestana@gmail.com>
Co-authored-by: command-bot <>

`secp256k1 v0.28.0` and `secp256k1-sys v0.9.0` were yanked because
building them fails for `aarch64-apple-darwin` targets.

Use the `secp256k1 v0.28.2` and `secp256k1-sys v0.9.2` patched versions
that build fine on ARM chipset MacOS.

In order to receive a username in `pallet-identity`, users have to,
among other things, provide a signature of the desired username. Right
now, there is an [extra encoding
step](https://github.com/paritytech/polkadot-sdk/blob/4ab078d6

/substrate/frame/identity/src/lib.rs#L1119)
when generating the payload to sign.

Encoding a `Vec` adds extra bytes related to the length, which changes
the payload. This is unnecessary and confusing as users expect the
payload to sign to be just the username bytes. This PR fixes this issue
by validating the signature directly against the username bytes.

---------

Signed-off-by: georgepisaltu <george.pisaltu@parity.io>

Added reference doc about:
- the pallet genesis config and genesis build, 
- runtime `genesis-builder` API,
- presets,
- interacting with the `chain-spec-builder` tool

I've added [minimal
runtime](https://github.com/paritytech/polkadot-sdk/tree/mku-chain-spec-guide/docs/sdk/src/reference_docs/chain_spec_runtime)
to demonstrate above topics.

I also sneaked in some little improvement to `chain-spec-builder` which
allows to parse output of the `list-presets` command.

---------

Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

This allows to use custom para_ids with cumulus-test-runtime. 

Zombienet is patching the genesis entries for `ParachainInfo`. This did
not work with `test-parachain` because it was using the `test_pallet`
for historic reasons I guess.

Depends on https://github.com/paritytech/polkadot-sdk/pull/4621.

Implemented the
[`XcmPaymentApi`](https://github.com/paritytech/polkadot-sdk/pull/3607)
and [`DryRunApi`](https://github.com/paritytech/polkadot-sdk/pull/3872)
on all system parachains.

More scenarios can be tested on both rococo and westend if all system
parachains implement this APIs.
The objective is for all XCM-enabled runtimes to implement them.
After demonstrating fee estimation in a UI on the testnets, come the
fellowship runtimes.

Step towards https://github.com/paritytech/polkadot-sdk/issues/690.

Currently, `chain-spec-builder` only creates a spec with `Live` chain
type. This PR adds the ability to specify it while keeping the same
default.

---------

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

Revived version of https://github.com/paritytech/substrate/pull/13311 .
Except Signature is not generic and is dictated by AuthorityId.

---------

Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Robert Hambrock <roberthambrock@gmail.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>

Fixes #4360 

Also rename: AllowedRenewals -> PotentialRenewals to avoid confusion of
future readers. (An entry in `AllowedRenewals` is not enough to allow a
renewal, the assignment also has to be complete, which is only checked
afterwards.)

- [x] Does not work with renewals as is - fix.
- [x] More tests
- [x] PR docs

Edit 1:
(Relevant blog post:
https://grillapp.net/12935/agile-coretime-pricing-explained-166522?ref=29715)

---------

Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: Dónal Murray <donal.murray@parity.io>
Co-authored-by: command-bot <>

marking it as release-able, attaching the same version number that is
attached to other binaries such as `polkadot` and `polkadot-parachain`.

I have more thoughts about the version number, though. The chain-spec
builder is mainly a user of the `sp-genesis-builder` api. So the
versioning should be such that it helps users know give a version of
`sp-genesis-builder` in their runtime, which version of
`chain-spec-builder` should they use?

With this, we can possibly alter the version number to always match
`sp-genesis-builder`.

Fixes https://github.com/paritytech/polkadot-sdk/issues/4352

- [x] Add to release artifacts ~~similar to
https://github.com/paritytech/polkadot-sdk/pull/4405~~ done here:
https://github.com/paritytech/polkadot-sdk/pull/4557

---------

Co-authored-by: Branislav Kontur <bkontur@gmail.com>

**Don't look at the commit history, it's confusing, as this branch is
based on another branch that was merged**

Fixes #598 
Also implements [RFC
#47](https://github.com/polkadot-fellows/RFCs/pull/47)

## Description

- Availability-recovery now first attempts to request the systematic
chunks for large POVs (which are the first ~n/3 chunks, which can
recover the full data without doing the costly reed-solomon decoding
process). This has a fallback of recovering from all chunks, if for some
reason the process fails. Additionally, backers are also used as a
backup for requesting the systematic chunks if the assigned validator is
not offering the chunk (each backer is only used for one systematic
chunk, to not overload them).
- Quite obviously, recovering from systematic chunks is much faster than
recovering from regular chunks (4000% faster as measured on my apple M2
Pro).
- Introduces a `ValidatorIndex` -> `ChunkIndex` mapping which is
different for every core, in order to avoid only querying the first n/3
validators over and over again in the same session. The mapping is the
one described in RFC 47.
- The mapping is feature-gated by the [NodeFeatures runtime
API](https://github.com/paritytech/polkadot-sdk/pull/2177) so that it
can only be enabled via a governance call once a sufficient majority of
validators have upgraded their client. If the feature is not enabled,
the mapping will be the identity mapping and backwards-compatibility
will be preserved.
- Adds a new chunk request protocol version (v2), which adds the
ChunkIndex to the response. This may or may not be checked against the
expected chunk index. For av-distribution and systematic recovery, this
will be checked, but for regular recovery, no. This is backwards
compatible. First, a v2 request is attempted. If that fails during
protocol negotiation, v1 is used.
- Systematic recovery is only attempted during approval-voting, where we
have easy access to the core_index. For disputes and collator
pov_recovery, regular chunk requests are used, just as before.

## Performance results

Some results from subsystem-bench:

with regular chunk recovery: CPU usage per block 39.82s
with recovery from backers: CPU usage per block 16.03s
with systematic recovery: CPU usage per block 19.07s

End-to-end results here:
https://github.com/paritytech/polkadot-sdk/issues/598#issuecomment-1792007099

#### TODO:

- [x] [RFC #47](https://github.com/polkadot-fellows/RFCs/pull/47)
- [x] merge https://github.com/paritytech/polkadot-sdk/pull/2177 and
rebase on top of those changes
- [x] merge https://github.com/paritytech/polkadot-sdk/pull/2771 and
rebase
- [x] add tests
- [x] preliminary performance measure on Versi: see
https://github.com/paritytech/polkadot-sdk/issues/598#issuecomment-1792007099
- [x] Rewrite the implementer's guide documentation
- [x] https://github.com/paritytech/polkadot-sdk/pull/3065 
- [x] https://github.com/paritytech/zombienet/issues/1705 and fix
zombienet tests
- [x] security audit
- [x] final versi test and performance measure

---------

Signed-off-by: alindima <alin@parity.io>
Co-authored-by: Javier Viola <javier@parity.io>

This PR removes deprecated code:
- The `RuntimeGenesisConfig` generic type parameter in
`GenericChainSpec` struct.
- `ChainSpec::from_genesis` method allowing to create chain-spec using
closure providing runtime genesis struct
- `GenesisSource::Factory` variant together with no longer needed
`GenesisSource`'s generic parameter `G` (which was intended to be a
runtime genesis struct).


https://github.com/paritytech/polkadot-sdk/blob/17b56fae/substrate/client/chain-spec/src/chain_spec.rs#L559-L563

Fixes https://github.com/paritytech/polkadot-sdk/issues/3748

Adds a subsystem benchmark for statements-distribution subsystem.

Results in CI (reference hw):
```
$ cargo bench -p polkadot-statement-distribution --bench statement-distribution-regression-bench --features subsystem-benchmarks

[Sent to peers] standart_deviation 0.07%
[Received from peers] standart_deviation 0.00%
[statement-distribution] standart_deviation 0.97%
[test-environment] standart_deviation 1.03%

Network usage, KiB                     total   per block
Received from peers                1088.0000    108.8000
Sent to peers                      1238.1800    123.8180

CPU usage, seconds                     total   per block
statement-distribution                0.3897      0.0390
test-environment                      0.4715      0.0472
```

## [0.5.0] - 2023-05-24

This is a small patch release that makes the `FindNode` command a bit
more robst:

- The `FindNode` command now retains the K (replication factor) best
results.
- The `FindNode` command has been updated to handle errors and
unexpected states without panicking.

### Changed

- kad: Refactor FindNode query, keep K best results and add tests
([#114](https://github.com/paritytech/litep2p/pull/114))

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>

# Umbrella Crate

The Polkadot-SDK "umbrella" is a crate that re-exports all other
published crates. This makes it
possible to have a very small `Cargo.toml` file that only has one
dependency, the umbrella
crate. This helps with selecting the right combination of crate
versions, since otherwise 3rd
party tools are needed to select a compatible set of versions.

## Features

The umbrella crate supports no-std builds and can therefore be used in
the runtime and node.
There are two main features: `runtime` and `node`. The `runtime` feature
enables all `no-std`
crates, while the `node` feature enables all `std` crates. It should be
used like any other
crate in the repo, with `default-features = false`.

For more fine-grained control, additionally, each crate can be enabled
selectively. The umbrella
exposes one feature per dependency. For example, if you only want to use
the `frame-support`
crate, you can enable the `frame-support` feature.

The umbrella exposes a few more general features:
- `tuples-96`: Needs to be enabled for runtimes that have more than 64
pallets.
- `serde`: Specifically enable `serde` en/decoding support.
- `experimental`: Experimental enable experimental features - should not
yet used in production.
- `with-tracing`: Enable tracing support.
- `try-runtime`, `runtime-benchmarks` and `std`: These follow the
standard conventions.
- `runtime`: As described above, enable all `no-std` crates.
- `node`: As described above, enable all `std` crates.
- There does *not* exist a dedicated docs feature. To generate docs,
enable the `runtime` and
`node` feature. For docs.rs the manifest contains specific configuration
to make it show up
  all re-exports.

There is a specific `zepter` check in place to ensure that the features
of the umbrella are
correctly configured. This check is run in CI and locally when running
`zepter`.

## Generation

The umbrella crate needs to be updated every time when a new crate is
added or removed from the
workspace. It is checked in CI by calling its generation script. The
generation script is
located in `./scripts/generate-umbrella.py` and needs dependency
`cargo_workspace`.

Example: `python3 scripts/generate-umbrella.py --sdk . --version 1.9.0`

## Usage

> Note: You can see a live example in the `staging-node-cli` and
`kitchensink-runtime` crates.

The umbrella crate can be added to your runtime crate like this:

`polkadot-sdk = { path = "../../../../umbrella", features = ["runtime"],
default-features =
false}`

or for a node:

`polkadot-sdk = { path = "../../../../umbrella", features = ["node"],
default-features = false
}`

In the code, it is then possible to bring all dependencies into scope
via:

`use polkadot_sdk::*;`

### Known Issues

The only known issue so far is the fact that the `use` statement brings
the dependencies only
into the outer module scope - not the global crate scope. For example,
the following code would
need to be adjusted:

```rust
use polkadot_sdk::*;

mod foo {
   // This does sadly not compile:
   frame_support::parameter_types! { }

   // Instead, we need to do this (or add an equivalent `use` statement):
   polkadot_sdk::frame_support::parameter_types! { }
}
```

Apart from this, no issues are known. There could be some bugs with how
macros locate their own
re-exports. Please compile issues that arise from using this crate.

## Dependencies

The umbrella crate re-exports all published crates, with a few
exceptions:
- Runtime crates like `rococo-runtime` etc are not exported. This
otherwise leads to very weird
  compile errors and should not be needed anyway.
- Example and fuzzing crates are not exported. This is currently
detected by checking the name
of the crate for these magic words. In the future, it will utilize
custom metadata, as it is
  done in the `rococo-runtime` crate.
- The umbrella crate itself. Should be obvious :)

## Follow Ups
- [ ] Re-writing the generator in Rust - the python script is at its
limit.
- [ ] Using custom metadata to exclude some crates instead of filtering
by names.
- [ ] Finding a way to setting the version properly. Currently its
locked in the CI script.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

@serban300

 could you please do the same for the MMR crate? Am not sure
what commit was released since there are no release tags in the repo.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

We need to use the `polkadot-ckb-merkle-mountain-range` dependency
published on `crates.io` in order to unblock the release of the
`sp-mmr-primitives` crate

The XCM docs were pretty much moved to the new rust docs format in
https://github.com/paritytech/polkadot-sdk/pull/2633, with the addition
of the XCM cookbook, which I plan to add more examples to shortly.

These docs were not mentioned in the polkadot-sdk rust docs, this PR
just mentions them there, so people can actually find them.

Third and final PR in the set, closes
https://github.com/paritytech/polkadot-sdk/issues/454.

Original PR: https://github.com/paritytech/polkadot-sdk/pull/2680

## Precursors:
- https://github.com/paritytech/polkadot-sdk/pull/3889.
- https://github.com/paritytech/polkadot-sdk/pull/3904.

## Follow up issues/improvements
- https://github.com/paritytech/polkadot-sdk/issues/4404

Overall changes are documented here (lot more visual 

):
https://hackmd.io/@ak0n/454-np-governance

## Summary of various roles 🤯
### Pallet Staking
**Nominator**: An account that directly stakes on `pallet-staking` and
nominates a set of validators.
**Stakers**: Common term for nominators and validators.
Virtual Stakers: Same as stakers, but they are keyless accounts and
their locks are managed by a pallet external to `pallet-staking`.

### Pallet Delegated Staking
**Agent**: An account that receives delegation from other accounts
(delegators) and stakes on their behalf. They are also Virtual Stakers
in `pallet-staking` where `pallet-delegated-staking` manages its locks.
**Delegator**: An account that delegates some funds to an agent.

### Pallet Nomination Pools
**Pool account**: Keyless account of a pool where funds are pooled.
Members pledge their funds towards the pools. These are going to become
`Agent` accounts in `pallet-delegated-staking`.
**Pool Members**: They are individual members of the pool who
contributed funds to it. They are also `Delegator` in
`pallet-delegated-staking`.

## Changes
### Multiple Stake strategies

**TransferStake**: The current nomination pool logic can be considered a
staking strategy where delegators transfer funds to pool and stake. In
this scenario, funds are locked in pool account, and users lose the
control of their funds.

**DelegateStake**: With this PR, we introduce a new staking strategy
where individual delegators delegate fund to pool. `Delegate` implies
funds are locked in delegator account itself. Important thing to note
is, pool does not have funds of its own, but it has authorization from
its members to use these funds for staking.

We extract out all the interaction of pool with staking interface into a
new trait `StakeStrategy`. This is the logic that varies between the
above two staking strategies. We use the trait `StakeStrategy` to
implement above two strategies: `TransferStake` and `DelegateStake`.

### NominationPool
Consumes an implementation of `StakeStrategy` instead of
`StakingInterface`. I have renamed it from `Staking` to `StakeAdapter`
to clarify the difference from the earlier used trait.

To enable delegation based staking in pool, Nomination pool can be
configured as:
```
type StakeAdapter = pallet_nomination_pools::adapter::DelegateStake<Self, DelegatedStaking>;
```

Note that with the following configuration, the changes in the PR are
no-op.
```
type StakeAdapter = pallet_nomination_pools::adapter::TransferStake<Self, Staking>;
```

## Deployment roadmap
Plan to enable this only in Westend. In production runtimes, we can keep
pool to use `TransferStake` which will be no functional change.

Once we have a full audit, we can enable this in Kusama followed by
Polkadot.

## TODO
- [x] Runtime level (Westend) migration for existing nomination pools.
- [x] Permissionless call/ pallet::tasks for claiming delegator funds.
- [x] Add/update benches.
- [x] Migration tests.
- [x] Storage flag to mark `DelegateStake` migration and integrity
checks to not allow `TransferStake` for migrated runtimes.

---------

Signed-off-by: Matteo Muraca <mmuraca247@gmail.com>
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Adrian Catangiu <adrian@parity.io>
Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Signed-off-by: divdeploy <chenguangxue@outlook.com>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: hongkuang <liurenhong@outlook.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: gemini132 <164285545+gemini132@users.noreply.github.com>
Co-authored-by: Matteo Muraca <56828990+muraca@users.noreply.github.com>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Alexandru Gheorghe <49718502+alexggh@users.noreply.github.com>
Co-authored-by: Alessandro Siniscalchi <asiniscalchi@gmail.com>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
Co-authored-by: Ross Bulat <ross@parity.io>
Co-authored-by: Serban Iorga <serban@parity.io>
Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com>
Co-authored-by: Sam Johnson <sam@durosoft.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: Javier Viola <363911+pepoviola@users.noreply.github.com>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
Co-authored-by: Dastan <88332432+dastansam@users.noreply.github.com>
Co-authored-by: Clara van Staden <claravanstaden64@gmail.com>
Co-authored-by: Ron <yrong1997@gmail.com>
Co-authored-by: Vincent Geddes <vincent@snowfork.com>
Co-authored-by: Svyatoslav Nikolsky <svyatonik@gmail.com>
Co-authored-by: Michal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
Co-authored-by: Dino Pačandi <3002868+Dinonard@users.noreply.github.com>
Co-authored-by: Andrei Eres <eresav@me.com>
Co-authored-by: Alin Dima <alin@parity.io>
Co-authored-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>
Co-authored-by: gupnik <nikhilgupta.iitk@gmail.com>
Co-authored-by: Vladimir Istyufeev <vladimir@parity.io>
Co-authored-by: Lulu <morgan@parity.io>
Co-authored-by: Juan Girini <juangirini@gmail.com>
Co-authored-by: Francisco Aguirre <franciscoaguirreperez@gmail.com>
Co-authored-by: Dónal Murray <donal.murray@parity.io>
Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com>
Co-authored-by: Kutsal Kaan Bilgin <kutsalbilgin@gmail.com>
Co-authored-by: Ermal Kaleci <ermalkaleci@gmail.com>
Co-authored-by: ordian <write@reusable.software>
Co-authored-by: divdeploy <166095818+divdeploy@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Sergej Sakac <73715684+Szegoo@users.noreply.github.com>
Co-authored-by: Squirrel <gilescope@gmail.com>
Co-authored-by: HongKuang <166261675+HongKuang@users.noreply.github.com>
Co-authored-by: Tsvetomir Dimitrov <tsvetomir@parity.io>
Co-authored-by: Egor_P <egor@parity.io>
Co-authored-by: Aaro Altonen <48052676+altonen@users.noreply.github.com>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: Léa Narzis <78718413+lean-apple@users.noreply.github.com>
Co-authored-by: Gonçalo Pestana <g6pestana@gmail.com>
Co-authored-by: georgepisaltu <52418509+georgepisaltu@users.noreply.github.com>
Co-authored-by: command-bot <>
Co-authored-by: PG Herveou <pgherveou@gmail.com>
Co-authored-by: jimwfs <wqq1479787@163.com>
Co-authored-by: jimwfs <169986508+jimwfs@users.noreply.github.com>
Co-authored-by: polka.dom <polkadotdom@gmail.com>

This implements the `CheckMetadataHash` extension as described in
[RFC78](https://polkadot-fellows.github.io/RFCs/approved/0078-merkleized-metadata.html).

Besides the signed extension, the `substrate-wasm-builder` is extended
to support generating the metadata-hash.

Closes: https://github.com/paritytech/polkadot-sdk/issues/291

---------

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

This PR introduces custom types / substrate wrappers for `Multiaddr`,
`multiaddr::Protocol`, `Multihash`, `ed25519::*` and supplementary types
like errors and iterators.

This is needed to unblock `libp2p` upgrade PR
https://github.com/paritytech/polkadot-sdk/pull/1631 after
https://github.com/paritytech/polkadot-sdk/pull/2944 was merged.
`libp2p` and `litep2p` currently depend on different versions of
`multiaddr` crate, and introduction of this "common ground" types is
needed to support independent version upgrades of `multiaddr` and
dependent crates in `libp2p` & `litep2p`.

While being just convenient to not tie versions of `libp2p` & `litep2p`
dependencies together, it's currently not even possible to keep `libp2p`
& `litep2p` dependencies updated to the same versions as `multiaddr` in
`libp2p` depends on `libp2p-identity` that we can't include as a
dependency of `litep2p`, which has it's own `PeerId` type. In the
future, to keep things updated on `litep2p` side, we will likely need to
fork `multiaddr` and make it use `litep2p` `PeerId` as a payload of
`/p2p/...` protocol.

With these changes, common code in substrate uses these custom types,
and `litep2p` & `libp2p` backends use corresponding libraries types.

Before relayer crates have been moved + merged, the `MetricsParams` type
has been created from a `substrate-relay` crate (binary) and hence it
has been setting the `substrate_relay_build_info` metic value properly -
to the binary version. Now it is created from the
`substrate-relay-helper` crate, which has the fixed (it isn't published)
version `0.1.0`, so our relay provides incorrect metric value. This
'breaks' our monitoring tools - we see that all relayers have that
incorrect version, which is not cool.

The idea is to have a global static variable (shame on me) that is
initialized by the binary during initialization like we do with the
logger initialization already. Was considering some alternative options:
- adding a separate argument to every relayer subcommand and propagating
it to the `MetricsParams::new()` causes a lot of changes and introduces
even more noise to the binary code, which is supposed to be as small as
possible in the new design. But I could do that if team thinks it is
better;
- adding a `structopt(skip) pub relayer_version: RelayerVersion`
argument to all subcommand params won't work, because it will be
initialized by default and `RelayerVersion` needs to reside in some util
crate (not the binary), so it'll have the wrong value again.

# Context

XCM docs are currently an md book hosted with github pages:
https://paritytech.github.io/xcm-docs/.
While that's fine, it's not in line with the work being done in the
polkadot-sdk docs.

# Main addition

This PR aims to fix that by bringing the docs back to this repo.
This does not have all the information currently present in the mdbook
xcm-docs but aims to be a good chunk of it and fully replace it over
time.

I also added the sections `guides` and `cookbook` which will be very
useful for users wanting to get into XCM.
For now I only added one example to the cookbook, but have ideas for
guides and more examples.
Having this docs be in rust docs is very useful for the cookbook.

# TODO

- [x] Use `FungibleAdapter`
- [x] Improve and relocate mock message queue
- [x] Fix license issue. Why does docs/sdk/ not have this problem? (Just
added the licenses)

# Next steps

- More examples in the cookbook
- End-to-end XCM guide with zombienet testing

---------

Co-authored-by: co...

This bump of versions:

https://github.com/paritytech/polkadot-sdk/pull/4409/files#diff-13ee4b2252c9e516a0547f2891aa2105c3ca71c6d7a1e682c69be97998dfc87eR11936

reintroduced a dependency to proc-macro-crate 2.0.0 which is suffering
from: https://github.com/bkchr/proc-macro-crate/pull/42 this, so bump
parity-scale-codec to a newer version to eliminate the bad
proc-macro-crate 2.0.0 dependency.

---------

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: command-bot <>

This is the second PR in preparation for
https://github.com/paritytech/polkadot-sdk/issues/454.

## Also see
- **Precursor** https://github.com/paritytech/polkadot-sdk/pull/3889.
- **Follow up** https://github.com/paritytech/polkadot-sdk/pull/3905.

Overall changes are documented here (lot more visual 

):
https://hackmd.io/@ak0n/454-np-governance

## Changes
### Delegation Interface
Provides delegation primitives for staking. 

Introduces two new roles:
- Agent: These are accounts who receive delegation from other accounts
(delegators) and stakes on behalf of them. The funds are held in
delegator accounts.
- Delegator: Accounts who delegate their funds to an agent authorising
them to use it for staking.

Supports
- A way for delegators to add or withdraw delegation to an agent.
- A way for an agent to slash a delegator during a slashing event.

### Pallet Delegated Staking
- Implements `DelegationInterface`.
- Lazy slashing: Any slashes to an Agent is posted in a ledger but not
immediately slashed. The agent can call
`DelegationInterface::delegator_slash` to slash the member and clear the
corresponding slash from its ledger.
- Consumes `StakingInterface` to provide `CoreStaking` features. In
reality, this will be `pallet-staking`.
- Ensures bookkeeping for agent and delegator are correct but leaves the
management of reward and slash logic upto the consumer of this pallet.
- While it does not expose any calls yet, it is written with the intent
of exposing these primitives via extrinsics.

## TODO
- [x] Improve unit tests in the pallet.
- [x] Separate slash reward perbill for rewarding the slash reporters?
- [x] Review if we should add more events.

---------

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Gonçalo Pestana <g6pestana@gmail.com>
Co-authored-by: georgepisaltu <52418509+georgepisaltu@users.noreply.github.com>

... to get rid of the macro debug logs like this from build output
```
[/home/alexggh/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fatality-proc-macro-0.1.0/src/types.rs:171:23] input = TokenStream [
    Literal {
        kind: Str,
        symbol: "Error while accessing runtime information",
        suffix: None,
        span: #0 bytes(6943..6986),
    },
]
```

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>

Bumps [nix](https://github.com/nix-rust/nix) from 0.27.1 to 0.28.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/nix-rust/nix/blob/master/CHANGELOG.md">nix's
changelog</a>.</em></p>
<blockquote>
<h2>[0.28.0] - 2024-02-24</h2>
<h3>Added</h3>
<ul>
<li>
<p>Added <code>mkdtemp</code> wrapper (<a
href="https://redirect.github.com/nix-rust/nix/pull/1297">#1297</a>)</p>
</li>
<li>
<p>Add associated constants <code>UTIME_OMIT</code>
<code>UTIME_NOW</code> for <code>TimeSpec</code>
(<a
href="https://redirect.github.com/nix-rust/nix/pull/1879">#1879</a>)</p>
</li>
<li>
<p>Added <code>EventFd</code> type. (<a
href="https://redirect.github.com/nix-rust/nix/pull/1945">#1945</a>)</p>
</li>
<li>
<ul>
<li>Added <code>impl From&lt;Signal&gt; for SigSet</code>.</li>
<li>Added <code>impl std::ops::BitOr for SigSet</code>.</li>
<li>Added <code>impl std::ops::BitOr for Signal</code>.</li>
<li>Added <code>impl std::ops::BitOr&lt;Signal&gt; for
SigSet</code></li>
</ul>
<p>(<a
href="https://redirect.github.com/nix-rust/nix/pull/1959">#1959</a>)</p>
</li>
<li>
<p>Added <code>TlsGetRecordType</code> control message type and
corresponding enum for
linux (<a
href="https://redirect.github.com/nix-rust/nix/pull/2065">#2065</a>)</p>
</li>
<li>
<p>Added <code>Ipv6HopLimit</code> to
<code>::nix::sys::socket::ControlMessage</code> for Linux,
MacOS, FreeBSD, DragonflyBSD, Android, iOS and Haiku.
(<a
href="https://redirect.github.com/nix-rust/nix/pull/2074">#2074</a>)</p>
</li>
<li>
<p>Added <code>Icmp</code> and <code>IcmpV6</code> to
<code>SockProtocol</code>
(<a
href="https://redirect.github.com/nix-rust/nix/pull/2103">#2103</a>)</p>
</li>
<li>
<p>Added rfork support for FreeBSD in <code>unistd</code>
(<a
href="https://redirect.github.com/nix-rust/nix/pull/2121">#2121</a>)</p>
</li>
<li>
<p>Added <code>MapFlags::map_hugetlb_with_size_log2</code> method for
Linux targets
(<a
href="https://redirect.github.com/nix-rust/nix/pull/2125">#2125</a>)</p>
</li>
<li>
<p>Added <code>mmap_anonymous</code> function
(<a
href="https://redirect.github.com/nix-rust/nix/pull/2127">#2127</a>)</p>
</li>
<li>
<p>Added <code>mips32r6</code> and <code>mips64r6</code> support for
signal, ioctl and ptrace
(<a
href="https://redirect.github.com/nix-rust/nix/pull/2138">#2138</a>)</p>
</li>
<li>
<p>Added <code>F_GETPATH</code> FcntlFlags entry on
Apple/NetBSD/DragonflyBSD for
<code>::nix::fcntl</code>. (<a
href="https://redirect.github.com/nix-rust/nix/pull/2142">#2142</a>)</p>
</li>
<li>
<p>Added <code>F_KINFO</code> FcntlFlags entry on FreeBSD for
<code>::nix::fcntl</code>.
(<a
href="https://redirect.github.com/nix-rust/nix/pull/2152">#2152</a>)</p>
</li>
<li>
<p>Added <code>F_GETPATH_NOFIRMLINK</code> and
<code>F_BARRIERFSYNC</code> FcntlFlags entry
on Apple for <code>::nix::fcntl</code>.
(<a
href="https://redirect.github.com/nix-rust/nix/pull/2155">#2155</a>)</p>
</li>
<li>
<p>Added newtype <code>Flock</code> to automatically unlock a held flock
upon drop.
Added <code>Flockable</code> trait to represent valid types for
<code>Flock</code>.
(<a
href="https://redirect.github.com/nix-rust/nix/pull/2170">#2170</a>)</p>
</li>
<li>
<p>Added <code>SetSockOpt</code> impls to enable Linux Kernel TLS on a
TCP socket and to
import TLS parameters. (<a
href="https://redirect.github.com/nix-rust/nix/pull/2175">#2175</a>)</p>
</li>
<li>
<ul>
<li>Added the <code>::nix::sys::socket::SocketTimestamp</code> enum for
configuring the
<code>TsClock</code> (a.k.a <code>SO_TS_CLOCK</code>) sockopt</li>
<li>Added FreeBSD's <code>ScmRealtime</code> and
<code>ScmMonotonic</code> as new options in
<code>::nix::sys::socket::ControlMessageOwned</code></li>
</ul>
<p>(<a
href="https://redirect.github.com/nix-rust/nix/pull/2187">#2187</a>)</p>
</li>
<li>
<p>Added new fanotify API: wrappers for <code>fanotify_init</code> and
<code>fanotify_mark</code>
(<a
href="https://redirect.github.com/nix-rust/nix/pull/2194">#2194</a>)</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/nix-rust/nix/commit/21ab06ef23de214174ddb039be5b5f08750d19e6"><code>21ab06e</code></a>
chore: release 0.8.0</li>
<li><a
href="https://github.com/nix-rust/nix/commit/595c6eb2515723eae1f1db1b9103c95f339380ec"><code>595c6eb</code></a>
chore: changelog for 0.28.0 (<a
href="https://redirect.github.com/nix-rust/nix/issues/2303">#2303</a>)</li>
<li><a
href="https://github.com/nix-rust/nix/commit/79b86a7dff071a2a59858d1c98e202fdf38ebd06"><code>79b86a7</code></a>
mmsg revert signature simplification (<a
href="https://redirect.github.com/nix-rust/nix/issues/2227">#2227</a>)</li>
<li><a
href="https://github.com/nix-rust/nix/commit/08e05a5f66d14d4f06288c9a725e87f473eba41a"><code>08e05a5</code></a>
refactor: remove redundant imports to fix CI (<a
href="https://redirect.github.com/nix-rust/nix/issues/2320">#2320</a>)</li>
<li><a
href="https://github.com/nix-rust/nix/commit/197f55b3ccbce3273bf6ce119d1a8541b5df5d66"><code>197f55b</code></a>
refactor: update nix (<a
href="https://redirect.github.com/nix-rust/nix/issues/2311">#2311</a>)</li>
<li><a
href="https://github.com/nix-rust/nix/commit/f34d00d353a8ebcb16eb8810b00d613e4a294925"><code>f34d00d</code></a>
chore(deps): try libc 0.2.153 (<a
href="https://redirect.github.com/nix-rust/nix/issues/2309">#2309</a>)</li>
<li><a
href="https://github.com/nix-rust/nix/commit/062f6c0f0306743a401138fc2612d59e818d5768"><code>062f6c0</code></a>
test: clean clippy needless_borrow false positive issue (<a
href="https://redirect.github.com/nix-rust/nix/issues/2310">#2310</a>)</li>
<li><a
href="https://github.com/nix-rust/nix/commit/ca173ff30cc42e0fc7a7ba2cf7c1fcbe61dccfa5"><code>ca173ff</code></a>
Fix the mount tests (<a
href="https://redirect.github.com/nix-rust/nix/issues/2269">#2269</a>)</li>
<li><a
href="https://github.com/nix-rust/nix/commit/0dfcf3228489a5060a53e69783a1eea6f43b11a9"><code>0dfcf32</code></a>
test: tweak test_fanotify to monitor ONLY 1 file (<a
href="https://redirect.github.com/nix-rust/nix/issues/2294">#2294</a>)</li>
<li><a
href="https://github.com/nix-rust/nix/commit/0fb346031f6ddc87a57377101a6063133a25e443"><code>0fb3460</code></a>
Temporarily disable CI on GNU hurd (<a
href="https://redirect.github.com/nix-rust/nix/issues/2307">#2307</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/nix-rust/nix/compare/v0.27.1...v0.28.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=nix&package-manager=cargo&previous-version=0.27.1&new-version=0.28.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

This PR bumps `proc-macro-crate` to the latest version.

In order to test a runtime from
https://github.com/polkadot-fellows/runtimes/ with the latest version of
polkadot-sdk one needs to use `cargo vendor` to extract all runtime
dependencies, patch them by hand and then build the runtime.

However at the moment 'vendored' builds fail due to
https://github.com/bkchr/proc-macro-crate/issues/48. To fix this
`proc-macro-crate` should be updated to version `3.0.1` or higher.

---------

Co-authored-by: command-bot <>

Reworks prospective-parachains so that we allow a number of unconnected
candidates (for which we don't know the parent candidate yet). Needed
for elastic scaling:
https://github.com/paritytech/polkadot-sdk/issues/3541. Without this,
candidate B will not be validated and backed until candidate A (its
parent) is validated and a backing statement reaches the validator.

Due to the high complexity of the subsystem, I rewrote parts of it so
that we don't concern ourselves with candidates which form cycles or
which form parachain forks. We now have "Fragment chains" instead of
"Fragment trees". This greatly simplifies some of the code and is a
compromise we can make. We just need to make sure that cycle-producing
parachains don't brick the relay chain and that fork-producing
parachains can still make some progress (on one core at least). The only
forks that are allowed are those on the relay chain, obviously.

Unconnected candidates are kept in the `CandidateStorage` and whenever a
new candidate is introduced, we try to repopulate the chain with as many
candidates as we can.

Also fixes https://github.com/paritytech/polkadot-sdk/issues/3219

Guide changes will be done as part of:
https://github.com/paritytech/polkadot-sdk/issues/3699

TODOs:

- [x] see if we can replace the `Cow` over the candidate commitments
with an `Arc` over the entire `ProspectiveCandidate`. It's only being
overwritten in unit tests. We can work around that.
- [x] finish fragment_chain unit tests
- [x] add more prospective-parachains subsystem tests
- [x] test with zombienet what happens if a parachain is creating cycles
(it should not brick the relay chain).
- [x] test with zombienet a parachain that is creating forks. it should
keep producing blocks from time to time (one bad collator should not DOS
the parachain, even if throughput decreases)
- [x] add some more logs and metrics
- [x] add prdoc and remove the "silent" label

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: Andrei Sandu <andrei-mihail@parity.io>

Extracting the logic for generating and verifying ancestry proofs from
this PR: https://github.com/paritytech/polkadot-sdk/pull/1903 with small
adjustments

@Lederstrumpf I added you as a co-author to each commit. Please let me
know if you want me to also associate an e-mail address with your name

The state-trie migration is completed on Rococo Asset-Hub as
double-checked
[here](https://github.com/paritytech/polkadot-sdk/issues/4174#issuecomment-2097895275).
Undeploying now.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Library `substrate-frame-cli` seems unused.

Last non-dependabot update to the folder was over [two years
ago](https://github.com/paritytech/polkadot-sdk/commits/master/substrate/utils/frame/frame-utilities-cli).

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

This PR adds two new CLI options to disable rate limiting for certain ip
addresses and whether to trust "proxy header".
After going back in forth I decided to use ip addr instead host because
we don't want rely on the host header which can be spoofed but another
solution is to resolve the ip addr from the socket to host name.

Example:

```bash
$ polkadot --rpc-rate-limit 10 --rpc-rate-limit-whitelisted-ips 127.0.0.1/8 --rpc-rate-limit-trust-proxy-headers
```

The ip addr is read from the HTTP proxy headers `Forwarded`,
`X-Forwarded-For` `X-Real-IP` if `--rpc-rate-limit-trust-proxy-headers`
is enabled if that is not enabled or the headers are not found then the
ip address is read from the socket.

//cc @BulatSaif can you test this and give some feedback on it?

This checks changed files against API surface changes against what the
prdoc says.

It will error if the detected semver change is greater than the one
listed in the prdoc. It will also error if any crates were touched but
not mentioned in the prdoc.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

# Context

Estimating fees for XCM execution and sending has been an area with bad
UX.
The addition of the
[XcmPaymentApi](https://github.com/paritytech/polkadot-sdk/pull/3607)
exposed the necessary components to be able to estimate XCM fees
correctly, however, that was not the full story.
The `XcmPaymentApi` works for estimating fees only if you know the
specific XCM you want to execute or send.
This is necessary but most UIs want to estimate the fees for extrinsics,
they don't necessarily know the XCM program that's executed by them.

# Main addition

A new runtime API is introduced, the `XcmDryRunApi`, that given an
extrinsic, or an XCM program, returns its effects:
- Execution result
- Local XCM (in the case of an extrinsic)
- Forwarded XCMs
- List of events

This API can be used on its own for dry-running purposes, for
double-checking or testing, but it mainly shines when used in
conjunction with the `XcmPaymentApi`.
UIs can use these two APIs to estimate transfers.

# How it works

New tests are added to exemplify how to incorporate both APIs.
There's a mock test just to make sure everything works under
`xcm-fee-payment-runtime-api`.
There's a real-world test using Westend and AssetHubWestend under
`cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs`.
Added both a test for a simple teleport between chains and a reserve
transfer asset between two parachains going through a reserve.

The steps to follow:
- Use `XcmDryRunApi::dry_run_extrinsic` to get local XCM program and
forwarded messages
- For each forwarded message
- Use `XcmPaymentApi::query_delivery_fee` LOCALLY to get the delivery
fees
- Use `XcmPaymentApi::query_xcm_weight` ON THE DESTINATION to get the
remote execution weight
- (optional) Use `XcmPaymentApi::query_acceptable_payment_assets` ON THE
DESTINATION to know on which assets the execution fees can be paid
- Use `XcmPaymentApi::query_weight_to_asset_fee` ON THE DESTINATION to
convert weight to the actual remote execution fees
- Use `XcmDryRunApi::dry_run_xcm` ON THE DESTINATION to know if a new
message will be forwarded, if so, continue

# Dear reviewer

The changes in this PR are grouped as follows, and in order of
importance:
- Addition of new runtime API
- Definition, mock and simple tests:
polkadot/xcm/xcm-fee-payment-runtime-api/*
- Implemented on Westend, Asset Hub Westend and Penpal, will implement
on every runtime in a following PR
- Addition of a new config item to the XCM executor for recording xcms
about to be executed
  - Definition: polkadot/xcm/xcm-executor/*
  - Implementation: polkadot/xcm/pallet-xcm/*
- had to update all runtime xcm_config.rs files with `type XcmRecorder =
XcmPallet;`
- Addition of a new trait for inspecting the messages in queues
  - Definition: polkadot/xcm/xcm-builder/src/routing.rs
  - Implemented it on all routers:
    - ChildParachainRouter: polkadot/runtime/common/src/xcm_sender.rs
- ParentAsUmp: cumulus/primitives/utility/src/lib.rs (piggybacked on
implementation in cumulus/pallets/parachain-system/src/lib.rs)
    - XcmpQueue: cumulus/pallets/xcmp-queue/src/lib.rs
    - Bridge: bridges/modules/xcm-bridge-hub-router/src/lib.rs
- More complicated and useful tests:
-
cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs

## Next steps

With this PR, Westend, AssetHubWestend, Rococo and AssetHubRococo have
the new API.
UIs can test on these runtimes to create better experiences around
cross-chain operations.

Next:
- Add XcmDryRunApi to all system parachains
- Integrate xcm fee estimation in all emulated tests
- Get this on the fellowship runtimes

---------

Co-authored-by: Adrian Catangiu <adrian@parity.io>

Split from #4374

This PR helps to reduce dependencies and align versions, which would
help to move them to workspace dep