- Send bridged WNDs: Penpal Rococo -> AH Rococo -> AH Westend
- Send bridged ROCs: Penpal Westend -> AH Westend -> AH Rococo

The tests send both ROCs and WNDs, for each direction the native asset
is only used to pay for the transport fees on the local AssetHub, and
are not sent over the bridge.

Including the native asset won't be necessary anymore once we get #4375.

---------

Signed-off-by: Adrian Catangiu <adrian@parity.io>
Co-authored-by: command-bot <>

This adds the new `SignedExtension` to Coretime Rococo and Coretime
Westend runtimes.

---------

Co-authored-by: Dónal Murray <donal.murray@parity.io>

On Westend Asset Hub, we allow Rococo Asset Hub to act as reserve for
any asset native to the Rococo or Ethereum ecosystems (practically
providing Westend access to Ethereum assets through double bridging:
W<>R<>Eth).

On Rococo Asset Hub, we allow Westend Asset Hub to act as reserve for
any asset native to the Westend ecosystem. We also allow Ethereum
contracts to act as reserves for the foreign assets identified by the
same respective contracts locations.

- [x] add emulated tests for various assets (native, trust-based,
foreign/bridged) going AHR -> AHW,
- [x] add equivalent tests for the other direction AHW -> AHR.

This PR is a prerequisite to doing the same for Polkadot<>Kusama bridge.

Partly fixes
https://github.com/paritytech/polkadot-sdk/pull/4890#discussion_r1655548633

Still the offchain API needs to be updated to hyper v1.0 and I opened an
issue for it, it's using low-level http body features that have been
removed

Co-authored-by: gupnik <mail.guptanikhil@gmail.com>

Introduce an optional auto-increment setup for the IDs of new assets.

---------

Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>

Enables the `request_revenue` and `notify_revenue` parts of [RFC 5 -
Coretime
Interface](https://polkadot-fellows.github.io/RFCs/approved/0005-coretime-interface.html)

TODO:
- [x] Finish first pass at implementation
- [x] ~~Need to explicitly burn uncollected and dropped revenue~~
Accumulate it instead
- [x] Confirm working on zombienet
- [x] Tests 
- [ ] Enable XCM `request_revenue` sending on Coretime chain on Kusama
and Polkadot

Fixes: #2209

---------

Co-authored-by: Dmitry Sinyavin <dmitry.sinyavin@parity.io>
Co-authored-by: command-bot <>
Co-authored-by: s0me0ne-unkn0wn <48632512+s0me0ne-unkn0wn@users.noreply.github.com>
Co-authored-by: Dónal Murray <donal.murray@parity.io>
Co-authored-by: Bastian Köcher <git@kchr.de>

This PR upgrades `litep2p` to the latest version and includes the two
fixes:

1. Enables incoming DHT record validation with `litep2p` network
backend.
2. Sets `TCP_NODELAY` flag on TCP & WS sockets in `litep2p` backend, as
it is currently done in `libp2p` backend.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Closes: https://github.com/paritytech/polkadot-sdk/issues/4298

This PR also merges `xcm-fee-payment-runtime-api` module to the
`xcm-runtime-api`.


## TODO

- [x] rename `convert` to `convert_location` and add new one
`convert_account` (opposite direction)
- [x] add to the all testnet runtimes
- [x] check polkadot-js if supports that automatically or if needs to be
added manually https://github.com/polkadot-js/api/pull/5917
- [ ] backport/patch for fellows and release (asap)

## Open questions
- [x] should we merge `xcm-runtime-api` and
`xcm-fee-payment-runtime-api` to the one module `xcm-runtime-api` ?

## Usage
Input:
 - `location:  VersionedLocation`
 
Output:
 - account_id bytes


![image](https://github.com/paritytech/polkadot-sdk/assets/8159517/4607b15a-77d2-462b-806c-606107776c0d)

---------

Co-authored-by: Bastian Köcher <git@kchr.de>

This is PR updates jsonrpsee v0.23 which mainly changes:
- Add `Extensions` which we now is using to get the connection id (used
by the rpc spec v2 impl)
- Update hyper to v1.0, http v1.0, soketto and related crates
(hyper::service::make_service_fn is removed)
- The subscription API for the client is modified to know why a
subscription was closed.

Full changelog here:
https://github.com/paritytech/jsonrpsee/releases/tag/v0.23.0

---------

Co-authored-by: Bastian Köcher <git@kchr.de>

A few refactorings to block import and block verification that should
not be controversial.

Block verification before block import is stateless by design as
described in https://substrate.stackexchange.com/a/1322/25 and the fact
that it wasn't yet I consider to be a bug. Some code that requires it
had to use `Mutex`, but I do not expect it to have a measurable
performance impact.

Similarly with block import checking whether block preconditions should
not be an exclusive operation, there is nothing fundamentally wrong with
checking a few competing blocks whose parent blocks exist at the same
time (and even import them concurrently later, though IIRC this is not
yet implemented either).

They were originally a part of
https://github.com/paritytech/polkadot-sdk/pull/4842 and upstreaming
will help us to reduce the size of the patch we need to apply on top of
upstream code at Subspace every time we upgrade. There are no new
features introduced here, just refactoring to get rid of unnecessary
requirements.

Upgrade libp2p to 0.52.4, including a fix: 

* Set Kademlia to server mode
(https://github.com/paritytech/substrate/pull/14703)

### TODO
- [x] Fix 3 zombienet tests failing:
  - [x] `zombienet-substrate-0002-validators-warp-sync`
- [ ]
~`zombienet-polkadot-functional-0005-parachains-disputes-past-session`~
The test is also flaky in other PRs and is not required for CI to
succeed.
  - [x] `zombienet-polkadot-functional-0009-approval-voting-coalescing`
- [x] Uncomment and update to the actual libp2p API tests in
[`substrate/client/network/src/protocol/notifications/handler.rs`](https://github.com/paritytech/polkadot-sdk/blob/7331f179/substrate/client/network/src/protocol/notifications/handler.rs#L1009).
- [x] When upgrading `multihash` crate as part of libp2p upgrade to
version v0.19.1, uncomment the conversion code at
https://github.com/paritytech/polkadot-sdk/blob/7547c494

/substrate/client/network/types/src/multihash.rs#L159
- [x] Perform a burn-in.

---------

Co-authored-by: Anton <anton.kalyaev@gmail.com>
Co-authored-by: command-bot <>
Co-authored-by: Dmitry Markin <dmitry@markin.tech>
Co-authored-by: Bastian Köcher <git@kchr.de>

Some primitives have impl Hex related traits enabled by `rustc-hex`
feature. People wanna use H256/H160 maybe need these trait impls

---------

Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <git@kchr.de>

After preparing in https://github.com/paritytech/polkadot-sdk/pull/4633,
we can lift also all internal dependencies up to the workspace.

This does not actually change anything, but uses `workspace = true` for
all dependencies. You can check it with:
```bash
git checkout -q $(git merge-base oty-lift-all-deps origin/master)
cargo tree -e features > master.out

git checkout -q oty-lift-all-deps
cargo tree -e features > new.out
diff master.out new.out
```

It did not yet lift 100% of dependencies, some inside of `target.*` or
some that had conflicting aliases introduced recently. But i will do
these together in a follow-up with CI checks.

Can be reproduced with [zepter](https://github.com/ggwpez/zepter/):
`zepter transpose d lift-to-workspace "regex:.*" --version-resolver
highest --skip-package "polkadot-sdk" --ignore-errors --fix`.

---------

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

I carried these things in a fork for a long time, I think wouldn't hurt
to have it upstream.

Originally submitted as part of
https://github.com/paritytech/polkadot-sdk/pull/1598 that went nowhere.

---------

Co-authored-by: Bastian Köcher <git@kchr.de>

Closes: https://github.com/paritytech/polkadot-sdk/issues/4748

Closes #3342

cc/ @liamaharon



TODO:

- [x] Improve docs.
- [x] Define public interface (See #3342).
  In case we define public calls to the pallet implementation:
  - Implement public calls.
  - Benchmarks.
  
polkadot address: 12gMhxHw8QjEwLQvnqsmMVY1z5gFa54vND74aMUbhhwN6mJR

---------

Co-authored-by: command-bot <>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>

Bumps
[curve25519-dalek](https://github.com/dalek-cryptography/curve25519-dalek)
from 4.1.2 to 4.1.3.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/dalek-cryptography/curve25519-dalek/commit/5312a0311ec40df95be953eacfa8a11b9a34bc54"><code>5312a03</code></a>
curve: Bump version to 4.1.3 (<a
href="https://redirect.github.com/dalek-cryptography/curve25519-dalek/issues/660">#660</a>)</li>
<li><a
href="https://github.com/dalek-cryptography/curve25519-dalek/commit/b4f9e4df92a4689fb59e312a21f940ba06ba7013"><code>b4f9e4d</code></a>
SECURITY: fix timing variability in backend/serial/u32/scalar.rs (<a
href="https://redirect.github.com/dalek-cryptography/curve25519-dalek/issues/661">#661</a>)</li>
<li><a
href="https://github.com/dalek-cryptography/curve25519-dalek/commit/415892acf1cdf9161bd6a4c99bc2f4cb8fae5e6a"><code>415892a</code></a>
SECURITY: fix timing variability in backend/serial/u64/scalar.rs (<a
href="https://redirect.github.com/dalek-cryptography/curve25519-dalek/issues/659">#659</a>)</li>
<li><a
href="https://github.com/dalek-cryptography/curve25519-dalek/commit/56bf398d0caed63ef1d1edfbd35eb5335132aba2"><code>56bf398</code></a>
Updates license field to valid SPDX format (<a
href="https://redirect.github.com/dalek-cryptography/curve25519-dalek/issues/647">#647</a>)</li>
<li><a
href="https://github.com/dalek-cryptography/curve25519-dalek/commit/9252fa5c0d09054fed4ac4d649e63c40fad7abaf"><code>9252fa5</code></a>
Mitigate check-cfg until MSRV 1.77 (<a
href="https://redirect.github.com/dalek-cryptography/curve25519-dalek/issues/652">#652</a>)</li>
<li><a
href="https://github.com/dalek-cryptography/curve25519-dalek/commit/1efe6a93b176c4389b78e81e52b2cf85d728aac6"><code>1efe6a9</code></a>
Fix a minor typo in signing.rs (<a
href="https://redirect.github.com/dalek-cryptography/curve25519-dalek/issues/649">#649</a>)</li>
<li><a
href="https://github.com/dalek-cryptography/curve25519-dalek/commit/cc3421a22fa7ee1f557cbe9243b450da53bbe962"><code>cc3421a</code></a>
Indicate that the rand_core feature is required (<a
href="https://redirect.github.com/dalek-cryptography/curve25519-dalek/issues/641">#641</a>)</li>
<li><a
href="https://github.com/dalek-cryptography/curve25519-dalek/commit/858c4ca8ae03d33fe8b71b4504c4d3f5ff5b45c0"><code>858c4ca</code></a>
Address new nightly clippy unnecessary qualifications (<a
href="https://redirect.github.com/dalek-cryptography/curve25519-dalek/issues/639">#639</a>)</li>
<li><a
href="https://github.com/dalek-cryptography/curve25519-dalek/commit/31ccb6705067d68782cb135e23c79b640a6a06ee"><code>31ccb67</code></a>
Remove platforms in favor using CARGO_CFG_TARGET_POINTER_WIDTH (<a
href="https://redirect.github.com/dalek-cryptography/curve25519-dalek/issues/636">#636</a>)</li>
<li><a
href="https://github.com/dalek-cryptography/curve25519-dalek/commit/19c7f4a5d5e577adc9cc65a837abef9ed7ebf0a4"><code>19c7f4a</code></a>
Fix new nightly redundant import lint warns (<a
href="https://redirect.github.com/dalek-cryptography/curve25519-dalek/issues/638">#638</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/dalek-cryptography/curve25519-dalek/compare/curve25519-4.1.2...curve25519-4.1.3">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=curve25519-dalek&package-manager=cargo&previous-version=4.1.2&new-version=4.1.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/paritytech/polkadot-sdk/network/alerts).

</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Previous attempt https://github.com/paritytech/substrate/pull/14314

Close #4443 

Ideally, we should move /health and /health/readiness to the prometheus
server but because it's was quite easy to implement on the RPC server
and that RPC server already exposes /health.

Manual tests on a polkadot node syncing:

```bash
➜ polkadot-sdk (na-fix-4443) ✗ curl -v localhost:9944/health
* Host localhost:9944 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:9944...
* connect to ::1 port 9944 from ::1 port 55024 failed: Connection refused
*   Trying 127.0.0.1:9944...
* Connected to localhost (127.0.0.1) port 9944
> GET /health HTTP/1.1
> Host: localhost:9944
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json; charset=utf-8
< content-length: 53
< date: Fri, 14 Jun 2024 16:12:23 GMT
<
* Connection #0 to host localhost left intact
{"peers":0,"isSyncing":false,"shouldHavePeers":false}%
➜ polkadot-sdk (na-fix-4443) ✗ curl -v localhost:9944/health/readiness
* Host localhost:9944 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:9944...
* connect to ::1 port 9944 from ::1 port 54328 failed: Connection refused
*   Trying 127.0.0.1:9944...
* Connected to localhost (127.0.0.1) port 9944
> GET /health/readiness HTTP/1.1
> Host: localhost:9944
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< content-type: application/json; charset=utf-8
< content-length: 0
< date: Fri, 14 Jun 2024 16:12:36 GMT
<
* Connection #0 to host localhost left intact
```

//cc @BulatSaif

 you may be interested in this..

---------

Co-authored-by: Bastian Köcher <git@kchr.de>

Implements most of
https://github.com/paritytech/polkadot-sdk/issues/1797

Core sharing (two parachains or more marachains scheduled on the same
core with the same `PartsOf57600` value) was not working correctly. The
expected behaviour is to have Backed and Included event in each block
for the paras sharing the core and the paras should take turns. E.g. for
two cores we expect: Backed(a); Included(a)+Backed(b);
Included(b)+Backed(a); etc. Instead of this each block contains just one
event and there are a lot of gaps (blocks w/o events) during the
session.

Core sharing should also work when collators are building collations
ahead of time

TODOs:

- [x] Add a zombienet test verifying that the behaviour mentioned above
works.
- [x] prdoc

---------

Co-authored-by: alindima <alin@parity.io>

Glutton currently is useful mostly for stress testing relay chain
validators. It is unusable for testing the collator networking and block
announcement and import scenarios. This PR resolves that by improving
glutton pallet to also buff up the blocks, up to the runtime configured
`BlockLength`.

### How it works
Includes an additional inherent in each parachain block. The `garbage`
argument passed to the inherent is filled with trash data. It's size is
computed by applying the newly introduced `block_length` percentage to
the maximum block length for mandatory dispatch class. After
https://github.com/paritytech/polkadot-sdk/pull/4765 is merged, the
length of inherent extrinsic will be added to the total block proof
size.

The remaining weight is burnt in `on_idle` as configured by the
`storage` percentage parameter.


TODO:
- [x] PRDoc
- [x] Readme update
- [x] Add tests

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>

Contains mainly changes/nits/refactors related to the relayer code
(`client-substrate` and `lib-substrate-relay`) migrated from the Bridges
V2 [branch](https://github.com/paritytech/polkadot-sdk/pull/4427).

Relates to:
https://github.com/paritytech/parity-bridges-common/issues/2976
Companion: https://github.com/paritytech/parity-bridges-common/pull/2988


## TODO
- [x] fix comments

## Questions
- [x] Do we need more testing for client V2 stuff? If so, how/what is
the ultimate test? @svyatonik


- [x] check
[comment](https://github.com/paritytech/polkadot-sdk/pull/4494#issuecomment-2117181144)
for more testing

---------

Co-authored-by: Svyatoslav Nikolsky <svyatonik@gmail.com>
Co-authored-by: Serban Iorga <serban@parity.io>

After using this tutorial in PBA, there was a few areas to improve it.
Moreover, I have:

- Improve `your_first_pallet`, link it in README, improve the parent
`guide` section.
- Updated the templates page, in light of recent efforts related to in
https://github.com/paritytech/polkadot-sdk/issues/3155
- Added small ref docs about metadata, completed the one about native
runtime, added one about host functions.
- Remove a lot of unfinished stuff from sdk-docs
- update diagram for `Hooks`

Addressing comment:
https://github.com/paritytech/polkadot-sdk/pull/4102/files#r1635502496

---------

Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

This branch propose to avoid clones in append by storing offset and size
in previous overlay depth.
That way on rollback we can just truncate and change size of existing
value.
To avoid copy it also means that :

- append on new overlay layer if there is an existing value: create a
new Append entry with previous offsets, and take memory of previous
overlay value.
- rollback on append: restore value by applying offsets and put it back
in previous overlay value
- commit on append: appended value overwrite previous value (is an empty
vec as the memory was taken). offsets of commited layer are dropped, if
there is offset in previous overlay layer they are maintained.
- set value (or remove) when append offsets are present: current
appended value is moved back to previous overlay value with offset
applied and current empty entry is overwrite (no offsets kept).

The modify mechanism is not needed anymore.
This branch lacks testing and break some existing genericity (bit of
duplicated code), but good to have to check direction.

Generally I am not sure if it is worth or we just should favor
differents directions (transients blob storage for instance), as the
current append mechanism is a bit tricky (having a variable length in
first position means we sometime need to insert in front of a vector).

Fix #30.

---------

Signed-off-by: Alexandru Vasile <alexandru.vasile@parity.io>
Co-authored-by: EgorPopelyaev <egor@parity.io>
Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com>
Co-authored-by: Liam Aharon <liam.aharon@hotmail.com>
Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Bastian Köcher <info@kchr.de>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

- unit tests for pov-recovery
- elastic scaling support (recovering multiple candidates in a single
relay chain block)
- also some small cleanups
- also switches to candidates_pending_availability in
`handle_empty_block_announce_data`

Fixes https://github.com/paritytech/polkadot-sdk/issues/3577

After https://github.com/paritytech/polkadot-sdk/pull/4097 is merged, we
should also add a zombienet test, similar to the existing
`0002-pov_recovery.toml` but which has a single collator using elastic
scaling on multiple cores.

take over #2941 
[Weights
compare](https://weights.tasty.limo/compare?unit=weight&ignore_errors=true&threshold=10&method=asymptotic&repo=polkadot-sdk&old=master&new=pg%2Fwasmi-to-v0.32.0-beta.7&path_pattern=substrate%2Fframe%2F**%2Fsrc%2Fweights.rs%2Cpolkadot%2Fruntime%2F*%2Fsrc%2Fweights%2F**%2F*.rs%2Cpolkadot%2Fbridges%2Fmodules%2F*%2Fsrc%2Fweights.rs%2Ccumulus%2F**%2Fweights%2F*.rs%2Ccumulus%2F**%2Fweights%2Fxcm%2F*.rs%2Ccumulus%2F**%2Fsrc%2Fweights.rs)

---------

Co-authored-by: command-bot <>
Co-authored-by: Alexander Theißen <alex.theissen@me.com>

Thanks for @xlc for the original seed info, I've just fixed it up a bit
and added example links.

I've moved the comparison between eth-rpc-api and frontier outside, as
it is opinionation. I think the content there was good but should live
in the README of the corresponding repos. No strong opinion, happy
either way.

---------

Co-authored-by: Bryan Chen <xlchen1291@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
Co-authored-by: Gonçalo Pestana <g6pestana@gmail.com>
Co-authored-by: command-bot <>

`secp256k1 v0.28.0` and `secp256k1-sys v0.9.0` were yanked because
building them fails for `aarch64-apple-darwin` targets.

Use the `secp256k1 v0.28.2` and `secp256k1-sys v0.9.2` patched versions
that build fine on ARM chipset MacOS.

In order to receive a username in `pallet-identity`, users have to,
among other things, provide a signature of the desired username. Right
now, there is an [extra encoding
step](https://github.com/paritytech/polkadot-sdk/blob/4ab078d6

/substrate/frame/identity/src/lib.rs#L1119)
when generating the payload to sign.

Encoding a `Vec` adds extra bytes related to the length, which changes
the payload. This is unnecessary and confusing as users expect the
payload to sign to be just the username bytes. This PR fixes this issue
by validating the signature directly against the username bytes.

---------

Signed-off-by: georgepisaltu <george.pisaltu@parity.io>

Added reference doc about:
- the pallet genesis config and genesis build, 
- runtime `genesis-builder` API,
- presets,
- interacting with the `chain-spec-builder` tool

I've added [minimal
runtime](https://github.com/paritytech/polkadot-sdk/tree/mku-chain-spec-guide/docs/sdk/src/reference_docs/chain_spec_runtime)
to demonstrate above topics.

I also sneaked in some little improvement to `chain-spec-builder` which
allows to parse output of the `list-presets` command.

---------

Co-authored-by: Alexandru Vasile <60601340+lexnv@users.noreply.github.com>
Co-authored-by: Sebastian Kunert <skunert49@gmail.com>

This allows to use custom para_ids with cumulus-test-runtime. 

Zombienet is patching the genesis entries for `ParachainInfo`. This did
not work with `test-parachain` because it was using the `test_pallet`
for historic reasons I guess.

Depends on https://github.com/paritytech/polkadot-sdk/pull/4621.

Implemented the
[`XcmPaymentApi`](https://github.com/paritytech/polkadot-sdk/pull/3607)
and [`DryRunApi`](https://github.com/paritytech/polkadot-sdk/pull/3872)
on all system parachains.

More scenarios can be tested on both rococo and westend if all system
parachains implement this APIs.
The objective is for all XCM-enabled runtimes to implement them.
After demonstrating fee estimation in a UI on the testnets, come the
fellowship runtimes.

Step towards https://github.com/paritytech/polkadot-sdk/issues/690.

Currently, `chain-spec-builder` only creates a spec with `Live` chain
type. This PR adds the ability to specify it while keeping the same
default.

---------

Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>

Revived version of https://github.com/paritytech/substrate/pull/13311 .
Except Signature is not generic and is dictated by AuthorityId.

---------

Co-authored-by: Davide Galassi <davxy@datawok.net>
Co-authored-by: Robert Hambrock <roberthambrock@gmail.com>
Co-authored-by: Adrian Catangiu <adrian@parity.io>

Fixes #4360 

Also rename: AllowedRenewals -> PotentialRenewals to avoid confusion of
future readers. (An entry in `AllowedRenewals` is not enough to allow a
renewal, the assignment also has to be complete, which is only checked
afterwards.)

- [x] Does not work with renewals as is - fix.
- [x] More tests
- [x] PR docs

Edit 1:
(Relevant blog post:
https://grillapp.net/12935/agile-coretime-pricing-explained-166522?ref=29715)

---------

Co-authored-by: eskimor <eskimor@no-such-url.com>
Co-authored-by: Dónal Murray <donal.murray@parity.io>
Co-authored-by: command-bot <>

marking it as release-able, attaching the same version number that is
attached to other binaries such as `polkadot` and `polkadot-parachain`.

I have more thoughts about the version number, though. The chain-spec
builder is mainly a user of the `sp-genesis-builder` api. So the
versioning should be such that it helps users know give a version of
`sp-genesis-builder` in their runtime, which version of
`chain-spec-builder` should they use?

With this, we can possibly alter the version number to always match
`sp-genesis-builder`.

Fixes https://github.com/paritytech/polkadot-sdk/issues/4352

- [x] Add to release artifacts ~~similar to
https://github.com/paritytech/polkadot-sdk/pull/4405~~ done here:
https://github.com/paritytech/polkadot-sdk/pull/4557

---------

Co-authored-by: Branislav Kontur <bkontur@gmail.com>

**Don't look at the commit history, it's confusing, as this branch is
based on another branch that was merged**

Fixes #598 
Also implements [RFC
#47](https://github.com/polkadot-fellows/RFCs/pull/47)

## Description

- Availability-recovery now first attempts to request the systematic
chunks for large POVs (which are the first ~n/3 chunks, which can
recover the full data without doing the costly reed-solomon decoding
process). This has a fallback of recovering from all chunks, if for some
reason the process fails. Additionally, backers are also used as a
backup for requesting the systematic chunks if the assigned validator is
not offering the chunk (each backer is only used for one systematic
chunk, to not overload them).
- Quite obviously, recovering from systematic chunks is much faster than
recovering from regular chunks (4000% faster as measured on my apple M2
Pro).
- Introduces a `ValidatorIndex` -> `ChunkIndex` mapping which is
different for every core, in order to avoid only querying the first n/3
validators over and over again in the same session. The mapping is the
one described in RFC 47.
- The mapping is feature-gated by the [NodeFeatures runtime
API](https://github.com/paritytech/polkadot-sdk/pull/2177) so that it
can only be enabled via a governance call once a sufficient majority of
validators have upgraded their client. If the feature is not enabled,
the mapping will be the identity mapping and backwards-compatibility
will be preserved.
- Adds a new chunk request protocol version (v2), which adds the
ChunkIndex to the response. This may or may not be checked against the
expected chunk index. For av-distribution and systematic recovery, this
will be checked, but for regular recovery, no. This is backwards
compatible. First, a v2 request is attempted. If that fails during
protocol negotiation, v1 is used.
- Systematic recovery is only attempted during approval-voting, where we
have easy access to the core_index. For disputes and collator
pov_recovery, regular chunk requests are used, just as before.

## Performance results

Some results from subsystem-bench:

with regular chunk recovery: CPU usage per block 39.82s
with recovery from backers: CPU usage per block 16.03s
with systematic recovery: CPU usage per block 19.07s

End-to-end results here:
https://github.com/paritytech/polkadot-sdk/issues/598#issuecomment-1792007099

#### TODO:

- [x] [RFC #47](https://github.com/polkadot-fellows/RFCs/pull/47)
- [x] merge https://github.com/paritytech/polkadot-sdk/pull/2177 and
rebase on top of those changes
- [x] merge https://github.com/paritytech/polkadot-sdk/pull/2771 and
rebase
- [x] add tests
- [x] preliminary performance measure on Versi: see
https://github.com/paritytech/polkadot-sdk/issues/598#issuecomment-1792007099
- [x] Rewrite the implementer's guide documentation
- [x] https://github.com/paritytech/polkadot-sdk/pull/3065 
- [x] https://github.com/paritytech/zombienet/issues/1705 and fix
zombienet tests
- [x] security audit
- [x] final versi test and performance measure

---------

Signed-off-by: alindima <alin@parity.io>
Co-authored-by: Javier Viola <javier@parity.io>