Since the fix for https://github.com/paritytech/polkadot-sdk/issues/4559
was to add a codeSubstitute to the rococo-parachains, we should update
the chain-specs in the repo.

The json diffs are not super easy to review, so if you want to manually
check:
- Run for example `jd -o people-rococo.diff
people-rococo-from-this-branch.json people-rococo-master.json`
- Check in the resulting diff file that only `codeSubstitutes` field is
changed

Signed-off-by: Alexandru Gheorghe <[email protected]>

This PR implements the `READ_ONLY` flag to be used as a `Callflag` in
the `call` function.
The flag indicates that the callee is restricted from modifying the
state during call execution.
It is equivalent to Ethereum's
[STATICCALL](https://eips.ethereum.org/EIPS/eip-214

).

---------

Co-authored-by: command-bot <>
Co-authored-by: Andrew Jones <[email protected]>
Co-authored-by: Alexander Theißen <[email protected]>

Co-authored-by: Oliver Tale-Yazdi <[email protected]>

`[check-runtime-migration` now takes more than 30 minutes. Quick fix
with increased timeout.

As requested
[here](https://github.com/paritytech/polkadot-sdk/issues/3155#issuecomment-2126934939).

The Dockerfiles are inspired by [this
one](https://github.com/paritytech/polkadot-sdk/blob/aa32faaebf64426becb2feeede347740eb7a3908/docker/dockerfiles/polkadot/polkadot_builder.Dockerfile).

- Addresses
[this](https://github.com/paritytech/polkadot-sdk/issues/3155#issuecomment-2126934939).
- Technical content got adopted from the existing [solochain
readme](https://github.com/paritytech/polkadot-sdk/tree/master/templates/solochain).
  - Updated some broken links there.
- The docker instructions will work after
https://github.com/paritytech/polkadot-sdk/pull/4637.
- See the [rendered
version](https://github.com/paritytech/polkadot-sdk/blob/rzadp/minimal-template-readme/templates/minimal/README.md

).

---------

Co-authored-by: gupnik <[email protected]>

This PR adds possibility to publish container images for the
`chain-spec-builder` binary on the regular basis.
Related to: https://github.com/paritytech/release-engineering/issues/190

Tested each bootnode with `--reserved-only` `--reserved-nodes`

Kusama
```
polkadot --chain kusama --base-path /tmp/node --reserved-only --reserved-nodes "/dns/kusama.bootnode.amforc.com/tcp/29999/wss/p2p/12D3KooWKvYf6qKaAF8UUDw3KsTwjHLnvkED23yxHbH3npMe8w4G" --no-hardware-benchmarks
polkadot --chain kusama --base-path /tmp/node --reserved-only --reserved-nodes "/dns/kusama.bootnode.amforc.com/tcp/30001/p2p/12D3KooWKvYf6qKaAF8UUDw3KsTwjHLnvkED23yxHbH3npMe8w4G" --no-hardware-benchmarks
```

Asset Hub Kusama
```
./polkadot-parachain --chain asset-hub-kusama --base-path /tmp/node --reserved-only --reserved-nodes "/dns/asset-hub-kusama.bootnode.amforc.com/tcp/29999/wss/p2p/12D3KooWHy1CPndZYphwdVqMb295KPC6LRt17Ae3zNSr7evzeF5a" --no-hardware-benchmarks
./polkadot-parachain --chain asset-hub-kusama --base-path /tmp/node --reserved-only --reserved-nodes "/dns/asset-hub-kusama.bootnode.amforc.com/tcp/30007/p2p/12D3KooWHy1CPndZYphwdVqMb295KPC6LRt17Ae3zNSr7evzeF5a" --no-hardware-benchmarks
```


Bridge Hub Kusama
```
./polkadot-parachain --chain bridge-hub-kusama --base-path /tmp/node --reserved-only --reserved-nodes "/dns/bridge-hub-kusama.bootnode.amforc.com/tcp/29999/wss/p2p/12D3KooWNyTBwRvCz1Ey2SgC1f3MvymhiAyLEa3cL8kU5gFH3V7Z" --no-hardware-benchmarks
./polkadot-parachain --chain bridge-hub-kusama --base-path /tmp/node --reserved-only --reserved-nodes "/dns/bridge-hub-kusama.bootnode.amforc.com/tcp/30010/p2p/12D3KooWNyTBwRvCz1Ey2SgC1f3MvymhiAyLEa3cL8kU5gFH3V7Z" --no-hardware-benchmarks
```

Coretime Kusama
```
./polkadot-parachain --chain coretime-kusama --base-path /tmp/node --reserved-only --reserved-nodes "/dns/coretime-kusama.bootnode.amforc.com/tcp/29999/wss/p2p/12D3KooWPrgxrrumrANp6Bp2SMEwMQHPHDbPzA1HbcrakZrbFi5P" --no-hardware-benchmarks
./polkadot-parachain --chain coretime-kusama --base-path /tmp/node --reserved-only --reserved-nodes "/dns/coretime-kusama.bootnode.amforc.com/tcp/30013/p2p/12D3KooWPrgxrrumrANp6Bp2SMEwMQHPHDbPzA1HbcrakZrbFi5P" --no-hardware-benchmarks
```

People Kusama
```
./polkadot-parachain --chain people-kusama --base-path /tmp/node --reserved-only --reserved-nodes "/dns/people-kusama.bootnode.amforc.com/tcp/29999/wss/p2p/12D3KooWPjzgKZe5jdG6TY4gwcFq8QxyyhqsYbQo6N29pwGePWLA" --no-hardware-benchmarks
./polkadot-parachain --chain people-kusama --base-path /tmp/node --reserved-only --reserved-nodes "/dns/people-kusama.bootnode.amforc.com/tcp/30004/p2p/12D3KooWPjzgKZe5jdG6TY4gwcFq8QxyyhqsYbQo6N29pwGePWLA" --no-hardware-benchmarks
```

People Westend
```
./polkadot-parachain --chain people-westend --base-path /tmp/node --reserved-only --reserved-nodes "/dns/people-westend.bootnode.amforc.com/tcp/29999/wss/p2p/12D3KooWE1btdwDhNpApg8BEe2QwJxdVDtz6a6BRhgTeUh9HMhWs" --no-hardware-benchmarks
./polkadot-parachain --chain people-westend --base-path /tmp/node --reserved-only --reserved-nodes "/dns/people-westend.bootnode.amforc.com/tcp/30016/p2p/12D3KooWE1btdwDhNpApg8BEe2QwJxdVDtz6a6BRhgTeUh9HMhWs" --no-hardware-benchmarks
```

Polkadot
```
polkadot --chain polkadot --base-path /tmp/node --reserved-only --reserved-nodes "/dns/polkadot.bootnode.amforc.com/tcp/29999/wss/p2p/12D3KooWT2HyZx5C6BBeLbCKhYG2SqJYuiu7sLMxGzUcQBko3BMr" --no-hardware-benchmarks
polkadot --chain polkadot --base-path /tmp/node --reserved-only --reserved-nodes "/dns/polkadot.bootnode.amforc.com/tcp/30001/p2p/12D3KooWT2HyZx5C6BBeLbCKhYG2SqJYuiu7sLMxGzUcQBko3BMr" --no-hardware-benchmarks
```

Asset Hub Polkadot
```
./polkadot-parachain --chain asset-hub-polkadot --base-path /tmp/node --reserved-only --reserved-nodes "/dns/asset-hub-polkadot.bootnode.amforc.com/tcp/29999/wss/p2p/12D3KooWDLxPXYnSHjNwq9ibqgxuzRni5VViuGNSjNe3ueqVgqE3" --no-hardware-benchmarks
./polkadot-parachain --chain asset-hub-polkadot --base-path /tmp/node --reserved-only --reserved-nodes "/dns/asset-hub-polkadot.bootnode.amforc.com/tcp/30007/p2p/12D3KooWDLxPXYnSHjNwq9ibqgxuzRni5VViuGNSjNe3ueqVgqE3" --no-hardware-benchmarks
```

Bridge Hub Polkadot
```
./polkadot-parachain --chain bridge-hub-polkadot --base-path /tmp/node --reserved-only --reserved-nodes "/dns/bridge-hub-polkadot.bootnode.amforc.com/tcp/29999/wss/p2p/12D3KooWGT5E56rAHfT5dY1pMLTrpAgV72yfDtD1Y5tPCHaTsifp" --no-hardware-benchmarks
./polkadot-parachain --chain bridge-hub-polkadot --base-path /tmp/node --reserved-only --reserved-nodes "/dns/bridge-hub-polkadot.bootnode.amforc.com/tcp/30010/p2p/12D3KooWGT5E56rAHfT5dY1pMLTrpAgV72yfDtD1Y5tPCHaTsifp" --no-hardware-benchmarks
```

Collectives Polkadot
```
./polkadot-parachain --chain collectives-polkadot --base-path /tmp/node --reserved-only --reserved-nodes "/dns/collectives-polkadot.bootnode.amforc.com/tcp/29999/wss/p2p/12D3KooWL6v6FHMtCP5VsiDbMHLRFiW6YBtv37BarpW3hLqnDski" --no-hardware-benchmarks
./polkadot-parachain --chain collectives-polkadot --base-path /tmp/node --reserved-only --reserved-nodes "/dns/collectives-polkadot.bootnode.amforc.com/tcp/30013/p2p/12D3KooWL6v6FHMtCP5VsiDbMHLRFiW6YBtv37BarpW3hLqnDski" --no-hardware-benchmarks
```

Makes all storage items in parachain-system public so that these can be
used by other pallets &/or runtimes.

---------

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: command-bot <>

## Runtime Apis
Introduces the following runtime apis to facilitate dapps and wallets in
integrating with the `DelegateStake` functionalities of the pools
(related: https://github.com/paritytech/polkadot-sdk/pull/3905

). These
apis are meant to support pool and member migration, as well as lazy
application of pending slashes of pool members.

```rust
fn pool_pending_slash(pool_id: PoolId) -> Balance;
fn member_pending_slash(member: AccountId) -> Balance;
fn pool_needs_delegate_migration(pool_id: PoolId) -> bool;
fn member_needs_delegate_migration(member: AccountId) -> bool;
```

## Refactors
- Introduces newtypes for `Agent`, `Delegator`, `Pool` and
`[Pool]Member`. And refactors `StakeAdapter` and `DelegationInterface`
to accept the above types. This will help make these apis typesafe
against using wrong account type.
- Fixing `DelegationInterface` apis to return optional (instead of
default value if key does not exist).
- Rename struct `Agent` that wraps `AgentLedger` to `AgentOuterLedger`
which is clearer (naming wise) and different from the newtype `Agent`.
- Cleaning up new Pool events (related to `Delegation` feature of pool).

---------

Signed-off-by: Matteo Muraca <[email protected]>
Signed-off-by: Alexandru Gheorghe <[email protected]>
Signed-off-by: Andrei Sandu <[email protected]>
Signed-off-by: Adrian Catangiu <[email protected]>
Signed-off-by: Alexandru Vasile <[email protected]>
Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Signed-off-by: divdeploy <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: hongkuang <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: gemini132 <[email protected]>
Co-authored-by: Matteo Muraca <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Alexandru Gheorghe <[email protected]>
Co-authored-by: Alessandro Siniscalchi <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>
Co-authored-by: Ross Bulat <[email protected]>
Co-authored-by: Serban Iorga <[email protected]>
Co-authored-by: s0me0ne-unkn0wn <[email protected]>
Co-authored-by: Sam Johnson <[email protected]>
Co-authored-by: Adrian Catangiu <[email protected]>
Co-authored-by: Javier Viola <[email protected]>
Co-authored-by: Alexandru Vasile <[email protected]>
Co-authored-by: Niklas Adolfsson <[email protected]>
Co-authored-by: Dastan <[email protected]>
Co-authored-by: Clara van Staden <[email protected]>
Co-authored-by: Ron <[email protected]>
Co-authored-by: Vincent Geddes <[email protected]>
Co-authored-by: Svyatoslav Nikolsky <[email protected]>
Co-authored-by: Michal Kucharczyk <[email protected]>
Co-authored-by: Dino Pačandi <[email protected]>
Co-authored-by: Andrei Eres <[email protected]>
Co-authored-by: Alin Dima <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Branislav Kontur <[email protected]>
Co-authored-by: Sebastian Kunert <[email protected]>
Co-authored-by: gupnik <[email protected]>
Co-authored-by: Vladimir Istyufeev <[email protected]>
Co-authored-by: Lulu <[email protected]>
Co-authored-by: Juan Girini <[email protected]>
Co-authored-by: Francisco Aguirre <[email protected]>
Co-authored-by: Dónal Murray <[email protected]>
Co-authored-by: Shawn Tabrizi <[email protected]>
Co-authored-by: Kutsal Kaan Bilgin <[email protected]>
Co-authored-by: Ermal Kaleci <[email protected]>
Co-authored-by: ordian <[email protected]>
Co-authored-by: divdeploy <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Sergej Sakac <[email protected]>
Co-authored-by: Squirrel <[email protected]>
Co-authored-by: HongKuang <[email protected]>
Co-authored-by: Tsvetomir Dimitrov <[email protected]>
Co-authored-by: Egor_P <[email protected]>
Co-authored-by: Aaro Altonen <[email protected]>
Co-authored-by: Dmitry Markin <[email protected]>
Co-authored-by: Alexandru Vasile <[email protected]>
Co-authored-by: Léa Narzis <[email protected]>
Co-authored-by: Gonçalo Pestana <[email protected]>
Co-authored-by: georgepisaltu <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: PG Herveou <[email protected]>
Co-authored-by: jimwfs <[email protected]>
Co-authored-by: jimwfs <[email protected]>
Co-authored-by: polka.dom <[email protected]>

Tested each bootnode with `--reserved-only --reserved-nodes`

Fixes https://github.com/paritytech/polkadot-sdk/issues/4552



---------

Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <[email protected]>

Depends on https://github.com/paritytech/polkadot-sdk/pull/4621.

Implemented the
[`XcmPaymentApi`](https://github.com/paritytech/polkadot-sdk/pull/3607)
and [`DryRunApi`](https://github.com/paritytech/polkadot-sdk/pull/3872)
on all system parachains.

More scenarios can be tested on both rococo and westend if all system
parachains implement this APIs.
The objective is for all XCM-enabled runtimes to implement them.
After demonstrating fee estimation in a UI on the testnets, come the
fellowship runtimes.

Step towards https://github.com/paritytech/polkadot-sdk/issues/690.

Addresses
[this](https://github.com/paritytech/polkadot-sdk/issues/3155#issuecomment-2134411391).

Validators already have been upgraded so they could already receive the
new `CollationWithParentHeadData` response when fetching collation.
However this is only sent by collators when the parachain has more than
1 core is assigned.

TODO:
- [x] PRDoc

---------

Signed-off-by: Andrei Sandu <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>

Currently, `chain-spec-builder` only creates a spec with `Live` chain
type. This PR adds the ability to specify it while keeping the same
default.

---------

Co-authored-by: Kian Paimani <[email protected]>

Fixes #4625.

Specifically, the `cfg` attribute `windows` refers to the compile target
and not the build environment, and in the case of cross-compilation, the
build environment and target can differ. However, the line modified is
related to documentation generation, so there should be no critical
issue with this change.

Revived version of https://github.com/paritytech/substrate/pull/13311

 .
Except Signature is not generic and is dictated by AuthorityId.

---------

Co-authored-by: Davide Galassi <[email protected]>
Co-authored-by: Robert Hambrock <[email protected]>
Co-authored-by: Adrian Catangiu <[email protected]>

Otherwise these tests are failing if you don't run with
`debug_assertions` enabled, which happens if you run tests locally in
release mode.

Fixes #4360 

Also rename: AllowedRenewals -> PotentialRenewals to avoid confusion of
future readers. (An entry in `AllowedRenewals` is not enough to allow a
renewal, the assignment also has to be complete, which is only checked
afterwards.)

- [x] Does not work with renewals as is - fix.
- [x] More tests
- [x] PR docs

Edit 1:
(Relevant blog post:
https://grillapp.net/12935/agile-coretime-pricing-explained-166522?ref=29715

)

---------

Co-authored-by: eskimor <[email protected]>
Co-authored-by: Dónal Murray <[email protected]>
Co-authored-by: command-bot <>

Follow-up to the new `XcmDryRunApi` runtime API introduced in
https://github.com/paritytech/polkadot-sdk/pull/3872.

Taking an extrinsic means the frontend has to sign first to dry-run and
once again to submit.
This is bad UX which is solved by taking an `origin` and a `call`.
This also has the benefit of being able to dry-run as any account, since
it needs no signature.

This is a breaking change since I changed `dry_run_extrinsic` to
`dry_run_call`, however, this API is still only on testnets.
The crates are bumped accordingly.

As a part of this PR, I changed the name of the API from `XcmDryRunApi`
to just `DryRunApi`, since it can be used for general dry-running :)

Step towards https://github.com/paritytech/polkadot-sdk/issues/690

.

Example of calling the API with PAPI, not the best code, just testing :)

```ts
// We just build a call, the arguments make it look very big though.
const call = localApi.tx.XcmPallet.transfer_assets({
  dest: XcmVersionedLocation.V4({ parents: 0, interior: XcmV4Junctions.X1(XcmV4Junction.Parachain(1000)) }),
  beneficiary: XcmVersionedLocation.V4({ parents: 0, interior: XcmV4Junctions.X1(XcmV4Junction.AccountId32({ network: undefined, id: Binary.fromBytes(encodeAccount(account.address)) })) }),
  weight_limit: XcmV3WeightLimit.Unlimited(),
  assets: XcmVersionedAssets.V4([{
    id: { parents: 0, interior: XcmV4Junctions.Here() },
    fun: XcmV3MultiassetFungibility.Fungible(1_000_000_000_000n) }
  ]),
  fee_asset_item: 0,
});
// We call the API passing in a signed origin 
const result = await localApi.apis.XcmDryRunApi.dry_run_call(
  WestendRuntimeOriginCaller.system(DispatchRawOrigin.Signed(account.address)),
  call.decodedCall
);
if (result.success && result.value.execution_result.success) {
  // We find the forwarded XCM we want. The first one going to AssetHub in this case.
  const xcmsToAssetHub = result.value.forwarded_xcms.find(([location, _]) => (
    location.type === "V4" &&
      location.value.parents === 0 &&
      location.value.interior.type === "X1"
      && location.value.interior.value.type === "Parachain"
      && location.value.interior.value.value === 1000
  ))!;

  // We can even find the delivery fees for that forwarded XCM.
  const deliveryFeesQuery = await localApi.apis.XcmPaymentApi.query_delivery_fees(xcmsToAssetHub[0], xcmsToAssetHub[1][0]);

  if (deliveryFeesQuery.success) {
    const amount = deliveryFeesQuery.value.type === "V4" && deliveryFeesQuery.value.value[0].fun.type === "Fungible" && deliveryFeesQuery.value.value[0].fun.value.valueOf() || 0n;
    // We store them in state somewhere.
    setDeliveryFees(formatAmount(BigInt(amount)));
  }
}
```

---------

Co-authored-by: Bastian Köcher <[email protected]>

Minor edit to a broken link for Rust Docs on the README.md

Co-authored-by: Kian Paimani <[email protected]>

marking it as release-able, attaching the same version number that is
attached to other binaries such as `polkadot` and `polkadot-parachain`.

I have more thoughts about the version number, though. The chain-spec
builder is mainly a user of the `sp-genesis-builder` api. So the
versioning should be such that it helps users know give a version of
`sp-genesis-builder` in their runtime, which version of
`chain-spec-builder` should they use?

With this, we can possibly alter the version number to always match
`sp-genesis-builder`.

Fixes https://github.com/paritytech/polkadot-sdk/issues/4352

- [x] Add to release artifacts ~~similar to
https://github.com/paritytech/polkadot-sdk/pull/4405~~ done here:
https://github.com/paritytech/polkadot-sdk/pull/4557



---------

Co-authored-by: Branislav Kontur <[email protected]>

Closes: https://github.com/paritytech/polkadot-sdk/issues/4354

This PR adds the steps to build and attach `frame-omni-bencher` and
`chain-spec-builder` binaries to the release draft

## TODO
- [x] add also chain-spec-builder binary
- [ ] ~~check/investigate Kian's comment: `chain spec builder. Ideally I
want it to match the version of the sp-genesis-builder crate`~~ see
[comment](https://github.com/paritytech/polkadot-sdk/pull/4518#issuecomment-2134731355)
- [ ] Backport to `[email protected]` release, so we can use it for next
fellows release: https://github.com/polkadot-fellows/runtimes/pull/324


- [ ] Backport to `[email protected]` release

---------

Co-authored-by: Branislav Kontur <[email protected]>

Step in https://github.com/paritytech/polkadot-sdk/issues/3688



Now that the `runtime` macro (Construct Runtime V2) has been
successfully deployed on Westend, this PR moves it out of the
experimental feature flag and makes it generally available for runtime
devs.

---------

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>

This detaches the templates from monorepo's workspace dependencies.

Currently the templates [re-use the monorepo's
dependencies](https://github.com/paritytech/polkadot-sdk-minimal-template/blob/bd8afe66ec566d61f36b0e3d731145741a9e9e19/Cargo.toml#L45-L58),
most of which are not needed.

The simplest approach is to specify versions directly and not use
workspace dependencies in the templates.

Another approach would be to programmatically filter dependencies that
are actually needed - but not sure if it's worth it, given that it would
complicate the synchronization job.

cc @Kianenigma @gupnik

This should make it more obvious that at instantiation of the
`MockValidationDataInherentDataProvider` the `para_id` needs to be
passed.

To understand with high granularity how many assignment tranches are
triggered before we concur that we have enough assignments.

This metric is important because the triggering of an assignment creates
a lot of work in the system for approving the candidate and gossiping
the necessary messages.

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>
Co-authored-by: ordian <[email protected]>

title: Improving `on_demand_assigner` emitted events

doc:
  - audience: Rutime User
description: OnDemandOrderPlaced event that is useful for indexers to
save data related to on demand orders. Check [discussion
here](https://substrate.stackexchange.com/questions/11366/ondemandassignmentprovider-ondemandorderplaced-event-was-removed/11389#11389

).

Closes #4254 

crates: [ 'runtime-parachain]

---------

Co-authored-by: Maciej <[email protected]>

Workaround for skipped but `required` github checks. The idea is to
trigger the workflow but filter out unaffected jobs or steps. See [ci_cd
998](https://github.com/paritytech/ci_cd/issues/988) for details

In `.github/workflows/check-changed-files.yml` there is a reusable
workflow thad does all the checks and publishes results as outputs.
Example usage:
```
jobs:
  changes:
    permissions:
      pull-requests: read
    uses: ./.github/workflows/check-changed-files.yml
  some-job:
    needs: changes
    if: ${{ needs.changes.outputs.rust }}
 ....... 
 ```

Adds a small CI check to match the existing Git deps agains a known-bad
list.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Fixes https://github.com/paritytech/polkadot-sdk/issues/3601

Since we print benchmark results manually, we don't need to save
benchmark_name anywhere, better just put the name inside `println!`.

**Don't look at the commit history, it's confusing, as this branch is
based on another branch that was merged**

Fixes #598 
Also implements [RFC
#47](https://github.com/polkadot-fellows/RFCs/pull/47)

## Description

- Availability-recovery now first attempts to request the systematic
chunks for large POVs (which are the first ~n/3 chunks, which can
recover the full data without doing the costly reed-solomon decoding
process). This has a fallback of recovering from all chunks, if for some
reason the process fails. Additionally, backers are also used as a
backup for requesting the systematic chunks if the assigned validator is
not offering the chunk (each backer is only used for one systematic
chunk, to not overload them).
- Quite obviously, recovering from systematic chunks is much faster than
recovering from regular chunks (4000% faster as measured on my apple M2
Pro).
- Introduces a `ValidatorIndex` -> `ChunkIndex` mapping which is
different for every core, in order to avoid only querying the first n/3
validators over and over again in the same session. The mapping is the
one described in RFC 47.
- The mapping is feature-gated by the [NodeFeatures runtime
API](https://github.com/paritytech/polkadot-sdk/pull/2177) so that it
can only be enabled via a governance call once a sufficient majority of
validators have upgraded their client. If the feature is not enabled,
the mapping will be the identity mapping and backwards-compatibility
will be preserved.
- Adds a new chunk request protocol version (v2), which adds the
ChunkIndex to the response. This may or may not be checked against the
expected chunk index. For av-distribution and systematic recovery, this
will be checked, but for regular recovery, no. This is backwards
compatible. First, a v2 request is attempted. If that fails during
protocol negotiation, v1 is used.
- Systematic recovery is only attempted during approval-voting, where we
have easy access to the core_index. For disputes and collator
pov_recovery, regular chunk requests are used, just as before.

## Performance results

Some results from subsystem-bench:

with regular chunk recovery: CPU usage per block 39.82s
with recovery from backers: CPU usage per block 16.03s
with systematic recovery: CPU usage per block 19.07s

End-to-end results here:
https://github.com/paritytech/polkadot-sdk/issues/598#issuecomment-1792007099

#### TODO:

- [x] [RFC #47](https://github.com/polkadot-fellows/RFCs/pull/47)
- [x] merge https://github.com/paritytech/polkadot-sdk/pull/2177 and
rebase on top of those changes
- [x] merge https://github.com/paritytech/polkadot-sdk/pull/2771 and
rebase
- [x] add tests
- [x] preliminary performance measure on Versi: see
https://github.com/paritytech/polkadot-sdk/issues/598#issuecomment-1792007099
- [x] Rewrite the implementer's guide documentation
- [x] https://github.com/paritytech/polkadot-sdk/pull/3065 
- [x] https://github.com/paritytech/zombienet/issues/1705

 and fix
zombienet tests
- [x] security audit
- [x] final versi test and performance measure

---------

Signed-off-by: alindima <[email protected]>
Co-authored-by: Javier Viola <[email protected]>

This detaches the templates from monorepo's workspace lints, so the
lints for the templates can evolve separately as needed.

Currently the templates [re-use the monorepo's
lints](https://github.com/paritytech/polkadot-sdk-minimal-template/blob/bd8afe66ec566d61f36b0e3d731145741a9e9e19/Cargo.toml#L16-L43)
which looks weird.

cc @Kianenigma @gupnik

This PR removes deprecated code:
- The `RuntimeGenesisConfig` generic type parameter in
`GenericChainSpec` struct.
- `ChainSpec::from_genesis` method allowing to create chain-spec using
closure providing runtime genesis struct
- `GenesisSource::Factory` variant together with no longer needed
`GenesisSource`'s generic parameter `G` (which was intended to be a
runtime genesis struct).


https://github.com/paritytech/polkadot-sdk/blob/17b56fae2d976a3df87f34076875de8c26da0355/substrate/client/chain-spec/src/chain_spec.rs#L559-L563