This branch propose to avoid clones in append by storing offset and size
in previous overlay depth.
That way on rollback we can just truncate and change size of existing
value.
To avoid copy it also means that :

- append on new overlay layer if there is an existing value: create a
new Append entry with previous offsets, and take memory of previous
overlay value.
- rollback on append: restore value by applying offsets and put it back
in previous overlay value
- commit on append: appended value overwrite previous value (is an empty
vec as the memory was taken). offsets of commited layer are dropped, if
there is offset in previous overlay layer they are maintained.
- set value (or remove) when append offsets are present: current
appended value is moved back to previous overlay value with offset
applied and current empty entry is overwrite (no offsets kept).

The modify mechanism is not needed anymore.
This branch lacks testing and break some existing genericity (bit of
duplicated code), but good to have to check direction.

Generally I am not sure if it is worth or we just should favor
differents directions (transients blob storage for instance), as the
current append mechanism is a bit tricky (having a variable length in
first position means we sometime need to insert in front of a vector).

Fix #30.

---------

Signed-off-by: Alexandru Vasile <[email protected]>
Co-authored-by: EgorPopelyaev <[email protected]>
Co-authored-by: Alexandru Vasile <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Branislav Kontur <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Sebastian Kunert <[email protected]>

## Issue

Currently, syncing parachains from scratch can lead to a very long
finalization time once they reach the tip of the chain. The problem is
that we try to finalize everything from 0 to the tip, which can be
thousands or even millions of blocks.

We finalize sequentially and try to compute displaced branches during
finalization. So for every block on the way, we compute an expensive
tree route.

## Proposed Improvements

In this PR, I propose improvements that solve this situation:

- **Skip tree route calculation if `leaves().len() == 1`:** This should
be enough for 90% of cases where there is only one leaf after sync.
- **Optimize finalization for long distances:** It can happen that the
parachain has imported some leaf and then receives a relay chain
notification with the finalized block. In that case, the previous
optimization will not trigger. A second mechanism should ensure that we
do not need to compute the full tree route. If the finalization distance
is long, we check the lowest common ancestor of all the leaves. If it is
above the to-be-finalized block, we know that there are no displaced
leaves. This is fast because forks are short and close to the tip, so we
can leverage the header cache.

## Alternative Approach

- The problem was introduced in #3962. Reverting that PR is another
possible strategy.
- We could store for every fork where it begins, however sounds a bit
more involved to me.


fixes #4614

... this is superfluous because set_reserved_peers implementation
already calls this method here:

https://github.com/paritytech/polkadot-sdk/blob/cdb297b15ad9c1d952c0501afaf6b764e5fd147c/substrate/client/network/src/protocol_controller.rs#L571

,
so the call just ends producing this warnings whenever we manipulate the
peers set.

```
Trying to remove unknown reserved node 12D3KooWRCePWvHoBbz9PSkw4aogtdVqkVDhiwpcHZCqh4hdPTXC from SetId(3)
peerset warnings (from different peers)
```

Signed-off-by: Alexandru Gheorghe <[email protected]>

- unit tests for pov-recovery
- elastic scaling support (recovering multiple candidates in a single
relay chain block)
- also some small cleanups
- also switches to candidates_pending_availability in
`handle_empty_block_announce_data`

Fixes https://github.com/paritytech/polkadot-sdk/issues/3577

After https://github.com/paritytech/polkadot-sdk/pull/4097 is merged, we
should also add a zombienet test, similar to the existing
`0002-pov_recovery.toml` but which has a single collator using elastic
scaling on multiple cores.

Add some debug logs to be able to identify the validators and parachains
that have most no-shows, this metric is valuable because it will help us
identify validators and parachains that regularly have this problem.

From the validator_index we can then query the on-chain information and
identify the exact validator that is causing the no-shows.

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>

- Addresses
[this](https://github.com/paritytech/polkadot-sdk/issues/3155#issuecomment-2126934939).
- Revamps the Readme, very similar to [the minimal
template](https://github.com/paritytech/polkadot-sdk/pull/4649).
- Changed `polkadot-launch` to `zombienet`, with instructions how to run
it.
- See the [rendered
version](https://github.com/paritytech/polkadot-sdk/blob/rzadp/parachain-template-readme/templates/parachain/README.md).

Update README.md to move the PSVM link under a "Tooling" section under the "Releases" section (#4734)

This update implements the suggestion from @Kianenigma mentioned in
https://github.com/paritytech/polkadot-sdk/pull/4718#issuecomment-2153777367

Replaces the "Other useful resources and tooling" section at the bottom
with a new (nicer) "Tooling" section just under the "Releases" section.

Small refactoring PR to improve the readability of the proc macros.
- small improvement in docs
- use new `let Some(..) else` expression
- removed extra indentations by early returns

Discovered during metadata v16 poc, extracted from:
https://github.com/paritytech/polkadot-sdk/pull/4358



---------

Signed-off-by: Alexandru Vasile <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: gupnik <[email protected]>

take over #2941 
[Weights
compare](https://weights.tasty.limo/compare?unit=weight&ignore_errors=true&threshold=10&method=asymptotic&repo=polkadot-sdk&old=master&new=pg%2Fwasmi-to-v0.32.0-beta.7&path_pattern=substrate%2Fframe%2F**%2Fsrc%2Fweights.rs%2Cpolkadot%2Fruntime%2F*%2Fsrc%2Fweights%2F**%2F*.rs%2Cpolkadot%2Fbridges%2Fmodules%2F*%2Fsrc%2Fweights.rs%2Ccumulus%2F**%2Fweights%2F*.rs%2Ccumulus%2F**%2Fweights%2Fxcm%2F*.rs%2Ccumulus%2F**%2Fsrc%2Fweights.rs

)

---------

Co-authored-by: command-bot <>
Co-authored-by: Alexander Theißen <[email protected]>

Closes: https://github.com/paritytech/parity-bridges-common/issues/2734

Thanks for @xlc for the original seed info, I've just fixed it up a bit
and added example links.

I've moved the comparison between eth-rpc-api and frontier outside, as
it is opinionation. I think the content there was good but should live
in the README of the corresponding repos. No strong opinion, happy
either way.

---------

Co-authored-by: Bryan Chen <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Gonçalo Pestana <[email protected]>
Co-authored-by: command-bot <>

<img width="484" alt="Screenshot 2024-05-30 at 10 30 41"
src="https://github.com/paritytech/polkadot-sdk/assets/5588131/92ece1d0-cda3-46cf-9f46-4026c565b52d

">

---------

Co-authored-by: Dónal Murray <[email protected]>

Co-authored-by: eskimor <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>

... when backing group is of size 1.

Signed-off-by: Alexandru Gheorghe <[email protected]>

Adds a link to the [Polkadot SDK Version
Manager](https://github.com/paritytech/psvm

) since this tool is not well
known, but very useful for developers using the Polkadot SDK.

---------

Co-authored-by: Bastian Köcher <[email protected]>

Part of https://github.com/paritytech/polkadot-sdk/issues/4324


We don't change but extend the existing cleanup strategy. 
- We still don't touch artifacts being stale less than 24h
- First time we attempt pruning only when we hit cache limit (10 GB)
- If somehow happened that after we hit 10 GB and least used artifact is
stale less than 24h we don't remove it.

---------

Co-authored-by: s0me0ne-unkn0wn <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>

The link to
[releases](https://github.com/paritytech/polkadot-sdk/releases) usually
takes you to a list with a bunch of drafts at the top so you have to
scroll.

[This link](https://github.com/paritytech/polkadot-sdk/releases/latest)
takes you directly to the latest release.

Nobody seems to maintain this and the job is disabled since months. I
think unless the Security team wants to pick this up we delete it for
now.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

`secp256k1 v0.28.0` and `secp256k1-sys v0.9.0` were yanked because
building them fails for `aarch64-apple-darwin` targets.

Use the `secp256k1 v0.28.2` and `secp256k1-sys v0.9.2` patched versions
that build fine on ARM chipset MacOS.

Inherited workspace dependencies cannot be renamed by the crate using
them (see [1](https://github.com/rust-lang/cargo/issues/12546),
[2](https://stackoverflow.com/questions/76792343/can-inherited-dependencies-in-rust-be-aliased-in-the-cargo-toml-file)).
Since we want to use inherited workspace dependencies everywhere, we
first need to unify all aliases that we use for a dependency throughout
the workspace.
The umbrella crate is currently excluded from this procedure, since it
should be able to export the crates by their original name without much
hassle.

For example: one crate may alias `parity-scale-codec` to `codec`, while
another crate does not alias it at all. After this change, all crates
have to use `codec` as name. The problematic combinations were:
- conflicting aliases: most crates aliases as `A` but some use `B`.
- missing alias: most of the crates alias a dep but some dont.
- superfluous alias: most crates dont alias a dep but some do.

The script that i used first determines whether most crates opted to
alias a dependency or not. From that info it decides whether to use an
alias or not. If it decided to use an alias, the most common one is used
everywhere.

To reproduce, i used
[this](https://github.com/ggwpez/substrate-scripts/blob/master/uniform-crate-alias.py)
python script in combination with
[this](https://github.com/ggwpez/zepter/blob/38ad10585fe98a5a86c1d2369738bc763a77057b/renames.json

)
error output from Zepter.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

In preparation for launching re-enabling
(https://github.com/paritytech/polkadot-sdk/issues/2418), we need to
adjust the disabling strategy of statement-distribution to use the relay
parent's state instead of the latest state (union of active leaves).
This will also ensure no raciness of getting the latest state vs
accepting statements from disabling validators at the cost of being more
lenient/potentially accepting more statements from disabled validators.

- [x] PRDoc

Follow up on #4414 to clean up the old storage.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

some minor text fixes.

This Dockerfile seems outdated - it currently fails to build (on my
machine).
I don't see it being built anywhere on CI.

I did a couple of tweaks to make it build.

---------

Co-authored-by: Alexander Samusev <[email protected]>

In order to receive a username in `pallet-identity`, users have to,
among other things, provide a signature of the desired username. Right
now, there is an [extra encoding
step](https://github.com/paritytech/polkadot-sdk/blob/4ab078d6754147ce731523292dd1882f8a7b5775/substrate/frame/identity/src/lib.rs#L1119

)
when generating the payload to sign.

Encoding a `Vec` adds extra bytes related to the length, which changes
the payload. This is unnecessary and confusing as users expect the
payload to sign to be just the username bytes. This PR fixes this issue
by validating the signature directly against the username bytes.

---------

Signed-off-by: georgepisaltu <[email protected]>

Added reference doc about:
- the pallet genesis config and genesis build, 
- runtime `genesis-builder` API,
- presets,
- interacting with the `chain-spec-builder` tool

I've added [minimal
runtime](https://github.com/paritytech/polkadot-sdk/tree/mku-chain-spec-guide/docs/sdk/src/reference_docs/chain_spec_runtime

)
to demonstrate above topics.

I also sneaked in some little improvement to `chain-spec-builder` which
allows to parse output of the `list-presets` command.

---------

Co-authored-by: Alexandru Vasile <[email protected]>
Co-authored-by: Sebastian Kunert <[email protected]>

This allows to use custom para_ids with cumulus-test-runtime. 

Zombienet is patching the genesis entries for `ParachainInfo`. This did
not work with `test-parachain` because it was using the `test_pallet`
for historic reasons I guess.

Since the fix for https://github.com/paritytech/polkadot-sdk/issues/4559
was to add a codeSubstitute to the rococo-parachains, we should update
the chain-specs in the repo.

The json diffs are not super easy to review, so if you want to manually
check:
- Run for example `jd -o people-rococo.diff
people-rococo-from-this-branch.json people-rococo-master.json`
- Check in the resulting diff file that only `codeSubstitutes` field is
changed

Signed-off-by: Alexandru Gheorghe <[email protected]>

This PR implements the `READ_ONLY` flag to be used as a `Callflag` in
the `call` function.
The flag indicates that the callee is restricted from modifying the
state during call execution.
It is equivalent to Ethereum's
[STATICCALL](https://eips.ethereum.org/EIPS/eip-214

).

---------

Co-authored-by: command-bot <>
Co-authored-by: Andrew Jones <[email protected]>
Co-authored-by: Alexander Theißen <[email protected]>

Co-authored-by: Oliver Tale-Yazdi <[email protected]>

`[check-runtime-migration` now takes more than 30 minutes. Quick fix
with increased timeout.

As requested
[here](https://github.com/paritytech/polkadot-sdk/issues/3155#issuecomment-2126934939).

The Dockerfiles are inspired by [this
one](https://github.com/paritytech/polkadot-sdk/blob/aa32faaebf64426becb2feeede347740eb7a3908/docker/dockerfiles/polkadot/polkadot_builder.Dockerfile).

- Addresses
[this](https://github.com/paritytech/polkadot-sdk/issues/3155#issuecomment-2126934939).
- Technical content got adopted from the existing [solochain
readme](https://github.com/paritytech/polkadot-sdk/tree/master/templates/solochain).
  - Updated some broken links there.
- The docker instructions will work after
https://github.com/paritytech/polkadot-sdk/pull/4637.
- See the [rendered
version](https://github.com/paritytech/polkadot-sdk/blob/rzadp/minimal-template-readme/templates/minimal/README.md

).

---------

Co-authored-by: gupnik <[email protected]>

This PR adds possibility to publish container images for the
`chain-spec-builder` binary on the regular basis.
Related to: https://github.com/paritytech/release-engineering/issues/190

Tested each bootnode with `--reserved-only` `--reserved-nodes`

Kusama
```
polkadot --chain kusama --base-path /tmp/node --reserved-only --reserved-nodes "/dns/kusama.bootnode.amforc.com/tcp/29999/wss/p2p/12D3KooWKvYf6qKaAF8UUDw3KsTwjHLnvkED23yxHbH3npMe8w4G" --no-hardware-benchmarks
polkadot --chain kusama --base-path /tmp/node --reserved-only --reserved-nodes "/dns/kusama.bootnode.amforc.com/tcp/30001/p2p/12D3KooWKvYf6qKaAF8UUDw3KsTwjHLnvkED23yxHbH3npMe8w4G" --no-hardware-benchmarks
```

Asset Hub Kusama
```
./polkadot-parachain --chain asset-hub-kusama --base-path /tmp/node --reserved-only --reserved-nodes "/dns/asset-hub-kusama.bootnode.amforc.com/tcp/29999/wss/p2p/12D3KooWHy1CPndZYphwdVqMb295KPC6LRt17Ae3zNSr7evzeF5a" --no-hardware-benchmarks
./polkadot-parachain --chain asset-hub-kusama --base-path /tmp/node --reserved-only --reserved-nodes "/dns/asset-hub-kusama.bootnode.amforc.com/tcp/30007/p2p/12D3KooWHy1CPndZYphwdVqMb295KPC6LRt17Ae3zNSr7evzeF5a" --no-hardware-benchmarks
```


Bridge Hub Kusama
```
./polkadot-parachain --chain bridge-hub-kusama --base-path /tmp/node --reserved-only --reserved-nodes "/dns/bridge-hub-kusama.bootnode.amforc.com/tcp/29999/wss/p2p/12D3KooWNyTBwRvCz1Ey2SgC1f3MvymhiAyLEa3cL8kU5gFH3V7Z" --no-hardware-benchmarks
./polkadot-parachain --chain bridge-hub-kusama --base-path /tmp/node --reserved-only --reserved-nodes "/dns/bridge-hub-kusama.bootnode.amforc.com/tcp/30010/p2p/12D3KooWNyTBwRvCz1Ey2SgC1f3MvymhiAyLEa3cL8kU5gFH3V7Z" --no-hardware-benchmarks
```

Coretime Kusama
```
./polkadot-parachain --chain coretime-kusama --base-path /tmp/node --reserved-only --reserved-nodes "/dns/coretime-kusama.bootnode.amforc.com/tcp/29999/wss/p2p/12D3KooWPrgxrrumrANp6Bp2SMEwMQHPHDbPzA1HbcrakZrbFi5P" --no-hardware-benchmarks
./polkadot-parachain --chain coretime-kusama --base-path /tmp/node --reserved-only --reserved-nodes "/dns/coretime-kusama.bootnode.amforc.com/tcp/30013/p2p/12D3KooWPrgxrrumrANp6Bp2SMEwMQHPHDbPzA1HbcrakZrbFi5P" --no-hardware-benchmarks
```

People Kusama
```
./polkadot-parachain --chain people-kusama --base-path /tmp/node --reserved-only --reserved-nodes "/dns/people-kusama.bootnode.amforc.com/tcp/29999/wss/p2p/12D3KooWPjzgKZe5jdG6TY4gwcFq8QxyyhqsYbQo6N29pwGePWLA" --no-hardware-benchmarks
./polkadot-parachain --chain people-kusama --base-path /tmp/node --reserved-only --reserved-nodes "/dns/people-kusama.bootnode.amforc.com/tcp/30004/p2p/12D3KooWPjzgKZe5jdG6TY4gwcFq8QxyyhqsYbQo6N29pwGePWLA" --no-hardware-benchmarks
```

People Westend
```
./polkadot-parachain --chain people-westend --base-path /tmp/node --reserved-only --reserved-nodes "/dns/people-westend.bootnode.amforc.com/tcp/29999/wss/p2p/12D3KooWE1btdwDhNpApg8BEe2QwJxdVDtz6a6BRhgTeUh9HMhWs" --no-hardware-benchmarks
./polkadot-parachain --chain people-westend --base-path /tmp/node --reserved-only --reserved-nodes "/dns/people-westend.bootnode.amforc.com/tcp/30016/p2p/12D3KooWE1btdwDhNpApg8BEe2QwJxdVDtz6a6BRhgTeUh9HMhWs" --no-hardware-benchmarks
```

Polkadot
```
polkadot --chain polkadot --base-path /tmp/node --reserved-only --reserved-nodes "/dns/polkadot.bootnode.amforc.com/tcp/29999/wss/p2p/12D3KooWT2HyZx5C6BBeLbCKhYG2SqJYuiu7sLMxGzUcQBko3BMr" --no-hardware-benchmarks
polkadot --chain polkadot --base-path /tmp/node --reserved-only --reserved-nodes "/dns/polkadot.bootnode.amforc.com/tcp/30001/p2p/12D3KooWT2HyZx5C6BBeLbCKhYG2SqJYuiu7sLMxGzUcQBko3BMr" --no-hardware-benchmarks
```

Asset Hub Polkadot
```
./polkadot-parachain --chain asset-hub-polkadot --base-path /tmp/node --reserved-only --reserved-nodes "/dns/asset-hub-polkadot.bootnode.amforc.com/tcp/29999/wss/p2p/12D3KooWDLxPXYnSHjNwq9ibqgxuzRni5VViuGNSjNe3ueqVgqE3" --no-hardware-benchmarks
./polkadot-parachain --chain asset-hub-polkadot --base-path /tmp/node --reserved-only --reserved-nodes "/dns/asset-hub-polkadot.bootnode.amforc.com/tcp/30007/p2p/12D3KooWDLxPXYnSHjNwq9ibqgxuzRni5VViuGNSjNe3ueqVgqE3" --no-hardware-benchmarks
```

Bridge Hub Polkadot
```
./polkadot-parachain --chain bridge-hub-polkadot --base-path /tmp/node --reserved-only --reserved-nodes "/dns/bridge-hub-polkadot.bootnode.amforc.com/tcp/29999/wss/p2p/12D3KooWGT5E56rAHfT5dY1pMLTrpAgV72yfDtD1Y5tPCHaTsifp" --no-hardware-benchmarks
./polkadot-parachain --chain bridge-hub-polkadot --base-path /tmp/node --reserved-only --reserved-nodes "/dns/bridge-hub-polkadot.bootnode.amforc.com/tcp/30010/p2p/12D3KooWGT5E56rAHfT5dY1pMLTrpAgV72yfDtD1Y5tPCHaTsifp" --no-hardware-benchmarks
```

Collectives Polkadot
```
./polkadot-parachain --chain collectives-polkadot --base-path /tmp/node --reserved-only --reserved-nodes "/dns/collectives-polkadot.bootnode.amforc.com/tcp/29999/wss/p2p/12D3KooWL6v6FHMtCP5VsiDbMHLRFiW6YBtv37BarpW3hLqnDski" --no-hardware-benchmarks
./polkadot-parachain --chain collectives-polkadot --base-path /tmp/node --reserved-only --reserved-nodes "/dns/collectives-polkadot.bootnode.amforc.com/tcp/30013/p2p/12D3KooWL6v6FHMtCP5VsiDbMHLRFiW6YBtv37BarpW3hLqnDski" --no-hardware-benchmarks
```

Makes all storage items in parachain-system public so that these can be
used by other pallets &/or runtimes.

---------

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: command-bot <>