..and other high level docs.


# Polling 

Please vote in the reactions of this PR 

- 👍 I agree to replace the website of this repo to
https://paritytech.github.io/polkadot-sdk/master/polkadot_sdk_docs/index.html
- 👎 Keep it as polkadot.network
- 🤷

 Different opinion

---------

Co-authored-by: Bastian Köcher <[email protected]>

This implements the `CheckMetadataHash` extension as described in
[RFC78](https://polkadot-fellows.github.io/RFCs/approved/0078-merkleized-metadata.html).

Besides the signed extension, the `substrate-wasm-builder` is extended
to support generating the metadata-hash.

Closes: https://github.com/paritytech/polkadot-sdk/issues/291

---------

Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: joe petrowski <[email protected]>
Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>

We are actually using an older version, where the log line is in a
different file.

This PR backports version bumps and reorganisation of the prdoc files
from the `1.12.0` release branch back to `master`

Closes #4366

This version includes the latest release of pjs/api
(https://github.com/polkadot-js/api/releases/tag/v11.1.1).
Thx!

This PR introduces custom types / substrate wrappers for `Multiaddr`,
`multiaddr::Protocol`, `Multihash`, `ed25519::*` and supplementary types
like errors and iterators.

This is needed to unblock `libp2p` upgrade PR
https://github.com/paritytech/polkadot-sdk/pull/1631 after
https://github.com/paritytech/polkadot-sdk/pull/2944 was merged.
`libp2p` and `litep2p` currently depend on different versions of
`multiaddr` crate, and introduction of this "common ground" types is
needed to support independent version upgrades of `multiaddr` and
dependent crates in `libp2p` & `litep2p`.

While being just convenient to not tie versions of `libp2p` & `litep2p`
dependencies together, it's currently not even possible to keep `libp2p`
& `litep2p` dependencies updated to the same versions as `multiaddr` in
`libp2p` depends on `libp2p-identity` that we can't include as a
dependency of `litep2p`, which has it's own `PeerId` type. In the
future, to keep things updated on `litep2p` side, we will likely need to
fork `multiaddr` and make it use `litep2p` `PeerId` as a payload of
`/p2p/...` protocol.

With these changes, common code in substrate uses these custom types,
and `litep2p` & `libp2p` backends use corresponding libraries types.

closes https://github.com/paritytech/parity-bridges-common/issues/2963

See issue above for rationale
I've been thinking about adding similar calls to other pallets, but:
- for parachains pallet I haven't been able to think of a case when we
will need that given how long referendum takes. I.e. if storage proof
format changes and we want to unstuck the bridge, it'll take a large a
few weeks to sync a single parachain header, then another weeks for
another and etc.
- for messages pallet I've made the similar call initially, but it just
changes a storage key (`OutboundLanes` and/or `InboundLanes`), so
there's no any logic here and it may be simply done using
`system.set_storage`.

---------

Co-authored-by: command-bot <>

closes https://github.com/paritytech/parity-bridges-common/issues/3000

Recently we've changed our bridge configuration for Rococo <> Westend
and our new relayer has started to submit transactions every ~ `30`
seconds. Eventually, it switches itself into limbo state, where it can't
submit more transactions - all `author_submitAndWatchExtrinsic` calls
are failing with the following error: `ERROR bridge Failed to send
transaction to BridgeHubRococo node: Call(ErrorObject { code:
ServerError(-32006), message: "Too many subscriptions on the
connection", data: Some(RawValue("Exceeded max limit of 1024")) })`.

Some links for those who want to explore:
- server side (node) has a strict limit on a number of active
subscriptions. It fails to open a new subscription if this limit is hit:
https://github.com/paritytech/jsonrpsee/blob/a4533966b997e83632509ad97eea010fc7c3efc0/server/src/middleware/rpc/layer/rpc_service.rs#L122-L132.
The limit is set to `1024` by default;
- internally this limit is a semaphore with `limit` permits:
https://github.com/paritytech/jsonrpsee/blob/a4533966b997e83632509ad97eea010fc7c3efc0/core/src/server/subscription.rs#L461-L485;
- semaphore permit is acquired in the first link;
- the permit is "returned" when the `SubscriptionSink` is dropped:
https://github.com/paritytech/jsonrpsee/blob/a4533966b997e83632509ad97eea010fc7c3efc0/core/src/server/subscription.rs#L310-L325;
- the `SubscriptionSink` is dropped when [this `polkadot-sdk`
function](https://github.com/paritytech/polkadot-sdk/blob/278486f9/substrate/client/rpc/src/utils.rs#L58-L94)
returns. In other words - when the connection is closed, the stream is
finished or internal subscription buffer limit is hit;
- the subscription has the internal buffer, so sending an item contains
of two steps: [reading an item from the underlying
stream](https://github.com/paritytech/polkadot-sdk/blob/278486f9/substrate/client/rpc/src/utils.rs#L125-L141)
and [sending it over the
connection](https://github.com/paritytech/polkadot-sdk/blob/278486f9/substrate/client/rpc/src/utils.rs#L111-L116);
- when the underlying stream is finished, the `inner_pipe_from_stream`
wants to ensure that all items are sent to the subscriber. So it: [waits
until the current send operation
completes](https://github.com/paritytech/polkadot-sdk/blob/278486f9/substrate/client/rpc/src/utils.rs#L146-L148)
and then [send all remaining items from the internal
buffer](https://github.com/paritytech/polkadot-sdk/blob/278486f9/substrate/client/rpc/src/utils.rs#L150-L155).
Once it is done, the function returns, the `SubscriptionSink` is
dropped, semaphore permit is dropped and we are ready to accept new
subscriptions;
- unfortunately, the code just calls the `pending_fut.await.is_err()` to
ensure that [the current send operation
completes](https://github.com/paritytech/polkadot-sdk/blob/278486f9/substrate/client/rpc/src/utils.rs#L146-L148).
But if there are no current send operation (which is normal), then the
`pending_fut` is set to terminated future and the `await` never
completes. Hence, no return from the function, no drop of
`SubscriptionSink`, no drop of semaphore permit, no new subscriptions
allowed (once number of susbcriptions hits the limit.

I've illustrated the issue with small test - you may ensure that if e.g.
the stream is initially empty, the
`subscription_is_dropped_when_stream_is_empty` will hang because
`pipe_from_stream` never exits.

Co-authored-by: Svyatoslav Nikolsky <[email protected]>

Implements https://github.com/paritytech/polkadot-sdk/issues/4429

Collators only need to maintain the implicit view for the paraid they
are collating on.
In this case, bypass prospective-parachains entirely. It's still useful
to use the GetMinimumRelayParents message from prospective-parachains
for validators, because the data is already present there.

This enables us to entirely remove the subsystem from collators, which
consumed resources needlessly

Aims to resolve https://github.com/paritytech/polkadot-sdk/issues/4167 

TODO:
- [x] fix unit tests

As per #3326, removes pallet::getter macro usage from
pallet-fast-unstake. The syntax `StorageItem::<T, I>::get()` should be
used instead.

cc @muraca

---------

Co-authored-by: Liam Aharon <[email protected]>

Replace usage of deprecated
`substrate_rpc_client::ChildStateApi::storage_keys` with
`substrate_rpc_client::ChildStateApi::storage_keys_paged`.

Required for successful scraping of Aleph Zero state.

Co-authored-by: jimwfs <[email protected]>

addresses https://github.com/paritytech/polkadot-sdk/issues/4440 (will
close once we have this in prod runtimes).
related: https://github.com/paritytech/polkadot-sdk/issues/2037.

An extra consumer reference is preventing pools to be destroyed. When a
pool is ready to be destroyed, we
can safely clear the consumer references if any. Notably, I only check
for one extra consumer reference since that is a known bug. Anything
more indicates possibly another issue and we probably don't want to
silently absorb those errors as well.

After this change, pools with extra consumer reference should be able to
destroy normally.

Changes the Ethereum client storage scope to public, so it can be set in
a migration.

When merged, we should backport to the all other release branches:

- [ ] release-crates-io-v1.7.0 - patch release the fellows BridgeHubs
runtimes https://github.com/paritytech/polkadot-sdk/pull/4504
- [ ] release-crates-io-v1.8.0 -
https://github.com/paritytech/polkadot-sdk/pull/4505
- [ ] release-crates-io-v1.9.0 -
https://github.com/paritytech/polkadot-sdk/pull/4506
- [ ] release-crates-io-v1.10.0 -
https://github.com/paritytech/polkadot-sdk/pull/4507
- [ ] release-crates-io-v1.11.0 -
https://github.com/paritytech/polkadot-sdk/pull/4508
- [ ] release-crates-io-v1.12.0 (commit soon)

Co-authored-by: command-bot <>

Before relayer crates have been moved + merged, the `MetricsParams` type
has been created from a `substrate-relay` crate (binary) and hence it
has been setting the `substrate_relay_build_info` metic value properly -
to the binary version. Now it is created from the
`substrate-relay-helper` crate, which has the fixed (it isn't published)
version `0.1.0`, so our relay provides incorrect metric value. This
'breaks' our monitoring tools - we see that all relayers have that
incorrect version, which is not cool.

The idea is to have a global static variable (shame on me) that is
initialized by the binary during initialization like we do with the
logger initialization already. Was considering some alternative options:
- adding a separate argument to every relayer subcommand and propagating
it to the `MetricsParams::new()` causes a lot of changes and introduces
even more noise to the binary code, which is supposed to be as small as
possible in the new design. But I could do that if team thinks it is
better;
- adding a `structopt(skip) pub relayer_version: RelayerVersion`
argument to all subcommand params won't work, because it will be
initialized by default and `RelayerVersion` needs to reside in some util
crate (not the binary), so it'll have the wrong value again.

Using Dynamic Parameters for contracts seems like a bad idea for now.

Given that we have benchmarks for each host function (in addition to our
extrinsics), parameter storage reads will be counted multiple times. We
will work on updates to the benchmarking framework to mitigate this
issue in future iterations.

---------

Co-authored-by: command-bot <>

resolves #3315

---------

Co-authored-by: doordashcon <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <[email protected]>

Snowbridge - Ethereum Client - Reject finalized updates without a sync committee in next store period (#4478)

While syncing Ethereum consensus updates to the Snowbridge Ethereum
light client, the syncing process stalled due to error
`InvalidSyncCommitteeUpdate` when importing the next sync committee for
period `1087`.

This bug manifested specifically because our light client checkpoint is
a few weeks old (submitted to governance weeks ago) and had to catchup
until a recent block. Since then, we have done thorough testing of the
catchup sync process.

### Symptoms
- Import next sync committee for period `1086` (essentially period
`1087`). Light client store period = `1086`.
- Import header in period `1087`. Light client store period = `1087`.
The current and next sync committee is not updated, and is now in an
outdated state. (current sync committee = `1086` and current sync
committee = `1087`, where it should be current sync committee = `1087`
and current sync committee = `None`)
- Import next sync committee for period `1087` (essentially period
`1088`) fails because the expected next sync committee's roots don't
match.

### Bug
The bug here is that the current and next sync committee's didn't
handover when an update in the next period was received.

### Fix
There are two possible fixes here:
1. Correctly handover sync committees when a header in the next period
is received.
2. Reject updates in the next period until the next sync committee
period is known.

We opted for solution 2, which is more conservative and requires less
changes.

### Polkadot-sdk versions
This fix should be backported in polkadot-sdk versions 1.7 and up.

Snowfork PR: https://github.com/Snowfork/polkadot-sdk/pull/145

---------

Co-authored-by: Vincent Geddes <[email protected]>

As per #3326, removes usage of the pallet::getter macro from the
democracy pallet. The syntax `StorageItem::<T, I>::get()` should be used
instead.

cc @muraca

Demote `Ignored block announcement because all validation slots for this
peer are occupied.` message to debug level.

This is mostly an indicator of somebody spamming the node or (more
likely) some node actively keeping up with the network but not
recognizing it's in a major sync mode, so sending zillions of block
announcements (have seen this on Versi).

This warning shouldn't be considered an error by the end user, so let's
make it debug.

Ref. https://github.com/paritytech/polkadot-sdk/issues/1929.

The bridge relay is **not** using `tokio`, while `jsonrpsee` does. To
make it work together, we are spawning a separate tokio task for every
jsonrpsee subscription, which holds a subscription reference. It looks
like we are not stopping those tasks when we no longer need it and when
there are more than `1024` active subscriptions, `jsonrpsee` stops
opening new subscriptions. This PR adds an `cancel` signal that is sent
to the background task when we no longer need a subscription.

This PR adds to the DHT only the peers that support the genesis/fork/kad
protocol.
Before this PR, any peer that supported the legacy `/kad/[id]` protocol
was added to the DHT.

This is the first step in removing the support for the legacy kad
protocols.

While I have adjusted unit tests to validate the appropriate behavior,
this still needs proper testing in our stack.

Part of https://github.com/paritytech/polkadot-sdk/issues/504.

cc @paritytech/networking

---------

Signed-off-by: Alexandru Vasile <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Re-applying #2302 after increasing the `MaxPageSize`.  

Remove `without_storage_info` from the XCMP queue pallet. Part of
https://github.com/paritytech/polkadot-sdk/issues/323

Changes:
- Limit the number of messages and signals a HRMP channel can have at
most.
- Limit the number of HRML channels.

A No-OP migration is put in place to ensure that all `BoundedVec`s still
decode and not truncate after upgrade. The storage version is thereby
bumped to 5 to have our tooling remind us to deploy that migration.

## Integration

If you see this error in your try-runtime-cli:  
```pre
Max message size for channel is too large. This means that the V5 migration can be front-run and an
attacker could place a large message just right before the migration to make other messages un-decodable.
Please either increase `MaxPageSize` or decrease the `max_message_size` for this channel. Channel max:
102400, MaxPageSize: 65535
```

Then increase the `MaxPageSize` of the `cumulus_pallet_xcmp_queue` to
something like this:
```rust
type MaxPageSize = ConstU32<{ 103 * 1024 }>;
```

There is currently no easy way for on-chain governance to adjust the
HRMP max message size of all channels, but it could be done:
https://github.com/paritytech/polkadot-sdk/issues/3145.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Francisco Aguirre <[email protected]>

As per #3326, removes pallet::getter usage from the bounties and
child-bounties pallets. The syntax `StorageItem::<T, I>::get()` should
be used instead.

Changes to one pallet involved changes in the other, so I figured it'd
be best to combine these two.

cc @muraca

---------

Co-authored-by: Bastian Köcher <[email protected]>

`cumulus-pallet-dmp-queue` is not needed anymore since
https://github.com/paritytech/polkadot-sdk/pull/1246.

The only logic that remains in the pallet is a lazy migration in the
[`on_idle`](https://github.com/paritytech/polkadot-sdk/blob/8d62c13b

/cumulus/pallets/dmp-queue/src/lib.rs#L158)
hook.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Fix after https://github.com/paritytech/polkadot-sdk/pull/4449

# Context

XCM docs are currently an md book hosted with github pages:
https://paritytech.github.io/xcm-docs/.
While that's fine, it's not in line with the work being done in the
polkadot-sdk docs.

# Main addition

This PR aims to fix that by bringing the docs back to this repo.
This does not have all the information currently present in the mdbook
xcm-docs but aims to be a good chunk of it and fully replace it over
time.

I also added the sections `guides` and `cookbook` which will be very
useful for users wanting to get into XCM.
For now I only added one example to the cookbook, but have ideas for
guides and more examples.
Having this docs be in rust docs is very useful for the cookbook.

# TODO

- [x] Use `FungibleAdapter`
- [x] Improve and relocate mock message queue
- [x] Fix license issue. Why does docs/sdk/ not have this problem? (Just
added the licenses)

# Next steps

- More examples in the cookbook
- End-to-end XCM guide with zombienet testing

---------

Co-authored-by: command-bot <>
Co-authored-by: joe petrowski <[email protected]>

Dear team, hello, 

Please consider this request to add bootnode endpoints to the chainspecs
of `coretime-westend` and `coretime-kusama`.

These bootnodes are operated by the Infrastructure Builders Programme as
part of the commitments of their members.

All these endpoints have been tested according to the procedure
indicated
[here](https://wiki.ibp.network/docs/consumers/bootnodes#testing-a-bootnode).

Additionally, please also consider some maintenance changes to
Gatotech's bootnode endpoint in several other chains.

Many thanks!!

Best regrds!

**_Milos_**

---------

Co-authored-by: hitchhooker <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Add RUNTIME_METADATA_HASH variable to
[`cache.extraEnv`](https://github.com/paritytech/forklift/tree/0.12.4?tab=readme-ov-file#cache)
in config.toml (setting available from forklift 0.12.4)

`ubuntu-latest` are free for public repo so it's better (and cheaper) to
use them for small jobs

This bump of versions:

https://github.com/paritytech/polkadot-sdk/pull/4409/files#diff-13ee4b2252c9e516a0547f2891aa2105c3ca71c6d7a1e682c69be97998dfc87eR11936

reintroduced a dependency to proc-macro-crate 2.0.0 which is suffering
from: https://github.com/bkchr/proc-macro-crate/pull/42 this, so bump
parity-scale-codec to a newer version to eliminate the bad
proc-macro-crate 2.0.0 dependency.

---------

Signed-off-by: Alexandru Gheorghe <[email protected]>
Co-authored-by: command-bot <>

https://github.com/paritytech/polkadot-sdk/pull/3166 made private
functions used in `spawn_tasks()` public but forgot to add them in
exported functions of the crate.

---------

Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

The storage item ExtrinsicIndex is already taken before the `finalize()`
in `note_finished_extrinsics()`, rendering it's always 0 in the log.
This commit fixes it by using the proper API for extrinsics count.

---------

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>