`*-host` and `*-port` are obsolete and we'll hopefully remove them in
the future (already WIP for Rococo <> Westend relayer)

Support async backing in `--dev` mode

This PR improve the relay mock `MockValidationDataInherentDataProvider`
to mach expectations of async backing runtimes.

* Add para_head in the mock relay proof
* Add relay slot in the mock relay proof 

fix https://github.com/paritytech/polkadot-sdk/issues/4437

This PR bumps `proc-macro-crate` to the latest version.

In order to test a runtime from
https://github.com/polkadot-fellows/runtimes/ with the latest version of
polkadot-sdk one needs to use `cargo vendor` to extract all runtime
dependencies, patch them by hand and then build the runtime.

However at the moment 'vendored' builds fail due to
https://github.com/bkchr/proc-macro-crate/issues/48. To fix this
`proc-macro-crate` should be updated to version `3.0.1` or higher.

---------

Co-authored-by: command-bot <>

- Moved check-runtime-migration from gitlab to github
- Moved `test-syscalls` and `cargo-check-all-benches` from gitlab to
github

Reworks prospective-parachains so that we allow a number of unconnected
candidates (for which we don't know the parent candidate yet). Needed
for elastic scaling:
https://github.com/paritytech/polkadot-sdk/issues/3541. Without this,
candidate B will not be validated and backed until candidate A (its
parent) is validated and a backing statement reaches the validator.

Due to the high complexity of the subsystem, I rewrote parts of it so
that we don't concern ourselves with candidates which form cycles or
which form parachain forks. We now have "Fragment chains" instead of
"Fragment trees". This greatly simplifies some of the code and is a
compromise we can make. We just need to make sure that cycle-producing
parachains don't brick the relay chain and that fork-producing
parachains can still make some progress (on one core at least). The only
forks that are allowed are those on the relay chain, obviously.

Unconnected candidates are kept in the `CandidateStorage` and whenever a
new candidate is introduced, we try to repopulate the chain with as many
candidates as we can.

Also fixes https://github.com/paritytech/polkadot-sdk/issues/3219

Guide changes will be done as part of:
https://github.com/paritytech/polkadot-sdk/issues/3699

TODOs:

- [x] see if we can replace the `Cow` over the candidate commitments
with an `Arc` over the entire `ProspectiveCandidate`. It's only being
overwritten in unit tests. We can work around that.
- [x] finish fragment_chain unit tests
- [x] add more prospective-parachains subsystem tests
- [x] test with zombienet what happens if a parachain is creating cycles
(it should not brick the relay chain).
- [x] test with zombienet a parachain that is creating forks. it should
keep producing blocks from time to time (one bad collator should not DOS
the parachain, even if throughput decreases)
- [x] add some more logs and metrics
- [x] add prdoc and remove the "silent" label

---------

Signed-off-by: Andrei Sandu <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>

PR adds forklift settings and forklift to test-github-actions

cc https://github.com/paritytech/ci_cd/issues/939

Currently the `CheckWeight` `SignedExtension` was tracking the size of
the proof and the extrinsic length separately. But in reality we need
one more check that ensures we don't hit the PoV limit with both
combined.

The rest of the logic remains unchanged. One scenario where the changes
make a difference is when we enter this branch:

https://github.com/paritytech/polkadot-sdk/blob/f34d8e3c

/substrate/frame/system/src/extensions/check_weight.rs#L185-L198

This was previously allowing to some extrinsics that is exceeding the
block limit but are withing the reserved area of `BlockWeights`. This
will now be caught by the later check I introduced. I think the new
behaviour makes sense, since the proof size dimension is designed for
parachains and they don't want to go over the limit and get rejected.


In the long run we should maybe get rid of `RuntimeBlockLength`
alltogether, however that would require a deprecation process and can
come at a later point.

---------

Co-authored-by: Adrian Catangiu <[email protected]>

Extracting the logic for generating and verifying ancestry proofs from
this PR: https://github.com/paritytech/polkadot-sdk/pull/1903 with small
adjustments

@Lederstrumpf I added you as a co-author to each commit. Please let me
know if you want me to also associate an e-mail address with your name

The state-trie migration is completed on Rococo Asset-Hub as
double-checked
[here](https://github.com/paritytech/polkadot-sdk/issues/4174#issuecomment-2097895275).
Undeploying now.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Library `substrate-frame-cli` seems unused.

Last non-dependabot update to the folder was over [two years
ago](https://github.com/paritytech/polkadot-sdk/commits/master/substrate/utils/frame/frame-utilities-cli).

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

#### Problem
`take()` consumes only 1 read worth of weight in
`single-block-migrations` example, while `take()`
[is](https://github.com/paritytech/polkadot-sdk/blob/master/substrate/frame/support/src/storage/unhashed.rs#L63)
`get() + kill()`, i.e should be 1 read + 1 write. I think this could
mislead developers who follow this example to write their migrations

---------

Co-authored-by: Bastian Köcher <[email protected]>

The block header is required to derive inherents for a relay chain next
block, this is useful in testing environments.

---------

Co-authored-by: Oliver Tale-Yazdi <[email protected]>

Closes: https://github.com/paritytech/polkadot-sdk/issues/4428

As per #3326, removes pallet::getter usage from the pallet
authority-discovery. The syntax `StorageItem::<T, I>::get()` should be
used instead.

cc @muraca

---------

Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

A part of #3326 

Removes all #[pallet::getter] usage from the contracts mock network
pallet. As the storage values were pub(super), read-only visibility was
lost external to the crate upon the removal of the macros. I have
implemented custom getters as a replacement, keeping the api the same.

If we care very much about consistency of the
storagevalue::<T>::get() syntax, the other option would be to set
the storage values to pub. Though I find preserving data authority
better myself.

@muraca

The fact that this takes two sessions to come into effect is not
obvious. Just added some docs to explain that.

Also tidied up uses of "broker chain" -> "coretime chain"

Closes #1961

Disable logging of rustls to get rid off the following log lines:
```
Sending fatal alert BadCertificate
```

Upstream also removed them: https://github.com/rustls/rustls/pull/1278


Closes: https://github.com/paritytech/polkadot-sdk/issues/3252

This PR adds two new CLI options to disable rate limiting for certain ip
addresses and whether to trust "proxy header".
After going back in forth I decided to use ip addr instead host because
we don't want rely on the host header which can be spoofed but another
solution is to resolve the ip addr from the socket to host name.

Example:

```bash
$ polkadot --rpc-rate-limit 10 --rpc-rate-limit-whitelisted-ips 127.0.0.1/8 --rpc-rate-limit-trust-proxy-headers
```

The ip addr is read from the HTTP proxy headers `Forwarded`,
`X-Forwarded-For` `X-Real-IP` if `--rpc-rate-limit-trust-proxy-headers`
is enabled if that is not enabled or the headers are not found then the
ip address is read from the socket.

//cc @BulatSaif can you test this and give some feedback on it?

This checks changed files against API surface changes against what the
prdoc says.

It will error if the detected semver change is greater than the one
listed in the prdoc. It will also error if any crates were touched but
not mentioned in the prdoc.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>

silent, because it'll be deployed with the
https://github.com/paritytech/polkadot-sdk/pull/4102, where this code
has been introduced

I've planned originally to avoid doing that check in the runtime code,
because it **may be** checked offchain. But actually, the check is quite
cheap and we could do that onchain too.

Adds an additional extrinsic call to the `pallet-balances` to _burn_
tokens.
Depending on the `keep_alive` flag, the call might or might not reap the
account.

Required modification of the _fungible's_ `Mutate` trait, `burn_from`
function to allow the `Preservation` argument.

**TODO**
- [x] run benchmarks & update weights
- [x] make sure prdoc is required & properly formatted

Related issue: https://github.com/paritytech/polkadot-sdk/issues/3943

---------

Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: command-bot <>

# Context

Estimating fees for XCM execution and sending has been an area with bad
UX.
The addition of the
[XcmPaymentApi](https://github.com/paritytech/polkadot-sdk/pull/3607)
exposed the necessary components to be able to estimate XCM fees
correctly, however, that was not the full story.
The `XcmPaymentApi` works for estimating fees only if you know the
specific XCM you want to execute or send.
This is necessary but most UIs want to estimate the fees for extrinsics,
they don't necessarily know the XCM program that's executed by them.

# Main addition

A new runtime API is introduced, the `XcmDryRunApi`, that given an
extrinsic, or an XCM program, returns its effects:
- Execution result
- Local XCM (in the case of an extrinsic)
- Forwarded XCMs
- List of events

This API can be used on its own for dry-running purposes, for
double-checking or testing, but it mainly shines when used in
conjunction with the `XcmPaymentApi`.
UIs can use these two APIs to estimate transfers.

# How it works

New tests are added to exemplify how to incorporate both APIs.
There's a mock test just to make sure everything works under
`xcm-fee-payment-runtime-api`.
There's a real-world test using Westend and AssetHubWestend under
`cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs`.
Added both a test for a simple teleport between chains and a reserve
transfer asset between two parachains going through a reserve.

The steps to follow:
- Use `XcmDryRunApi::dry_run_extrinsic` to get local XCM program and
forwarded messages
- For each forwarded message
- Use `XcmPaymentApi::query_delivery_fee` LOCALLY to get the delivery
fees
- Use `XcmPaymentApi::query_xcm_weight` ON THE DESTINATION to get the
remote execution weight
- (optional) Use `XcmPaymentApi::query_acceptable_payment_assets` ON THE
DESTINATION to know on which assets the execution fees can be paid
- Use `XcmPaymentApi::query_weight_to_asset_fee` ON THE DESTINATION to
convert weight to the actual remote execution fees
- Use `XcmDryRunApi::dry_run_xcm` ON THE DESTINATION to know if a new
message will be forwarded, if so, continue

# Dear reviewer

The changes in this PR are grouped as follows, and in order of
importance:
- Addition of new runtime API
- Definition, mock and simple tests:
polkadot/xcm/xcm-fee-payment-runtime-api/*
- Implemented on Westend, Asset Hub Westend and Penpal, will implement
on every runtime in a following PR
- Addition of a new config item to the XCM executor for recording xcms
about to be executed
  - Definition: polkadot/xcm/xcm-executor/*
  - Implementation: polkadot/xcm/pallet-xcm/*
- had to update all runtime xcm_config.rs files with `type XcmRecorder =
XcmPallet;`
- Addition of a new trait for inspecting the messages in queues
  - Definition: polkadot/xcm/xcm-builder/src/routing.rs
  - Implemented it on all routers:
    - ChildParachainRouter: polkadot/runtime/common/src/xcm_sender.rs
- ParentAsUmp: cumulus/primitives/utility/src/lib.rs (piggybacked on
implementation in cumulus/pallets/parachain-system/src/lib.rs)
    - XcmpQueue: cumulus/pallets/xcmp-queue/src/lib.rs
    - Bridge: bridges/modules/xcm-bridge-hub-router/src/lib.rs
- More complicated and useful tests:
-
cumulus/parachains/integration-tests/emulated/tests/assets/asset-hub-westend/src/tests/xcm_fee_estimation.rs

## Next steps

With this PR, Westend, AssetHubWestend, Rococo and AssetHubRococo have
the new API.
UIs can test on these runtimes to create better experiences around
cross-chain operations.

Next:
- Add XcmDryRunApi to all system parachains
- Integrate xcm fee estimation in all emulated tests
- Get this on the fellowship runtimes

---------

Co-authored-by: Adrian Catangiu <[email protected]>

Addressing comment:
https://github.com/paritytech/polkadot-sdk/pull/3455#issuecomment-2094829076

---------

Co-authored-by: command-bot <>

Mention that offenders are no longer chilled and suggest node operators
and nominators to monitor their nodes/nominees closely.

---------

Co-authored-by: Maciej <[email protected]>

Added manual jobs for code coverage (triggered via `codecov-start` job):
 - **codecov-start** - initialize Codecov report for commit/pr
- **test-linux-stable-codecov** - perform `nextest run` and upload
coverage data parts
- **codecov-finish** - finalize uploading of data parts and generate
Codecov report

Coverage requires code to be built with `-C instrument-coverage` which
causes build errors (e .g. ```error[E0275]: overflow evaluating the
requirement `<mock::Test as pallet::Config>::KeyOwnerProof == _\` ```,
seems like related to
[2641](https://github.com/paritytech/polkadot-sdk/issues/2641)) and
unstable tests behavior
([example](https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6004731)).
This is where we'll nee the developers assistance

closing [[polkadot-sdk] Add code coverage
#902](https://github.com/paritytech/ci_cd/issues/902)

Generated with the script from
https://github.com/paritytech/polkadot-sdk/pull/2931

Edit: previously linked to a very similar PR

Closes: https://github.com/paritytech/polkadot-sdk/issues/4003 (please
see for the problem description)

## TODO
- [x] add more tests covering `WrapVersion` corner cases (e.g. para has
lower version, ...)
- [x] regenerate benchmarks `runtime_parachains::hrmp` (fix for Rococo
is here: https://github.com/paritytech/polkadot-sdk/pull/4332)

## Questions / possible improvements
- [ ] A `WrapVersion` implementation for `pallet_xcm` initiates version
discovery with
[note_unknown_version](https://github.com/paritytech/polkadot-sdk/blob/master/polkadot/xcm/pallet-xcm/src/lib.rs#L2527C5-L2527C25),
there is possibility to avoid this overhead in this HRMP case to create
new `WrapVersion` adapter for `pallet_xcm` which would not use
`note_unknown_version`. Is it worth to do it or not?
- [ ] There's a possibility to decouple XCM functionality from the HRMP
pallet, allowing any relay chain to generate its own notifications. This
approach wouldn't restrict notifications solely to the XCM. However,
it's uncertain whether it's worthwhile or desirable to do so? It means
making HRMP pallet more generic. E.g. hiding HRMP notifications behind
some trait:
	```
	trait HrmpNotifications {

		fn on_channel_open_request(
			sender: ParaId,
			proposed_max_capacity: u32,
			proposed_max_message_size: u32) -> primitives::DownwardMessage;

fn on_channel_accepted(recipient: ParaId) ->
primitives::DownwardMessage;

fn on_channel_closing(initiator: ParaId, sender: ParaId, recipient:
ParaId) -> primitives::DownwardMessage;
	}
	```
and then we could have whatever adapter, `impl HrmpNotifications for
VersionedXcmHrmpNotifications {...}`,
	```
	impl parachains_hrmp::Config for Runtime {
	..
		type HrmpNotifications = VersionedXcmHrmpNotifications;
	..
	}
	```

---------

Co-authored-by: command-bot <>
Co-authored-by: Bastian Köcher <[email protected]>

Improve Create release draft workflow + templates for the free notes and docker images sections in the notes (#4371)

This PR has the following changes:

- New templates for the free notes and docker images sections in the
release notes. There is going to be a section for the manual additions
to the release notes + a section with the links to the docker images for
`polkadot` and `polkadot-parachain` binaries at the end of the release
draft.
- Fix for matrix section in the Create release draft flow (adds the
release environment variable)
- Reduction of the message which is posted to the announcement chats, as
the current one with the full release notes text is too big.

Split from #4374

This PR helps to reduce dependencies and align versions, which would
help to move them to workspace dep

Needed for: https://github.com/polkadot-fellows/runtimes/issues/248

This PR introduces a new type `TypeWithDefault<T, D: Get<T>>` to be able
to provide a custom default for any type. This can, then, be used to
provide the nonce type that returns the current block number as the
default, to avoid replay of immortal transactions.

---------

Co-authored-by: Oliver Tale-Yazdi <[email protected]>

When a staker unbonds and withdraws, it is possible that their stash
will contain less currency than the existential deposit. If that
happens, their stash is reaped. But if the existential deposit is zero,
the reap is not triggered. This PR adjusts `pallet_staking` to reap a
stash in the special case that the stash value is zero and the
existential deposit is zero.

This change is important for blockchains built on substrate that require
an existential deposit of zero, becuase it conserves valued storage
space.

There are two places in which ledgers are checked to determine if their
value is less than the existential deposit and they should be reaped: in
the methods `do_withdraw_unbonded` and `reap_stash`. When the check is
made, the condition `ledger_total == 0` is also checked. If
`ledger_total` is zero, then it must be below any existential deposit
greater than zero and equal to an existential deposit of 0.

I added a new test for each method to confirm the change behaves as
expected.

Closes https://github.com/paritytech/polkadot-sdk/issues/4340

---------

Co-authored-by: command-bot <>

Related to
https://github.com/paritytech/parity-bridges-common/issues/2962
Relay companion:
https://github.com/paritytech/parity-bridges-common/pull/2978

Example usage:
```
./target/release/substrate-relay relay-parachain-head rococo-to-bridge-hub-westend \
    --source-host localhost --source-port 9942 \
    --target-host localhost --target-port 8945 --target-signer //Alice \
    --at-relay-block 61
```