This reverts commit 805d54dd.

https://github.com/paritytech/polkadot-sdk/pull/3166

 made private
functions used in `spawn_tasks()` public but forgot to add them in
exported functions of the crate.

---------

Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

The storage item ExtrinsicIndex is already taken before the `finalize()`
in `note_finished_extrinsics()`, rendering it's always 0 in the log.
This commit fixes it by using the proper API for extrinsics count.

---------

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>

This is for adding onfinality polkadot bootnode. Please correct me if
this is not the right place for adding a new bootnode

This is the second PR in preparation for
https://github.com/paritytech/polkadot-sdk/issues/454.

## Also see
- **Precursor** https://github.com/paritytech/polkadot-sdk/pull/3889.
- **Follow up** https://github.com/paritytech/polkadot-sdk/pull/3905.

Overall changes are documented here (lot more visual 😍):
https://hackmd.io/@ak0n/454-np-governance



## Changes
### Delegation Interface
Provides delegation primitives for staking. 

Introduces two new roles:
- Agent: These are accounts who receive delegation from other accounts
(delegators) and stakes on behalf of them. The funds are held in
delegator accounts.
- Delegator: Accounts who delegate their funds to an agent authorising
them to use it for staking.

Supports
- A way for delegators to add or withdraw delegation to an agent.
- A way for an agent to slash a delegator during a slashing event.

### Pallet Delegated Staking
- Implements `DelegationInterface`.
- Lazy slashing: Any slashes to an Agent is posted in a ledger but not
immediately slashed. The agent can call
`DelegationInterface::delegator_slash` to slash the member and clear the
corresponding slash from its ledger.
- Consumes `StakingInterface` to provide `CoreStaking` features. In
reality, this will be `pallet-staking`.
- Ensures bookkeeping for agent and delegator are correct but leaves the
management of reward and slash logic upto the consumer of this pallet.
- While it does not expose any calls yet, it is written with the intent
of exposing these primitives via extrinsics.

## TODO
- [x] Improve unit tests in the pallet.
- [x] Separate slash reward perbill for rewarding the slash reporters?
- [x] Review if we should add more events.

---------

Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Gonçalo Pestana <[email protected]>
Co-authored-by: georgepisaltu <[email protected]>

Link was dead

---------

Co-authored-by: Bastian Köcher <[email protected]>

Add small fixes, reorg debug definition and add more verbose debug logs
in k8s logs reading.
Thx!

Please add RadiumBlock bootnodes to the spec files for Westend Coretime
and Westend People

---------

Co-authored-by: Dónal Murray <[email protected]>

... to get rid of the macro debug logs like this from build output
```
[/home/alexggh/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fatality-proc-macro-0.1.0/src/types.rs:171:23] input = TokenStream [
    Literal {
        kind: Str,
        symbol: "Error while accessing runtime information",
        suffix: None,
        span: #0 bytes(6943..6986),
    },
]
```

Signed-off-by: Alexandru Gheorghe <[email protected]>

This PR changes the fork calculation and pruning algorithm to enable
future block header pruning. It's required because the previous
algorithm relied on the block header persistence. It follows the
[related
discussion](https://github.com/paritytech/polkadot-sdk/issues/1570

)

The previous code contained this comment describing the situation:
```
	/// Note a block height finalized, displacing all leaves with number less than the finalized
	/// block's.
	///
	/// Although it would be more technically correct to also prune out leaves at the
	/// same number as the finalized block, but with different hashes, the current behavior
	/// is simpler and our assumptions about how finalization works means that those leaves
	/// will be pruned soon afterwards anyway.
	pub fn finalize_height(&mut self, number: N) -> FinalizationOutcome<H, N> {
```

The previous algorithm relied on the existing block headers to prune
forks later and to enable block header pruning we need to clear all
obsolete forks right after the block finalization to not depend on the
related block headers in the future.

---------

Co-authored-by: Bastian Köcher <[email protected]>

A small step towards
https://forum.polkadot.network/t/polkadot-parachain-omni-node-gathering-ideas-and-feedback/7823


and #4352.

Many parachains use `camelCase` and/or `PascalCase`ing for their chain
spec extension. Sometimes the only reason polkadot-parachain cannot sync
them is because it cannot parse the chain spec extension. This PR
relaxes the requirement for the extension to be camel case.

---------

Co-authored-by: Branislav Kontur <[email protected]>

Bumps [nix](https://github.com/nix-rust/nix) from 0.27.1 to 0.28.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/nix-rust/nix/blob/master/CHANGELOG.md">nix's
changelog</a>.</em></p>
<blockquote>
<h2>[0.28.0] - 2024-02-24</h2>
<h3>Added</h3>
<ul>
<li>
<p>Added <code>mkdtemp</code> wrapper (<a
href="https://redirect.github.com/nix-rust/nix/pull/1297">#1297</a>)</p>
</li>
<li>
<p>Add associated constants <code>UTIME_OMIT</code>
<code>UTIME_NOW</code> for <code>TimeSpec</code>
(<a
href="https://redirect.github.com/nix-rust/nix/pull/1879">#1879</a>)</p>
</li>
<li>
<p>Added <code>EventFd</code> type. (<a
href="https://redirect.github.com/nix-rust/nix/pull/1945">#1945</a>)</p>
</li>
<li>
<ul>
<li>Added <code>impl From&lt;Signal&gt; for SigSet</code>.</li>
<li>Added <code>impl std::ops::BitOr for SigSet</code>.</li>
<li>Added <code>impl std::ops::BitOr for Signal</code>.</li>
<li>Added <code>impl std::ops::BitOr&lt;Signal&gt; for
SigSet</code></li>
</ul>
<p>(<a
href="https://redirect.github.com/nix-rust/nix/pull/1959">#1959</a>)</p>
</li>
<li>
<p>Added <code>TlsGetRecordType</code> control message type and
corresponding enum for
linux (<a
href="https://redirect.github.com/nix-rust/nix/pull/2065">#2065</a>)</p>
</li>
<li>
<p>Added <code>Ipv6HopLimit</code> to
<code>::nix::sys::socket::ControlMessage</code> for Linux,
MacOS, FreeBSD, DragonflyBSD, Android, iOS and Haiku.
(<a
href="https://redirect.github.com/nix-rust/nix/pull/2074">#2074</a>)</p>
</li>
<li>
<p>Added <code>Icmp</code> and <code>IcmpV6</code> to
<code>SockProtocol</code>
(<a
href="https://redirect.github.com/nix-rust/nix/pull/2103">#2103</a>)</p>
</li>
<li>
<p>Added rfork support for FreeBSD in <code>unistd</code>
(<a
href="https://redirect.github.com/nix-rust/nix/pull/2121">#2121</a>)</p>
</li>
<li>
<p>Added <code>MapFlags::map_hugetlb_with_size_log2</code> method for
Linux targets
(<a
href="https://redirect.github.com/nix-rust/nix/pull/2125">#2125</a>)</p>
</li>
<li>
<p>Added <code>mmap_anonymous</code> function
(<a
href="https://redirect.github.com/nix-rust/nix/pull/2127">#2127</a>)</p>
</li>
<li>
<p>Added <code>mips32r6</code> and <code>mips64r6</code> support for
signal, ioctl and ptrace
(<a
href="https://redirect.github.com/nix-rust/nix/pull/2138">#2138</a>)</p>
</li>
<li>
<p>Added <code>F_GETPATH</code> FcntlFlags entry on
Apple/NetBSD/DragonflyBSD for
<code>::nix::fcntl</code>. (<a
href="https://redirect.github.com/nix-rust/nix/pull/2142">#2142</a>)</p>
</li>
<li>
<p>Added <code>F_KINFO</code> FcntlFlags entry on FreeBSD for
<code>::nix::fcntl</code>.
(<a
href="https://redirect.github.com/nix-rust/nix/pull/2152">#2152</a>)</p>
</li>
<li>
<p>Added <code>F_GETPATH_NOFIRMLINK</code> and
<code>F_BARRIERFSYNC</code> FcntlFlags entry
on Apple for <code>::nix::fcntl</code>.
(<a
href="https://redirect.github.com/nix-rust/nix/pull/2155">#2155</a>)</p>
</li>
<li>
<p>Added newtype <code>Flock</code> to automatically unlock a held flock
upon drop.
Added <code>Flockable</code> trait to represent valid types for
<code>Flock</code>.
(<a
href="https://redirect.github.com/nix-rust/nix/pull/2170">#2170</a>)</p>
</li>
<li>
<p>Added <code>SetSockOpt</code> impls to enable Linux Kernel TLS on a
TCP socket and to
import TLS parameters. (<a
href="https://redirect.github.com/nix-rust/nix/pull/2175">#2175</a>)</p>
</li>
<li>
<ul>
<li>Added the <code>::nix::sys::socket::SocketTimestamp</code> enum for
configuring the
<code>TsClock</code> (a.k.a <code>SO_TS_CLOCK</code>) sockopt</li>
<li>Added FreeBSD's <code>ScmRealtime</code> and
<code>ScmMonotonic</code> as new options in
<code>::nix::sys::socket::ControlMessageOwned</code></li>
</ul>
<p>(<a
href="https://redirect.github.com/nix-rust/nix/pull/2187">#2187</a>)</p>
</li>
<li>
<p>Added new fanotify API: wrappers for <code>fanotify_init</code> and
<code>fanotify_mark</code>
(<a
href="https://redirect.github.com/nix-rust/nix/pull/2194">#2194</a>)</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/nix-rust/nix/commit/21ab06ef23de214174ddb039be5b5f08750d19e6"><code>21ab06e</code></a>
chore: release 0.8.0</li>
<li><a
href="https://github.com/nix-rust/nix/commit/595c6eb2515723eae1f1db1b9103c95f339380ec"><code>595c6eb</code></a>
chore: changelog for 0.28.0 (<a
href="https://redirect.github.com/nix-rust/nix/issues/2303">#2303</a>)</li>
<li><a
href="https://github.com/nix-rust/nix/commit/79b86a7dff071a2a59858d1c98e202fdf38ebd06"><code>79b86a7</code></a>
mmsg revert signature simplification (<a
href="https://redirect.github.com/nix-rust/nix/issues/2227">#2227</a>)</li>
<li><a
href="https://github.com/nix-rust/nix/commit/08e05a5f66d14d4f06288c9a725e87f473eba41a"><code>08e05a5</code></a>
refactor: remove redundant imports to fix CI (<a
href="https://redirect.github.com/nix-rust/nix/issues/2320">#2320</a>)</li>
<li><a
href="https://github.com/nix-rust/nix/commit/197f55b3ccbce3273bf6ce119d1a8541b5df5d66"><code>197f55b</code></a>
refactor: update nix (<a
href="https://redirect.github.com/nix-rust/nix/issues/2311">#2311</a>)</li>
<li><a
href="https://github.com/nix-rust/nix/commit/f34d00d353a8ebcb16eb8810b00d613e4a294925"><code>f34d00d</code></a>
chore(deps): try libc 0.2.153 (<a
href="https://redirect.github.com/nix-rust/nix/issues/2309">#2309</a>)</li>
<li><a
href="https://github.com/nix-rust/nix/commit/062f6c0f0306743a401138fc2612d59e818d5768"><code>062f6c0</code></a>
test: clean clippy needless_borrow false positive issue (<a
href="https://redirect.github.com/nix-rust/nix/issues/2310">#2310</a>)</li>
<li><a
href="https://github.com/nix-rust/nix/commit/ca173ff30cc42e0fc7a7ba2cf7c1fcbe61dccfa5"><code>ca173ff</code></a>
Fix the mount tests (<a
href="https://redirect.github.com/nix-rust/nix/issues/2269">#2269</a>)</li>
<li><a
href="https://github.com/nix-rust/nix/commit/0dfcf3228489a5060a53e69783a1eea6f43b11a9"><code>0dfcf32</code></a>
test: tweak test_fanotify to monitor ONLY 1 file (<a
href="https://redirect.github.com/nix-rust/nix/issues/2294">#2294</a>)</li>
<li><a
href="https://github.com/nix-rust/nix/commit/0fb346031f6ddc87a57377101a6063133a25e443"><code>0fb3460</code></a>
Temporarily disable CI on GNU hurd (<a
href="https://redirect.github.com/nix-rust/nix/issues/2307">#2307</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/nix-rust/nix/compare/v0.27.1...v0.28.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=nix&package-manager=cargo&previous-version=0.27.1&new-version=0.28.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores

)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

related to
https://github.com/paritytech/parity-bridges-common/issues/2962
Usage example:
```sh
RUST_LOG=runtime=trace,rpc=trace,bridge=trace \
        ./target/release/substrate-relay relay-messages-range bridge-hub-rococo-to-bridge-hub-westend \
        --source-host localhost \
        --source-port 8943 \
        --source-version-mode Auto \
        --source-signer //Eve \
        --source-transactions-mortality 4 \
        --target-host localhost \
        --target-port 8945 \
        --target-version-mode Auto \
        --target-signer //Eve \
        --target-transactions-mortality 4 \
        --lane 00000002 \
        --at-source-block 34 \
        --messages-start 1 \
        --messages-end 1
INFO bridge Connecting to BridgeHubRococo node at ws://localhost:8943
INFO bridge Connecting to BridgeHubWestend node at ws://localhost:8945
TRACE bridge Refined weight of BridgeHubRococo->BridgeHubWestend message [0, 0, 0, 2]/1: at-source: Weight(ref_time: 0, proof_size: 0), at-target: Weight(ref_time: 452953993, proof_size: 0)
TRACE bridge Sent transaction to BridgeHubWestend node: 0x38552f4db6bc78baecb52ebd2f7d103b1c919c16b83129dc083bf01b7281955b
TRACE bridge BridgeHubWestend transaction 0x38552f4db6bc78baecb52ebd2f7d103b1c919c16b83129dc083bf01b7281955b has been included in block: (0x29a20bdca8726df0b32af9067290b7fc0a886908da3a30f3db60a6ea52be4604, 0)
TRACE bridge BridgeHubWestend transaction 0x38552f4db6bc78baecb52ebd2f7d103b1c919c16b83129dc083bf01b7281955b has been finalized at block: 0x29a20bdca8726df0b32af9067290b7fc0a886908da3a30f3db60a6ea52be4604
```

Github currently [doesn't support creating
subfolders](https://github.com/orgs/community/discussions/18055) in
`.github/workflows` folder. PR unifies naming in this folder. The idea
is that all files ideally should have naming
`<stage>-<short_descriptive_name>.yml`. The stage names are taken from
gitlab and needed only for logical division.
Changes in actions:
 - `check-licences` runs on self-hosted runners
- `check-workspace` and `check-markdown` are moved to `checks-quick.yml`
 - `test-linux-stable-int` moved to a new file `tests-linux-stable.yml`
- `fmt-check.yml` file is removed, formatting is checked in
`checks-quick.yml`

`*-host` and `*-port` are obsolete and we'll hopefully remove them in
the future (already WIP for Rococo <> Westend relayer)

Support async backing in `--dev` mode

This PR improve the relay mock `MockValidationDataInherentDataProvider`
to mach expectations of async backing runtimes.

* Add para_head in the mock relay proof
* Add relay slot in the mock relay proof 

fix https://github.com/paritytech/polkadot-sdk/issues/4437

This PR bumps `proc-macro-crate` to the latest version.

In order to test a runtime from
https://github.com/polkadot-fellows/runtimes/ with the latest version of
polkadot-sdk one needs to use `cargo vendor` to extract all runtime
dependencies, patch them by hand and then build the runtime.

However at the moment 'vendored' builds fail due to
https://github.com/bkchr/proc-macro-crate/issues/48. To fix this
`proc-macro-crate` should be updated to version `3.0.1` or higher.

---------

Co-authored-by: command-bot <>

- Moved check-runtime-migration from gitlab to github
- Moved `test-syscalls` and `cargo-check-all-benches` from gitlab to
github

Reworks prospective-parachains so that we allow a number of unconnected
candidates (for which we don't know the parent candidate yet). Needed
for elastic scaling:
https://github.com/paritytech/polkadot-sdk/issues/3541. Without this,
candidate B will not be validated and backed until candidate A (its
parent) is validated and a backing statement reaches the validator.

Due to the high complexity of the subsystem, I rewrote parts of it so
that we don't concern ourselves with candidates which form cycles or
which form parachain forks. We now have "Fragment chains" instead of
"Fragment trees". This greatly simplifies some of the code and is a
compromise we can make. We just need to make sure that cycle-producing
parachains don't brick the relay chain and that fork-producing
parachains can still make some progress (on one core at least). The only
forks that are allowed are those on the relay chain, obviously.

Unconnected candidates are kept in the `CandidateStorage` and whenever a
new candidate is introduced, we try to repopulate the chain with as many
candidates as we can.

Also fixes https://github.com/paritytech/polkadot-sdk/issues/3219

Guide changes will be done as part of:
https://github.com/paritytech/polkadot-sdk/issues/3699



TODOs:

- [x] see if we can replace the `Cow` over the candidate commitments
with an `Arc` over the entire `ProspectiveCandidate`. It's only being
overwritten in unit tests. We can work around that.
- [x] finish fragment_chain unit tests
- [x] add more prospective-parachains subsystem tests
- [x] test with zombienet what happens if a parachain is creating cycles
(it should not brick the relay chain).
- [x] test with zombienet a parachain that is creating forks. it should
keep producing blocks from time to time (one bad collator should not DOS
the parachain, even if throughput decreases)
- [x] add some more logs and metrics
- [x] add prdoc and remove the "silent" label

---------

Signed-off-by: Andrei Sandu <[email protected]>
Co-authored-by: Andrei Sandu <[email protected]>

PR adds forklift settings and forklift to test-github-actions

cc https://github.com/paritytech/ci_cd/issues/939

Currently the `CheckWeight` `SignedExtension` was tracking the size of
the proof and the extrinsic length separately. But in reality we need
one more check that ensures we don't hit the PoV limit with both
combined.

The rest of the logic remains unchanged. One scenario where the changes
make a difference is when we enter this branch:

https://github.com/paritytech/polkadot-sdk/blob/f34d8e3cf033e2a22a41b505c437972a5dc83d78/substrate/frame/system/src/extensions/check_weight.rs#L185-L198



This was previously allowing to some extrinsics that is exceeding the
block limit but are withing the reserved area of `BlockWeights`. This
will now be caught by the later check I introduced. I think the new
behaviour makes sense, since the proof size dimension is designed for
parachains and they don't want to go over the limit and get rejected.


In the long run we should maybe get rid of `RuntimeBlockLength`
alltogether, however that would require a deprecation process and can
come at a later point.

---------

Co-authored-by: Adrian Catangiu <[email protected]>

Extracting the logic for generating and verifying ancestry proofs from
this PR: https://github.com/paritytech/polkadot-sdk/pull/1903 with small
adjustments

@Lederstrumpf I added you as a co-author to each commit. Please let me
know if you want me to also associate an e-mail address with your name

The state-trie migration is completed on Rococo Asset-Hub as
double-checked
[here](https://github.com/paritytech/polkadot-sdk/issues/4174#issuecomment-2097895275

).
Undeploying now.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

Library `substrate-frame-cli` seems unused.

Last non-dependabot update to the folder was over [two years
ago](https://github.com/paritytech/polkadot-sdk/commits/master/substrate/utils/frame/frame-utilities-cli

).

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

#### Problem
`take()` consumes only 1 read worth of weight in
`single-block-migrations` example, while `take()`
[is](https://github.com/paritytech/polkadot-sdk/blob/master/substrate/frame/support/src/storage/unhashed.rs#L63

)
`get() + kill()`, i.e should be 1 read + 1 write. I think this could
mislead developers who follow this example to write their migrations

---------

Co-authored-by: Bastian Köcher <[email protected]>

The block header is required to derive inherents for a relay chain next
block, this is useful in testing environments.

---------

Co-authored-by: Oliver Tale-Yazdi <[email protected]>

Closes: https://github.com/paritytech/polkadot-sdk/issues/4428

As per #3326, removes pallet::getter usage from the pallet
authority-discovery. The syntax `StorageItem::<T, I>::get()` should be
used instead.

cc @muraca

---------

Co-authored-by: Liam Aharon <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

A part of #3326 

Removes all #[pallet::getter] usage from the contracts mock network
pallet. As the storage values were pub(super), read-only visibility was
lost external to the crate upon the removal of the macros. I have
implemented custom getters as a replacement, keeping the api the same.

If we care very much about consistency of the
storagevalue::<T>::get() syntax, the other option would be to set
the storage values to pub. Though I find preserving data authority
better myself.

@muraca

The fact that this takes two sessions to come into effect is not
obvious. Just added some docs to explain that.

Also tidied up uses of "broker chain" -> "coretime chain"

Closes #1961

Disable logging of rustls to get rid off the following log lines:
```
Sending fatal alert BadCertificate
```

Upstream also removed them: https://github.com/rustls/rustls/pull/1278


Closes: https://github.com/paritytech/polkadot-sdk/issues/3252