**Overview:**
Adding an extra malus variant focusing on disputing finalized blocks. It
will:
- wrap around approval-voting
- listen to `OverseerSignal::BlockFinalized` and when encountered start
a dispute for the `dispute_offset`th ancestor
- simply pass through all other messages and signals

Add zombienet tests testing various edgecases:
- disputing freshly finalized blocks
- disputing stale finalized blocks
- disputing eagerly pruned finalized blocks (might be separate PR)

**TODO:**
- [x] Register new malus variant
- [x] Simple pass through wrapper (approval-voting)
- [x] Simple network definition
- [x] Listen to block finalizations
- [x] Fetch ancestor hash
- [x] Fetch session index
- [x] Fetch candidate
- [x] Construct and send dispute message
- [x] zndsl test 1 checking that disputes on fresh finalizations resolve
valid Closes #1365
- [x] zndsl test 2 checking that disputes for too old finalized blocks
are not possible Closes #1364
- [ ] zndsl test 3 checking that disputes for candidates with eagerly
pruned relay parent state are handled correctly #1359 (deferred to a
separate PR)
- [x] Unit tests for new malus variant (testing cli etc)
- [x] Clean/streamline error handling
- [ ] ~~Ensure it tests properly on session boundaries~~

---------

Co-authored-by: Javier Viola <[email protected]>
Co-authored-by: Marcin S. <[email protected]>
Co-authored-by: Tsvetomir Dimitrov <[email protected]>

## Overview

This PR aligns the `spec_version` formatting to the [recent
changes](https://github.com/polkadot-fellows/runtimes/pull/26/files#diff-efa4caeb17487ecb13d8f5eb7863c3241d84afa2e73fbf25909a2ca89df0f362R142

)
made for the Polkadot/Kusama runtimes.

It also backports the latest version `v1.4.0` bumps as `1_004_000`.

## Details

During the switch from `v0.9` to `v1.x`, the format of the
`spec_version` was modified from: `(M)m_ppp` for a runtime considered on
version `M.m.pp`. For instance `0.9.42` had a `spec_version` of `9420`.

With the transition to `v1.x`, the format was changed to a bigger number
(still `u32`) formatted as `MM_mm_ppp` where `1.2.3` would be stored as
`01_02_003`.

This PR aligns the format with that has been introduced in the
fellowship repo: `MMM_mmm_ppp`.

---------

Co-authored-by: Bastian Köcher <[email protected]>

# Description

Documentation for the TxPause and SafeMode pallets.

Based on reading and the notes from the following related PRs:
- https://github.com/paritytech/substrate/pull/12092



I believe this also completes the checklist to be able to close the
related PRs:
- Close #302 
- Close #274

---------

Co-authored-by: joe petrowski <[email protected]>

This PR removes some `ChainSpec` types which are not necessary
(left-overs from #1256). Currently `ChainSpec` does not have to be
generic over the specific `RuntimeGenesisConfig`, it is enough to use
single type for all:

https://github.com/paritytech/polkadot-sdk/blob/9f787018857660440182142adc806954d7d07709/cumulus/polkadot-parachain/src/chain_spec/mod.rs#L53-L54


related to: https://github.com/paritytech/polkadot-sdk/issues/25

---------

Co-authored-by: command-bot <>

Add `build-push-image-polkadot-debug` job to needs since we changed to
use the polkadot-debug image from the current branch for cumulus test
and we need to be sure that the image is ready. Fix issues like
https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4481059

cc: @bkchr



Co-authored-by: Bastian Köcher <[email protected]>

Otherwise the user may runs into weird errors if there is already a db.

Include fix for `0002-parachains-upgrade-smoke-test` test.

The following members have been added:

```pre
cumulus/parachains/integration-tests/emulated/chains/parachains/testing/penpal
cumulus/parachains/testnets-common
polkadot/node/tracking-allocator
substrate/frame/examples/frame-crate
```

CI check can be added after
https://github.com/paritytech/pipeline-scripts/pull/105

 is merged.

---------

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

The last issue blocking the removal of the Polkadot and Kusama system
parachains from the repo in #1737 is the dependency on the runtime code
through the RuntimeApi in `polkadot-parachain`.

This PR introduces two fake runtimes to satisfy the build requirements
and changes the `new_partial` function to make it not be generic over
the runtimes.
The reason for the second runtime is the different Aura keys used in
Polkadot Asset Hub, as the impl for AuraApi depends on this type.
If this changes the `RuntimeApi` generic could be removed completely
from all functions in `services.rs` and and generic type parameters in
`services.rs` and specified as a concrete type to TFullClient`.

---------

Co-authored-by: Bastian Köcher <[email protected]>

The FastUnstake pallet tests were previously directly modifying storage
items in the pallet-staking to alter exposure. However, due to the
introduction of the [new paged exposure
feature](https://github.com/paritytech/polkadot-sdk/pull/1189) these
tests were not testing against correct storage items. This issue
resulted in a bug that I didn't catch, which has been addressed in [this
fix](https://github.com/paritytech/polkadot-sdk/pull/2369).

This PR introduces a modification to how the pallet-fast-unstake handles
exposure. It now utilizes `pallet-staking::EraInfo` to set or mutate
Exposures.

Use non-zero weight limit in benchmarking `pallet_xcm::execute()` so the
given XCM actually executes.

# Description

Needed this in my code as part of the larger data structure.

---------

Co-authored-by: command-bot <>

Printing the `unpin_worker_sender` included the entire stacktrace and
that is a little bit too verbose.

After #2001 the `fork_choice` strategy may remain uninitialized in the
block import pipeline which uses relay-chain verifier.
Refer to
https://github.com/paritytech/polkadot-sdk/pull/2430#issuecomment-1823979813
for some further discussion.

DMP queue migration is complete and the pallet should be removed to fix
the check runtime upgrade CI.

<img width="1501" alt="Screenshot 2023-11-24 at 16 05 37"
src="https://github.com/paritytech/polkadot-sdk/assets/16665596/ee1da6bb-2756-4423-8085-1e4c73553ad5">

<img width="1501" alt="Screenshot 2023-11-24 at 16 06 44"
src="https://github.com/paritytech/polkadot-sdk/assets/16665596/03f3e88f-aed8-4eaa-aab6-7998f72258be">

I noticed that this test broke at some point. The parachain nodes should
only acquire their blocks from the relay chain. But they were connecting
to their peers and started fetching blocks from there.

In this test I now take additional measures so we check that each nodes
really uses pov-recovery to get the blocks.

closes #2194 

cc @mrcnski



---------

Co-authored-by: Bastian Köcher <[email protected]>

This PR converts unsafe math operations to safe in the staking pallet.
 
Closes https://github.com/paritytech/polkadot-sdk/issues/2431



---------

Co-authored-by: Ankan <[email protected]>

On extrinsics/call, ensure local XCM execution is complete/successful.
Otherwise, fail the extrinsic so that state changes don't get committed
to the db.

Added regression tests that fail without the fix.

fixes #2237

---------

Co-authored-by: Adrian Catangiu <[email protected]>

Amends some staking pallet docs, and deprecation comment, to adjust to
the latest controller deprecation state.

Note, do we need the `README.md` file, which is a duplicate of the
pallet docs? Docs would be easier to maintain, and less ambiguity for
devs to refer to, if we had one source of truth in the generated pallet
docs.

Broken after #2446.

This moves the macro related re-exports to `__private` to make it more
obvious for downstream users that they are using an internal api.

---------

Co-authored-by: command-bot <>

CI: Disable runtime upgrade spec name check on Westend Asset Hub and fix Staking pallet migration (#2447)

Westend Asset Hub currently failing because the spec name is being
changed next runtime upgrade
(https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4413125).

This also fixes an idempotency issue with a staking pallet migration.
Similar issues will be caught automatically now that we've also updated
to try-runtime-cli v0.5.0 which checks for idempotency issues.

This also enables try-state checks running in the CI.

Collectives Westend is missing an `on-chain-release-build` feature flag.

This PR removes `#[macro_use]` from the service module in each of the
Substrate nodes in the repo.

* Parachain Template
* Polkadot Parachain
* Minimal Node
* Node Template
* Kitchen Sink Node

IDK why this annotation was present, maybe from when we had the
`new_partial!` macro?

---------

Co-authored-by: Joshy Orndorff <[email protected]>

The out-dated version (bad tag) of [polkadot
image](https://github.com/paritytech/polkadot-sdk/blob/ede4a362622dfa69eb167eaa876246b1289f4b41/.gitlab/pipeline/zombienet/cumulus.yml#L31)
([docker
info](https://hub.docker.com/layers/paritypr/polkadot-debug/master/images/sha256:adb1658052cf671b50c90d5cece5c7a131efa1a95978249bd5cb85a5ad654f7a?context=explore))
was used. This PR fixes this.

See also:
https://github.com/paritytech/polkadot-sdk/pull/2411#issuecomment-1822632724



Also adds an abstraction that allows asynchronous backends to be passed to `ChainApiSubsystem`
---------

Co-authored-by: Sebastian Kunert <[email protected]>

Add the possibility to inject additional key-values in the
sproof-builder that generates the relay root that gets stored in
parachain-system.

Rationale: pallets that verify additional storage items (not those
verified by parachain-system) from the relay should be able to proof
against the relay root that gets stored in parachain-system. This PR
allows to create provide additional nibles that can later be used for
verifiability in other pallets

The `SkipCheckifFeeless::IDENTIFIER` became transparent (ie was whatever
the inner signed ext was). This PR just makes the `TypeInfo` transparent
too, so that libraries that use said info to decode the data (ie subxt)
can behave identically whether or not the `SkipCheckifFeeless` wrapper
is used or not.

---------

Co-authored-by: Oliver Tale-Yazdi <[email protected]>

Fully-qualified path was not being used in benchmark macro for one of
the cases. This PR fixes this and removes the unnecessary import in a
couple of files, which I believe was done to fix this issue.

Docker images from paritypr are also used in testnets. PR reverts docker
tags naming to a more convenient.

cc @PierreBesson

Deprecates `RewardDestination::Controller` variant.

- [x] `RewardDestination::Controller` annotated with `#[deprecated]`.
- [x] `Controller` variant is now handled the same way as `Stash` in
`payout_stakers`.
- [x] `set_payee` errors if `RewardDestination::Controller` is provided.
- [x] Added `update_payee` call to lazily migrate
`RewardDestination::Controller` `Payee` storage entries to
`RewardDestination::Account(controller)` .
- [x] `payout_stakers_dead_controller` has been removed from benches &
weights - was not used.
- [x] Tests no longer use `RewardDestination::Controller`.

---------

Co-authored-by: command-bot <>
Co-authored-by: Gonçalo Pestana <[email protected]>
Co-authored-by: georgepisaltu <[email protected]>

This is follow-up for:
https://github.com/paritytech/polkadot-sdk/pull/2001

Block import queue for `test-parachain` (`cumulus-test-service`) shall
use delayed best block feature.

This should fixed broken zombienet tests.

This updates the tick runtime and polkadot-parachain collator to use
async backing.

This action will trigger when a merge queue is created and will add a
positive status check under the `review-bot` account, allowing the PR to
pass the required check for the merge.

---------

Co-authored-by: Chevdor <[email protected]>

Adding on top of the new builder pattern for creating XCM programs, I'm
adding some more APIs:

```rust
let paying_fees: Xcm<()> = Xcm::builder() // Only allow paying for fees
  .withdraw_asset() // First instruction has to load the holding register
  .buy_execution() // Second instruction has to be `buy_execution`
  .build();

let paying_fees_invalid: Xcm<()> = Xcm::builder()
  .withdraw_asset()
  .build(); // Invalid, need to pay for fees

let not_paying_fees: Xcm<()> = Xcm::builder_unpaid()
  .unpaid_execution() // Needed
  .withdraw_asset()
  .deposit_asset()
  .build();

let all_goes: Xcm<()> = Xcm::builder_unsafe() // You can do anything
  .withdraw_asset()
  .deposit_asset()
  .build();
```

The invalid bits are because the methods don't even exist on the types
that you'd want to call them on.

---------

Co-authored-by: command-bot <>

This PR adds the `delayed_best_block` flag to `ParachainBlockImport`. If
not set, the `params.fork_choice` is not updated (to
`ForkChoiceStrategy::Custom`) during the block import.

When `delayed_best_block` is set to `false` all parachain blocks on the
[longest
fork](https://github.com/paritytech/polkadot-sdk/blob/552be4800d9e4b480f79a300fc531783e04be099/substrate/client/service/src/client/client.rs#L708-L709

)
will be notified as the best block, allowing transaction pool to be
updated with every imported block.

Otherwise imported blocks will not be notified as best blocks
(`fork_choice=ForkChoiceStrategy::Custom(false)`), and transaction pool
would be updated only with best block received from relay-chain.

Improvement for: #1202

---------

Co-authored-by: Bastian Köcher <[email protected]>