Skip to content
  1. Mar 06, 2024
  3. Mar 05, 2024
  5. Mar 04, 2024
  7. Mar 03, 2024
  9. Mar 02, 2024
    • gupnik's avatar
      Remove `as frame_system::DefaultConfig` from the required syntax in `derive_impl` (#3505) · cdc8d197
      gupnik authored 
      Step in https://github.com/paritytech/polkadot-sdk/issues/171

This PR removes the need to specify `as [disambiguation_path]` for cases
where the trait definition resides within the same scope as default impl
path.

For example, in the following macro invocation
```rust
#[derive_impl(frame_system::config_preludes::TestDefaultConfig as frame_system::DefaultConfig)]
impl frame_system::Config for Runtime {
   ...
}
```
the trait `DefaultConfig` lies within the `frame_system` scope and
`TestDefaultConfig` impls the `DefaultConfig` trait. Using this
information, we can compute the disambiguation path internally, thus
removing the need of an explicit specification.

In cases where the trait lies outside this scope, we would still need to
specify it explicitly, but this should take care of most (if not all)
uses of `derive_impl` within FRAME's context.
      cdc8d197
  11. Mar 01, 2024
    • Alin Dima's avatar
      provisioner: allow multiple cores assigned to the same para (#3233) · 62b78a16
      Alin Dima authored 
      https://github.com/paritytech/polkadot-sdk/issues/3130

builds on top of https://github.com/paritytech/polkadot-sdk/pull/3160

Processes the availability cores and builds a record of how many
candidates it should request from prospective-parachains and their
predecessors.
Tries to supply as many candidates as the runtime can back. Note that
the runtime changes to back multiple candidates per para are not yet
done, but this paves the way for it.

The following backing/inclusion policy is assumed:
1. the runtime will never back candidates of the same para which don't
form a chain with the already backed candidates. Even if the others are
still pending availability. We're optimistic that they won't time out
and we don't want to back parachain forks (as the complexity would be
huge).
2. if a candidate is timed out of the core before being included, all of
its successors occupying a core will be evicted.
3. only the candidates which are made available and form a chain
starting from the on-chain para head may be included/enacted and cleared
from the cores. In other words, if para head is at A and the cores are
occupied by B->C->D, and B and D are made available, only B will be
included and its core cleared. C and D will remain on the cores awaiting
for C to be made available or timed out. As point (2) above already
says, if C is timed out, D will also be dropped.
4. The runtime will deduplicate candidates which form a cycle. For
example if the provisioner supplies candidates A->B->A, the runtime will
only back A (as the state output will be the same)

Note that if a candidate is timed out, we don't guarantee that in the
next relay chain block the block author will be able to fill all of the
timed out cores of the para. That increases complexity by a lot.
Instead, the provisioner will supply N candidates where N is the number
of candidates timed out, but doesn't include their successors which will
be also deleted by the runtime. This'll be backfilled in the next relay
chain block.

Adjacent changes:
- Also fixes: https://github.com/paritytech/polkadot-sdk/issues/3141
- For non prospective-parachains, don't supply multiple candidates per
para (we can't have elastic scaling without prospective parachains
enabled). paras_inherent should already sanitise this input but it's
more efficient this way.

Note: all of these changes are backwards-compatible with the
non-elastic-scaling scenario (one core per para).
      62b78a16
    • Andrei Eres's avatar
      subsystem-bench: add regression tests for availability read and write (#3311) · f0e589d7
      Andrei Eres authored 
      ### What's been done
- `subsystem-bench` has been split into two parts: a cli benchmark
runner and a library.
- The cli runner is quite simple. It just allows us to run `.yaml` based
test sequences. Now it should only be used to run benchmarks during
development.
- The library is used in the cli runner and in regression tests. Some
code is changed to make the library independent of the runner.
- Added first regression tests for availability read and write that
replicate existing test sequences.

### How we run regression tests
- Regression tests are simply rust integration tests without the
harnesses.
- They should only be compiled under the `subsystem-benchmarks` feature
to prevent them from running with other tests.
- This doesn't work when running tests with `nextest` in CI, so
additional filters have been added to the `nextest` runs.
- Each benchmark run takes a different time in the beginning, so we
"warm up" the tests until their CPU usage differs by only 1%.
- After the warm-up, we run the benchmarks a few more times and compare
the average with the exception using a precision.

### What is still wrong?
- I haven't managed to set up approval voting tests. The spread of their
results is too large and can't be narrowed down in a reasonable amount
of time in the warm-up phase.
- The tests start an unconfigurable prometheus endpoint inside, which
causes errors because they use the same 9999 port. I disable it with a
flag, but I think it's better to extract the endpoint launching outside
the test, as we already do with `valgrind` and `pyroscope`. But we still
use `prometheus` inside the tests.

### Future work
* https://github.com/paritytech/polkadot-sdk/issues/3528
* https://github.com/paritytech/polkadot-sdk/issues/3529
* https://github.com/paritytech/polkadot-sdk/issues/3530
* https://github.com/paritytech/polkadot-sdk/issues/3531



---------

Co-authored-by: default avatarAlexander Samusev <[email protected]>
      f0e589d7
    • Xiliang Chen's avatar
      make SelfParaId a metadata constant (#3517) · 6f81a4a0
      Xiliang Chen authored 
      expose para id via metadata

#2116 is blocked by this
      6f81a4a0
    • Dmitry Markin's avatar
      Fix crash of synced parachain node run with `--sync=warp` (#3523) · a1b57a8c
      Dmitry Markin authored 
      Fixes https://github.com/paritytech/polkadot-sdk/issues/3496.
      a1b57a8c
    • Egor_P's avatar
      [Backport] Node version and spec_version bumps and ordering of the prdoc files from 1.8.0 (#3508) · 59b26614
      Egor_P authored 
      This PR backports Node version and `spec_version` bumps to `1.8.0` from
the latest release and orders prdoc files related to it.
      59b26614
    • Francisco Aguirre's avatar
      Add `claim_assets` extrinsic to `pallet-xcm` (#3403) · 65088668
      Francisco Aguirre authored 
      If an XCM execution fails or ends with leftover assets, these will be
trapped.
In order to claim them, a custom XCM has to be executed, with the
`ClaimAsset` instruction.
However, arbitrary XCM execution is not allowed everywhere yet and XCM
itself is still not easy enough to use for users out there with trapped
assets.
This new extrinsic in `pallet-xcm` will allow these users to easily
claim their assets, without concerning themselves with writing arbitrary
XCMs.

Part of fixing https://github.com/paritytech/polkadot-sdk/issues/3495



---------

Co-authored-by: command-bot <>
Co-authored-by: default avatarAdrian Catangiu <[email protected]>
      65088668
  13. Feb 29, 2024
  15. Feb 28, 2024
    • Alexandru Gheorghe's avatar
      Fixup multi-collator parachain transition to async backing (#3510) · 833bafdb
      Alexandru Gheorghe authored 
      

Fixing: 
```
Verification failed for block 0x07bbf1e04121d70a4bdb21cc055132b53ac2390fa95c4d05497fc91b1e8bf7f5 received from (12D3KooWJzLd8skcAgA24EcJey7aJAhYctfUxWGjSP5Usk9wbpPZ): "Header 0x07bbf1e04121d70a4bdb21cc055132b53ac2390fa95c4d05497fc91b1e8bf7f5 rejected: too far in the future"   
```

---------

Signed-off-by: default avatarAlexandru Gheorghe <[email protected]>
Co-authored-by: default avatarDmitry Sinyavin <[email protected]>
Co-authored-by: default avatars0me0ne-unkn0wn <[email protected]>
Co-authored-by: default avatarBastian Köcher <[email protected]>
      833bafdb
    • Oliver Tale-Yazdi's avatar
      Multi-Block-Migrations, `poll` hook and new System callbacks (#1781) · eefd5fe4
      Oliver Tale-Yazdi authored 
      This MR is the merge of
https://github.com/paritytech/substrate/pull/14414 and
https://github.com/paritytech/substrate/pull/14275. It implements
[RFC#13](https://github.com/polkadot-fellows/RFCs/pull/13), closes
https://github.com/paritytech/polkadot-sdk/issues/198.

----- 

This Merge request introduces three major topicals:

1. Multi-Block-Migrations
1. New pallet `poll` hook for periodic service work
1. Replacement hooks for `on_initialize` and `on_finalize` in cases
where `poll` cannot be used

and some more general changes to FRAME.  
The changes for each topical span over multiple crates. They are listed
in topical order below.

# 1.) Multi-Block-Migrations

Multi-Block-Migrations are facilitated by creating `pallet_migrations`
and configuring `System::Config::MultiBlockMigrator` to point to it.
Executive picks this up and triggers one step of the migrations pallet
per block.
The chain is in lockdown mode for as long as an MBM is ongoing.
Executive does this by polling `MultiBlockMigrator::ongoing` and not
allowing any transaction in a block, if true.

A MBM is defined through trait `SteppedMigration`. A condensed version
looks like this:
```rust
/// A migration that can proceed in multiple steps.
pub trait SteppedMigration {
	type Cursor: FullCodec + MaxEncodedLen;
	type Identifier: FullCodec + MaxEncodedLen;

	fn id() -> Self::Identifier;

	fn max_steps() -> Option<u32>;

	fn step(
		cursor: Option<Self::Cursor>,
		meter: &mut WeightMeter,
	) -> Result<Option<Self::Cursor>, SteppedMigrationError>;
}
```

`pallet_migrations` can be configured with an aggregated tuple of these
migrations. It then starts to migrate them one-by-one on the next
runtime upgrade.
Two things are important here:
- 1. Doing another runtime upgrade while MBMs are ongoing is not a good
idea and can lead to messed up state.
- 2. **Pallet Migrations MUST BE CONFIGURED IN `System::Config`,
otherwise it is not used.**

The pallet supports an `UpgradeStatusHandler` that can be used to notify
external logic of upgrade start/finish (for example to pause XCM
dispatch).

Error recovery is very limited in the case that a migration errors or
times out (exceeds its `max_steps`). Currently the runtime dev can
decide in `FailedMigrationHandler::failed` how to handle this. One
follow-up would be to pair this with the `SafeMode` pallet and enact
safe mode when an upgrade fails, to allow governance to rescue the
chain. This is currently not possible, since governance is not
`Mandatory`.

## Runtime API

- `Core`: `initialize_block` now returns `ExtrinsicInclusionMode` to
inform the Block Author whether they can push transactions.

### Integration

Add it to your runtime implementation of `Core` and `BlockBuilder`:
```patch
diff --git a/runtime/src/lib.rs b/runtime/src/lib.rs
@@ impl_runtime_apis! {
	impl sp_block_builder::Core<Block> for Runtime {
-		fn initialize_block(header: &<Block as BlockT>::Header) {
+		fn initialize_block(header: &<Block as BlockT>::Header) -> RuntimeExecutiveMode {
			Executive::initialize_block(header)
		}

		...
	}
```

# 2.) `poll` hook

A new pallet hook is introduced: `poll`. `Poll` is intended to replace
mostly all usage of `on_initialize`.
The reason for this is that any code that can be called from
`on_initialize` cannot be migrated through an MBM. Currently there is no
way to statically check this; the implication is to use `on_initialize`
as rarely as possible.
Failing to do so can result in broken storage invariants.

The implementation of the poll hook depends on the `Runtime API` changes
that are explained above.

# 3.) Hard-Deadline callbacks

Three new callbacks are introduced and configured on `System::Config`:
`PreInherents`, `PostInherents` and `PostTransactions`.
These hooks are meant as replacement for `on_initialize` and
`on_finalize` in cases where the code that runs cannot be moved to
`poll`.
The reason for this is to make the usage of HD-code (hard deadline) more
explicit - again to prevent broken invariants by MBMs.

# 4.) FRAME (general changes)

## `frame_system` pallet

A new memorize storage item `InherentsApplied` is added. It is used by
executive to track whether inherents have already been applied.
Executive and can then execute the MBMs directly between inherents and
transactions.

The `Config` gets five new items:
- `SingleBlockMigrations` this is the new way of configuring migrations
that run in a single block. Previously they were defined as last generic
argument of `Executive`. This shift is brings all central configuration
about migrations closer into view of the developer (migrations that are
configured in `Executive` will still work for now but is deprecated).
- `MultiBlockMigrator` this can be configured to an engine that drives
MBMs. One example would be the `pallet_migrations`. Note that this is
only the engine; the exact MBMs are injected into the engine.
- `PreInherents` a callback that executes after `on_initialize` but
before inherents.
- `PostInherents` a callback that executes after all inherents ran
(including MBMs and `poll`).
- `PostTransactions` in symmetry to `PreInherents`, this one is called
before `on_finalize` but after all transactions.

A sane default is to set all of these to `()`. Example diff suitable for
any chain:
```patch
@@ impl frame_system::Config for Test {
 	type MaxConsumers = ConstU32<16>;
+	type SingleBlockMigrations = ();
+	type MultiBlockMigrator = ();
+	type PreInherents = ();
+	type PostInherents = ();
+	type PostTransactions = ();
 }
```

An overview of how the block execution now looks like is here. The same
graph is also in the rust doc.

<details><summary>Block Execution Flow</summary>
<p>

![Screenshot 2023-12-04 at 19 11
29](https://github.com/paritytech/polkadot-sdk/assets/10380170/e88a80c4-ef11-4faa-8df5-8b33a724c054)

</p>
</details> 

## Inherent Order

Moved to https://github.com/paritytech/polkadot-sdk/pull/2154



---------------


## TODO

- [ ] Check that `try-runtime` still works
- [ ] Ensure backwards compatibility with old Runtime APIs
- [x] Consume weight correctly
- [x] Cleanup

---------

Signed-off-by: default avatarOliver Tale-Yazdi <[email protected]>
Co-authored-by: default avatarLiam Aharon <[email protected]>
Co-authored-by: default avatarJuan Girini <[email protected]>
Co-authored-by: command-bot <>
Co-authored-by: default avatarFrancisco Aguirre <[email protected]>
Co-authored-by: default avatarGavin Wood <[email protected]>
Co-authored-by: default avatarBastian Köcher <[email protected]>
      eefd5fe4
    • Clara van Staden's avatar
      Snowbridge - Extract Ethereum Chain ID (#3501) · 576681b8
      Clara van Staden authored 
      While adding runtime tests to
https://github.com/polkadot-fellows/runtimes/pull/130, I noticed the
Ethereum chain ID was hardcoded. For Kusama + Polkadot, the Ethereum
chain ID should 1 (Mainnet), whereas on Rococo it is 11155111 (Sepolia).

This PR also updates the Snowbridge crates versions to the current
versions on crates.io.

---------

Co-authored-by: claravanstaden <Cats 4 life!>
      576681b8
    • maksimryndin's avatar
      PVF: re-preparing artifact on failed runtime construction (#3187) · 42613667
      maksimryndin authored 
      resolve https://github.com/paritytech/polkadot-sdk/issues/3139

- [x] use a distinguishable error for `execute_artifact`
- [x] remove artifact in case of a `RuntimeConstruction` error during
the execution
- [x] augment the `validate_candidate_with_retry` of `ValidationBackend`
with the case of retriable `RuntimeConstruction` error during the
execution
- [x] update the book
(https://paritytech.github.io/polkadot-sdk/book/node/utility/pvf-host-and-workers.html#retrying-execution-requests

)
- [x] add a test
- [x] run zombienet tests

---------

Co-authored-by: default avatars0me0ne-unkn0wn <[email protected]>
      42613667
    • Kian Paimani's avatar
      Add documentation around FRAME Offchain workers (#3463) · 14530269
      Kian Paimani authored 
      - deprecation companion:
https://github.com/substrate-developer-hub/substrate-docs/pull/2136
- inspired by
https://substrate.stackexchange.com/questions/11058/how-can-i-create-ocw-that-wont-activates-every-block-but-will-activates-only-w/11060#11060



---------

Co-authored-by: default avatarSergej Sakac <[email protected]>
      14530269
    • Oliver Tale-Yazdi's avatar
      [prdoc] Optional SemVer bumps and Docs (#3441) · f4eedceb
      Oliver Tale-Yazdi authored 
      

Changes:
- Add an optional `bump` field to the crates in a prdoc.
- Explain the cargo semver interpretation for <1 versions in the release
doc.

---------

Signed-off-by: default avatarOliver Tale-Yazdi <[email protected]>
      f4eedceb
    • Alexandru Vasile's avatar
      rpc-v2/tx/tests: Add transaction broadcast tests and check propagated tx status (#3193) · f1b2189e
      Alexandru Vasile authored 
      This PR adds tests for the `transaction_broadcast` method.


The testing needs to coordinate the following components:
- The `TestApi` marks transactions as invalid and implements
`ChainApi::validate_transaction`
- this is what dictates if a transaction is valid or not and is called
from within the `BasicPool`
- The `BasicPool` which maintains the transactions and implements
`submit_and_watch` needed by the tx broadcast to submit the transaction
- The status of the transaction pool is exposed by mocking the BasicPool
- The `ChainHeadMockClient` which mocks the
`BlockchainEvents::import_notification_stream` needed by the tx
broadcast to know to which blocks the transaction is submitted

The following changes have been added to the substrate testing to
accommodate this:
- `TestApi` gets ` remove_invalid`, counterpart to `add_invalid` to
ensure an invalid transaction can become valid again; as well as a
priority setter for extrinsics
- `BasicPool` test constructor is extended with options for the
`PoolRotator`
- this mechanism is needed because transactions are banned for 30mins
(default) after they are declared invalid
  - testing bypasses this by providing a `Duration::ZERO`

### Testing Scenarios

- Capture the status of the transaction as it is normally broadcasted
- `transaction_stop` is valid while the transaction is in progress
- A future transaction is handled when the dependencies are completed
- Try to resubmit the transaction at a later block (currently invalid)
- An invalid transaction status is propagated; the transaction is marked
as temporarily banned; then the ban expires and transaction is
resubmitted
  
This builds on top of:
https://github.com/paritytech/polkadot-sdk/pull/3079
Part of: https://github.com/paritytech/polkadot-sdk/issues/3084



cc @paritytech/subxt-team

---------

Signed-off-by: default avatarAlexandru Vasile <[email protected]>
Co-authored-by: default avatarJames Wilson <[email protected]>
      f1b2189e