Skip to content
  1. Apr 26, 2024
  3. Apr 10, 2024
  5. Apr 09, 2024
  7. Apr 08, 2024
    • Oliver Tale-Yazdi's avatar
      [FRAME] Runtime Omni Bencher (#3512) · 9543d314
      Oliver Tale-Yazdi authored 
      This MR contains two major changes and some maintenance cleanup.  

## 1. Free Standing Pallet Benchmark Runner

Closes https://github.com/paritytech/polkadot-sdk/issues/3045, depends
on your runtime exposing the `GenesisBuilderApi` (like
https://github.com/paritytech/polkadot-sdk/pull/1492).

Introduces a new binary crate: `frame-omni-bencher`.  
It allows to directly benchmark a WASM blob - without needing a node or
chain spec.

This makes it much easier to generate pallet weights and should allow us
to remove bloaty code from the node.
It should work for all FRAME runtimes that dont use 3rd party host calls
or non `BlakeTwo256` block hashing (basically all polkadot parachains
should work).

It is 100% backwards compatible with the old CLI args, when the `v1`
compatibility command is used. This is done to allow for forwards
compatible addition of new commands.

### Example (full example in the Rust docs)

Installing the CLI:
```sh
cargo install --locked --path substrate/utils/frame/omni-bencher
frame-omni-bencher --help
```

Building the Westend runtime:
```sh
cargo build -p westend-runtime --release --features runtime-benchmarks
```

Benchmarking the runtime:
```sh
frame-omni-bencher v1 benchmark pallet --runtime target/release/wbuild/westend-runtime/westend_runtime.compact.compressed.wasm --all
```

## 2. Building the Benchmark Genesis State in the Runtime

Closes https://github.com/paritytech/polkadot-sdk/issues/2664

This adds `--runtime` and `--genesis-builder=none|runtime|spec`
arguments to the `benchmark pallet` command to make it possible to
generate the genesis storage by the runtime. This can be used with both
the node and the freestanding benchmark runners. It utilizes the new
`GenesisBuilder` RA and depends on having
https://github.com/paritytech/polkadot-sdk/pull/3412

 deployed.

## 3. Simpler args for `PalletCmd::run`

You can do three things here to integrate the changes into your node:
- nothing: old code keeps working as before but emits a deprecated
warning
- delete: remove the pallet benchmarking code from your node and use the
omni-bencher instead
- patch: apply the patch below and keep using as currently. This emits a
deprecated warning at runtime, since it uses the old way to generate a
genesis state, but is the smallest change.

```patch
runner.sync_run(|config| cmd
-    .run::<HashingFor<Block>, ReclaimHostFunctions>(config)
+    .run_with_spec::<HashingFor<Block>, ReclaimHostFunctions>(Some(config.chain_spec))
)
```

## 4. Maintenance Change
- `pallet-nis` get a `BenchmarkSetup` config item to prepare its
counterparty asset.
- Add percent progress print when running benchmarks.
- Dont immediately exit on benchmark error but try to run as many as
possible and print errors last.

---------

Signed-off-by: default avatarOliver Tale-Yazdi <[email protected]>
Co-authored-by: default avatarLiam Aharon <[email protected]>
      9543d314
  9. Apr 02, 2024
  11. Mar 26, 2024
    • Bastian Köcher's avatar
      westend: `SignedPhase` is a constant (#3646) · 0c15d887
      Bastian Köcher authored 
      

In preparation for the merkleized metadata, we need to ensure that
constants are actually constant. If we want to test the unsigned phase
we could for example just disable signed voter. Or we add some extra
mechanism to the pallet to disable the signed phase from time to time.

---------

Co-authored-by: default avatarAnkan <[email protected]>
Co-authored-by: default avatarKian Paimani <[email protected]>
      0c15d887
    • Dcompoze's avatar
      Fix spelling mistakes across the whole repository (#3808) · 002d9260
      Dcompoze authored 
      **Update:** Pushed additional changes based on the review comments.

**This pull request fixes various spelling mistakes in this
repository.**

Most of the changes are contained in the first **3** commits:

- `Fix spelling mistakes in comments and docs`

- `Fix spelling mistakes in test names`

- `Fix spelling mistakes in error messages, panic messages, logs and
tracing`

Other source code spelling mistakes are separated into individual
commits for easier reviewing:

- `Fix the spelling of 'authority'`

- `Fix the spelling of 'REASONABLE_HEADERS_IN_JUSTIFICATION_ANCESTRY'`

- `Fix the spelling of 'prev_enqueud_messages'`

- `Fix the spelling of 'endpoint'`

- `Fix the spelling of 'children'`

- `Fix the spelling of 'PenpalSiblingSovereignAccount'`

- `Fix the spelling of 'PenpalSudoAccount'`

- `Fix the spelling of 'insufficient'`

- `Fix the spelling of 'PalletXcmExtrinsicsBenchmark'`

- `Fix the spelling of 'subtracted'`

- `Fix the spelling of 'CandidatePendingAvailability'`

- `Fix the spelling of 'exclusive'`

- `Fix the spelling of 'until'`

- `Fix the spelling of 'discriminator'`

- `Fix the spelling of 'nonexistent'`

- `Fix the spelling of 'subsystem'`

- `Fix the spelling of 'indices'`

- `Fix the spelling of 'committed'`

- `Fix the spelling of 'topology'`

- `Fix the spelling of 'response'`

- `Fix the spelling of 'beneficiary'`

- `Fix the spelling of 'formatted'`

- `Fix the spelling of 'UNKNOWN_PROOF_REQUEST'`

- `Fix the spelling of 'succeeded'`

- `Fix the spelling of 'reopened'`

- `Fix the spelling of 'proposer'`

- `Fix the spelling of 'InstantiationNonce'`

- `Fix the spelling of 'depositor'`

- `Fix the spelling of 'expiration'`

- `Fix the spelling of 'phantom'`

- `Fix the spelling of 'AggregatedKeyValue'`

- `Fix the spelling of 'randomness'`

- `Fix the spelling of 'defendant'`

- `Fix the spelling of 'AquaticMammal'`

- `Fix the spelling of 'transactions'`

- `Fix the spelling of 'PassingTracingSubscriber'`

- `Fix the spelling of 'TxSignaturePayload'`

- `Fix the spelling of 'versioning'`

- `Fix the spelling of 'descendant'`

- `Fix the spelling of 'overridden'`

- `Fix the spelling of 'network'`

Let me know if this structure is adequate.

**Note:** The usage of the words `Merkle`, `Merkelize`, `Merklization`,
`Merkelization`, `Merkleization`, is somewhat inconsistent but I left it
as it is.

~~**Note:** In some places the term `Receival` is used to refer to
message reception, IMO `Reception` is the correct word here, but I left
it as it is.~~

~~**Note:** In some places the term `Overlayed` is used instead of the
more acceptable version `Overlaid` but I also left it as it is.~~

~~**Note:** In some places the term `Applyable` is used instead of the
correct version `Applicable` but I also left it as it is.~~

**Note:** Some usage of British vs American english e.g. `judgement` vs
`judgment`, `initialise` vs `initialize`, `optimise` vs `optimize` etc.
are both present in different places, but I suppose that's
understandable given the number of contributors.

~~**Note:** There is a spelling mistake in `.github/CODEOWNERS` but it
triggers errors in CI when I make changes to it, so I left it as it
is.~~
      002d9260
  13. Mar 15, 2024
  15. Mar 13, 2024
  17. Mar 04, 2024
    • Gavin Wood's avatar
      FRAME: Create `TransactionExtension` as a replacement for `SignedExtension` (#2280) · fd5f9292
      Gavin Wood authored 
      Closes #2160

First part of [Extrinsic
Horizon](https://github.com/paritytech/polkadot-sdk/issues/2415

)

Introduces a new trait `TransactionExtension` to replace
`SignedExtension`. Introduce the idea of transactions which obey the
runtime's extensions and have according Extension data (né Extra data)
yet do not have hard-coded signatures.

Deprecate the terminology of "Unsigned" when used for
transactions/extrinsics owing to there now being "proper" unsigned
transactions which obey the extension framework and "old-style" unsigned
which do not. Instead we have __*General*__ for the former and
__*Bare*__ for the latter. (Ultimately, the latter will be phased out as
a type of transaction, and Bare will only be used for Inherents.)

Types of extrinsic are now therefore:
- Bare (no hardcoded signature, no Extra data; used to be known as
"Unsigned")
- Bare transactions (deprecated): Gossiped, validated with
`ValidateUnsigned` (deprecated) and the `_bare_compat` bits of
`TransactionExtension` (deprecated).
  - Inherents: Not gossiped, validated with `ProvideInherent`.
- Extended (Extra data): Gossiped, validated via `TransactionExtension`.
  - Signed transactions (with a hardcoded signature).
  - General transactions (without a hardcoded signature).

`TransactionExtension` differs from `SignedExtension` because:
- A signature on the underlying transaction may validly not be present.
- It may alter the origin during validation.
- `pre_dispatch` is renamed to `prepare` and need not contain the checks
present in `validate`.
- `validate` and `prepare` is passed an `Origin` rather than a
`AccountId`.
- `validate` may pass arbitrary information into `prepare` via a new
user-specifiable type `Val`.
- `AdditionalSigned`/`additional_signed` is renamed to
`Implicit`/`implicit`. It is encoded *for the entire transaction* and
passed in to each extension as a new argument to `validate`. This
facilitates the ability of extensions to acts as underlying crypto.

There is a new `DispatchTransaction` trait which contains only default
function impls and is impl'ed for any `TransactionExtension` impler. It
provides several utility functions which reduce some of the tedium from
using `TransactionExtension` (indeed, none of its regular functions
should now need to be called directly).

Three transaction version discriminator ("versions") are now
permissible:
- 0b000000100: Bare (used to be called "Unsigned"): contains Signature
or Extra (extension data). After bare transactions are no longer
supported, this will strictly identify an Inherents only.
- 0b100000100: Old-school "Signed" Transaction: contains Signature and
Extra (extension data).
- 0b010000100: New-school "General" Transaction: contains Extra
(extension data), but no Signature.

For the New-school General Transaction, it becomes trivial for authors
to publish extensions to the mechanism for authorizing an Origin, e.g.
through new kinds of key-signing schemes, ZK proofs, pallet state,
mutations over pre-authenticated origins or any combination of the
above.

## Code Migration

### NOW: Getting it to build

Wrap your `SignedExtension`s in `AsTransactionExtension`. This should be
accompanied by renaming your aggregate type in line with the new
terminology. E.g. Before:

```rust
/// The SignedExtension to the basic transaction logic.
pub type SignedExtra = (
	/* snip */
	MySpecialSignedExtension,
);
/// Unchecked extrinsic type as expected by this runtime.
pub type UncheckedExtrinsic =
	generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, SignedExtra>;
```

After:

```rust
/// The extension to the basic transaction logic.
pub type TxExtension = (
	/* snip */
	AsTransactionExtension<MySpecialSignedExtension>,
);
/// Unchecked extrinsic type as expected by this runtime.
pub type UncheckedExtrinsic =
	generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, TxExtension>;
```

You'll also need to alter any transaction building logic to add a
`.into()` to make the conversion happen. E.g. Before:

```rust
fn construct_extrinsic(
		/* snip */
) -> UncheckedExtrinsic {
	let extra: SignedExtra = (
		/* snip */
		MySpecialSignedExtension::new(/* snip */),
	);
	let payload = SignedPayload::new(call.clone(), extra.clone()).unwrap();
	let signature = payload.using_encoded(|e| sender.sign(e));
	UncheckedExtrinsic::new_signed(
		/* snip */
		Signature::Sr25519(signature),
		extra,
	)
}
```

After:

```rust
fn construct_extrinsic(
		/* snip */
) -> UncheckedExtrinsic {
	let tx_ext: TxExtension = (
		/* snip */
		MySpecialSignedExtension::new(/* snip */).into(),
	);
	let payload = SignedPayload::new(call.clone(), tx_ext.clone()).unwrap();
	let signature = payload.using_encoded(|e| sender.sign(e));
	UncheckedExtrinsic::new_signed(
		/* snip */
		Signature::Sr25519(signature),
		tx_ext,
	)
}
```

### SOON: Migrating to `TransactionExtension`

Most `SignedExtension`s can be trivially converted to become a
`TransactionExtension`. There are a few things to know.

- Instead of a single trait like `SignedExtension`, you should now
implement two traits individually: `TransactionExtensionBase` and
`TransactionExtension`.
- Weights are now a thing and must be provided via the new function `fn
weight`.

#### `TransactionExtensionBase`

This trait takes care of anything which is not dependent on types
specific to your runtime, most notably `Call`.

- `AdditionalSigned`/`additional_signed` is renamed to
`Implicit`/`implicit`.
- Weight must be returned by implementing the `weight` function. If your
extension is associated with a pallet, you'll probably want to do this
via the pallet's existing benchmarking infrastructure.

#### `TransactionExtension`

Generally:
- `pre_dispatch` is now `prepare` and you *should not reexecute the
`validate` functionality in there*!
- You don't get an account ID any more; you get an origin instead. If
you need to presume an account ID, then you can use the trait function
`AsSystemOriginSigner::as_system_origin_signer`.
- You get an additional ticket, similar to `Pre`, called `Val`. This
defines data which is passed from `validate` into `prepare`. This is
important since you should not be duplicating logic from `validate` to
`prepare`, you need a way of passing your working from the former into
the latter. This is it.
- This trait takes two type parameters: `Call` and `Context`. `Call` is
the runtime call type which used to be an associated type; you can just
move it to become a type parameter for your trait impl. `Context` is not
currently used and you can safely implement over it as an unbounded
type.
- There's no `AccountId` associated type any more. Just remove it.

Regarding `validate`:
- You get three new parameters in `validate`; all can be ignored when
migrating from `SignedExtension`.
- `validate` returns a tuple on success; the second item in the tuple is
the new ticket type `Self::Val` which gets passed in to `prepare`. If
you use any information extracted during `validate` (off-chain and
on-chain, non-mutating) in `prepare` (on-chain, mutating) then you can
pass it through with this. For the tuple's last item, just return the
`origin` argument.

Regarding `prepare`:
- This is renamed from `pre_dispatch`, but there is one change:
- FUNCTIONALITY TO VALIDATE THE TRANSACTION NEED NOT BE DUPLICATED FROM
`validate`!!
- (This is different to `SignedExtension` which was required to run the
same checks in `pre_dispatch` as in `validate`.)

Regarding `post_dispatch`:
- Since there are no unsigned transactions handled by
`TransactionExtension`, `Pre` is always defined, so the first parameter
is `Self::Pre` rather than `Option<Self::Pre>`.

If you make use of `SignedExtension::validate_unsigned` or
`SignedExtension::pre_dispatch_unsigned`, then:
- Just use the regular versions of these functions instead.
- Have your logic execute in the case that the `origin` is `None`.
- Ensure your transaction creation logic creates a General Transaction
rather than a Bare Transaction; this means having to include all
`TransactionExtension`s' data.
- `ValidateUnsigned` can still be used (for now) if you need to be able
to construct transactions which contain none of the extension data,
however these will be phased out in stage 2 of the Transactions Horizon,
so you should consider moving to an extension-centric design.

## TODO

- [x] Introduce `CheckSignature` impl of `TransactionExtension` to
ensure it's possible to have crypto be done wholly in a
`TransactionExtension`.
- [x] Deprecate `SignedExtension` and move all uses in codebase to
`TransactionExtension`.
  - [x] `ChargeTransactionPayment`
  - [x] `DummyExtension`
  - [x] `ChargeAssetTxPayment` (asset-tx-payment)
  - [x] `ChargeAssetTxPayment` (asset-conversion-tx-payment)
  - [x] `CheckWeight`
  - [x] `CheckTxVersion`
  - [x] `CheckSpecVersion`
  - [x] `CheckNonce`
  - [x] `CheckNonZeroSender`
  - [x] `CheckMortality`
  - [x] `CheckGenesis`
  - [x] `CheckOnlySudoAccount`
  - [x] `WatchDummy`
  - [x] `PrevalidateAttests`
  - [x] `GenericSignedExtension`
  - [x] `SignedExtension` (chain-polkadot-bulletin)
  - [x] `RefundSignedExtensionAdapter`
- [x] Implement `fn weight` across the board.
- [ ] Go through all pre-existing extensions which assume an account
signer and explicitly handle the possibility of another kind of origin.
- [x] `CheckNonce` should probably succeed in the case of a non-account
origin.
- [x] `CheckNonZeroSender` should succeed in the case of a non-account
origin.
- [x] `ChargeTransactionPayment` and family should fail in the case of a
non-account origin.
  - [ ] 
- [x] Fix any broken tests.

---------

Signed-off-by: default avatargeorgepisaltu <[email protected]>
Signed-off-by: default avatarAlexandru Vasile <[email protected]>
Signed-off-by: default avatardependabot[bot] <[email protected]>
Signed-off-by: default avatarOliver Tale-Yazdi <[email protected]>
Signed-off-by: default avatarAlexandru Gheorghe <[email protected]>
Signed-off-by: default avatarAndrei Sandu <[email protected]>
Co-authored-by: default avatarNikhil Gupta <[email protected]>
Co-authored-by: default avatargeorgepisaltu <[email protected]>
Co-authored-by: default avatarChevdor <[email protected]>
Co-authored-by: default avatarBastian Köcher <[email protected]>
Co-authored-by: default avatarMaciej <[email protected]>
Co-authored-by: default avatarJavier Viola <[email protected]>
Co-authored-by: default avatarMarcin S. <[email protected]>
Co-authored-by: default avatarTsvetomir Dimitrov <[email protected]>
Co-authored-by: default avatarJavier Bullrich <[email protected]>
Co-authored-by: default avatarKoute <[email protected]>
Co-authored-by: default avatarAdrian Catangiu <[email protected]>
Co-authored-by: Vladimir Istyufeev's avatarVladimir Istyufeev <[email protected]>
Co-authored-by: default avatarRoss Bulat <[email protected]>
Co-authored-by: default avatarGonçalo Pestana <[email protected]>
Co-authored-by: default avatarLiam Aharon <[email protected]>
Co-authored-by: default avatarSvyatoslav Nikolsky <[email protected]>
Co-authored-by: default avatarAndré Silva <[email protected]>
Co-authored-by: default avatarOliver Tale-Yazdi <[email protected]>
Co-authored-by: default avatars0me0ne-unkn0wn <[email protected]>
Co-authored-by: default avatarordian <[email protected]>
Co-authored-by: default avatarSebastian Kunert <[email protected]>
Co-authored-by: default avatarAaro Altonen <[email protected]>
Co-authored-by: default avatarDmitry Markin <[email protected]>
Co-authored-by: default avatarAlexandru Vasile <[email protected]>
Co-authored-by: default avatarAlexander Samusev <[email protected]>
Co-authored-by: default avatarJulian Eager <[email protected]>
Co-authored-by: default avatarMichal Kucharczyk <[email protected]>
Co-authored-by: default avatarDavide Galassi <[email protected]>
Co-authored-by: default avatarDónal Murray <[email protected]>
Co-authored-by: default avataryjh <[email protected]>
Co-authored-by: default avatarTom Mi <[email protected]>
Co-authored-by: default avatardependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: default avatarWill | Paradox | ParaNodes.io <[email protected]>
Co-authored-by: default avatarBastian Köcher <[email protected]>
Co-authored-by: default avatarJoshy Orndorff <[email protected]>
Co-authored-by: default avatarJoshy Orndorff <[email protected]>
Co-authored-by: default avatarPG Herveou <[email protected]>
Co-authored-by: default avatarAlexander Theißen <[email protected]>
Co-authored-by: default avatarKian Paimani <[email protected]>
Co-authored-by: default avatarJuan Girini <[email protected]>
Co-authored-by: default avatarbader y <[email protected]>
Co-authored-by: default avatarJames Wilson <[email protected]>
Co-authored-by: default avatarjoe petrowski <[email protected]>
Co-authored-by: default avatarasynchronous rob <[email protected]>
Co-authored-by: default avatarParth <[email protected]>
Co-authored-by: default avatarAndrew Jones <[email protected]>
Co-authored-by: default avatarJonathan Udd <[email protected]>
Co-authored-by: default avatarSerban Iorga <[email protected]>
Co-authored-by: default avatarEgor_P <[email protected]>
Co-authored-by: default avatarBranislav Kontur <[email protected]>
Co-authored-by: default avatarEvgeny Snitko <[email protected]>
Co-authored-by: default avatarJust van Stam <[email protected]>
Co-authored-by: default avatarFrancisco Aguirre <[email protected]>
Co-authored-by: default avatargupnik <[email protected]>
Co-authored-by: default avatardzmitry-lahoda <[email protected]>
Co-authored-by: default avatarzhiqiangxu <[email protected]>
Co-authored-by: default avatarNazar Mokrynskyi <[email protected]>
Co-authored-by: default avatarAnwesh <[email protected]>
Co-authored-by: default avatarcheme <[email protected]>
Co-authored-by: default avatarSam Johnson <[email protected]>
Co-authored-by: default avatarkianenigma <[email protected]>
Co-authored-by: default avatarJegor Sidorenko <[email protected]>
Co-authored-by: default avatarMuharem <[email protected]>
Co-authored-by: default avatarjoepetrowski <[email protected]>
Co-authored-by: default avatarAlexandru Gheorghe <[email protected]>
Co-authored-by: default avatarGabriel Facco de Arruda <[email protected]>
Co-authored-by: default avatarSquirrel <[email protected]>
Co-authored-by: default avatarAndrei Sandu <[email protected]>
Co-authored-by: default avatargeorgepisaltu <[email protected]>
Co-authored-by: command-bot <>
      fd5f9292
  19. Feb 28, 2024
  21. Feb 12, 2024
  23. Feb 08, 2024
    • Gonçalo Pestana's avatar
      Fixes `TotalValueLocked` out of sync in nomination pools (#3052) · aac07af0
      Gonçalo Pestana authored 
      The `TotalLockedValue` storage value in nomination pools pallet may get
out of sync if the staking pallet does implicit withdrawal of unlocking
chunks belonging to a bonded pool stash. This fix is based on a new
method in the `OnStakingUpdate` traits, `on_withdraw`, which allows the
nomination pools pallet to adjust the `TotalLockedValue` every time
there is an implicit or explicit withdrawal from a bonded pool's stash.

This PR also adds a migration that checks and updates the on-chain TVL
if it got out of sync due to the bug this PR fixes.

**Changes to `trait OnStakingUpdate`**

In order for staking to notify the nomination pools pallet that chunks
where withdrew, we add a new method, `on_withdraw` to the
`OnStakingUpdate` trait. The nomination pools pallet filters the
withdraws that are related to bonded pool accounts and updates the
`TotalValueLocked` accordingly.

**Others**
- Adds try-state checks to the EPM/staking e2e tests
- Adds tests for auto withdrawing in the context of nomination pools

**To-do**
- [x] check if we need a migration to fix the current `TotalValueLocked`
(run try-runtime)
- [x] migrations to fix the current on-chain TVL value 

 ✅ **Kusama**:
```
TotalValueLocked: 99.4559 kKSM
TotalValueLocked (calculated) 99.4559 kKSM
```
⚠️ **Westend**:
```
TotalValueLocked: 18.4060 kWND
TotalValueLocked (calculated) 18.4050 kWND
```
**Polkadot**: TVL not released yet.

Closes https://github.com/paritytech/polkadot-sdk/issues/3055



---------

Co-authored-by: command-bot <>
Co-authored-by: default avatarRoss Bulat <[email protected]>
Co-authored-by: default avatarDónal Murray <[email protected]>
      aac07af0
  25. Feb 06, 2024
    • Squirrel's avatar
      sp-std -> core (#3199) · bc2e5e1f
      Squirrel authored 
      First in a series of PRs that reduces our use of sp-std with a view to
deprecating it.

This is just looking at /substrate and moving some of the references
from `sp-std` to `core`.
These particular changes should be uncontroversial.

Where macros are used `::core` should be used to remove any ambiguity.

part of https://github.com/paritytech/polkadot-sdk/issues/2101
      bc2e5e1f
  27. Jan 31, 2024
    • Branislav Kontur's avatar
      [frame] `#[pallet::composite_enum]` improved variant count handling + removed... · bb8ddc46
      Branislav Kontur authored 
      [frame] `#[pallet::composite_enum]` improved variant count handling + removed `pallet_balances`'s `MaxHolds` config (#2657)

I started this investigation/issue based on @liamaharon question
[here](https://github.com/paritytech/polkadot-sdk/pull/1801#discussion_r1410452499).

## Problem

The `pallet_balances` integrity test should correctly detect that the
runtime has correct distinct `HoldReasons` variant count. I assume the
same situation exists for RuntimeFreezeReason.

It is not a critical problem, if we set `MaxHolds` with a sufficiently
large value, everything should be ok. However, in this case, the
integrity_test check becomes less useful.

**Situation for "any" runtime:**
- `HoldReason` enums from different pallets:
```rust
        /// from pallet_nis
        #[pallet::composite_enum]
	pub enum HoldReason {
		NftReceipt,
	}

        /// from pallet_preimage
        #[pallet::composite_enum]
	pub enum HoldReason {
		Preimage,
	}

        // from pallet_state-trie-migration
        #[pallet::composite_enum]
	pub enum HoldReason {
		SlashForContinueMigrate,
		SlashForMigrateCustomTop,
		SlashForMigrateCustomChild,
	}
```

- generated `RuntimeHoldReason` enum looks like:
```rust
pub enum RuntimeHoldReason {

    #[codec(index = 32u8)]
    Preimage(pallet_preimage::HoldReason),

    #[codec(index = 38u8)]
    Nis(pallet_nis::HoldReason),

    #[codec(index = 42u8)]
    StateTrieMigration(pallet_state_trie_migration::HoldReason),
}
```

- composite enum `RuntimeHoldReason` variant count is detected as `3`
- we set `type MaxHolds = ConstU32<3>`
- `pallet_balances::integrity_test` is ok with `3`(at least 3)

However, the real problem can occur in a live runtime where some
functionality might stop working. This is due to a total of 5 distinct
hold reasons (for pallets with multi-instance support, it is even more),
and not all of them can be used because of an incorrect `MaxHolds`,
which is deemed acceptable according to the `integrity_test`:
  ```
  // pseudo-code - if we try to call all of these:

T::Currency::hold(&pallet_nis::HoldReason::NftReceipt.into(),
&nft_owner, deposit)?;
T::Currency::hold(&pallet_preimage::HoldReason::Preimage.into(),
&nft_owner, deposit)?;

T::Currency::hold(&pallet_state_trie_migration::HoldReason::SlashForContinueMigrate.into(),
&nft_owner, deposit)?;

  // With `type MaxHolds = ConstU32<3>` these two will fail

T::Currency::hold(&pallet_state_trie_migration::HoldReason::SlashForMigrateCustomTop.into(),
&nft_owner, deposit)?;

T::Currency::hold(&pallet_state_trie_migration::HoldReason::SlashForMigrateCustomChild.into(),
&nft_owner, deposit)?;
  ```  


## Solutions

A macro `#[pallet::*]` expansion is extended of `VariantCount`
implementation for the `#[pallet::composite_enum]` enum type. This
expansion generates the `VariantCount` implementation for pallets'
`HoldReason`, `FreezeReason`, `LockId`, and `SlashReason`. Enum variants
must be plain enum values without fields to ensure a deterministic
count.

The composite runtime enum, `RuntimeHoldReason` and
`RuntimeFreezeReason`, now sets `VariantCount::VARIANT_COUNT` as the sum
of pallets' enum `VariantCount::VARIANT_COUNT`:
```rust
#[frame_support::pallet(dev_mode)]
mod module_single_instance {

	#[pallet::composite_enum]
	pub enum HoldReason {
		ModuleSingleInstanceReason1,
		ModuleSingleInstanceReason2,
	}
...
}

#[frame_support::pallet(dev_mode)]
mod module_multi_instance {

	#[pallet::composite_enum]
	pub enum HoldReason<I: 'static = ()> {
		ModuleMultiInstanceReason1,
		ModuleMultiInstanceReason2,
		ModuleMultiInstanceReason3,
	}
...
}


impl self::sp_api_hidden_includes_construct_runtime::hidden_include::traits::VariantCount
    for RuntimeHoldReason
{
    const VARIANT_COUNT: u32 = 0
        + module_single_instance::HoldReason::VARIANT_COUNT
        + module_multi_instance::HoldReason::<module_multi_instance::Instance1>::VARIANT_COUNT
        + module_multi_instance::HoldReason::<module_multi_instance::Instance2>::VARIANT_COUNT
        + module_multi_instance::HoldReason::<module_multi_instance::Instance3>::VARIANT_COUNT;
}
```

In addition, `MaxHolds` is removed (as suggested
[here](https://github.com/paritytech/polkadot-sdk/pull/2657#discussion_r1443324573))
from `pallet_balances`, and its `Holds` are now bounded to
`RuntimeHoldReason::VARIANT_COUNT`. Therefore, there is no need to let
the runtime specify `MaxHolds`.


## For reviewers

Relevant changes can be found here:
- `substrate/frame/support/procedural/src/lib.rs` 
-  `substrate/frame/support/procedural/src/pallet/parse/composite.rs`
-  `substrate/frame/support/procedural/src/pallet/expand/composite.rs`
-
`substrate/frame/support/procedural/src/construct_runtime/expand/composite_helper.rs`
-
`substrate/frame/support/procedural/src/construct_runtime/expand/hold_reason.rs`
-
`substrate/frame/support/procedural/src/construct_runtime/expand/freeze_reason.rs`
- `substrate/frame/support/src/traits/misc.rs`

And the rest of the files is just about removed `MaxHolds` from
`pallet_balances`

## Next steps

Do the same for `MaxFreezes`
https://github.com/paritytech/polkadot-sdk/issues/2997

.

---------

Co-authored-by: command-bot <>
Co-authored-by: default avatarBastian Köcher <[email protected]>
Co-authored-by: default avatarDónal Murray <[email protected]>
Co-authored-by: default avatargupnik <[email protected]>
      bb8ddc46
  29. Jan 26, 2024
  31. Jan 22, 2024
  33. Dec 18, 2023
  35. Dec 15, 2023
  37. Dec 13, 2023
  39. Dec 12, 2023
    • Ross Bulat's avatar
      Staking: Add `deprecate_controller_batch` AdminOrigin call (#2589) · 048a9c27
      Ross Bulat authored 
      

Partially Addresses #2500

Adds a `deprecate_controller_batch` call to the staking pallet that is
callable by `Root` and `StakingAdmin`. To be used for controller account
deprecation and removed thereafter. Adds
`MaxControllersDeprecationBatch` pallet constant that defines max
possible deprecations per call.

- [x] Add `deprecate_controller_batch` call, and
`MaxControllersInDeprecationBatch` constant.
- [x] Add tests, benchmark, weights. Tests that weight is only consumed
if unique pair.
- [x] Adds `StakingAdmin` origin to staking's `AdminOrigin` type in
westend runtime.
- [x] Determined that worst case 5,900 deprecations does fit into
`maxBlock` `proofSize` and `refTime` in both normal and operational
thresholds, meaning we can deprecate all controllers for each network in
one call.

## Block Weights

By querying `consts.system.blockWeights` we can see that the
`deprecate_controller_batch` weights fit within the `normal` threshold
on Polkadot.

#### `controller_deprecation_batch` where i = 5900:
#### Ref time: 69,933,325,300
#### Proof size: 21,040,390

### Polkadot 

```
// consts.query.blockWeights

maxBlock: {
        refTime: 2,000,000,000,000
        proofSize: 18,446,744,073,709,551,615
}
normal: {
 maxExtrinsic: {
	refTime: 1,479,873,955,000
	proofSize: 13,650,590,614,545,068,195
 }
 maxTotal: {
	refTime: 1,500,000,000,000
	proofSize: 13,835,058,055,282,163,711
 }
}
```

### Kusama

```
// consts.query.blockWeights

  maxBlock: {
    refTime: 2,000,000,000,000
    proofSize: 18,446,744,073,709,551,615
  }
    normal: {
      maxExtrinsic: {
        refTime: 1,479,875,294,000
        proofSize: 13,650,590,614,545,068,195
      }
      maxTotal: {
        refTime: 1,500,000,000,000
        proofSize: 13,835,058,055,282,163,711
      }
}
```

---------

Co-authored-by: command-bot <>
Co-authored-by: default avatarGonçalo Pestana <[email protected]>
      048a9c27
  41. Dec 01, 2023
  43. Nov 30, 2023
  45. Nov 28, 2023
  47. Nov 01, 2023
    • Ankan's avatar
      [NPoS] Paging reward payouts in order to scale rewardable nominators (#1189) · 00b85c51
      Ankan authored 
      helps https://github.com/paritytech/polkadot-sdk/issues/439.
closes https://github.com/paritytech/polkadot-sdk/issues/473.

PR link in the older substrate repository:
https://github.com/paritytech/substrate/pull/13498.

# Context
Rewards payout is processed today in a single block and limited to
`MaxNominatorRewardedPerValidator`. This number is currently 512 on both
Kusama and Polkadot.

This PR tries to scale the nominators payout to an unlimited count in a
multi-block fashion. Exposures are stored in pages, with each page
capped to a certain number (`MaxExposurePageSize`). Starting out, this
number would be the same as `MaxNominatorRewardedPerValidator`, but
eventually, this number can be lowered through new runtime upgrades to
limit the rewardeable nominators per dispatched call instruction.

The changes in the PR are backward compatible.

## How payouts would work like after this change
Staking exposes two calls, 1) the existing `payout_stakers` and 2)
`payout_stakers_by_page`.

### payout_stakers
This remains backward compatible with no signature change. If for a
given era a validator has multiple pages, they can call `payout_stakers`
multiple times. The pages are executed in an ascending sequence and the
runtime takes care of preventing double claims.

### payout_stakers_by_page
Very similar to `payout_stakers` but also accepts an extra param
`page_index`. An account can choose to payout rewards only for an
explicitly passed `page_index`.

**Lets look at an example scenario**
Given an active validator on Kusama had 1100 nominators,
`MaxExposurePageSize` set to 512 for Era e. In order to pay out rewards
to all nominators, the caller would need to call `payout_stakers` 3
times.

- `payout_stakers(origin, stash, e)` => will pay the first 512
nominators.
- `payout_stakers(origin, stash, e)` => will pay the second set of 512
nominators.
- `payout_stakers(origin, stash, e)` => will pay the last set of 76
nominators.
...
- `payout_stakers(origin, stash, e)` => calling it the 4th time would
return an error `InvalidPage`.

The above calls can also be replaced by `payout_stakers_by_page` and
passing a `page_index` explicitly.

## Commission note
Validator commission is paid out in chunks across all the pages where
each commission chunk is proportional to the total stake of the current
page. This implies higher the total stake of a page, higher will be the
commission. If all the pages of a validator's single era are paid out,
the sum of commission paid to the validator across all pages should be
equal to what the commission would have been if we had a non-paged
exposure.

### Migration Note
Strictly speaking, we did not need to bump our storage version since
there is no migration of storage in this PR. But it is still useful to
mark a storage upgrade for the following reasons:

- New storage items are introduced in this PR while some older storage
items are deprecated.
- For the next `HistoryDepth` eras, the exposure would be incrementally
migrated to its corresponding paged storage item.
- Runtimes using staking pallet would strictly need to wait at least
`HistoryDepth` eras with current upgraded version (14) for the migration
to complete. At some era `E` such that `E >
era_at_which_V14_gets_into_effect + HistoryDepth`, we will upgrade to
version X which will remove the deprecated storage items.
In other words, it is a strict requirement that E<sub>x</sub> -
E<sub>14</sub> > `HistoryDepth`, where
E<sub>x</sub> = Era at which deprecated storages are removed from
runtime,
E<sub>14</sub> = Era at which runtime is upgraded to version 14.
- For Polkadot and Kusama, there is a [tracker
ticket](https://github.com/paritytech/polkadot-sdk/issues/433) to clean
up the deprecated storage items.

### Storage Changes

#### Added
- ErasStakersOverview
- ClaimedRewards
- ErasStakersPaged

#### Deprecated
The following can be cleaned up after 84 eras which is tracked
[here](https://github.com/paritytech/polkadot-sdk/issues/433).

- ErasStakers.
- ErasStakersClipped.
- StakingLedger.claimed_rewards, renamed to
StakingLedger.legacy_claimed_rewards.

### Config Changes
- Renamed MaxNominatorRewardedPerValidator to MaxExposurePageSize.

### TODO
- [x] Tracker ticket for cleaning up the old code after 84 eras.
- [x] Add companion.
- [x] Redo benchmarks before merge.
- [x] Add Changelog for pallet_staking.
- [x] Pallet should be configurable to enable/disable paged rewards.
- [x] Commission payouts are distributed across pages.
- [x] Review documentation thoroughly.
- [x] Rename `MaxNominatorRewardedPerValidator` ->
`MaxExposurePageSize`.
- [x] NMap for `ErasStakersPaged`.
- [x] Deprecate ErasStakers.
- [x] Integrity tests.

### Followup issues
[Runtime api for deprecated ErasStakers storage
item](https://github.com/paritytech/polkadot-sdk/issues/426

)

---------

Co-authored-by: default avatarJavier Viola <[email protected]>
Co-authored-by: default avatarRoss Bulat <[email protected]>
Co-authored-by: command-bot <>
      00b85c51
  49. Oct 24, 2023
  51. Oct 20, 2023
    • Bastian Köcher's avatar
      `xcm`: Change `TypeInfo::path` to not include `staging` (#1948) · f3bf5c1a
      Bastian Köcher authored 
      The `xcm` crate was renamed to `staging-xcm` to be able to publish it to
crates.io as someone as squatted `xcm`. The problem with this rename is
that the `TypeInfo` includes the crate name which ultimately lands in
the metadata. The metadata is consumed by downstream users like
`polkadot-js` or people building on top of `polkadot-js`. These people
are using the entire `path` to find the type in the type registry. Thus,
their code would break as the type path would now be [`staging_xcm`,
`VersionedXcm`] instead of [`xcm`, `VersionedXcm`]. This pull request
fixes this by renaming the path segment `staging_xcm` to `xcm`.

This requires: https://github.com/paritytech/scale-info/pull/197



---------

Co-authored-by: default avatarFrancisco Aguirre <[email protected]>
      f3bf5c1a
  53. Oct 15, 2023
    • Gonçalo Pestana's avatar
      Refactor staking ledger (#1484) · 8ee4042c
      Gonçalo Pestana authored 
      This PR refactors the staking ledger logic to encapsulate all reads and
mutations of `Ledger`, `Bonded`, `Payee` and stake locks within the
`StakingLedger` struct implementation.

With these changes, all the reads and mutations to the `Ledger`, `Payee`
and `Bonded` storage map should be done through the methods exposed by
StakingLedger to ensure the data and lock consistency of the operations.
The new introduced methods that mutate and read Ledger are:

- `ledger.update()`: inserts/updates a staking ledger in storage;
updates staking locks accordingly (and ledger.bond(), which is synthatic
sugar for ledger.update())
- `ledger.kill()`: removes all Bonded and StakingLedger related data for
a given ledger; updates staking locks accordingly;
`StakingLedger::get(account)`: queries both the `Bonded` and `Ledger`
storages and returns a `Option<StakingLedger>`. The pallet impl exposes
fn ledger(account) as synthatic sugar for `StakingLedger::get(account)`.

Retrieving a ledger with `StakingLedger::get()` can be done by providing
either a stash or controller account. The input must be wrapped in a
`StakingAccount` variant (Stash or Controller) which is treated
accordingly. This simplifies the caller API but will eventually be
deprecated once we completely get rid of the controller account in
staking. However, this refactor will help with the work necessary when
completely removing the controller.

Other goals:

- No logical changes have been introduced in this PR;
- No breaking changes or updates in wallets required;
- No new storage items or need to perform storage migrations;
- Centralise the changes to bonds and ledger updates to simplify the
OnStakingUpdate updates to the target list (related to
https://github.com/paritytech/polkadot-sdk/issues/443)

Note: it would be great to prevent or at least raise a warning if
`Ledger<T>`, `Payee<T>` and `Bonded<T>` storage types are accessed
outside the `StakingLedger` implementation. This PR should not get
blocked by that feature, but there's a tracking issue here
https://github.com/paritytech/polkadot-sdk/issues/149

Related and step towards
https://github.com/paritytech/polkadot-sdk/issues/443
      8ee4042c
  55. Oct 01, 2023
  57. Sep 22, 2023
  59. Sep 18, 2023
    • Gonçalo Pestana's avatar
      Implements a variable deposit base calculation for EPM signed submissions (#1547) · 614aa31b
      Gonçalo Pestana authored 
      **Note**: This is a lift-and-shift PR from the old substrate and
polkadot repos, both PRs have been reviewed and audited
(https://github.com/paritytech/substrate/pull/13983,
https://github.com/paritytech/polkadot/pull/7140)

---

This PR implements a generic `BaseDeposit` calculation for signed
submissions, based on the size of the submission queue.

It adds a new associated type to EPM's config, `type SignedDepositBase`,
that implements `Convert<usize, BalanceOf<T>>`, which is used to
calculate the base deposit for signed submissions based on the size of
the signed submissions queue.

`struct GeometricDepositBase<Balance, Fixed, Inc>` implements the
convert trait so that the deposit value increases as a geometric
progression. The deposit base is calculated by `deposit_base =
fixed_deposit_base * (1 + increase_factor)^n`, where `n` is the term of
the progression (i.e. the number of signed submissions in the queue).
`Fixed` and `Inc` generic params are getters for `Balance` and
`IncreaseFactor` to compute the geometric progression. If
`IncreaseFactor = 0`, then the signed deposit is constant and equal to
`Fixed` regardless of the size of the queue.

### Runtime configs

In Kusama, the progression with 10% increase without changing the
current signed fixed deposit is: (term == size of the queue)

Term 1: `1,333,333,332,000`
Term 2: `1,333,333,332,000 * 1.10 = 1,466,666,665,200`
Term 3: `1,333,333,332,000 * 1.10^2 = 1,613,333,331,200`
Term 4: `1,333,333,332,000 * 1.10^3 = 1,774,666,664,320`
Term 5: `1,333,333,332,000 * 1.10^4 = 1,952,133,330,752`
Term 6: `1,333,333,332,000 * 1.10^5 = 2,147,346,663,827.20`
Term 7: `1,333,333,332,000 * 1.10^6 = 2,362,081,330,210.92`
Term 8: `1,333,333,332,000 * 1.10^7 = 2,598,289,463,231.01`
Term 9: `1,333,333,332,000 * 1.10^8 = 2,858,118,409,554.11`
Term 10: `1,333,333,332,000 * 1.10^9 = 3,143,930,250,509.52`

Westend:

Term 1: `2,000,000,000,000`
Term 2: `2,000,000,000,000 * 1.10 = 2,200,000,000,000`
Term 3: `2,000,000,000,000 * 1.10^2 = 2,420,000,000,000`
Term 4: `2,000,000,000,000 * 1.10^3 = 2,662,000,000,000`
Term 5: `2,000,000,000,000 * 1.10^4 = 2,928,200,000,000`
Term 6: `2,000,000,000,000 * 1.10^5 = 3,221,020,000,000`
Term 7: `2,000,000,000,000 * 1.10^6 = 3,543,122,000,000`
Term 8: `2,000,000,000,000 * 1.10^7 = 3,897,434,200,000`
Term 9: `2,000,000,000,000 * 1.10^8 = 4,287,177,620,000`
Term 10: `2,000,000,000,000 * 1.10^9 = 4,715,895,382,000`

and in Polkadot, the deposit increase is disabled in the current state
of the PR, as the increase factor is 0% -- so nothing changes from the
current behaviour.

Closes https://github.com/paritytech-secops/srlabs_findings/issues/189
      614aa31b
  61. Sep 11, 2023
  63. Aug 31, 2023
  65. Aug 30, 2023
  67. Aug 29, 2023
  69. Aug 28, 2023
  71. Aug 25, 2023
  73. Aug 24, 2023
    • juangirini's avatar
      Restructure `frame_benchmarking` macro related exports (#14787) · bc2b3d9d
      juangirini authored 
      

* make reexports private

* make reexports private 2

* make reexports private for runtime-benchmarking

* make reexports private for try-runtime

* fix for try-runtime

* make reexports private for tests

* fmt

* make reexports private for tests

* make reexports private for experimental

* fix beefy

* fix ui test

* fix ui test

* fix benches

* ".git/.scripts/commands/fmt/fmt.sh"

* fix contracts use

* wip

* wip

* do not reexport sp_api::metadata_ir

* fix CI checks

* fix support tests

* ".git/.scripts/commands/fmt/fmt.sh"

* Update frame/support/src/lib.rs

Co-authored-by: default avatarBastian Köcher <[email protected]>

* import codec directly

* fmt

* wip

* wip

* remove benchmarking private exports

* fix zepter warning

* fix imports

* partially fix node-cli tests

* fix node-cli tests

* fix node-cli build

* fix zepter warning

* fix test

* fix ui test

* fix ci

* remove unnecessary imports

* add import back

* add import back

---------

Co-authored-by: command-bot <>
Co-authored-by: default avatarBastian Köcher <[email protected]>
      bc2b3d9d