Skip to content
  1. Jan 23, 2024
    • Niklas Adolfsson's avatar
      rpc: backpressured RPC server (bump jsonrpsee 0.20) (#1313) · e16ef086
      Niklas Adolfsson authored 
      This is a rather big change in jsonrpsee, the major things in this bump
are:
- Server backpressure (the subscription impls are modified to deal with
that)
- Allow custom error types / return types (remove jsonrpsee::core::Error
and jsonrpee::core::CallError)
- Bug fixes (graceful shutdown in particular not used by substrate
anyway)
   - Less dependencies for the clients in particular
   - Return type requires Clone in method call responses
   - Moved to tokio channels
   - Async subscription API (not used in this PR)

Major changes in this PR:
- The subscriptions are now bounded and if subscription can't keep up
with the server it is dropped
- CLI: add parameter to configure the jsonrpc server bounded message
buffer (default is 64)
- Add our own subscription helper to deal with the unbounded streams in
substrate

The most important things in this PR to review is the added helpers
functions in `substrate/client/rpc/src/utils.rs` and the rest is pretty
much chore.

Regarding the "bounded buffer limit" it may cause the server to handle
the JSON-RPC calls
slower than before.

The message size limit is bounded by "--rpc-response-size" thus "by
default 10MB * 64 = 640MB"
but the subscription message size is not covered by this limit and could
be capped as well.

Hopefully the last release prior to 1.0, sorry in advance for a big PR

Previous attempt: https://github.com/paritytech/substrate/pull/13992

Resolves https://github.com/paritytech/polkadot-sdk/issues/748, resolves
https://github.com/paritytech/polkadot-sdk/issues/627
      e16ef086
  3. Jan 22, 2024
  5. Jan 18, 2024
  7. Jan 15, 2024
  9. Jan 12, 2024
    • Dmitry Markin's avatar
      Extract warp sync strategy from `ChainSync` (#2467) · 5208bed7
      Dmitry Markin authored 
      

Extract `WarpSync` (and `StateSync` as part of warp sync) from
`ChainSync` as independent syncing strategy called by `SyncingEngine`.
Introduce `SyncingStrategy` enum as a proxy between `SyncingEngine` and
specific syncing strategies.

## Limitations
Gap sync is kept in `ChainSync` for now because it shares the same set
of peers as block syncing implementation in `ChainSync`. Extraction of a
common context responsible for peer management in syncing strategies
able to run in parallel is planned for a follow-up PR.

## Further improvements
A possibility of conversion of `SyncingStartegy` into a trait should be
evaluated. The main stopper for this is that different strategies need
to communicate different actions to `SyncingEngine` and respond to
different events / provide different APIs (e.g., requesting
justifications is only possible via `ChainSync` and not through
`WarpSync`; `SendWarpProofRequest` action is only relevant to
`WarpSync`, etc.)

---------

Co-authored-by: default avatarAaro Altonen <[email protected]>
      5208bed7
    • Serban Iorga's avatar
      Kitchensink: Fix pallet_mmr config (#2919) · 868788a5
      Serban Iorga authored 
      Related to https://github.com/paritytech/polkadot-sdk/issues/2787

Fixes `pallet_mmr::Config` for the kitchensink runtime
      868788a5
  11. Jan 11, 2024
    • Sebastian Kunert's avatar
      Cumulus test service cleanup (#2887) · c93f5aba
      Sebastian Kunert authored 
      closes #2567 

Followup for https://github.com/paritytech/polkadot-sdk/pull/2331

This PR contains multiple internal cleanups:

1. This gets rid of the functionality in `generate_genesis_block` which
was only used in one benchmark
2. Fixed `transaction_pool` and `transaction_throughput` benchmarks
failing since they require a tokio runtime now.
3. Removed `parachain_id` CLI option from the test parachain
4. Removed `expect` call from `RuntimeResolver`
      c93f5aba
  13. Jan 10, 2024
    • Nazar Mokrynskyi's avatar
      Improve storage monitor API (#2899) · af2e30e3
      Nazar Mokrynskyi authored 
      This removes the need to unnecessarily provide a very specific data
structure `DatabaseSource` and removes huge `sc-client-db` dependency
from storage monitor. It is now possible to use storage monitor with any
path.

P.S. I still strongly dislike that it pulls `clap` dependency for such a
small feature, but many other crates do as well, so nothing special
here.
      af2e30e3
    • joe petrowski's avatar
      Unique Usernames in Identity Pallet (#2651) · d1f678c0
      joe petrowski authored 
      

This PR allows _username authorities_ to issue unique usernames that
correspond with an account. It also provides two-way lookup, that is
from `AccountId` to a single, "primary" `Username` (alongside
`Registration`) and multiple unique `Username`s to an `AccountId`.

Key features:

- Username Authorities added (and removed) via privileged origin.
- Authorities have a `suffix` and an `allocation`. They can grant up to
`allocation` usernames. Their `suffix` will be appended to the usernames
that they issue. A suffix may be up to 7 characters long.
- Users can ask an authority to grant them a username. This will take
the form `myusername.suffix`. The entire name (including suffix) must be
less than or equal to 32 alphanumeric characters.
- Users can approve a username for themselves in one of two ways (that
is, authorities cannot grant them arbitrarily):
- Pre-sign the entire username (including suffix) with a secret key that
corresponds to their `AccountId` (for keyed accounts, obviously); or
- Accept the username after it has been granted by an authority (it will
be queued until accepted) (for non-keyed accounts like pure proxies or
multisigs).
- The system does not require any funds or deposits. Users without an
identity will be given a default one (presumably all fields set to
`None`). If they update this info, they will need to place the normal
storage deposit.
- If a user does not have any username, their first one will be set as
`Primary`, and their `AccountId` will map to that one. If they get
subsequent usernames, they can choose which one to be their primary via
`set_primary_username`.
- There are some state cleanup functions to remove expired usernames
that have not been accepted and dangling usernames whose owners have
called `clear_identity`.

TODO:

- [x] Add migration to runtimes
- [x] Probably do off-chain migration into People Chain genesis
- [x] Address a few TODO questions in code (please review)

---------

Co-authored-by: default avatarLiam Aharon <[email protected]>
Co-authored-by: default avatarGonçalo Pestana <[email protected]>
Co-authored-by: default avatarOliver Tale-Yazdi <[email protected]>
Co-authored-by: default avatarDónal Murray <[email protected]>
      d1f678c0
  15. Jan 09, 2024
  17. Jan 07, 2024
  19. Jan 06, 2024
  21. Jan 04, 2024
  23. Dec 21, 2023
  25. Dec 20, 2023
  27. Dec 19, 2023
  29. Dec 15, 2023
  31. Dec 13, 2023
  33. Dec 12, 2023
    • Ross Bulat's avatar
      Staking: Add `deprecate_controller_batch` AdminOrigin call (#2589) · 048a9c27
      Ross Bulat authored 
      

Partially Addresses #2500

Adds a `deprecate_controller_batch` call to the staking pallet that is
callable by `Root` and `StakingAdmin`. To be used for controller account
deprecation and removed thereafter. Adds
`MaxControllersDeprecationBatch` pallet constant that defines max
possible deprecations per call.

- [x] Add `deprecate_controller_batch` call, and
`MaxControllersInDeprecationBatch` constant.
- [x] Add tests, benchmark, weights. Tests that weight is only consumed
if unique pair.
- [x] Adds `StakingAdmin` origin to staking's `AdminOrigin` type in
westend runtime.
- [x] Determined that worst case 5,900 deprecations does fit into
`maxBlock` `proofSize` and `refTime` in both normal and operational
thresholds, meaning we can deprecate all controllers for each network in
one call.

## Block Weights

By querying `consts.system.blockWeights` we can see that the
`deprecate_controller_batch` weights fit within the `normal` threshold
on Polkadot.

#### `controller_deprecation_batch` where i = 5900:
#### Ref time: 69,933,325,300
#### Proof size: 21,040,390

### Polkadot 

```
// consts.query.blockWeights

maxBlock: {
        refTime: 2,000,000,000,000
        proofSize: 18,446,744,073,709,551,615
}
normal: {
 maxExtrinsic: {
	refTime: 1,479,873,955,000
	proofSize: 13,650,590,614,545,068,195
 }
 maxTotal: {
	refTime: 1,500,000,000,000
	proofSize: 13,835,058,055,282,163,711
 }
}
```

### Kusama

```
// consts.query.blockWeights

  maxBlock: {
    refTime: 2,000,000,000,000
    proofSize: 18,446,744,073,709,551,615
  }
    normal: {
      maxExtrinsic: {
        refTime: 1,479,875,294,000
        proofSize: 13,650,590,614,545,068,195
      }
      maxTotal: {
        refTime: 1,500,000,000,000
        proofSize: 13,835,058,055,282,163,711
      }
}
```

---------

Co-authored-by: command-bot <>
Co-authored-by: default avatarGonçalo Pestana <[email protected]>
      048a9c27
  35. Dec 11, 2023
  37. Dec 08, 2023
    • Sam Johnson's avatar
      Tasks: general system for recognizing and executing service work (#1343) · ac3f14d2
      Sam Johnson authored 
      `polkadot-sdk` version of original tasks PR located here:
https://github.com/paritytech/substrate/pull/14329

Fixes #206

## Status
- [x] Generic `Task` trait
- [x] `RuntimeTask` aggregated enum, compatible with
`construct_runtime!`
- [x] Casting between `Task` and `RuntimeTask` without needing `dyn` or
`Box`
- [x] Tasks Example pallet
- [x] Runtime tests for Tasks example pallet
- [x] Parsing for task-related macros
- [x] Retrofit parsing to make macros optional
- [x] Expansion for task-related macros
- [x] Adds support for args in tasks
- [x] Retrofit tasks example pallet to use macros instead of manual
syntax
- [x] Weights
- [x] Cleanup
- [x] UI tests
- [x] Docs

## Target Syntax
Adapted from
https://github.com/paritytech/polkadot-sdk/issues/206#issue-1865172283



```rust
// NOTE: this enum is optional and is auto-generated by the other macros if not present
#[pallet::task]
pub enum Task<T: Config> {
    AddNumberIntoTotal {
        i: u32,
    }
}

/// Some running total.
#[pallet::storage]
pub(super) type Total<T: Config<I>, I: 'static = ()> =
StorageValue<_, (u32, u32), ValueQuery>;

/// Numbers to be added into the total.
#[pallet::storage]
pub(super) type Numbers<T: Config<I>, I: 'static = ()> =
StorageMap<_, Twox64Concat, u32, u32, OptionQuery>;

#[pallet::tasks_experimental]
impl<T: Config<I>, I: 'static> Pallet<T, I> {
	/// Add a pair of numbers into the totals and remove them.
	#[pallet::task_list(Numbers::<T, I>::iter_keys())]
	#[pallet::task_condition(|i| Numbers::<T, I>::contains_key(i))]
	#[pallet::task_index(0)]
	pub fn add_number_into_total(i: u32) -> DispatchResult {
		let v = Numbers::<T, I>::take(i).ok_or(Error::<T, I>::NotFound)?;
		Total::<T, I>::mutate(|(total_keys, total_values)| {
			*total_keys += i;
			*total_values += v;
		});
		Ok(())
	}
}
```

---------

Co-authored-by: default avatarNikhil Gupta <[email protected]>
Co-authored-by: default avatarkianenigma <[email protected]>
Co-authored-by: Nikhil Gupta <>
Co-authored-by: default avatarGavin Wood <[email protected]>
Co-authored-by: default avatarOliver Tale-Yazdi <[email protected]>
Co-authored-by: default avatargupnik <[email protected]>
      ac3f14d2
  39. Dec 07, 2023
  41. Dec 06, 2023
  43. Dec 05, 2023
  45. Dec 01, 2023
  47. Nov 30, 2023
  49. Nov 29, 2023
  51. Nov 28, 2023
    • Aaro Altonen's avatar
      Rework the event system of `sc-network` (#1370) · e71c484d
      Aaro Altonen authored 
      This commit introduces a new concept called `NotificationService` which
allows Polkadot protocols to communicate with the underlying
notification protocol implementation directly, without routing events
through `NetworkWorker`. This implies that each protocol has its own
service which it uses to communicate with remote peers and that each
`NotificationService` is unique with respect to the underlying
notification protocol, meaning `NotificationService` for the transaction
protocol can only be used to send and receive transaction-related
notifications.

The `NotificationService` concept introduces two additional benefits:
  * allow protocols to start using custom handshakes
  * allow protocols to accept/reject inbound peers

Previously the validation of inbound connections was solely the
responsibility of `ProtocolController`. This caused issues with light
peers and `SyncingEngine` as `ProtocolController` would accept more
peers than `SyncingEngine` could accept which caused peers to have
differing views of their own states. `SyncingEngine` would reject excess
peers but these rejections were not properly communicated to those peers
causing them to assume that they were accepted.

With `NotificationService`, the local handshake is not sent to remote
peer if peer is rejected which allows it to detect that it was rejected.

This commit also deprecates the use of `NetworkEventStream` for all
notification-related events and going forward only DHT events are
provided through `NetworkEventStream`. If protocols wish to follow each
other's events, they must introduce additional abtractions, as is done
for GRANDPA and transactions protocols by following the syncing protocol
through `SyncEventStream`.

Fixes https://github.com/paritytech/polkadot-sdk/issues/512
Fixes https://github.com/paritytech/polkadot-sdk/issues/514
Fixes https://github.com/paritytech/polkadot-sdk/issues/515
Fixes https://github.com/paritytech/polkadot-sdk/issues/554
Fixes https://github.com/paritytech/polkadot-sdk/issues/556

---
These changes are transferred from
https://github.com/paritytech/substrate/pull/14197

 but there are no
functional changes compared to that PR

---------

Co-authored-by: default avatarDmitry Markin <[email protected]>
Co-authored-by: default avatarAlexandru Vasile <[email protected]>
      e71c484d
  53. Nov 22, 2023
  55. Nov 17, 2023
  57. Nov 14, 2023
  59. Nov 13, 2023
    • gupnik's avatar
      Adds syntax for marking calls feeless (#1926) · 60c77a2e
      gupnik authored 
      Fixes https://github.com/paritytech/polkadot-sdk/issues/1725



This PR adds the following changes:
1. An attribute `pallet::feeless_if` that can be optionally attached to
a call like so:
```rust
#[pallet::feeless_if(|_origin: &OriginFor<T>, something: &u32| -> bool {
	*something == 0
})]
pub fn do_something(origin: OriginFor<T>, something: u32) -> DispatchResult {
     ....
}
```
The closure passed accepts references to arguments as specified in the
call fn. It returns a boolean that denotes the conditions required for
this call to be "feeless".

2. A signed extension `SkipCheckIfFeeless<T: SignedExtension>` that
wraps a transaction payment processor such as
`pallet_transaction_payment::ChargeTransactionPayment`. It checks for
all calls annotated with `pallet::feeless_if` to see if the conditions
are met. If so, the wrapped signed extension is not called, essentially
making the call feeless.

In order to use this, you can simply replace your existing signed
extension that manages transaction payment like so:
```diff
- pallet_transaction_payment::ChargeTransactionPayment<Runtime>,
+ pallet_skip_feeless_payment::SkipCheckIfFeeless<
+	Runtime,
+	pallet_transaction_payment::ChargeTransactionPayment<Runtime>,
+ >,
```

### Todo
- [x] Tests
- [x] Docs
- [x] Prdoc

---------

Co-authored-by: Nikhil Gupta <>
Co-authored-by: default avatarOliver Tale-Yazdi <[email protected]>
Co-authored-by: default avatarFrancisco Aguirre <[email protected]>
Co-authored-by: default avatarLiam Aharon <[email protected]>
      60c77a2e
  61. Nov 06, 2023
  63. Nov 05, 2023