Skip to content
  1. Feb 01, 2024
  3. Jan 26, 2024
  5. Jan 23, 2024
    • Niklas Adolfsson's avatar
      rpc: backpressured RPC server (bump jsonrpsee 0.20) (#1313) · e16ef086
      Niklas Adolfsson authored 
      This is a rather big change in jsonrpsee, the major things in this bump
are:
- Server backpressure (the subscription impls are modified to deal with
that)
- Allow custom error types / return types (remove jsonrpsee::core::Error
and jsonrpee::core::CallError)
- Bug fixes (graceful shutdown in particular not used by substrate
anyway)
   - Less dependencies for the clients in particular
   - Return type requires Clone in method call responses
   - Moved to tokio channels
   - Async subscription API (not used in this PR)

Major changes in this PR:
- The subscriptions are now bounded and if subscription can't keep up
with the server it is dropped
- CLI: add parameter to configure the jsonrpc server bounded message
buffer (default is 64)
- Add our own subscription helper to deal with the unbounded streams in
substrate

The most important things in this PR to review is the added helpers
functions in `substrate/client/rpc/src/utils.rs` and the rest is pretty
much chore.

Regarding the "bounded buffer limit" it may cause the server to handle
the JSON-RPC calls
slower than before.

The message size limit is bounded by "--rpc-response-size" thus "by
default 10MB * 64 = 640MB"
but the subscription message size is not covered by this limit and could
be capped as well.

Hopefully the last release prior to 1.0, sorry in advance for a big PR

Previous attempt: https://github.com/paritytech/substrate/pull/13992

Resolves https://github.com/paritytech/polkadot-sdk/issues/748, resolves
https://github.com/paritytech/polkadot-sdk/issues/627
      e16ef086
  7. Jan 22, 2024
  9. Jan 12, 2024
    • Dmitry Markin's avatar
      Extract warp sync strategy from `ChainSync` (#2467) · 5208bed7
      Dmitry Markin authored 
      

Extract `WarpSync` (and `StateSync` as part of warp sync) from
`ChainSync` as independent syncing strategy called by `SyncingEngine`.
Introduce `SyncingStrategy` enum as a proxy between `SyncingEngine` and
specific syncing strategies.

## Limitations
Gap sync is kept in `ChainSync` for now because it shares the same set
of peers as block syncing implementation in `ChainSync`. Extraction of a
common context responsible for peer management in syncing strategies
able to run in parallel is planned for a follow-up PR.

## Further improvements
A possibility of conversion of `SyncingStartegy` into a trait should be
evaluated. The main stopper for this is that different strategies need
to communicate different actions to `SyncingEngine` and respond to
different events / provide different APIs (e.g., requesting
justifications is only possible via `ChainSync` and not through
`WarpSync`; `SendWarpProofRequest` action is only relevant to
`WarpSync`, etc.)

---------

Co-authored-by: default avatarAaro Altonen <[email protected]>
      5208bed7
  11. Jan 10, 2024
    • Alin Dima's avatar
      add fallback request for req-response protocols (#2771) · f2a750ee
      Alin Dima authored 
      Previously, it was only possible to retry the same request on a
different protocol name that had the exact same binary payloads.

Introduce a way of trying a different request on a different protocol if
the first one fails with Unsupported protocol.

This helps with adding new req-response versions in polkadot while
preserving compatibility with unupgraded nodes.

The way req-response protocols were bumped previously was that they were
bundled with some other notifications protocol upgrade, like for async
backing (but that is more complicated, especially if the feature does
not require any changes to a notifications protocol). Will be needed for
implementing https://github.com/polkadot-fellows/RFCs/pull/47

TODO:
- [x]  add tests
- [x] add guidance docs in polkadot about req-response protocol
versioning
      f2a750ee
  13. Jan 08, 2024
  15. Jan 07, 2024
  17. Jan 04, 2024
  19. Dec 27, 2023
  21. Dec 18, 2023
  23. Dec 15, 2023
  25. Dec 14, 2023
  27. Dec 13, 2023
  29. Dec 11, 2023
  31. Dec 07, 2023
  33. Dec 06, 2023
  35. Dec 01, 2023
  37. Nov 30, 2023
  39. Nov 28, 2023
    • Aaro Altonen's avatar
      Rework the event system of `sc-network` (#1370) · e71c484d
      Aaro Altonen authored 
      This commit introduces a new concept called `NotificationService` which
allows Polkadot protocols to communicate with the underlying
notification protocol implementation directly, without routing events
through `NetworkWorker`. This implies that each protocol has its own
service which it uses to communicate with remote peers and that each
`NotificationService` is unique with respect to the underlying
notification protocol, meaning `NotificationService` for the transaction
protocol can only be used to send and receive transaction-related
notifications.

The `NotificationService` concept introduces two additional benefits:
  * allow protocols to start using custom handshakes
  * allow protocols to accept/reject inbound peers

Previously the validation of inbound connections was solely the
responsibility of `ProtocolController`. This caused issues with light
peers and `SyncingEngine` as `ProtocolController` would accept more
peers than `SyncingEngine` could accept which caused peers to have
differing views of their own states. `SyncingEngine` would reject excess
peers but these rejections were not properly communicated to those peers
causing them to assume that they were accepted.

With `NotificationService`, the local handshake is not sent to remote
peer if peer is rejected which allows it to detect that it was rejected.

This commit also deprecates the use of `NetworkEventStream` for all
notification-related events and going forward only DHT events are
provided through `NetworkEventStream`. If protocols wish to follow each
other's events, they must introduce additional abtractions, as is done
for GRANDPA and transactions protocols by following the syncing protocol
through `SyncEventStream`.

Fixes https://github.com/paritytech/polkadot-sdk/issues/512
Fixes https://github.com/paritytech/polkadot-sdk/issues/514
Fixes https://github.com/paritytech/polkadot-sdk/issues/515
Fixes https://github.com/paritytech/polkadot-sdk/issues/554
Fixes https://github.com/paritytech/polkadot-sdk/issues/556

---
These changes are transferred from
https://github.com/paritytech/substrate/pull/14197

 but there are no
functional changes compared to that PR

---------

Co-authored-by: default avatarDmitry Markin <[email protected]>
Co-authored-by: default avatarAlexandru Vasile <[email protected]>
      e71c484d
  41. Nov 23, 2023
  43. Nov 17, 2023
    • Serban Iorga's avatar
      Beefy: small fixes (#2378) · 3ab2bc9f
      Serban Iorga authored 
      Related to #2285

- save the state of the BEEFY gadget after processing a finality proof.
We need this in order to avoid skipping blocks.
- avoid reprocessing the old state when not necessary
      3ab2bc9f
  45. Nov 13, 2023
    • Bastian Köcher's avatar
      pallet-grandpa: Remove `GRANDPA_AUTHORITIES_KEY` (#2181) · ebcf0a0f
      Bastian Köcher authored 
      

Remove the `GRANDPA_AUTHORITIES_KEY` key and its usage. Apparently this
was used in the early days to communicate the grandpa authorities to the
node. However, we have now a runtime api that does this for us. So, this
pull request is moving from the custom managed storage item to a FRAME
managed storage item.

This pr also includes a migration for doing the switch on a running
chain.

---------

Co-authored-by: default avatarDavide Galassi <[email protected]>
      ebcf0a0f
  47. Nov 07, 2023
  49. Nov 06, 2023
  51. Nov 03, 2023
    • Bastian Köcher's avatar
      `sc-block-builder`: Remove `BlockBuilderProvider` (#2099) · ca5f1056
      Bastian Köcher authored 
      The `BlockBuilderProvider` was a trait that was defined in
`sc-block-builder`. The trait was implemented for `Client`. This
basically meant that you needed to import `sc-block-builder` any way to
have access to the block builder. So, this trait was not providing any
real value. This pull request is removing the said trait. Instead of the
trait it introduces a builder for creating a `BlockBuilder`. The builder
currently has the quite fabulous name `BlockBuilderBuilder` (I'm open to
any better name 😅

). The rest of the pull request is about
replacing the old trait with the new builder.

# Downstream code changes

If you used `new_block` or `new_block_at` before you now need to switch
it over to the new `BlockBuilderBuilder` pattern:

```rust
// `new` requires a type that implements `CallApiAt`. 
let mut block_builder = BlockBuilderBuilder::new(client)
                // Then you need to specify the hash of the parent block the block will be build on top of
		.on_parent_block(at)
                // The block builder also needs the block number of the parent block. 
                // Here it is fetched from the given `client` using the `HeaderBackend`
                // However, there also exists `with_parent_block_number` for directly passing the number
		.fetch_parent_block_number(client)
		.unwrap()
                // Enable proof recording if required. This call is optional.
		.enable_proof_recording()
                // Pass the digests. This call is optional.
                .with_inherent_digests(digests)
		.build()
		.expect("Creates new block builder");
```

---------

Co-authored-by: default avatarSebastian Kunert <[email protected]>
Co-authored-by: command-bot <>
      ca5f1056
  53. Nov 01, 2023
    • Dmitry Markin's avatar
      Move syncing code from `sc-network-common` to `sc-network-sync` (#1912) · 1cd6acdf
      Dmitry Markin authored 
      This PR moves syncing-related code from `sc-network-common` to
`sc-network-sync`.

Unfortunately, some parts are tightly integrated with networking, so
they were left in `sc-network-common` for now:

1. `SyncMode` in `common/src/sync.rs` (used in `NetworkConfiguration`).
2. `BlockAnnouncesHandshake`, `BlockRequest`, `BlockResponse`, etc. in
`common/src/sync/message.rs` (used in `src/protocol.rs` and
`src/protocol/message.rs`).

More substantial refactoring is needed to decouple syncing and
networking completely, including getting rid of the hardcoded sync
protocol.

## Release notes

Move syncing-related code from `sc-network-common` to `sc-network-sync`.
Delete `ChainSync` trait as it's never used (the only implementation is
accessed directly from `SyncingEngine` and exposes a lot of public
methods that are not part of the trait). Some new trait(s) for syncing
will likely be introduced as part of Sync 2.0 refactoring to represent
syncing strategies.
      1cd6acdf
  55. Oct 27, 2023
    • juangirini's avatar
      feat: FRAME umbrella crate. (#1337) · 43415ef5
      juangirini authored 
      ### Original PR https://github.com/paritytech/substrate/pull/14137

This PR brings in the first version of the "_`frame` umbrella crate_".
This crate is intended to serve two purposes:

1. documentation
2. easier development with frame. Ideally, we want most users to be able
to build a frame-based pallet and runtime using just `frame` (plus
`scale-codec` and `scale-info`).

The crate is not finalized and is not yet intended for external use.
Therefore, the version is set to `0.0.1-dev`, this PR is `silent`, and
the entire crate is hidden behind the `experimental` flag. The main
intention in merging it early on is to be able to iterate on it in the
rest of
[`developer-hub`](https://github.com/paritytech/polkadot-sdk-docs/)
efforts.

The public API of the `frame` crate is at the moment as follows: 

```
pub mod frame
pub use frame::log
pub use frame::pallet
pub mod frame::arithmetic
pub use frame::arithmetic::<<sp_arithmetic::*>>
pub use frame::arithmetic::<<sp_arithmetic::traits::*>>
pub mod frame::deps
pub use frame::deps::codec
pub use frame::deps::frame_executive
pub use frame::deps::frame_support
pub use frame::deps::frame_system
pub use frame::deps::scale_info
pub use frame::deps::sp_api
pub use frame::deps::sp_arithmetic
pub use frame::deps::sp_block_builder
pub use frame::deps::sp_consensus_aura
pub use frame::deps::sp_consensus_grandpa
pub use frame::deps::sp_core
pub use frame::deps::sp_inherents
pub use frame::deps::sp_io
pub use frame::deps::sp_offchain
pub use frame::deps::sp_runtime
pub use frame::deps::sp_std
pub use frame::deps::sp_version
pub mod frame::derive
pub use frame::derive::CloneNoBound
pub use frame::derive::Debug
pub use frame::derive::Debug
pub use frame::derive::DebugNoBound
pub use frame::derive::Decode
pub use frame::derive::Decode
pub use frame::derive::DefaultNoBound
pub use frame::derive::Encode
pub use frame::derive::Encode
pub use frame::derive::EqNoBound
pub use frame::derive::PartialEqNoBound
pub use frame::derive::RuntimeDebug
pub use frame::derive::RuntimeDebugNoBound
pub use frame::derive::TypeInfo
pub use frame::derive::TypeInfo
pub mod frame::prelude
pub use frame::prelude::<<frame_support::pallet_prelude::*>>
pub use frame::prelude::<<frame_system::pallet_prelude::*>>
pub use frame::prelude::<<sp_std::prelude::*>>
pub use frame::prelude::CloneNoBound
pub use frame::prelude::Debug
pub use frame::prelude::Debug
pub use frame::prelude::DebugNoBound
pub use frame::prelude::Decode
pub use frame::prelude::Decode
pub use frame::prelude::DefaultNoBound
pub use frame::prelude::Encode
pub use frame::prelude::Encode
pub use frame::prelude::EqNoBound
pub use frame::prelude::PartialEqNoBound
pub use frame::prelude::RuntimeDebug
pub use frame::prelude::RuntimeDebugNoBound
pub use frame::prelude::TypeInfo
pub use frame::prelude::TypeInfo
pub use frame::prelude::frame_system
pub mod frame::primitives
pub use frame::primitives::BlakeTwo256
pub use frame::primitives::H160
pub use frame::primitives::H256
pub use frame::primitives::H512
pub use frame::primitives::Hash
pub use frame::primitives::Keccak256
pub use frame::primitives::U256
pub use frame::primitives::U512
pub mod frame::runtime
pub mod frame::runtime::apis
pub use frame::runtime::apis::<<frame_system_rpc_runtime_api::*>>
pub use frame::runtime::apis::<<sp_api::*>>
pub use frame::runtime::apis::<<sp_block_builder::*>>
pub use frame::runtime::apis::<<sp_consensus_aura::*>>
pub use frame::runtime::apis::<<sp_consensus_grandpa::*>>
pub use frame::runtime::apis::<<sp_offchain::*>>
pub use frame::runtime::apis::<<sp_session::runtime_api::*>>
pub use frame::runtime::apis::<<sp_transaction_pool::runtime_api::*>>
pub use frame::runtime::apis::ApplyExtrinsicResult
pub use frame::runtime::apis::CheckInherentsResult
pub use frame::runtime::apis::InherentData
pub use frame::runtime::apis::OpaqueMetadata
pub use frame::runtime::apis::impl_runtime_apis
pub use frame::runtime::apis::sp_api
pub mod frame::runtime::prelude
pub use frame::runtime::prelude::<<frame_executive::*>>
pub use frame::runtime::prelude::ConstBool
pub use frame::runtime::prelude::ConstI128
pub use frame::runtime::prelude::ConstI16
pub use frame::runtime::prelude::ConstI32
pub use frame::runtime::prelude::ConstI64
pub use frame::runtime::prelude::ConstI8
pub use frame::runtime::prelude::ConstU128
pub use frame::runtime::prelude::ConstU16
pub use frame::runtime::prelude::ConstU32
pub use frame::runtime::prelude::ConstU64
pub use frame::runtime::prelude::ConstU8
pub use frame::runtime::prelude::NativeVersion
pub use frame::runtime::prelude::RuntimeVersion
pub use frame::runtime::prelude::construct_runtime
pub use frame::runtime::prelude::create_runtime_str
pub use frame::runtime::prelude::derive_impl
pub use frame::runtime::prelude::frame_support
pub use frame::runtime::prelude::ord_parameter_types
pub use frame::runtime::prelude::parameter_types
pub use frame::runtime::prelude::runtime_version
pub mod frame::runtime::testing_prelude
pub use frame::runtime::testing_prelude::BuildStorage
pub use frame::runtime::testing_prelude::Storage
pub mod frame::runtime::types_common
pub type frame::runtime::types_common::AccountId = <<frame::runtime::types_common::Signature as sp_runtime::traits::Verify>::Signer as sp_runtime::traits::IdentifyAccount>::AccountId
pub type frame::runtime::types_common::BlockNumber = u32
pub type frame::runtime::types_common::BlockOf<T, Extra> = sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<frame::runtime::types_common::BlockNumber, sp_runtime::traits::BlakeTwo256>, sp_runtime::generic::unchecked_extrinsic::UncheckedExtrinsic<sp_runtime::multiaddress::MultiAddress<frame::runtime::types_common::AccountId, ()>, <T as frame_system::pallet::Config>::RuntimeCall, frame::runtime::types_common::Signature, Extra>>
pub type frame::runtime::types_common::OpaqueBlock = sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<frame::runtime::types_common::BlockNumber, sp_runtime::traits::BlakeTwo256>, sp_runtime::OpaqueExtrinsic>
pub type frame::runtime::types_common::Signature = sp_runtime::MultiSignature
pub type frame::runtime::types_common::SystemSignedExtensionsOf<T> = (frame_system::extensions::check_non_zero_sender::CheckNonZeroSender<T>, frame_system::extensions::check_spec_version::CheckSpecVersion<T>, frame_system::extensions::check_tx_version::CheckTxVersion<T>, frame_system::extensions::check_genesis::CheckGenesis<T>, frame_system::extensions::check_mortality::CheckMortality<T>, frame_system::extensions::check_nonce::CheckNonce<T>, frame_system::extensions::check_weight::CheckWeight<T>)
pub mod frame::testing_prelude
pub use frame::testing_prelude::<<frame_executive::*>>
pub use frame::testing_prelude::<<frame_system::mocking::*>>
pub use frame::testing_prelude::BuildStorage
pub use frame::testing_prelude::ConstBool
pub use frame::testing_prelude::ConstI128
pub use frame::testing_prelude::ConstI16
pub use frame::testing_prelude::ConstI32
pub use frame::testing_prelude::ConstI64
pub use frame::testing_prelude::ConstI8
pub use frame::testing_prelude::ConstU128
pub use frame::testing_prelude::ConstU16
pub use frame::testing_prelude::ConstU32
pub use frame::testing_prelude::ConstU64
pub use frame::testing_prelude::ConstU8
pub use frame::testing_prelude::NativeVersion
pub use frame::testing_prelude::RuntimeVersion
pub use frame::testing_prelude::Storage
pub use frame::testing_prelude::TestState
pub use frame::testing_prelude::assert_err
pub use frame::testing_prelude::assert_err_ignore_postinfo
pub use frame::testing_prelude::assert_error_encoded_size
pub use frame::testing_prelude::assert_noop
pub use frame::testing_prelude::assert_ok
pub use frame::testing_prelude::assert_storage_noop
pub use frame::testing_prelude::construct_runtime
pub use frame::testing_prelude::create_runtime_str
pub use frame::testing_prelude::derive_impl
pub use frame::testing_prelude::frame_support
pub use frame::testing_prelude::frame_system
pub use frame::testing_prelude::if_std
pub use frame::testing_prelude::ord_parameter_types
pub use frame::testing_prelude::parameter_types
pub use frame::testing_prelude::runtime_version
pub use frame::testing_prelude::storage_alias
pub mod frame::traits
pub use frame::traits::<<frame_support::traits::*>>
pub use frame::traits::<<sp_runtime::traits::*>>
```

---

The road to full stabilization is

- [ ] https://github.com/paritytech/polkadot-sdk/issues/127


- [ ] have a more intentional version bump, as opposed to the current bi
weekly force-major-bump
- [ ] revise the internal API of `frame`, especially what goes into the
`prelude`s.
- [ ] migrate all internal pallets and runtime to use `frame`

---------

Co-authored-by: default avatarkianenigma <[email protected]>
Co-authored-by: default avatarKian Paimani <[email protected]>
Co-authored-by: default avatarOliver Tale-Yazdi <[email protected]>
Co-authored-by: default avatarFrancisco Aguirre <[email protected]>
      43415ef5
  57. Oct 16, 2023
    • Adrian Catangiu's avatar
      sc-consensus-beefy: fix initialization when state is unavailable (#1888) · 646ecd0e
      Adrian Catangiu authored 
      Fix situation where BEEFY initial validator set could not be determined.

If state is unavailable at BEEFY genesis block to get initial validator
set, get the info from header digests. For this, we need to walk back
the chain starting from BEEFY genesis looking for the BEEFY digest
announcing the active validator set for that respective session.

This commit fixes a silly bug where walking back the chain was stopped
when reaching BEEFY genesis block, which is incorrect when BEEFY genesis
is not session boundary block. When BEEFY genesis is set to some random
block within a session, we need to walk back to the start of the session
to see the validator set announcement.

Added regression test for this fix.

Fixes https://github.com/paritytech/polkadot-sdk/issues/1885



Signed-off-by: default avatarAdrian Catangiu <[email protected]>
      646ecd0e
  59. Oct 13, 2023
    • Adrian Catangiu's avatar
      sc-consensus-beefy: improve gossip logic (#1852) · 83206000
      Adrian Catangiu authored 
      - Remove cached messages used for deduplication in `GossipValidator`
since they're already deduplicated in upper layer `NetworkGossip`.
- Add cache for "justified rounds" to quickly discard any further (even
if potentially different) justifications at the gossip level, once a
valid one (for a respective round) is submitted to the worker.
- Add short-circuit in worker `finalize()` method to not attempt to
finalize same block multiple times (for example when we get
justifications for same block from multiple components like
block-import, gossip or on-demand).
- Change a test which had A LOT of latency in syncing blocks for some
weird reason and would only run after ~150seconds. It now runs
instantly.

Fixes https://github.com/paritytech/polkadot-sdk/issues/1728
      83206000
  61. Sep 29, 2023
  63. Sep 27, 2023
    • Michal Kucharczyk's avatar
      `BlockId` removal: `tx-pool` refactor (#1678) · ab3a3bc2
      Michal Kucharczyk authored 
      

It changes following APIs:
- trait `ChainApi`
-- `validate_transaction`

- trait `TransactionPool` 
--`submit_at`
--`submit_one`
--`submit_and_watch`

and some implementation details, in particular:
- impl `Pool` 
--`submit_at`
--`resubmit_at`
--`submit_one`
--`submit_and_watch`
--`prune_known`
--`prune`
--`prune_tags`
--`resolve_block_number`
--`verify`
--`verify_one`

- revalidation queue

All tests are also adjusted.

---------

Co-authored-by: command-bot <>
Co-authored-by: default avatarBastian Köcher <[email protected]>
      ab3a3bc2
  65. Sep 17, 2023
    • Davide Galassi's avatar
      Babe epoch newtype (#1596) · c7dbfc21
      Davide Galassi authored 
      Removal of verbatim duplication of BABE's `Epoch` struct in the client.

I think is better to have one single definition and wrap the primitive
`Epoch` in a newtype (required because we need to implement the `Epoch`
trait).
      c7dbfc21
  67. Sep 15, 2023
  69. Sep 12, 2023
  71. Sep 11, 2023
  73. Sep 05, 2023
  75. Sep 04, 2023
    • Chevdor's avatar
      Markdown linter (#1309) · a30092ab
      Chevdor authored 
      

* Add markdown linting

- add linter default rules
- adapt rules to current code
- fix the code for linting to pass
- add CI check

fix #1243

* Fix markdown for Substrate
* Fix tooling install
* Fix workflow
* Add documentation
* Remove trailing spaces
* Update .github/.markdownlint.yaml

Co-authored-by: default avatarOliver Tale-Yazdi <[email protected]>
* Fix mangled markdown/lists
* Fix captalization issues on known words
      a30092ab