- Mar 19, 2024
-
-
Davide Galassi authoredIntroduces `CryptoBytes` type defined as: ```rust pub struct CryptoBytes<const N: usize, Tag = ()>(pub [u8; N], PhantomData<fn() -> Tag>); ``` The type implements a bunch of methods and traits which are typically expected from a byte array newtype (NOTE: some of the methods and trait implementations IMO are a bit redundant, but I decided to maintain them all to not change too much stuff in this PR) It also introduces two (generic) typical consumers of `CryptoBytes`: `PublicBytes` and `SignatureBytes`. ```rust pub struct PublicTag; pub PublicBytes<const N: usize, CryptoTag> = CryptoBytes<N, (PublicTag, CryptoTag)>; pub struct SignatureTag; pub SignatureBytes<const N: usize, CryptoTag> = CryptoBytes<N, (SignatureTag, CryptoTag)>; ``` Both of them use a tag to differentiate the two types at a higher level. Downstream specializations will further specialize using a dedicated crypto tag. For example in ECDSA: ```rust pub struct EcdsaTag; pub type Public = PublicBytes<PUBLIC_KEY_SERIALIZED_SIZE, EcdsaTag>; pub type Signature = PublicBytes<PUBLIC_KEY_SERIALIZED_SIZE, EcdsaTag>; ``` Overall we have a cleaner and most importantly **consistent** code for all the types involved All these details are opaque to the end user which can use `Public` and `Signature` for the cryptos as before
-
- Mar 13, 2024
-
-
georgepisaltu authored
Revert "FRAME: Create `TransactionExtension` as a replacement for `SignedExtension` (#2280)" (#3665) This PR reverts #2280 which introduced `TransactionExtension` to replace `SignedExtension`. As a result of the discussion [here](https://github.com/paritytech/polkadot-sdk/pull/3623#issuecomment-1986789700), the changes will be reverted for now with plans to reintroduce the concept in the future. --------- Signed-off-by:georgepisaltu <[email protected]>
-
- Mar 04, 2024
-
-
Gavin Wood authored
Closes #2160 First part of [Extrinsic Horizon](https://github.com/paritytech/polkadot-sdk/issues/2415) Introduces a new trait `TransactionExtension` to replace `SignedExtension`. Introduce the idea of transactions which obey the runtime's extensions and have according Extension data (né Extra data) yet do not have hard-coded signatures. Deprecate the terminology of "Unsigned" when used for transactions/extrinsics owing to there now being "proper" unsigned transactions which obey the extension framework and "old-style" unsigned which do not. Instead we have __*General*__ for the former and __*Bare*__ for the latter. (Ultimately, the latter will be phased out as a type of transaction, and Bare will only be used for Inherents.) Types of extrinsic are now therefore: - Bare (no hardcoded signature, no Extra data; used to be known as "Unsigned") - Bare transactions (deprecated): Gossiped, validated with `ValidateUnsigned` (deprecated) and the `_bare_compat` bits of `TransactionExtension` (deprecated). - Inherents: Not gossiped, validated with `ProvideInherent`. - Extended (Extra data): Gossiped, validated via `TransactionExtension`. - Signed transactions (with a hardcoded signature). - General transactions (without a hardcoded signature). `TransactionExtension` differs from `SignedExtension` because: - A signature on the underlying transaction may validly not be present. - It may alter the origin during validation. - `pre_dispatch` is renamed to `prepare` and need not contain the checks present in `validate`. - `validate` and `prepare` is passed an `Origin` rather than a `AccountId`. - `validate` may pass arbitrary information into `prepare` via a new user-specifiable type `Val`. - `AdditionalSigned`/`additional_signed` is renamed to `Implicit`/`implicit`. It is encoded *for the entire transaction* and passed in to each extension as a new argument to `validate`. This facilitates the ability of extensions to acts as underlying crypto. There is a new `DispatchTransaction` trait which contains only default function impls and is impl'ed for any `TransactionExtension` impler. It provides several utility functions which reduce some of the tedium from using `TransactionExtension` (indeed, none of its regular functions should now need to be called directly). Three transaction version discriminator ("versions") are now permissible: - 0b000000100: Bare (used to be called "Unsigned"): contains Signature or Extra (extension data). After bare transactions are no longer supported, this will strictly identify an Inherents only. - 0b100000100: Old-school "Signed" Transaction: contains Signature and Extra (extension data). - 0b010000100: New-school "General" Transaction: contains Extra (extension data), but no Signature. For the New-school General Transaction, it becomes trivial for authors to publish extensions to the mechanism for authorizing an Origin, e.g. through new kinds of key-signing schemes, ZK proofs, pallet state, mutations over pre-authenticated origins or any combination of the above. ## Code Migration ### NOW: Getting it to build Wrap your `SignedExtension`s in `AsTransactionExtension`. This should be accompanied by renaming your aggregate type in line with the new terminology. E.g. Before: ```rust /// The SignedExtension to the basic transaction logic. pub type SignedExtra = ( /* snip */ MySpecialSignedExtension, ); /// Unchecked extrinsic type as expected by this runtime. pub type UncheckedExtrinsic = generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, SignedExtra>; ``` After: ```rust /// The extension to the basic transaction logic. pub type TxExtension = ( /* snip */ AsTransactionExtension<MySpecialSignedExtension>, ); /// Unchecked extrinsic type as expected by this runtime. pub type UncheckedExtrinsic = generic::UncheckedExtrinsic<Address, RuntimeCall, Signature, TxExtension>; ``` You'll also need to alter any transaction building logic to add a `.into()` to make the conversion happen. E.g. Before: ```rust fn construct_extrinsic( /* snip */ ) -> UncheckedExtrinsic { let extra: SignedExtra = ( /* snip */ MySpecialSignedExtension::new(/* snip */), ); let payload = SignedPayload::new(call.clone(), extra.clone()).unwrap(); let signature = payload.using_encoded(|e| sender.sign(e)); UncheckedExtrinsic::new_signed( /* snip */ Signature::Sr25519(signature), extra, ) } ``` After: ```rust fn construct_extrinsic( /* snip */ ) -> UncheckedExtrinsic { let tx_ext: TxExtension = ( /* snip */ MySpecialSignedExtension::new(/* snip */).into(), ); let payload = SignedPayload::new(call.clone(), tx_ext.clone()).unwrap(); let signature = payload.using_encoded(|e| sender.sign(e)); UncheckedExtrinsic::new_signed( /* snip */ Signature::Sr25519(signature), tx_ext, ) } ``` ### SOON: Migrating to `TransactionExtension` Most `SignedExtension`s can be trivially converted to become a `TransactionExtension`. There are a few things to know. - Instead of a single trait like `SignedExtension`, you should now implement two traits individually: `TransactionExtensionBase` and `TransactionExtension`. - Weights are now a thing and must be provided via the new function `fn weight`. #### `TransactionExtensionBase` This trait takes care of anything which is not dependent on types specific to your runtime, most notably `Call`. - `AdditionalSigned`/`additional_signed` is renamed to `Implicit`/`implicit`. - Weight must be returned by implementing the `weight` function. If your extension is associated with a pallet, you'll probably want to do this via the pallet's existing benchmarking infrastructure. #### `TransactionExtension` Generally: - `pre_dispatch` is now `prepare` and you *should not reexecute the `validate` functionality in there*! - You don't get an account ID any more; you get an origin instead. If you need to presume an account ID, then you can use the trait function `AsSystemOriginSigner::as_system_origin_signer`. - You get an additional ticket, similar to `Pre`, called `Val`. This defines data which is passed from `validate` into `prepare`. This is important since you should not be duplicating logic from `validate` to `prepare`, you need a way of passing your working from the former into the latter. This is it. - This trait takes two type parameters: `Call` and `Context`. `Call` is the runtime call type which used to be an associated type; you can just move it to become a type parameter for your trait impl. `Context` is not currently used and you can safely implement over it as an unbounded type. - There's no `AccountId` associated type any more. Just remove it. Regarding `validate`: - You get three new parameters in `validate`; all can be ignored when migrating from `SignedExtension`. - `validate` returns a tuple on success; the second item in the tuple is the new ticket type `Self::Val` which gets passed in to `prepare`. If you use any information extracted during `validate` (off-chain and on-chain, non-mutating) in `prepare` (on-chain, mutating) then you can pass it through with this. For the tuple's last item, just return the `origin` argument. Regarding `prepare`: - This is renamed from `pre_dispatch`, but there is one change: - FUNCTIONALITY TO VALIDATE THE TRANSACTION NEED NOT BE DUPLICATED FROM `validate`!! - (This is different to `SignedExtension` which was required to run the same checks in `pre_dispatch` as in `validate`.) Regarding `post_dispatch`: - Since there are no unsigned transactions handled by `TransactionExtension`, `Pre` is always defined, so the first parameter is `Self::Pre` rather than `Option<Self::Pre>`. If you make use of `SignedExtension::validate_unsigned` or `SignedExtension::pre_dispatch_unsigned`, then: - Just use the regular versions of these functions instead. - Have your logic execute in the case that the `origin` is `None`. - Ensure your transaction creation logic creates a General Transaction rather than a Bare Transaction; this means having to include all `TransactionExtension`s' data. - `ValidateUnsigned` can still be used (for now) if you need to be able to construct transactions which contain none of the extension data, however these will be phased out in stage 2 of the Transactions Horizon, so you should consider moving to an extension-centric design. ## TODO - [x] Introduce `CheckSignature` impl of `TransactionExtension` to ensure it's possible to have crypto be done wholly in a `TransactionExtension`. - [x] Deprecate `SignedExtension` and move all uses in codebase to `TransactionExtension`. - [x] `ChargeTransactionPayment` - [x] `DummyExtension` - [x] `ChargeAssetTxPayment` (asset-tx-payment) - [x] `ChargeAssetTxPayment` (asset-conversion-tx-payment) - [x] `CheckWeight` - [x] `CheckTxVersion` - [x] `CheckSpecVersion` - [x] `CheckNonce` - [x] `CheckNonZeroSender` - [x] `CheckMortality` - [x] `CheckGenesis` - [x] `CheckOnlySudoAccount` - [x] `WatchDummy` - [x] `PrevalidateAttests` - [x] `GenericSignedExtension` - [x] `SignedExtension` (chain-polkadot-bulletin) - [x] `RefundSignedExtensionAdapter` - [x] Implement `fn weight` across the board. - [ ] Go through all pre-existing extensions which assume an account signer and explicitly handle the possibility of another kind of origin. - [x] `CheckNonce` should probably succeed in the case of a non-account origin. - [x] `CheckNonZeroSender` should succeed in the case of a non-account origin. - [x] `ChargeTransactionPayment` and family should fail in the case of a non-account origin. - [ ] - [x] Fix any broken tests. --------- Signed-off-by:Alexandru Vasile <[email protected]> Signed-off-by:
dependabot[bot] <[email protected]> Signed-off-by:
Oliver Tale-Yazdi <[email protected]> Signed-off-by:
Alexandru Gheorghe <[email protected]> Signed-off-by:
Andrei Sandu <[email protected]> Co-authored-by:
Nikhil Gupta <[email protected]> Co-authored-by:
georgepisaltu <[email protected]> Co-authored-by:
Chevdor <[email protected]> Co-authored-by:
Bastian Köcher <[email protected]> Co-authored-by:
Maciej <[email protected]> Co-authored-by:
Javier Viola <[email protected]> Co-authored-by:
Marcin S. <[email protected]> Co-authored-by:
Tsvetomir Dimitrov <[email protected]> Co-authored-by:
Javier Bullrich <[email protected]> Co-authored-by:
Koute <[email protected]> Co-authored-by:
Adrian Catangiu <[email protected]> Co-authored-by:
Vladimir Istyufeev <[email protected]>
Co-authored-by: Ross Bulat <[email protected]> Co-authored-by:
Gonçalo Pestana <[email protected]> Co-authored-by:
Liam Aharon <[email protected]> Co-authored-by:
Svyatoslav Nikolsky <[email protected]> Co-authored-by:
André Silva <[email protected]> Co-authored-by:
s0me0ne-unkn0wn <[email protected]> Co-authored-by:
ordian <[email protected]> Co-authored-by:
Sebastian Kunert <[email protected]> Co-authored-by:
Aaro Altonen <[email protected]> Co-authored-by:
Dmitry Markin <[email protected]> Co-authored-by:
Alexandru Vasile <[email protected]> Co-authored-by:
Alexander Samusev <[email protected]> Co-authored-by:
Julian Eager <[email protected]> Co-authored-by:
Michal Kucharczyk <[email protected]> Co-authored-by:
Davide Galassi <[email protected]> Co-authored-by:
Dónal Murray <[email protected]> Co-authored-by:
yjh <[email protected]> Co-authored-by:
Tom Mi <[email protected]> Co-authored-by:
dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by:
Will | Paradox | ParaNodes.io <[email protected]> Co-authored-by:
Bastian Köcher <[email protected]> Co-authored-by:
Joshy Orndorff <[email protected]> Co-authored-by:
Joshy Orndorff <[email protected]> Co-authored-by:
PG Herveou <[email protected]> Co-authored-by:
Alexander Theißen <[email protected]> Co-authored-by:
Kian Paimani <[email protected]> Co-authored-by:
Juan Girini <[email protected]> Co-authored-by:
bader y <[email protected]> Co-authored-by:
James Wilson <[email protected]> Co-authored-by:
joe petrowski <[email protected]> Co-authored-by:
asynchronous rob <[email protected]> Co-authored-by:
Parth <[email protected]> Co-authored-by:
Andrew Jones <[email protected]> Co-authored-by:
Jonathan Udd <[email protected]> Co-authored-by:
Serban Iorga <[email protected]> Co-authored-by:
Egor_P <[email protected]> Co-authored-by:
Branislav Kontur <[email protected]> Co-authored-by:
Evgeny Snitko <[email protected]> Co-authored-by:
Just van Stam <[email protected]> Co-authored-by:
Francisco Aguirre <[email protected]> Co-authored-by:
gupnik <[email protected]> Co-authored-by:
dzmitry-lahoda <[email protected]> Co-authored-by:
zhiqiangxu <[email protected]> Co-authored-by:
Nazar Mokrynskyi <[email protected]> Co-authored-by:
Anwesh <[email protected]> Co-authored-by:
cheme <[email protected]> Co-authored-by:
Sam Johnson <[email protected]> Co-authored-by:
kianenigma <[email protected]> Co-authored-by:
Jegor Sidorenko <[email protected]> Co-authored-by:
Muharem <[email protected]> Co-authored-by:
joepetrowski <[email protected]> Co-authored-by:
Alexandru Gheorghe <[email protected]> Co-authored-by:
Gabriel Facco de Arruda <[email protected]> Co-authored-by:
Squirrel <[email protected]> Co-authored-by:
Andrei Sandu <[email protected]> Co-authored-by:
-
- Feb 29, 2024
-
-
Andrei Sandu authored
Signed-off-by: -
Tsvetomir Dimitrov authored
This PR removes `AssignmentProviderConfig` and uses the corresponding ondemand parameters from `HostConfiguration` instead. Additionally `scheduling_lookahead` and all coretime/ondemand related parameters are extracted in a separate struct - `SchedulerParams`. The most relevant commit from the PR is [this one](https://github.com/paritytech/polkadot-sdk/pull/3181/commits/830bc0f5). Fixes https://github.com/paritytech/polkadot-sdk/issues/2268 --------- Co-authored-by: command-bot <>
-
- Feb 28, 2024
-
-
Oliver Tale-Yazdi authoredThis MR is the merge of https://github.com/paritytech/substrate/pull/14414 and https://github.com/paritytech/substrate/pull/14275. It implements [RFC#13](https://github.com/polkadot-fellows/RFCs/pull/13), closes https://github.com/paritytech/polkadot-sdk/issues/198. ----- This Merge request introduces three major topicals: 1. Multi-Block-Migrations 1. New pallet `poll` hook for periodic service work 1. Replacement hooks for `on_initialize` and `on_finalize` in cases where `poll` cannot be used and some more general changes to FRAME. The changes for each topical span over multiple crates. They are listed in topical order below. # 1.) Multi-Block-Migrations Multi-Block-Migrations are facilitated by creating `pallet_migrations` and configuring `System::Config::MultiBlockMigrator` to point to it. Executive picks this up and triggers one step of the migrations pallet per block. The chain is in lockdown mode for as long as an MBM is ongoing. Executive does this by polling `MultiBlockMigrator::ongoing` and not allowing any transaction in a block, if true. A MBM is defined through trait `SteppedMigration`. A condensed version looks like this: ```rust /// A migration that can proceed in multiple steps. pub trait SteppedMigration { type Cursor: FullCodec + MaxEncodedLen; type Identifier: FullCodec + MaxEncodedLen; fn id() -> Self::Identifier; fn max_steps() -> Option<u32>; fn step( cursor: Option<Self::Cursor>, meter: &mut WeightMeter, ) -> Result<Option<Self::Cursor>, SteppedMigrationError>; } ``` `pallet_migrations` can be configured with an aggregated tuple of these migrations. It then starts to migrate them one-by-one on the next runtime upgrade. Two things are important here: - 1. Doing another runtime upgrade while MBMs are ongoing is not a good idea and can lead to messed up state. - 2. **Pallet Migrations MUST BE CONFIGURED IN `System::Config`, otherwise it is not used.** The pallet supports an `UpgradeStatusHandler` that can be used to notify external logic of upgrade start/finish (for example to pause XCM dispatch). Error recovery is very limited in the case that a migration errors or times out (exceeds its `max_steps`). Currently the runtime dev can decide in `FailedMigrationHandler::failed` how to handle this. One follow-up would be to pair this with the `SafeMode` pallet and enact safe mode when an upgrade fails, to allow governance to rescue the chain. This is currently not possible, since governance is not `Mandatory`. ## Runtime API - `Core`: `initialize_block` now returns `ExtrinsicInclusionMode` to inform the Block Author whether they can push transactions. ### Integration Add it to your runtime implementation of `Core` and `BlockBuilder`: ```patch diff --git a/runtime/src/lib.rs b/runtime/src/lib.rs @@ impl_runtime_apis! { impl sp_block_builder::Core<Block> for Runtime { - fn initialize_block(header: &<Block as BlockT>::Header) { + fn initialize_block(header: &<Block as BlockT>::Header) -> RuntimeExecutiveMode { Executive::initialize_block(header) } ... } ``` # 2.) `poll` hook A new pallet hook is introduced: `poll`. `Poll` is intended to replace mostly all usage of `on_initialize`. The reason for this is that any code that can be called from `on_initialize` cannot be migrated through an MBM. Currently there is no way to statically check this; the implication is to use `on_initialize` as rarely as possible. Failing to do so can result in broken storage invariants. The implementation of the poll hook depends on the `Runtime API` changes that are explained above. # 3.) Hard-Deadline callbacks Three new callbacks are introduced and configured on `System::Config`: `PreInherents`, `PostInherents` and `PostTransactions`. These hooks are meant as replacement for `on_initialize` and `on_finalize` in cases where the code that runs cannot be moved to `poll`. The reason for this is to make the usage of HD-code (hard deadline) more explicit - again to prevent broken invariants by MBMs. # 4.) FRAME (general changes) ## `frame_system` pallet A new memorize storage item `InherentsApplied` is added. It is used by executive to track whether inherents have already been applied. Executive and can then execute the MBMs directly between inherents and transactions. The `Config` gets five new items: - `SingleBlockMigrations` this is the new way of configuring migrations that run in a single block. Previously they were defined as last generic argument of `Executive`. This shift is brings all central configuration about migrations closer into view of the developer (migrations that are configured in `Executive` will still work for now but is deprecated). - `MultiBlockMigrator` this can be configured to an engine that drives MBMs. One example would be the `pallet_migrations`. Note that this is only the engine; the exact MBMs are injected into the engine. - `PreInherents` a callback that executes after `on_initialize` but before inherents. - `PostInherents` a callback that executes after all inherents ran (including MBMs and `poll`). - `PostTransactions` in symmetry to `PreInherents`, this one is called before `on_finalize` but after all transactions. A sane default is to set all of these to `()`. Example diff suitable for any chain: ```patch @@ impl frame_system::Config for Test { type MaxConsumers = ConstU32<16>; + type SingleBlockMigrations = (); + type MultiBlockMigrator = (); + type PreInherents = (); + type PostInherents = (); + type PostTransactions = (); } ``` An overview of how the block execution now looks like is here. The same graph is also in the rust doc. <details><summary>Block Execution Flow</summary> <p>  </p> </details> ## Inherent Order Moved to https://github.com/paritytech/polkadot-sdk/pull/2154 --------------- ## TODO - [ ] Check that `try-runtime` still works - [ ] Ensure backwards compatibility with old Runtime APIs - [x] Consume weight correctly - [x] Cleanup --------- Signed-off-by:Gavin Wood <[email protected]> Co-authored-by:
-
Liam Aharon authoredCloses https://github.com/paritytech/polkadot-sdk-docs/issues/55 - Changes 'current storage version' terminology to less ambiguous 'in-code storage version' (suggestion by @ggwpez ) - Adds a new example pallet `pallet-example-single-block-migrations` - Adds a new reference doc to replace https://docs.substrate.io/maintain/runtime-upgrades/ (temporarily living in the pallet while we wait for developer hub PR to merge) - Adds documentation for the `storage_alias` macro - Improves `trait Hooks` docs - Improves `trait GetStorageVersion` docs - Update the suggested patterns for using `VersionedMigration`, so that version unchecked migrations are never exported - Prevents accidental usage of version unchecked migrations in runtimes https://github.com/paritytech/substrate/pull/14421#discussion_r1255467895 - Unversioned migration code is kept inside `mod version_unchecked`, versioned code is kept in `pub mod versioned` - It is necessary to use modules to limit visibility because the inner migration must be `pub`. See https://github.com/rust-lang/rust/issues/30905 and https://internals.rust-lang.org/t/lang-team-minutes-private-in-public-rules/4504/40 for more. ### todo - [x] move to reference docs to proper place within sdk-docs (now that https://github.com/paritytech/polkadot-sdk/pull/2102 is merged) - [x] prdoc --------- Co-authored-by:
-
maksimryndin authored
resolve https://github.com/paritytech/polkadot-sdk/issues/3116 a follow-up on https://github.com/paritytech/polkadot-sdk/pull/3061#pullrequestreview-1847530265: - [x] reuse collator overseer builder for polkadot-node and collator - [x] run zombienet test (0001-parachains-smoke-test.toml) - [x] make wasm build errors more user-friendly for an easier problem detection when using different toolchains in Rust --------- Co-authored-by:
-
- Feb 02, 2024
-
-
Bastian Köcher authored
This change is mainly for people running the local variants. They can directly start with async backing. --------- Signed-off-by:
-
- Jan 29, 2024
-
-
Alejandro Martinez Andres authored
After some conversations around the topic of launching a community staging network ( [forum post 1](https://forum.polkadot.network/t/a-new-test-network-for-polkadot/4325), [forum post 2](https://forum.polkadot.network/t/the-new-polkadot-community-testnet/4956) ). And Erin having introduced the community testnet in the [last episode of AAG](https://www.youtube.com/watch?v=uzJEGVG78_E&t=1612s). I would like to include the `paseo` options as one of the chains available for launching using the polkadot node. Note that the chain spec implementation is not yet included at this stage. UPDATE: Merged master with rebase enabled and introduced quite some noise in this PR. Relevant changes are in: - `polkadot/node/service/chain-specs/paseo.json` - `polkadot/node/service/src/chain_spec.rs` - `polkadot/cli/src/command.rs` --------- Signed-off-by:
Sergej Sakac <[email protected]> Co-authored-by:
Adel Arja <[email protected]> Co-authored-by:
Francisco Gamundi <[email protected]> Co-authored-by:
-
s0me0ne-unkn0wn authored
Currently, collators and their alongside nodes spin up a full-scale overseer running a bunch of subsystems that are not needed if the node is not a validator. That was considered to be harmless; however, we've got problems with unused subsystems getting stalled for a reason not currently known, resulting in the overseer exiting and bringing down the whole node. This PR aims to only run needed subsystems on such nodes, replacing the rest with `DummySubsystem`. It also enables collator-optimized availability recovery subsystem implementation. Partially solves #1730.
-
- Jan 12, 2024
-
-
Dmitry Markin authored
Extract `WarpSync` (and `StateSync` as part of warp sync) from `ChainSync` as independent syncing strategy called by `SyncingEngine`. Introduce `SyncingStrategy` enum as a proxy between `SyncingEngine` and specific syncing strategies. ## Limitations Gap sync is kept in `ChainSync` for now because it shares the same set of peers as block syncing implementation in `ChainSync`. Extraction of a common context responsible for peer management in syncing strategies able to run in parallel is planned for a follow-up PR. ## Further improvements A possibility of conversion of `SyncingStartegy` into a trait should be evaluated. The main stopper for this is that different strategies need to communicate different actions to `SyncingEngine` and respond to different events / provide different APIs (e.g., requesting justifications is only possible via `ChainSync` and not through `WarpSync`; `SendWarpProofRequest` action is only relevant to `WarpSync`, etc.) --------- Co-authored-by:
-
- Jan 04, 2024
-
-
Marcin S. authored
Please let me know if this would be a better UX. Should we have specific links in the error message?
-
- Dec 18, 2023
-
-
Branislav Kontur authored
This PR contains just a few clippy fixes and nits, which are, however, relaxed by workspace clippy settings here: https://github.com/paritytech/polkadot-sdk/blob/master/Cargo.toml#L483-L506 --------- Co-authored-by:
Dmitry Sinyavin <[email protected]> Co-authored-by:
-
- Dec 13, 2023
-
-
Alexandru Gheorghe authored
Initial implementation for the plan discussed here: https://github.com/paritytech/polkadot-sdk/issues/701 Built on top of https://github.com/paritytech/polkadot-sdk/pull/1178 v0: https://github.com/paritytech/polkadot/pull/7554, ## Overall idea When approval-voting checks a candidate and is ready to advertise the approval, defer it in a per-relay chain block until we either have MAX_APPROVAL_COALESCE_COUNT candidates to sign or a candidate has stayed MAX_APPROVALS_COALESCE_TICKS in the queue, in both cases we sign what candidates we have available. This should allow us to reduce the number of approvals messages we have to create/send/verify. The parameters are configurable, so we should find some values that balance: - Security of the network: Delaying broadcasting of an approval shouldn't but the finality at risk and to make sure that never happens we won't delay sending a vote if we are past 2/3 from the no-show time. - Scalability of the network: MAX_APPROVAL_COALESCE_COUNT = 1 & MAX_APPROVALS_COALESCE_TICKS =0, is what we have now and we know from the measurements we did on versi, it bottlenecks approval-distribution/approval-voting when increase significantly the number of validators and parachains - Block storage: In case of disputes we have to import this votes on chain and that increase the necessary storage with MAX_APPROVAL_COALESCE_COUNT * CandidateHash per vote. Given that disputes are not the normal way of the network functioning and we will limit MAX_APPROVAL_COALESCE_COUNT in the single digits numbers, this should be good enough. Alternatively, we could try to create a better way to store this on-chain through indirection, if that's needed. ## Other fixes: - Fixed the fact that we were sending random assignments to non-validators, that was wrong because those won't do anything with it and they won't gossip it either because they do not have a grid topology set, so we would waste the random assignments. - Added metrics to be able to debug potential no-shows and mis-processing of approvals/assignments. ## TODO: - [x] Get feedback, that this is moving in the right direction. @ordian @sandreim @eskimor @burdges, let me know what you think. - [x] More and more testing. - [x] Test in versi. - [x] Make MAX_APPROVAL_COALESCE_COUNT & MAX_APPROVAL_COALESCE_WAIT_MILLIS a parachain host configuration. - [x] Make sure the backwards compatibility works correctly - [x] Make sure this direction is compatible with other streams of work: https://github.com/paritytech/polkadot-sdk/issues/635 & https://github.com/paritytech/polkadot-sdk/issues/742 - [x] Final versi burn-in before merging --------- Signed-off-by:
-
- Dec 05, 2023
-
-
Marcin S. authored
Co-authored-by:
-
- Nov 29, 2023
-
-
Bastian Köcher authored
The `BlockNumber` and `Hash` are fixed types any way.
-
- Nov 28, 2023
-
-
Aaro Altonen authored
This commit introduces a new concept called `NotificationService` which allows Polkadot protocols to communicate with the underlying notification protocol implementation directly, without routing events through `NetworkWorker`. This implies that each protocol has its own service which it uses to communicate with remote peers and that each `NotificationService` is unique with respect to the underlying notification protocol, meaning `NotificationService` for the transaction protocol can only be used to send and receive transaction-related notifications. The `NotificationService` concept introduces two additional benefits: * allow protocols to start using custom handshakes * allow protocols to accept/reject inbound peers Previously the validation of inbound connections was solely the responsibility of `ProtocolController`. This caused issues with light peers and `SyncingEngine` as `ProtocolController` would accept more peers than `SyncingEngine` could accept which caused peers to have differing views of their own states. `SyncingEngine` would reject excess peers but these rejections were not properly communicated to those peers causing them to assume that they were accepted. With `NotificationService`, the local handshake is not sent to remote peer if peer is rejected which allows it to detect that it was rejected. This commit also deprecates the use of `NetworkEventStream` for all notification-related events and going forward only DHT events are provided through `NetworkEventStream`. If protocols wish to follow each other's events, they must introduce additional abtractions, as is done for GRANDPA and transactions protocols by following the syncing protocol through `SyncEventStream`. Fixes https://github.com/paritytech/polkadot-sdk/issues/512 Fixes https://github.com/paritytech/polkadot-sdk/issues/514 Fixes https://github.com/paritytech/polkadot-sdk/issues/515 Fixes https://github.com/paritytech/polkadot-sdk/issues/554 Fixes https://github.com/paritytech/polkadot-sdk/issues/556 --- These changes are transferred from https://github.com/paritytech/substrate/pull/14197 but there are no functional changes compared to that PR --------- Co-authored-by:
-
André Silva authored
This was never used and we probably don't need it anyway.
-
s0me0ne-unkn0wn authored
Co-authored-by:
Vladimir Istyufeev <[email protected]> -
André Silva authored
Currently the polkadot node will backoff from block authoring if finality starts lagging. This PR disables this mechanism on production networks (polkadot and kusama) and adds a flags to optionally force enabling it.
-
- Nov 23, 2023
-
-
Bastian Köcher authored
This moves the macro related re-exports to `__private` to make it more obvious for downstream users that they are using an internal api. --------- Co-authored-by: command-bot <>
-
- Nov 08, 2023
-
-
Bastian Köcher authored
One for local networks with `fast-runtime` feature activated (1 minute sessions) and one without the feature activated that will be the default that runs with 1 hour long sessions.
-
- Nov 06, 2023
-
-
Andrei Sandu authored
**_PR migrated from https://github.com/paritytech/polkadot/pull/6782_** This PR will upgrade the network protocol to version 3 -> VStaging which will later be renamed to V3. This version introduces a new kind of assignment certificate that will be used for tranche0 assignments. Instead of issuing/importing one tranche0 assignment per candidate, there will be just one certificate per relay chain block per validator. However, we will not be sending out the new assignment certificates, yet. So everything should work exactly as before. Once the majority of the validators have been upgraded to the new protocol version we will enable the new certificates (starting at a specific relay chain block) with a new client update. There are still a few things that need to be done: - [x] Use bitfield instead of Vec<CandidateIndex>: https://github.com/paritytech/polkadot/pull/6802 - [x] Fix existing approval-distribution and approval-voting tests - [x] Fix bitfield-distribution and statement-distribution tests - [x] Fix network bridge tests - [x] Implement todos in the code - [x] Add tests to cover new code - [x] Update metrics - [x] Remove the approval distribution aggression levels: TBD PR - [x] Parachains DB migration - [x] Test network protocol upgrade on Versi - [x] Versi Load test - [x] Add Zombienet test - [x] Documentation updates - [x] Fix for sending DistributeAssignment for each candidate claimed by a v2 assignment (warning: Importing locally an already known assignment) - [x] Fix AcceptedDuplicate - [x] Fix DB migration so that we can still keep old data. - [x] Final Versi burn in --------- Signed-off-by:
-
- Nov 05, 2023
-
-
Michal Kucharczyk authoredThis PR prepares chains specs for _native-runtime-free_ world. This PR has following changes: - `substrate`: - adds support for: - JSON based `GenesisConfig` to `ChainSpec` allowing interaction with runtime `GenesisBuilder` API. - interacting with arbitrary runtime wasm blob to[ `chain-spec-builder`](https://github.com/paritytech/substrate/blob/3ef576eaeb3f42610e85daecc464961cf1295570/bin/utils/chain-spec-builder/src/lib.rs#L46) command line util, - removes [`code`](https://github.com/paritytech/substrate/blob/3ef576eaeb3f42610e85daecc464961cf1295570/frame/system/src/lib.rs#L660) from `system_pallet` - adds `code` to the `ChainSpec` - deprecates [`ChainSpec::from_genesis`](https://github.com/paritytech/substrate/blob/3ef576eaeb3f42610e85daecc464961cf1295570/client/chain-spec/src/chain_spec.rs#L263), but also changes the signature of this method extending it with `code` argument. [`ChainSpec::builder()`](https://github.com/paritytech/substrate/blob/20bee680ed098be7239cf7a6b804cd4de267983e/client/chain-spec/src/chain_spec.rs#L507) should be used instead. - `polkadot`: - all references to `RuntimeGenesisConfig` in `node/service` are removed, - all `(kusama|polkadot|versi|rococo|wococo)_(staging|dev)_genesis_config` functions now return the JSON patch for default runtime `GenesisConfig`, - `ChainSpecBuilder` is used, `ChainSpec::from_genesis` is removed, - `cumulus`: - `ChainSpecBuilder` is used, `ChainSpec::from_genesis` is removed, - _JSON_ patch configuration used instead of `RuntimeGenesisConfig struct` in all chain specs. --------- Co-authored-by: command-bot <> Co-authored-by:
Kevin Krone <[email protected]> Co-authored-by:
-
- Nov 03, 2023
-
-
Alexandru Gheorghe authored
The check_hardware functions does not give us too much information as to what is failing, so let's return the list of failed metrics, so that callers can print it. This would make debugging easier, rather than try to guess which dimension is actually failing. Signed-off-by:
-
- Oct 24, 2023
-
-
Marcin S. authored
-
- Oct 23, 2023
-
-
ordian authored
Fixes #768.
-
- Oct 18, 2023
-
-
Serban Iorga authored
Fellowship companion: https://github.com/polkadot-fellows/runtimes/pull/65 This starts the BEEFY client by default for Polkadot nodes. Governance/sudo call is later required to enable/start consensus. Part of https://github.com/paritytech/parity-bridges-common/issues/2420
-
- Oct 12, 2023
-
-
Anton Vilhelm Ásgeirsson authored
# Description In a couple of cases, there were links pointing to the w3f github pages domain. In other instances, there were links pointing to the old polkadot repo's github pages. Both of these are now pointing to the relevant links in https://paritytech.github.io/polkadot-sdk/book/index.html. These changes were made specifically because the w3f github pages returns a 404, and while fixing the links, the old polkadot repo links were touched up as well even if they do redirect properly. This shouldn't affect anything as these are documentation link changes only.
-
- Sep 28, 2023
-
-
Michal Kucharczyk authored
[`RococoGenesisExt`](https://github.com/paritytech/polkadot-sdk/blob/a414ea75 /polkadot/node/service/src/chain_spec.rs#L152-L171) is removed. It was the hack to allow overwriting `EpochDurationInBlocks`. Removal of `RococGenesisExt` prevents from manipulating the state to change the runtime constants. Changes: - Environment variable which controls the `time::EpochDurationInBlocks` value was added: `ROCOCO_EPOCH_DURATION` (epoch duration will be set to the value of env), - `10,100,600` versions of rococo-runtime are built in CI and put into `polkadot-debug` docker image. `rococo-runtime` building examples: - to build runtime for `versi_staging_testnet` which had EpochDurationInBlocks set to 100: ``` ROCOCO_EPOCH_DURATION=100 cargo build --features=fast-runtime -p rococo-runtime ``` - to build runtime for `wococo_development` ``` ROCOCO_EPOCH_DURATION=10 cargo build --features=fast-runtime -p rococo-runtime ``` - to build `versi-staging` chain spec: ``` ROCOCO_EPOCH_DURATION=100 cargo run -p polkadot --features=fast-runtime -- build-spec --chain versi-staging --raw ``` - to build `wococo-dev` chain spec: ``` ROCOCO_EPOCH_DURATION=10 cargo run -p polkadot --features=fast-runtime -- build-spec --chain wococo-dev --raw ``` It is also possible to change the epoch duration by replacing the `code` field in the chain spec with the hex dump of pre-built runtime wasm blob (because the epoch duration is hard-coded into wasm blob). --------- Co-authored-by:
-
- Sep 27, 2023
-
-
Alejandro Martinez Andres authored
Migrating [PR from the archived polkadot repo](https://github.com/paritytech/polkadot/pull/7272) As per https://github.com/paritytech/polkadot/pull/7272#issuecomment-1559240466, the changes in this MR include the following pallets into [x] Rococo and [x] Westend runtimes: pallet_conviction_voting pallet_referenda pallet_ranked_collective pallet_custom_origins pallet_whitelist And only for westend-runtime: pallet_treasury Following [Kusama runtime config](https://github.com/paritytech/polkadot/tree/dbae30efe080a1d41fe54ef4da8af47614c9ca93/runtime/kusama/src) as a baseline. Benchmarking of the following pallets done for both Rococo and Westend: pallet_conviction_voting pallet_referenda pallet_ranked_collective (only on Rococo) pallet_whitelist And only for Westend: pallet_treasury Removed Gov1 from Rococo as in https://github.com/paritytech/polkadot/pull/6701 Rococo Gov1 storage will be cleaned in a different PR - [issue ](https://github.com/paritytech/polkadot-sdk/issues/1618) --------- Co-authored-by: -
Chris Sosnin authored
- Async-backing related primitives are stable `primitives::v6` - Async-backing API is now part of `api_version(7)` - It's enabled on Rococo and Westend runtimes --------- Signed-off-by:
-
- Sep 19, 2023
-
-
Bastian Köcher authored
This pull request removes the Polkadot and Kusama native runtime from the polkadot node. This brings some implications with it: There are no more kusama/polkadot-dev chain specs available. We will need to write some tooling in the fellowship repo to provide them easily. The try-runtime job for polkadot & kusama is not available anymore as we don't have the dev chain specs anymore. Certain benchmarking commands will also not work until we migrate them to use a runtime api. Some crates in utils are still depending on the polkadot/kusama native runtime that will also need to be fixed. Port of: https://github.com/paritytech/polkadot/pull/7467
-
- Sep 15, 2023
-
-
Rahul Subramaniyam authored
Submit the outstanding PRs from the old repos(these were already reviewed and approved before the repo rorg, but not yet submitted): Main PR: https://github.com/paritytech/substrate/pull/14014 Companion PRs: https://github.com/paritytech/polkadot/pull/7134, https://github.com/paritytech/cumulus/pull/2489 The changes in the PR: 1. ChainSync currently calls into the block request handler directly. Instead, move the block request handler behind a trait. This allows new protocols to be plugged into ChainSync. 2. BuildNetworkParams is changed so that custom relay protocol implementations can be (optionally) passed in during network creation time. If custom protocol is not specified, it defaults to the existing block handler 3. BlockServer and BlockDownloader traits are introduced for the protocol implementation. The existing block handler has been changed to implement these traits 4. Other changes: [X] Make TxHash serializable. This is needed for exchanging the serialized hash in the relay protocol messages [X] Clean up types no longer used(OpaqueBlockRequest, OpaqueBlockResponse) --------- Co-authored-by:
-
- Sep 05, 2023
-
-
Dmitry Markin authored
-
- Aug 29, 2023
-
-
Bastian Köcher authored
* Fix `test-rustdoc` * ".git/.scripts/commands/fmt/fmt.sh" --------- Co-authored-by: command-bot <>
-
- Aug 28, 2023
-
-
ordian authored
* deps: replace lru with schnellru * bring the peace to the galaxy
-
- Aug 22, 2023
-
-
Adrian Catangiu authored
* runtime: add BEEFY and MMR to Westend Signed-off-by:
-
- Aug 18, 2023
-
-
asynchronous rob authored
* inclusion emulator logic for asynchronous backing (#4790) * initial stab at candidate_context * fmt * docs & more TODOs * some cleanups * reframe as inclusion_emulator * documentations yes * update types * add constraint modifications * watermark * produce modifications * v2 primitives: re-export all v1 for consistency * vstaging primitives * emulator constraints: handle code upgrades * produce outbound HRMP modifications * stack. * method for applying modifications * method just for sanity-checking modifications * fragments produce modifications, not prospectives * make linear * add some TODOs * remove stacking; handle code upgrades * take `fragment` private * reintroduce stacking. * fragment constructor * add TODO * allow validating fragments against future constraints * docs * relay-parent number and min code size checks * check code upgrade restriction * check max hrmp per candidate * fmt * remove GoAhead logic because it wasn't helpful * docs on code upgrade failure * test stacking * test modifications against constraints * fmt * test fragments * descending or duplicate test * fmt * remove unused imports in vstaging * wrong primitives * spellcheck * Runtime changes for Asynchronous Backing (#4786) * inclusion: utility for allowed relay-parents * inclusion: use prev number instead of prev hash * track most recent context of paras * inclusion: accept previous relay-parents * update dmp advancement rule for async backing * fmt * add a comment about validation outputs * clean up a couple of TODOs * weights * fix weights * fmt * Resolve dmp todo * Restore inclusion tests * Restore paras_inherent tests * MostRecentContext test * Benchmark for new paras dispatchable * Prepare check_validation_outputs for upgrade * cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_paras.rs * cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_paras.rs * cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_paras.rs * cargo run --quiet --profile=production --features=runtime-benchmarks -- benchmark --chain=rococo-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/rococo/src/weights/runtime_parachains_paras.rs * Implementers guide changes * More tests for allowed relay parents * Add a github issue link * Compute group index based on relay parent * Storage migration * Move allowed parents tracker to shared * Compile error * Get group assigned to core at the next block * Test group assignment * fmt * Error instead of panic * Update guide * Extend doc-comment * Update runtime/parachains/src/shared.rs Co-authored-by:
Chris Sosnin <[email protected]> Co-authored-by:
Parity Bot <[email protected]> Co-authored-by:
Chris Sosnin <[email protected]> * Prospective Parachains Subsystem (#4913) * docs and skeleton * subsystem skeleton * main loop * fragment tree basics & fmt * begin fragment trees & view * flesh out more of view update logic * further flesh out update logic * some refcount functions for fragment trees * add fatal/non-fatal errors * use non-fatal results * clear up some TODOs * ideal format for scheduling info * add a bunch of TODOs * some more fluff * extract fragment graph to submodule * begin fragment graph API * trees, not graphs * improve docs * scope and constructor for trees * add some test TODOs * limit max ancestors and store constraints * constructor * constraints: fix bug in HRMP watermarks * fragment tree population logic * set::retain * extract population logic * implement add_and_populate * fmt * add some TODOs in tests * implement child-selection * strip out old stuff based on wrong assumptions * use fatality * implement pruning * remove unused ancestor constraints * fragment tree instantiation * remove outdated comment * add message/request types and skeleton for handling * fmt * implement handle_candidate_seconded * candidate storage: handle backed * implement handle_candidate_backed * implement answer_get_backable_candidate * remove async where not needed * implement fetch_ancestry * add logic for run_iteration * add some docs * remove global allow(unused), fix warnings * make spellcheck happy (despite English) * fmt * bump Cargo.lock * replace tracing with gum * introduce PopulateFrom trait * implement GetHypotheticalDepths * revise docs slightly * first fragment tree scope test * more scope tests * test add_candidate * fmt * test retain * refactor test code * test populate is recursive * test contiguity of depth 0 is maintained * add_and_populate tests * cycle tests * remove PopulateFrom trait * fmt * test hypothetical depths (non-recursive) * have CandidateSeconded return membership * tree membership requests * Add a ProspectiveParachainsSubsystem struct * add a staging API for base constraints * add a `From` impl * add runtime API for staging_validity_constraints * implement fetch_base_constraints * implement `fetch_upcoming_paras` * remove reconstruction of candidate receipt; no obvious usecase * fmt * export message to broader module * remove last TODO * correctly export * fix compilation and add GetMinimumRelayParent request * make provisioner into a real subsystem with proper mesage bounds * fmt * fix ChannelsOut in overseer test * fix overseer tests * fix again * fmt * Integrate prospective parachains subsystem into backing: Part 1 (#5557) * BEGIN ASYNC candidate-backing CHANGES * rename & document modes * answer prospective validation data requests * GetMinimumRelayParents request is now plural * implement an implicit view utility for backing subsystems * implicit-view: get allowed relay parents * refactorings and improvements to implicit view * add some TODOs for tests * split implicit view updates into 2 functions * backing: define State to prepare for functional refactor * add some docs * backing: implement bones of new leaf activation logic * backing: create per-relay-parent-states * use new handle_active_leaves_update * begin extracting logic from CandidateBackingJob * mostly extract statement import from job logic * handle statement imports outside of job logic * do some TODO planning for prospective parachains integration * finish rewriting backing subsystem in functional style * add prospective parachains mode to relay parent entries * fmt * add a RejectedByProspectiveParachains error * notify prospective parachains of seconded and backed candidates * always validate candidates exhaustively in backing. * return persisted_validation_data from validation * handle rejections by prospective parachains * implement seconding sanity check * invoke validate_and_second * Alter statement table to allow multiple seconded messages per validator * refactor backing to have statements carry PVD * clean up all warnings * Add tests for implicit view * Improve doc comments * Prospective parachains mode based on Runtime API version * Add a TODO * Rework seconding_sanity_check * Iterate over responses * Update backing tests * collator-protocol: load PVD from runtime * Fix validator side tests * Update statement-distribution to fetch PVD * Fix statement-distribution tests * Backing tests with prospective paras #1 * fix per_relay_parent pruning in backing * Test multiple leaves * Test seconding sanity check * Import statement order Before creating an entry in `PerCandidateState` map wait for the approval from the prospective parachains * Add a test for correct state updates * Second multiple candidates per relay parent test * Add backing tests with prospective paras * Second more than one test without prospective paras * Add a test for prospective para blocks * Update malus * typos Co-authored-by:
Robert Klotzner <[email protected]> * Version 3 is already live. * Fix tests (#6055) * Fix backing tests * Fix warnings. * fmt * collator-protocol: asynchronous backing changes (#5740) * Draft collator side changes * Start working on collations management * Handle peer's view change * Versioning on advertising * Versioned collation fetching request * Handle versioned messages * Improve docs for collation requests * Add spans * Add request receiver to overseer * Fix collator side tests * Extract relay parent mode to lib * Validator side draft * Add more checks for advertisement * Request pvd based on async backing mode * review * Validator side improvements * Make old tests green * More fixes * Collator side tests draft * Send collation test * fmt * Collator side network protocol versioning * cleanup * merge artifacts * Validator side net protocol versioning * Remove fragment tree membership request * Resolve todo * Collator side core state test * Improve net protocol compatibility * Validator side tests * more improvements * style fixes * downgrade log * Track implicit assignments * Limit the number of seconded candidates per para * Add a sanity check * Handle fetched candidate * fix tests * Retry fetch * Guard against dequeueing while already fetching * Reintegrate connection management * Timeout on advertisements * fmt * spellcheck * update tests after merge * validator assignment fixes for backing and collator protocol (#6158) * Rename depth->ancestry len in tests * Refactor group assignments * Remove implicit assignments * backing: consider occupied core assignments * Track a single para on validator side * Refactor prospective parachains mode request (#6179) * Extract prospective parachains mode into util * Skip activations depending on the mode * backing: don't send backed candidate to provisioner (#6185) * backing: introduce `CanSecond` request for advertisements filtering (#6225) * Drop BoundToRelayParent * draft changes * fix backing tests * Fix genesis ancestry * Fix validator side tests * more tests * cargo generate-lockfile * Implement `StagingValidityConstraints` Runtime API method (#6258) * Implement StagingValidityConstraints * spellcheck * fix ump params * Update hrmp comment * Introduce ump per candidate limit * hypothetical earliest block * refactor primitives usage * hypothetical earliest block number test * fix build * Prepare the Runtime for asynchronous backing upgrade (#6287) * Introduce async backing params to runtime config * fix cumulus config * use config * finish runtimes * Introduce new staging API * Update collator protocol * Update provisioner * Update prospective parachains * Update backing * Move async backing params lower in the config * make naming consistent * misc * Use real prospective parachains subsystem (#6407) * Backport `HypotheticalFrontier` into the feature branch (#6605) * implement more general HypotheticalFrontier * fmt * drop unneeded request Co-authored-by:
Bradley Olson <[email protected]> Co-authored-by:
alexgparity <[email protected]> * Asynchronous backing statement distribution: Take III (#5999) * add notification types for v2 statement-distribution * improve protocol docs * add empty vstaging module * fmt * add backed candidate packet request types * start putting down structure of new logic * handle activated leaf * some sanity-checking on outbound statements * fmt * update vstaging share to use statements with PVD * tiny refactor, candidate_hash location * import local statements * refactor statement import * first stab at broadcast logic * fmt * fill out some TODOs * start on handling incoming * split off session info into separate map * start in on a knowledge tracker * address some grumbles * format * missed comment * some docs for direct * add note on slashing * amend * simplify 'direct' code * finish up the 'direct' logic * add a bunch of tests for the direct-in-group logic * rename 'direct' to 'cluster', begin a candidate_entry module * distill candidate_entry * start in on a statement-store module * some utilities for the statement store * rewrite 'send_statement_direct' using new tools * filter sending logic on peers which have the relay-parent in their view. * some more logic for handling incoming statements * req/res: BackedCandidatePacket -> AttestedCandidate + tweaks * add a `validated_in_group` bitfield to BackedCandidateInventory * BackedCandidateInventory -> Manifest * start in on requester module * add outgoing request for attested candidate * add a priority mechanism for requester * some request dispatch logic * add seconded mask to tagged-request * amend manifest to hold group index * handle errors and set up scaffold for response validation * validate attested candidate responses * requester -> requests * add some utilities for manipulating requests * begin integrating requester * start grid module * tiny * refactor grid topology to expose more info to subsystems * fix grid_topology test * fix overseer test * implement topology group-based view construction logic * fmt * flesh out grid slightly more * add indexed groups utility * integrate Groups into per-session info * refactor statement store to borrow Groups * implement manifest knowledge utility * add a test for topology setup * don't send to group members * test for conflicting manifests * manifest knowledge tests * fmt * rename field * garbage collection for grid tracker * routines for finding correct/incorrect advertisers * add manifest import logic * tweak naming * more tests for manifest import * add comment * rework candidates into a view-wide tracker * fmt * start writing boilerplate for grid sending * fmt * some more group boilerplate * refactor handling of topology and authority IDs * fmt * send statements directly to grid peers where possible * send to cluster only if statement belongs to cluster * improve handling of cluster statements * handle incoming statements along the grid * API for introduction of candidates into the tree * backing: use new prospective parachains API * fmt prospective parachains changes * fmt statement-dist * fix condition * get ready for tracking importable candidates * prospective parachains: add Cow logic * incomplete and complete hypothetical candidates * remove keep_if_unneeded * fmt * implement more general HypotheticalFrontier * fmt, cleanup * add a by_parent_hash index to candidate tracker * more framework for future code * utilities for getting all hypothetical candidates for frontier * track origin in statement store * fmt * requests should return peer * apply post-confirmation reckoning * flesh out import/announce/circulate logic on new statements * adjust * adjust TODO comment * fix backing tests * update statement-distribution to use new indexedvec * fmt * query hypothetical candidates * implement `note_importable_under` * extract common utility of fragment tree updates * add a helper function for getting statements unknown by backing * import fresh statements to backing * send announcements and acknowledgements over grid * provide freshly importable statements also avoid tracking backed candidates in statement distribution * do not issue requests on newly importable candidates * add TODO for later when confirming candidate * write a routine for handling backed candidate notifications * simplify grid substantially * add some test TODOs * handle confirmed candidates & grid announcements * finish implementing manifest handling, including follow up statements * send follow-up statements when acknowledging freshly backed * fmt * handle incoming acknowledgements * a little DRYing * wire up network messages to handlers * fmt * some skeleton code for peer view update handling * more peer view skeleton stuff * Fix async backing statement distribution tests (#6621) * Fix compile errors in tests * Cargo fmt * Resolve some todos in async backing statement-distribution branch (#6482) * Implement `remove_by_relay_parent` * Extract `minimum_votes` to shared primitives. * Add `can_send_statements_received_with_prejudice` test * Fix test * Update docstrings * Cargo fmt * Fix compile error * Fix compile errors in tests * Cargo fmt * Add module docs; write `test_priority_ordering` (first draft) * Fix `test_priority_ordering` * Move `insert_or_update_priority`: `Drop` -> `set_cluster_priority` * Address review comments * Remove `Entry::get_mut` * fix test compilation * add a TODO for a test * clean up a couple of TODOs * implement sending pending cluster statements * refactor utility function for sending acknowledgement and statements * mostly implement catching peers up via grid * Fix clippy error * alter grid to track all pending statements * fix more TODOs and format * tweak a TODO in requests * some logic for dispatching requests * fmt * skeleton for response receiving * Async backing statement distribution: cluster tests (#6678) * Add `pending_statements_set_when_receiving_fresh_statements` * Add `pending_statements_updated_when_sending_statements` test * fix up * fmt * update TODO * rework seconded mask in requests * change doc * change unhandledresponse not to borrow request manager * only accept responses sufficient to back * finish implementing response handling * extract statement filter to protocol crate * rework requests: use statement filter in network protocol * dispatch cluster requests correctly * rework cluster statement sending * implement request answering * fmt * only send confirmed candidate statement messages on unified relay-parent * Fix Tests In Statement Distribution Branch * Async Backing: Integrate `vstaging` of statement distribution into `lib.rs` (#6715) * Integrate `handle_active_leaves_update` * Integrate `share_local_statement`/`handle_backed_candidate_message` * Start hooking up request/response flow * Finish hooking up request/response flow * Limit number of parallel requests in responder * Fix test compilation errors * Fix missing check for prospective parachains mode * Fix some more compile errors * clean up some review comments * clean up warnings * Async backing statement distribution: grid tests (#6673) * Add `manifest_import_returns_ok_true` test * cargo fmt * Add pending_communication_receiving_manifest_on_confirmed_candidate * Add `senders_can_provide_manifests_in_acknowledgement` test * Add a couple of tests for pending statements * Add `pending_statements_cleared_when_sending` test * Add `pending_statements_respect_remote_knowledge` test * Refactor group creation in tests * Clarify docs * Address some review comments * Make some clarifications * Fix post-merge errors * Clarify test `senders_can_provide_manifests_in_acknowledgement` * Try writing `pending_statements_are_updated_after_manifest_exchange` * Document "seconding limit" and `reject_overflowing_manifests` test * Test that seconding counts are not updated for validators on error * Fix tests * Fix manifest exchange test * Add more tests in `requests.rs` (#6707) This resolves remaining TODOs in this file. * remove outdated inventory terminology * Async backing statement distribution: `Candidates` tests (#6658) * Async Backing: Fix clippy errors in statement distribution branch (#6720) * Integrate `handle_active_leaves_update` * Integrate `share_local_statement`/`handle_backed_candidate_message` * Start hooking up request/response flow * Finish hooking up request/response flow * Limit number of parallel requests in responder * Fix test compilation errors * Fix missing check for prospective parachains mode * Fix some more compile errors * Async Backing: Fix clippy errors in statement distribution branch * Fix some more clippy lints * add tests module * fix warnings in existing tests * create basic test harness * create a test state struct * fmt * create empty cluster & grid modules for tests * some TODOs for cluster test suite * describe test-suite for grid logic * describe request test suite * fix seconding-limit bug * Remove extraneous `pub` This somehow made it into my clippy PR. * Fix some test compile warnings * Remove some unneeded `allow`s * adapt some new test helpers from Marcin * add helper for activating a gossip topology * add utility for signing statements * helpers for connecting/disconnecting peers * round out network utilities * fmt * fix bug in initializing validator-meta * fix compilation * implement first cluster test * TODOs for incoming request tests * Remove unneeded `make_committed_candidate` helper * fmt * some more tests for cluster * add a TODO about grid senders * integrate inbound req/res into test harness * polish off initial cluster test suite * keep introduce candidate request * fix tests after introduce candidate request * fmt * Add grid protocol to module docs * Fix comments * Test `backed_in_path_only: true` * Update node/network/protocol/src/lib.rs Co-authored-by:
Marcin S <[email protected]> Co-authored-by:
Anthony Alaribe <[email protected]> Co-authored-by:
Anthony Alaribe <[email protected]> Co-authored-by:
alindima <[email protected]> * Update collation generation for asynchronous backing (#7405) * break candidate receipt construction and distribution into own function * update implementers' guide to include SubmitCollation * implement SubmitCollation for collation-generation * fmt * fix test compilation & remove unnecessary submodule * add some TODOs for a test suite. * Update roadmap/implementers-guide/src/types/overseer-protocol.md Co-authored-by:
Robert Klotzner <[email protected]> Co-authored-by:
Mattia L.V. Bradascio <[email protected]> Co-authored-by:
BradleyOlson64 <[email protected]> Co-authored-by:
-
