- Nov 03, 2020
-
-
Bastian Köcher authored
-
- Nov 02, 2020
-
-
Bastian Köcher authored
- Set the parachains configuration in Rococo genesis - Don't stop the overseer when a subsystem job is stopped - Several small code changes
-
ordian authored
* utils: remove unused error * av-store: do not exit early on errors * av-store: revert logging change on master * av-store: add a test
-
asynchronous rob authored* Commit my changes * some backing fixes * indentation * fix backing tests * tweak includability rules * comment * Update node/core/backing/src/lib.rs Co-authored-by:
Bastian Köcher <[email protected]> * Update node/core/backing/src/lib.rs Co-authored-by:
Bastian Köcher <[email protected]> Co-authored-by:
-
asynchronous rob authored
* fix: ensure candidate validation gets code based on occupied core assumption * guide: runtime API for historical validation code * add historical runtime API * integrate into runtime API subsystem * remove blocked TODO * fix service build: enable notifications protocol only under real overseer * Update node/subsystem/src/messages.rs Co-authored-by:
Sergei Shulepov <[email protected]> * fix compilation Co-authored-by:
Robert Habermeier <[email protected]> Co-authored-by:
-
Sergey Pepyakin authored
* UMP: Update the impl guide * UMP: Incorporate XCM related changes into the guide * UMP: Data structures and configuration * UMP: Initial plumbing * UMP: Data layout * UMP: Acceptance criteria & enactment * UMP: Fix dispatcher bug and add the test for it * UMP: Constrain the maximum size of an UMP message This commit addresses the UMP part of https://github.com/paritytech/polkadot/issues/1869 * Fix failing test due to misconfiguration * Make the type of RelayDispatchQueueSize be more apparent in the guide * Revert renaming `max_upward_queue_capacity` to `max_upward_queue_count` * convert spaces to tabs Co-authored-by:
Bernhard Schuster <[email protected]> * Update runtime/parachains/src/router/ump.rs Co-authored-by:
-
- Nov 01, 2020
-
-
Bastian Köcher authored
* Fixes bug that collator wasn't sending `Declare` message * Apply suggestions from code review I'm an idiot
-
- Oct 31, 2020
-
-
Bastian Köcher authored
* start working on building the real overseer Unfortunately, this fails to compile right now due to an upstream failure to compile which is probably brought on by a recent upgrade to rustc v1.47. * fill in AllSubsystems internal constructors * replace fn make_metrics with Metrics::attempt_to_register * update to account for #1740 * remove Metrics::register, rename Metrics::attempt_to_register * add 'static bounds to real_overseer type params * pass authority_discovery and network_service to real_overseer It's not straightforwardly obvious that this is the best way to handle the case when there is no authority discovery service, but it seems to be the best option available at the moment. * select a proper database configuration for the availability store db * use subdirectory for av-store database path * apply Basti's patch which avoids needing to parameterize everything on Block * simplify path extraction * get all tests to compile * Fix Prometheus double-registry error for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476: ```rust Some(registry) => Self::try_register(registry).map_err(|err| { eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err); err }), ``` That pointed out where the registration was failing, which led to this fix. The test still doesn't pass, but it now fails in a new and different way! * authorities must have authority discovery, but not necessarily overseer handlers * fix broken SpawnedSubsystem impls detailed logging determined that using the `Box::new` style of future generation, the `self.run` method was never being called, leading to dropped receivers / closed senders for those subsystems, causing the overseer to shut down immediately. This is not the final fix needed to get things working properly, but it's a good start. * use prometheus properly Prometheus lets us register simple counters, which aren't very interesting. It also allows us to register CounterVecs, which are. With a CounterVec, you can provide a set of labels, which can later be used to filter the counts. We were using them wrong, though. This pattern was repeated in a variety of places in the code: ```rust // panics with an cardinality mismatch let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?; my_counter.with_label_values(&["succeeded"]).inc() ``` The problem is that the labels provided in the constructor are not the set of legal values which can be annotated, but a set of individual label names which can have individual, arbitrary values. This commit fixes that. * get av-store subsystem to actually run properly and not die on first signal * typo fix: incomming -> incoming * don't disable authority discovery in test nodes * Fix rococo-v1 missing session keys * Update node/core/av-store/Cargo.toml * try dummying out av-store on non-full-nodes * overseer and subsystems are required only for full nodes * Reduce the amount of warnings on browser target * Fix two more warnings * InclusionInherent should actually have an Inherent module on rococo * Ancestry: don't return genesis' parent hash * Update Cargo.lock * fix broken test * update test script: specify chainspec as script argument * Apply suggestions from code review Co-authored-by:Peter Goodspeed-Niklaus <[email protected]> Co-authored-by:
Andronik Ordian <[email protected]> Co-authored-by:
Fedor Sakharov <[email protected]> Co-authored-by:
Robert Habermeier <[email protected]> Co-authored-by:
Peter Goodspeed-Niklaus <[email protected]> Co-authored-by:
Max Inden <[email protected]> Co-authored-by:
Sergey Shulepov <[email protected]> Co-authored-by:
-
- Oct 29, 2020
-
-
Sergey Pepyakin authored
* remove pending TODO after the DMP impl merge * DMP: Update the impl guide * DMP: Incorporate XCM related changes into the guide This is the DMP related part of https://github.com/paritytech/polkadot/issues/1702
-
- Oct 28, 2020
-
-
Fedor Sakharov authored
* Change SpawnedSubsystem type to log subsystem errors * Remove clone
-
Sergey Pepyakin authored
-
Sergey Pepyakin authored
-
Sergey Pepyakin authored
* DMP: data structures and plumbing * DMP: Implement DMP logic in the router module DMP: Integrate DMP parts into the inclusion module * DMP: Introduce the max size limit for the size of a downward message * DMP: Runtime API for accessing inbound messages * OCD Small clean ups * DMP: fix the naming of the error * DMP: add caution about a non-existent recipient
-
Peter Goodspeed-Niklaus authored
* start working on building the real overseer Unfortunately, this fails to compile right now due to an upstream failure to compile which is probably brought on by a recent upgrade to rustc v1.47. * fill in AllSubsystems internal constructors * replace fn make_metrics with Metrics::attempt_to_register * update to account for #1740 * remove Metrics::register, rename Metrics::attempt_to_register * add 'static bounds to real_overseer type params * pass authority_discovery and network_service to real_overseer It's not straightforwardly obvious that this is the best way to handle the case when there is no authority discovery service, but it seems to be the best option available at the moment. * select a proper database configuration for the availability store db * use subdirectory for av-store database path * apply Basti's patch which avoids needing to parameterize everything on Block * simplify path extraction * get all tests to compile * Fix Prometheus double-registry error for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476: ```rust Some(registry) => Self::try_register(registry).map_err(|err| { eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err); err }), ``` That pointed out where the registration was failing, which led to this fix. The test still doesn't pass, but it now fails in a new and different way! * authorities must have authority discovery, but not necessarily overseer handlers * fix broken SpawnedSubsystem impls detailed logging determined that using the `Box::new` style of future generation, the `self.run` method was never being called, leading to dropped receivers / closed senders for those subsystems, causing the overseer to shut down immediately. This is not the final fix needed to get things working properly, but it's a good start. * use prometheus properly Prometheus lets us register simple counters, which aren't very interesting. It also allows us to register CounterVecs, which are. With a CounterVec, you can provide a set of labels, which can later be used to filter the counts. We were using them wrong, though. This pattern was repeated in a variety of places in the code: ```rust // panics with an cardinality mismatch let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?; my_counter.with_label_values(&["succeeded"]).inc() ``` The problem is that the labels provided in the constructor are not the set of legal values which can be annotated, but a set of individual label names which can have individual, arbitrary values. This commit fixes that. * get av-store subsystem to actually run properly and not die on first signal * typo fix: incomming -> incoming * don't disable authority discovery in test nodes * Fix rococo-v1 missing session keys * Update node/core/av-store/Cargo.toml * try dummying out av-store on non-full-nodes * overseer and subsystems are required only for full nodes * Reduce the amount of warnings on browser target * Fix two more warnings * InclusionInherent should actually have an Inherent module on rococo * Ancestry: don't return genesis' parent hash * Update Cargo.lock * fix broken test * update test script: specify chainspec as script argument * Apply suggestions from code review Co-authored-by:
-
- Oct 27, 2020
-
-
Bastian Köcher authored
The proposer wasn't registered in prometheus and thus, we could not see its metrics.
-
Fedor Sakharov authored
-
Bernhard Schuster authored
-
- Oct 24, 2020
-
-
Sergey Pepyakin authored
* annoying whitespaces * update guide Add `CheckValidationOutputs` runtime api and also change the candidate-validation stuff * promote ValidationOutputs to global primitives i.e. move it from node specific primitives to global v1 primitives. This will be needed when we share it later in the runtime inclusion module * refactor acceptance checks in the inclusion module factor out the common code to share it during the block inclusion and for the forthcoming CheckValidationOutputs runtime api. Also note that the acceptance criteria was updated to incorporate checks that exist now in candidate-validation * plumb the runtime api outside * extract validation_data from ValidationOutputs * use runtime-api to check validation outputs apart from that refactor, update docs and tidy a bit * Update the maxium code size This is to fix a test that performs an upgrade.
-
Fedor Sakharov authored
* Initial commit * Move tests to separate module * Move timestamps to the newtype * Change idx name * Use Duration for consts and update chunk records * Ordering::Equal * Update node/core/av-store/src/lib.rs Co-authored-by:
-
- Oct 21, 2020
-
-
Cecile Tonglet authored
* WIP * WIP * WIP * WIP * WIP * WIP * Update node/service/src/lib.rs Co-authored-by:
-
- Oct 15, 2020
-
-
ordian authored
-
- Oct 09, 2020
-
-
Rakan Alhneiti authored
* Fix keystore types * Use SyncCryptoStorePtr * Borrow keystore * Fix unused imports * Fix polkadot service * Fix bitfield-distribution tests * Fix indentation * Fix backing tests * Fix tests * Fix provisioner tests * Removed SyncCryptoStorePtr * Fix services * Address PR feedback * Address PR feedback - 2 * Update CryptoStorePtr imports to be from sp_keystore * Typo * Fix CryptoStore import * Document the reason behind using filesystem keystore * Remove VALIDATORS * Fix duplicate dependency * Mark sp-keystore as optional * Fix availability distribution * Fix call to sign_with * Fix keystore usage * Remove tokio and fix parachains Cargo config * Typos * Fix keystore dereferencing * Fix CryptoStore import * Fix provisioner * Fix node backing * Update services * Cleanup dependencies * Use sync_keystore * Fix node service * Fix node service - 2 * Fix node service - 3 * Rename CryptoStorePtr to SyncCryptoStorePtr * "Update Substrate" * Apply suggestions from code review * Update node/core/backing/Cargo.toml * Update primitives/src/v0.rs Co-authored-by:
-
- Oct 06, 2020
-
-
ordian authored
* chain-api subsystem: implement BlockHeader messsage * update the guide
-
ordian authored
* stupid, but it compiles * redo * cleanup * add ValidatorDiscovery to msgs * sketch network bridge code * ConnectToAuthorities instead of validators * more stuff * cleanup * more stuff * complete ConnectToAuthoritiesState * Update node/network/bridge/src/lib.rs Co-authored-by:
-
- Oct 01, 2020
-
-
ordian authored
* overseer metrics: messages relayed * provisioner metrics: cosmetic changes * candidate selection metrics: cosmetic changes * availability bitfields metrics * availability distribution metrics * PoV distribution metrics * statement-distribution: small simplification * statement-distribution: extract log target into a const * statement-distribution: metrics * address review nits
-
- Sep 29, 2020
-
-
ordian authored
-
- Sep 23, 2020
-
-
ordian authored
* provisioner: remove tokio from dev-dependencies * provisioner: use futures_timer instead
-
- Sep 11, 2020
-
-
Cecile Tonglet authored
* Initial commit Forked at: d04e4492 Parent branch: origin/master * Validation: don't detect STDIN closing when running in process (#1695)
-
- Sep 10, 2020
-
-
Fedor Sakharov authored
* WIP * The initial implementation of the collator side. * Improve comments * Multiple collation requests * Add more tests and comments to validator side * Add comments, remove dead code * Apply suggestions from code review Co-authored-by:
-
- Sep 09, 2020
-
-
Cecile Tonglet authored
* Initial commit Forked at: bf7ccb84 Parent branch: origin/master * Propagate test mode all the way down to ValidationPool * Update validation/src/validation_service/mod.rs * Fix test * WIP Forked at: bf7ccb84 Parent branch: origin/master * Update service/src/lib.rs Co-authored-by:
-
- Sep 08, 2020
-
-
Peter Goodspeed-Niklaus authored
* choose the straightforward candidate selection algorithm for now * add draft implementation of candidate selection * fix typo in summary * more properly report misbehaving collators * describe how CandidateSelection subsystem becomes aware of candidates * revise candidate selection / collator protocol interaction pattern * implement rest of candidate selection per the guide * review: resolve nits * start writing test suite, harness * implement first test * add second test * implement third test Co-authored-by:
-
- Aug 29, 2020
-
-
ordian authored
* update kvdb-rocksdb to 0.9.1 * update Cargo.lock * "Update Substrate" * resolve merge conflicts * resolve merge conflicts t. 2 Co-authored-by: parity-processbot <>
-
- Aug 20, 2020
-
-
ordian authored
* overseer: add ExternalRequest to Event * proposer: wait for the hash to be activated * update comments * overseer: handle unbounded growth of listeners map * overseer: fix compilation * overseer: clean up dead listeners * overseer: cosmetic changes * overseer: cosmetic changes t.2 * overseer: add debug_assertions * overseer: fix formatting
-
- Aug 19, 2020
-
-
asynchronous rob authored
* extract collator assignment from cores * check required collator * test and fix checks
-
- Aug 18, 2020
-
-
asynchronous rob authored
* update primitives * correct parent_head field * make hrmp field pub * refactor validation data: runtime * refactor validation data: messages * add arguments to full_validation_data runtime API * port runtime API * mostly port over candidate validation * remove some parameters from ValidationParams * guide: update candidate validation * update candidate outputs * update ValidationOutputs in primitives * port over candidate validation * add a new test for no-transient behavior * update util runtime API wrappers * candidate backing * fix missing imports * change some fields of validation data around * runtime API impl * update candidate validation * fix backing tests * grumbles from review * fix av-store tests * fix some more crates * fix provisioner tests * fix availability distribution tests * port collation-generation to new validation data * fix overseer tests * Update roadmap/implementers-guide/src/node/utility/candidate-validation.md Co-authored-by:
-
ordian authored
* service-new: cosmetic changes * overseer: draft of prometheus metrics * metrics: update active_leaves metrics * metrics: extract into functions * metrics: resolve XXX * metrics: it's ugly, but it works * Bump Substrate * metrics: move a bunch of code around * Bumb substrate again * metrics: fix a warning * fix a warning in runtime * metrics: statements signed * metrics: statements impl RegisterMetrics * metrics: refactor Metrics trait * metrics: add Metrics assoc type to JobTrait * metrics: move Metrics trait to util * metrics: fix overseer * metrics: fix backing * metrics: fix candidate validation * metrics: derive Default * metrics: docs * metrics: add stubs for other subsystems * metrics: add more stubs and fix compilation * metrics: fix doctest * metrics: move to subsystem * metrics: fix candidate validation * metrics: bitfield signing * metrics: av store * metrics: chain API * metrics: runtime API * metrics: stub for avad * metrics: candidates seconded * metrics: ok I gave up * metrics: provisioner * metrics: remove a clone by requiring Metrics: Sync * metrics: YAGNI * metrics: remove another TODO * metrics: for later * metrics: add parachain_ prefix * metrics: s/signed_statement/signed_statements * utils: add a comment for job metrics * metrics: address review comments * metrics: oops * metrics: make sure to save files before commit
😅 * use _total suffix for requests metrics Co-authored-by:Gav Wood <[email protected]> Co-authored-by:
Ashley Ruglys <[email protected]> Co-authored-by:
-
- Aug 17, 2020
-
-
ordian authored
* subsystem: add spawn_blocking to SubsystemContext * candidate-validation: use spawn_blocking for exhaustive tasks
-
- Aug 07, 2020
-
-
Peter Goodspeed-Niklaus authored
* break out subsystem-util and subsystem-test-helpers into individual crates * cause all packages to check successfully
-
- Aug 06, 2020
-
-
Peter Goodspeed-Niklaus authored
* sketch out provisioner basics * handle provisionable data * stub out select_inherent_data * split runtime APIs into sub-chapters to improve linkability * explain SignedAvailabilityBitfield semantics * add internal link to further documentation * some more work figuring out how the provisioner can do its thing * fix broken link * don't import enum variants where it's one layer deep * make request_availability_cores a free fn in util * document more precisely what should happen on block production * finish first-draft implementation of provisioner * start working on the full and proper backed candidate selection rule * Pass number of block under construction via RequestInherentData * Revert "Pass number of block under construction via RequestInherentData" This reverts commit 850fe62c. That initially looked like the better approach--it spent the time budget for fetching the block number in the proposer, instead of the provisioner, and that felt more appropriate--but it turns out not to be obvious how to get the block number of the block under construction from within the proposer. The Chain API may be less ideal, but it should be easier to implement. * wip: get the block under production from the Chain API * add ChainApiMessage to AllMessages * don't break the run loop if a provisionable data channel closes * clone only those backed candidates which are coherent * propagate chain_api subsystem through various locations * add delegated_subsystem! macro to ease delegating subsystems Unfortunately, it doesn't work right: ``` error[E0446]: private type `CandidateBackingJob` in public interface --> node/core/backing/src/lib.rs:775:1 | 86 | struct CandidateBackingJob { | - `CandidateBackingJob` declared as private ... 775 | delegated_subsystem!(CandidateBackingJob as CandidateBackingSubsystem); | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ can't leak private type ``` I'm not sure precisely what's going wrong, here; I suspect the problem is the use of `$job as JobTrait>::RunArgs` and `::ToJob`; the failure would be that it's not reifying the types to verify that the actual types are public, but instead referring to them via `CandidateBackingJob`, which is in fact private; that privacy is the point. Going to see if I can generic my way out of this, but we may be headed for a quick revert here. * fix delegated_subsystem The invocation is a bit more verbose than I'd prefer, but it's also more explicit about what types need to be public. I'll take it as a win. * add provisioning subsystem; reduce public interface of provisioner * deny missing docs in provisioner * refactor core selection per code review suggestion This is twice as much code when measured by line, but IMO it is in fact somewhat clearer to read, so overall a win. Also adds an improved rule for selecting availability bitfields, which (unlike the previous implementation) guarantees that the appropriate postconditions hold there. * fix bad merge double-declaration * update guide with (hopefully) complete provisioner candidate selection procedure * clarify candidate selection algorithm * Revert "clarify candidate selection algorithm" This reverts commit c68a02ac. * clarify candidate selection algorithm * update provisioner to implement candidate selection per the guide * add test that no more than one bitfield is selected per validator * add test that each selected bitfield corresponds to an occupied core * add test that more set bits win conflicts * add macro for specializing runtime requests; specailize all runtime requests * add tests harness for select_candidates tests * add first real select_candidates test, fix test_harness * add mock overseer and test that success is possible * add test that the candidate selection algorithm picks the right ones * make candidate selection test somewhat more stringent
-
- Aug 04, 2020
-
-
ordian authored
* propagate chain_api subsystem through various locations * add ChainApi to AllMessages Co-authored-by:
-
