1. Oct 09, 2020
    • Rakan Alhneiti's avatar
      Update to work with async keystore – Companion PR for #7000 (#1740) · bc7d1322
      Rakan Alhneiti authored
      
      
      * Fix keystore types
      
      * Use SyncCryptoStorePtr
      
      * Borrow keystore
      
      * Fix unused imports
      
      * Fix polkadot service
      
      * Fix bitfield-distribution tests
      
      * Fix indentation
      
      * Fix backing tests
      
      * Fix tests
      
      * Fix provisioner tests
      
      * Removed SyncCryptoStorePtr
      
      * Fix services
      
      * Address PR feedback
      
      * Address PR feedback - 2
      
      * Update CryptoStorePtr imports to be from sp_keystore
      
      * Typo
      
      * Fix CryptoStore import
      
      * Document the reason behind using filesystem keystore
      
      * Remove VALIDATORS
      
      * Fix duplicate dependency
      
      * Mark sp-keystore as optional
      
      * Fix availability distribution
      
      * Fix call to sign_with
      
      * Fix keystore usage
      
      * Remove tokio and fix parachains Cargo config
      
      * Typos
      
      * Fix keystore dereferencing
      
      * Fix CryptoStore import
      
      * Fix provisioner
      
      * Fix node backing
      
      * Update services
      
      * Cleanup dependencies
      
      * Use sync_keystore
      
      * Fix node service
      
      * Fix node service - 2
      
      * Fix node service - 3
      
      * Rename CryptoStorePtr to SyncCryptoStorePtr
      
      * "Update Substrate"
      
      * Apply suggestions from code review
      
      * Update node/core/backing/Cargo.toml
      
      * Update primitives/src/v0.rs
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      
      * Fix wasm build
      
      * Update Cargo.lock
      
      Co-authored-by: parity-processbot <>
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      bc7d1322
  2. Oct 01, 2020
    • ordian's avatar
      implement remaining subsystem metrics (#1770) · 5bb296bf
      ordian authored
      * overseer metrics: messages relayed
      
      * provisioner metrics: cosmetic changes
      
      * candidate selection metrics: cosmetic changes
      
      * availability bitfields metrics
      
      * availability distribution metrics
      
      * PoV distribution metrics
      
      * statement-distribution: small simplification
      
      * statement-distribution: extract log target into a const
      
      * statement-distribution: metrics
      
      * address review nits
      5bb296bf
  3. Sep 29, 2020
  4. Sep 23, 2020
  5. Aug 18, 2020
    • asynchronous rob's avatar
      Implement validation data refactor (#1585) · 8c881e45
      asynchronous rob authored
      
      
      * update primitives
      
      * correct parent_head field
      
      * make hrmp field pub
      
      * refactor validation data: runtime
      
      * refactor validation data: messages
      
      * add arguments to full_validation_data runtime API
      
      * port runtime API
      
      * mostly port over candidate validation
      
      * remove some parameters from ValidationParams
      
      * guide: update candidate validation
      
      * update candidate outputs
      
      * update ValidationOutputs in primitives
      
      * port over candidate validation
      
      * add a new test for no-transient behavior
      
      * update util runtime API wrappers
      
      * candidate backing
      
      * fix missing imports
      
      * change some fields of validation data around
      
      * runtime API impl
      
      * update candidate validation
      
      * fix backing tests
      
      * grumbles from review
      
      * fix av-store tests
      
      * fix some more crates
      
      * fix provisioner tests
      
      * fix availability distribution tests
      
      * port collation-generation to new validation data
      
      * fix overseer tests
      
      * Update roadmap/implementers-guide/src/node/utility/candidate-validation.md
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      8c881e45
    • ordian's avatar
      initial prometheus metrics (#1536) · 804958ab
      ordian authored
      * service-new: cosmetic changes
      
      * overseer: draft of prometheus metrics
      
      * metrics: update active_leaves metrics
      
      * metrics: extract into functions
      
      * metrics: resolve XXX
      
      * metrics: it's ugly, but it works
      
      * Bump Substrate
      
      * metrics: move a bunch of code around
      
      * Bumb substrate again
      
      * metrics: fix a warning
      
      * fix a warning in runtime
      
      * metrics: statements signed
      
      * metrics: statements impl RegisterMetrics
      
      * metrics: refactor Metrics trait
      
      * metrics: add Metrics assoc type to JobTrait
      
      * metrics: move Metrics trait to util
      
      * metrics: fix overseer
      
      * metrics: fix backing
      
      * metrics: fix candidate validation
      
      * metrics: derive Default
      
      * metrics: docs
      
      * metrics: add stubs for other subsystems
      
      * metrics: add more stubs and fix compilation
      
      * metrics: fix doctest
      
      * metrics: move to subsystem
      
      * metrics: fix candidate validation
      
      * metrics: bitfield signing
      
      * metrics: av store
      
      * metrics: chain API
      
      * metrics: runtime API
      
      * metrics: stub for avad
      
      * metrics: candidates seconded
      
      * metrics: ok I gave up
      
      * metrics: provisioner
      
      * metrics: remove a clone by requiring Metrics: Sync
      
      * metrics: YAGNI
      
      * metrics: remove another TODO
      
      * metrics: for later
      
      * metrics: add parachain_ prefix
      
      * metrics: s/signed_statement/signed_statements
      
      * utils: add a comment for job metrics
      
      * metrics: address review comments
      
      * metrics: oops
      
      * metrics: make sure to save files before commit 😅
      
      
      
      * use _total suffix for requests metrics
      
      Co-authored-by: default avatarMax Inden <[email protected]>
      
      * metrics: add tests for overseer
      
      * update Cargo.lock
      
      * overseer: add a test for CollationGeneration
      
      * collation-generation: impl metrics
      
      * collation-generation: use kebab-case for name
      
      * collation-generation: add a constructor
      
      Co-authored-by: default avatarGav Wood <[email protected]>
      Co-authored-by: default avatarAshley Ruglys <[email protected]>
      Co-authored-by: default avatarMax Inden <[email protected]>
      804958ab
  6. Aug 07, 2020
  7. Aug 06, 2020
    • Peter Goodspeed-Niklaus's avatar
      implement provisioner (#1473) · a39d8803
      Peter Goodspeed-Niklaus authored
      * sketch out provisioner basics
      
      * handle provisionable data
      
      * stub out select_inherent_data
      
      * split runtime APIs into sub-chapters to improve linkability
      
      * explain SignedAvailabilityBitfield semantics
      
      * add internal link to further documentation
      
      * some more work figuring out how the provisioner can do its thing
      
      * fix broken link
      
      * don't import enum variants where it's one layer deep
      
      * make request_availability_cores a free fn in util
      
      * document more precisely what should happen on block production
      
      * finish first-draft implementation of provisioner
      
      * start working on the full and proper backed candidate selection rule
      
      * Pass number of block under construction via RequestInherentData
      
      * Revert "Pass number of block under construction via RequestInherentData"
      
      This reverts commit 850fe62c.
      
      That initially looked like the better approach--it spent the time
      budget for fetching the block number in the proposer, instead of
      the provisioner, and that felt more appropriate--but it turns out
      not to be obvious how to get the block number of the block under
      construction from within the proposer. The Chain API may be less
      ideal, but it should be easier to implement.
      
      * wip: get the block under production from the Chain API
      
      * add ChainApiMessage to AllMessages
      
      * don't break the run loop if a provisionable data channel closes
      
      * clone only those backed candidates which are coherent
      
      * propagate chain_api subsystem through various locations
      
      * add delegated_subsystem! macro to ease delegating subsystems
      
      Unfortunately, it doesn't work right:
      
      ```
      error[E0446]: private type `CandidateBackingJob` in public interface
         --> node/core/backing/src/lib.rs:775:1
          |
      86  | struct CandidateBackingJob {
          | - `CandidateBackingJob` declared as private
      ...
      775 | delegated_subsystem!(CandidateBackingJob as CandidateBackingSubsystem);
          | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ can't leak private type
      ```
      
      I'm not sure precisely what's going wrong, here; I suspect the problem is
      the use of `$job as JobTrait>::RunArgs` and `::ToJob`; the failure would be
      that it's not reifying the types to verify that the actual types are public,
      but instead referring to them via `CandidateBackingJob`, which is in fact private;
      that privacy is the point.
      
      Going to see if I can generic my way out of this, but we may be headed for a
      quick revert here.
      
      * fix delegated_subsystem
      
      The invocation is a bit more verbose than I'd prefer, but it's also
      more explicit about what types need to be public. I'll take it as a win.
      
      * add provisioning subsystem; reduce public interface of provisioner
      
      * deny missing docs in provisioner
      
      * refactor core selection per code review suggestion
      
      This is twice as much code when measured by line, but IMO it is
      in fact somewhat clearer to read, so overall a win.
      
      Also adds an improved rule for selecting availability bitfields,
      which (unlike the previous implementation) guarantees that the
      appropriate postconditions hold there.
      
      * fix bad merge double-declaration
      
      * update guide with (hopefully) complete provisioner candidate selection procedure
      
      * clarify candidate selection algorithm
      
      * Revert "clarify candidate selection algorithm"
      
      This reverts commit c68a02ac.
      
      * clarify candidate selection algorithm
      
      * update provisioner to implement candidate selection per the guide
      
      * add test that no more than one bitfield is selected per validator
      
      * add test that each selected bitfield corresponds to an occupied core
      
      * add test that more set bits win conflicts
      
      * add macro for specializing runtime requests; specailize all runtime requests
      
      * add tests harness for select_candidates tests
      
      * add first real select_candidates test, fix test_harness
      
      * add mock overseer and test that success is possible
      
      * add test that the candidate selection algorithm picks the right ones
      
      * make candidate selection test somewhat more stringent
      a39d8803