1. 28 Oct, 2020 2 commits
    • Fedor Sakharov's avatar
      Change SpawnedSubsystem type to log subsystem errors (#1878) · d34585dd
      Fedor Sakharov authored
      * Change SpawnedSubsystem type to log subsystem errors
      
      * Remove clone
      d34585dd
    • Peter Goodspeed-Niklaus's avatar
      start working on building the real overseer (#1795) · 798f781f
      Peter Goodspeed-Niklaus authored
      
      
      * start working on building the real overseer
      
      Unfortunately, this fails to compile right now due to an upstream
      failure to compile which is probably brought on by a recent upgrade
      to rustc v1.47.
      
      * fill in AllSubsystems internal constructors
      
      * replace fn make_metrics with Metrics::attempt_to_register
      
      * update to account for #1740
      
      * remove Metrics::register, rename Metrics::attempt_to_register
      
      * add 'static bounds to real_overseer type params
      
      * pass authority_discovery and network_service to real_overseer
      
      It's not straightforwardly obvious that this is the best way to handle
      the case when there is no authority discovery service, but it seems
      to be the best option available at the moment.
      
      * select a proper database configuration for the availability store db
      
      * use subdirectory for av-store database path
      
      * apply Basti's patch which avoids needing to parameterize everything on Block
      
      * simplify path extraction
      
      * get all tests to compile
      
      * Fix Prometheus double-registry error
      
      for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476:
      
      ```rust
      Some(registry) => Self::try_register(registry).map_err(|err| {
      	eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err);
      	err
      }),
      ```
      
      That pointed out where the registration was failing, which led to
      this fix. The test still doesn't pass, but it now fails in a new
      and different way!
      
      * authorities must have authority discovery, but not necessarily overseer handlers
      
      * fix broken SpawnedSubsystem impls
      
      detailed logging determined that using the `Box::new` style of
      future generation, the `self.run` method was never being called,
      leading to dropped receivers / closed senders for those subsystems,
      causing the overseer to shut down immediately.
      
      This is not the final fix needed to get things working properly,
      but it's a good start.
      
      * use prometheus properly
      
      Prometheus lets us register simple counters, which aren't very
      interesting. It also allows us to register CounterVecs, which are.
      With a CounterVec, you can provide a set of labels, which can
      later be used to filter the counts.
      
      We were using them wrong, though. This pattern was repeated in a
      variety of places in the code:
      
      ```rust
      // panics with an cardinality mismatch
      let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?;
      my_counter.with_label_values(&["succeeded"]).inc()
      ```
      
      The problem is that the labels provided in the constructor are not
      the set of legal values which can be annotated, but a set of individual
      label names which can have individual, arbitrary values.
      
      This commit fixes that.
      
      * get av-store subsystem to actually run properly and not die on first signal
      
      * typo fix: incomming -> incoming
      
      * don't disable authority discovery in test nodes
      
      * Fix rococo-v1 missing session keys
      
      * Update node/core/av-store/Cargo.toml
      
      * try dummying out av-store on non-full-nodes
      
      * overseer and subsystems are required only for full nodes
      
      * Reduce the amount of warnings on browser target
      
      * Fix two more warnings
      
      * InclusionInherent should actually have an Inherent module on rococo
      
      * Ancestry: don't return genesis' parent hash
      
      * Update Cargo.lock
      
      * fix broken test
      
      * update test script: specify chainspec as script argument
      
      * Apply suggestions from code review
      
      Co-authored-by: default avatarBastian Köcher <bkchr@users.noreply.github.com>
      
      * Update node/service/src/lib.rs
      
      Co-authored-by: default avatarBastian Köcher <bkchr@users.noreply.github.com>
      
      * node/service/src/lib: Return error via ? operator
      
      * post-merge blues
      
      * add is_collator flag
      
      * prevent occasional av-store test panic
      
      * simplify fix; expand application
      
      * run authority_discovery in Role::Discover when collating
      
      * distinguish between proposer closed channel errors
      
      * add IsCollator enum, remove is_collator CLI flag
      
      * improve formatting
      
      * remove nop loop
      
      * Fix some stuff
      
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      Co-authored-by: Bastian Köcher's avatarBastian Köcher <git@kchr.de>
      Co-authored-by: Fedor Sakharov's avatarFedor Sakharov <fedor.sakharov@gmail.com>
      Co-authored-by: default avatarRobert Habermeier <robert@Roberts-MBP.lan1>
      Co-authored-by: default avatarBastian Köcher <bkchr@users.noreply.github.com>
      Co-authored-by: default avatarMax Inden <mail@max-inden.de>
      798f781f
  2. 27 Oct, 2020 1 commit
  3. 09 Oct, 2020 1 commit
    • Rakan Alhneiti's avatar
      Update to work with async keystore – Companion PR for #7000 (#1740) · bc7d1322
      Rakan Alhneiti authored
      
      
      * Fix keystore types
      
      * Use SyncCryptoStorePtr
      
      * Borrow keystore
      
      * Fix unused imports
      
      * Fix polkadot service
      
      * Fix bitfield-distribution tests
      
      * Fix indentation
      
      * Fix backing tests
      
      * Fix tests
      
      * Fix provisioner tests
      
      * Removed SyncCryptoStorePtr
      
      * Fix services
      
      * Address PR feedback
      
      * Address PR feedback - 2
      
      * Update CryptoStorePtr imports to be from sp_keystore
      
      * Typo
      
      * Fix CryptoStore import
      
      * Document the reason behind using filesystem keystore
      
      * Remove VALIDATORS
      
      * Fix duplicate dependency
      
      * Mark sp-keystore as optional
      
      * Fix availability distribution
      
      * Fix call to sign_with
      
      * Fix keystore usage
      
      * Remove tokio and fix parachains Cargo config
      
      * Typos
      
      * Fix keystore dereferencing
      
      * Fix CryptoStore import
      
      * Fix provisioner
      
      * Fix node backing
      
      * Update services
      
      * Cleanup dependencies
      
      * Use sync_keystore
      
      * Fix node service
      
      * Fix node service - 2
      
      * Fix node service - 3
      
      * Rename CryptoStorePtr to SyncCryptoStorePtr
      
      * "Update Substrate"
      
      * Apply suggestions from code review
      
      * Update node/core/backing/Cargo.toml
      
      * Update primitives/src/v0.rs
      
      Co-authored-by: default avatarBastian Köcher <bkchr@users.noreply.github.com>
      
      * Fix wasm build
      
      * Update Cargo.lock
      
      Co-authored-by: parity-processbot <>
      Co-authored-by: default avatarBastian Köcher <bkchr@users.noreply.github.com>
      bc7d1322
  4. 01 Oct, 2020 1 commit
    • Andronik Ordian's avatar
      implement remaining subsystem metrics (#1770) · 5bb296bf
      Andronik Ordian authored
      * overseer metrics: messages relayed
      
      * provisioner metrics: cosmetic changes
      
      * candidate selection metrics: cosmetic changes
      
      * availability bitfields metrics
      
      * availability distribution metrics
      
      * PoV distribution metrics
      
      * statement-distribution: small simplification
      
      * statement-distribution: extract log target into a const
      
      * statement-distribution: metrics
      
      * address review nits
      5bb296bf
  5. 29 Sep, 2020 1 commit
  6. 10 Sep, 2020 1 commit
  7. 31 Aug, 2020 1 commit
  8. 20 Aug, 2020 1 commit
  9. 18 Aug, 2020 1 commit
    • Andronik Ordian's avatar
      initial prometheus metrics (#1536) · 804958ab
      Andronik Ordian authored
      * service-new: cosmetic changes
      
      * overseer: draft of prometheus metrics
      
      * metrics: update active_leaves metrics
      
      * metrics: extract into functions
      
      * metrics: resolve XXX
      
      * metrics: it's ugly, but it works
      
      * Bump Substrate
      
      * metrics: move a bunch of code around
      
      * Bumb substrate again
      
      * metrics: fix a warning
      
      * fix a warning in runtime
      
      * metrics: statements signed
      
      * metrics: statements impl RegisterMetrics
      
      * metrics: refactor Metrics trait
      
      * metrics: add Metrics assoc type to JobTrait
      
      * metrics: move Metrics trait to util
      
      * metrics: fix overseer
      
      * metrics: fix backing
      
      * metrics: fix candidate validation
      
      * metrics: derive Default
      
      * metrics: docs
      
      * metrics: add stubs for other subsystems
      
      * metrics: add more stubs and fix compilation
      
      * metrics: fix doctest
      
      * metrics: move to subsystem
      
      * metrics: fix candidate validation
      
      * metrics: bitfield signing
      
      * metrics: av store
      
      * metrics: chain API
      
      * metrics: runtime API
      
      * metrics: stub for avad
      
      * metrics: candidates seconded
      
      * metrics: ok I gave up
      
      * metrics: provisioner
      
      * metrics: remove a clone by requiring Metrics: Sync
      
      * metrics: YAGNI
      
      * metrics: remove another TODO
      
      * metrics: for later
      
      * metrics: add parachain_ prefix
      
      * metrics: s/signed_statement/signed_statements
      
      * utils: add a comment for job metrics
      
      * metrics: address review comments
      
      * metrics: oops
      
      * metrics: make sure to save files before commit 😅
      
      
      
      * use _total suffix for requests metrics
      
      Co-authored-by: default avatarMax Inden <mail@max-inden.de>
      
      * metrics: add tests for overseer
      
      * update Cargo.lock
      
      * overseer: add a test for CollationGeneration
      
      * collation-generation: impl metrics
      
      * collation-generation: use kebab-case for name
      
      * collation-generation: add a constructor
      
      Co-authored-by: default avatarGav Wood <gavin@parity.io>
      Co-authored-by: default avatarAshley Ruglys <ashley.ruglys@gmail.com>
      Co-authored-by: default avatarMax Inden <mail@max-inden.de>
      804958ab
  10. 12 Aug, 2020 1 commit
    • asynchronous rob's avatar
      Network bridge refactoring impl (#1537) · 1e9c0540
      asynchronous rob authored
      * update networking types
      
      * port over overseer-protocol message types
      
      * Add the collation protocol to network bridge
      
      * message sending
      
      * stub for ConnectToValidators
      
      * add some helper traits and methods to protocol types
      
      * add collator protocol message
      
      * leaves-updating
      
      * peer connection and disconnection
      
      * add utilities for dispatching multiple events
      
      * implement message handling
      
      * add an observedrole enum with equality and no sentry nodes
      
      * derive partial-eq on network bridge event
      
      * add PartialEq impls for network message types
      
      * add Into implementation for observedrole
      
      * port over existing network bridge tests
      
      * add some more tests
      
      * port bitfield distribution
      
      * port over bitfield distribution tests
      
      * add codec indices
      
      * port PoV distribution
      
      * port over PoV distribution tests
      
      * port over statement distribution
      
      * port over statement distribution tests
      
      * update overseer and service-new
      
      * address review comments
      
      * port availability distribution
      
      * port over availability distribution tests
      1e9c0540
  11. 10 Aug, 2020 1 commit
  12. 07 Aug, 2020 1 commit
  13. 28 Jul, 2020 1 commit
    • asynchronous rob's avatar
      Add new Runtime API messages and make runtime API request fallible (#1485) · 7ed4cb9c
      asynchronous rob authored
      * polkadot-subsystem: update runtime API message types
      
      * update all networking subsystems to use fallible runtime APIs
      
      * fix bitfield-signing and make it use new runtime APIs
      
      * port candidate-backing to handle runtime API errors and new types
      
      * remove old runtime API messages
      
      * remove unused imports
      
      * fix grumbles
      
      * fix backing tests
      7ed4cb9c
  14. 27 Jul, 2020 2 commits
    • Fedor Sakharov's avatar
      Availability store subsystem (#1404) · b838b386
      Fedor Sakharov authored
      * Initial commit
      
      * WIP
      
      * Make atomic transactions
      
      * Remove pruning code
      
      * Fix build and add a Nop to bridge
      
      * Fixes from review
      
      * Move config struct around for clarity
      
      * Rename constructor and warn on missing docs
      
      * Fix a test and rename a message
      
      * Fix some more reviews
      
      * Obviously failed to rebase cleanly
      b838b386
    • Peter Goodspeed-Niklaus's avatar
      add ActiveLeavesUpdate, remove StartWork, StopWork (#1458) · 12a02556
      Peter Goodspeed-Niklaus authored
      * add ActiveLeavesUpdate, remove StartWork, StopWork
      
      * replace StartWork, StopWork in subsystem crate tests
      
      * mechanically update OverseerSignal in other modules
      
      * convert overseer to take advantage of new multi-hash update abilities
      
      Note: this does not yet convert the tests; some of the tests now freeze:
      
      test tests::overseer_start_stop_works ... test tests::overseer_start_stop_works has been running for over 60 seconds
      test tests::overseer_finalize_works ... test tests::overseer_finalize_works has been running for over 60 seconds
      
      * fix broken overseer tests
      
      * manually impl PartialEq for ActiveLeavesUpdate, rm trait Equivalent
      
      This cleans up the code a bit and makes it easier in the future to
      do the right thing when comparing ALUs.
      
      * use target in all network bridge logging
      
      * reduce spamming of  and
      12a02556
  15. 26 Jul, 2020 1 commit
  16. 23 Jul, 2020 1 commit
    • Bernhard Schuster's avatar
      implement bitfield distribution subsystem (#1368) · 08fcbda4
      Bernhard Schuster authored
      
      
      * feat bitfield distribution
      
      * feat bitfield distribution part 2
      
      * pair programming with rustc & cargo
      
      * lets go
      
      * move bitfield-distribution to the node/network folder
      
      * shape shifting
      
      * lunchtime
      
      * ignore the two fn recursion for now
      
      * step by step
      
      * triplesteps
      
      * bandaid commit
      
      * unordered futures magic
      
      * chore
      
      * reword markdown
      
      * clarify
      
      * lacks abortable processing impl details
      
      * slimify
      
      * fix: warnings and avoid ctx.clone() improve comments
      
      * review comments
      
      * fix details
      
      * make sure outgoing messages are tracked
      
      * fix name
      
      * fix subsystem
      
      * partial test impl
      
      * relax context bounds
      
      * test
      
      * X
      
      * X
      
      * initial test
      
      * fix relay_message not tracked when origin is self
      
      * fix/guide: grammar
      
      Co-authored-by: asynchronous rob's avatarRobert Habermeier <rphmeier@gmail.com>
      
      * work around missing Eq+PartialEq
      
      * fix: add missing message to provisioner
      
      * unify per_job to job_data
      
      * fix/review: part one
      
      * fix/review: more grumbles
      
      * fix/review: track incoming messages per peer
      
      * fix/review: extract fn, avoid nested matches
      
      * fix/review: more tests, simplify test
      
      * fix/review: extend tests to cover more cases
      
      * chore/rename: Tracker -> ProtocolState
      
      * chore check and comment rewording
      
      * feat test: invalid peer message
      
      * remove ignored test cases and unused macros
      
      * fix master merge fallout + warnings
      
      Co-authored-by: asynchronous rob's avatarRobert Habermeier <rphmeier@gmail.com>
      08fcbda4