1. Nov 05, 2020
    • ordian's avatar
      more resilient subsystems (#1908) · 9758a180
      ordian authored
      * backing: extract log target
      
      * bitfield-signing: extract log target
      
      * utils: fix a typo
      
      * provisioner: extract log target
      
      * candidate selection: remove unused error variant
      
      * bitfield-distribution: change the return type of run
      
      * pov-distribution: extract log target
      
      * collator-protocol: simplify runtime request
      
      * collation-generation: do not exit early on error
      
      * collation-generation: do not exit on double init
      
      * collator-protocol: do not exit on errors and rename LOG_TARGET
      
      * collator-protocol: a workaround for ununused imports warning
      
      * Update node/network/bitfield-distribution/src/lib.rs
      
      * collation-generation: elevate warn! to error!
      
      * collator-protocol: fix imports
      
      * post merge fix
      
      * fix compilation
      9758a180
  2. Nov 03, 2020
  3. Nov 01, 2020
  4. Oct 31, 2020
    • Bastian Köcher's avatar
      Adds test parachain adder collator (#1864) · 8cadebb7
      Bastian Köcher authored
      
      
      * start working on building the real overseer
      
      Unfortunately, this fails to compile right now due to an upstream
      failure to compile which is probably brought on by a recent upgrade
      to rustc v1.47.
      
      * fill in AllSubsystems internal constructors
      
      * replace fn make_metrics with Metrics::attempt_to_register
      
      * update to account for #1740
      
      * remove Metrics::register, rename Metrics::attempt_to_register
      
      * add 'static bounds to real_overseer type params
      
      * pass authority_discovery and network_service to real_overseer
      
      It's not straightforwardly obvious that this is the best way to handle
      the case when there is no authority discovery service, but it seems
      to be the best option available at the moment.
      
      * select a proper database configuration for the availability store db
      
      * use subdirectory for av-store database path
      
      * apply Basti's patch which avoids needing to parameterize everything on Block
      
      * simplify path extraction
      
      * get all tests to compile
      
      * Fix Prometheus double-registry error
      
      for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476:
      
      ```rust
      Some(registry) => Self::try_register(registry).map_err(|err| {
      	eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err);
      	err
      }),
      ```
      
      That pointed out where the registration was failing, which led to
      this fix. The test still doesn't pass, but it now fails in a new
      and different way!
      
      * authorities must have authority discovery, but not necessarily overseer handlers
      
      * fix broken SpawnedSubsystem impls
      
      detailed logging determined that using the `Box::new` style of
      future generation, the `self.run` method was never being called,
      leading to dropped receivers / closed senders for those subsystems,
      causing the overseer to shut down immediately.
      
      This is not the final fix needed to get things working properly,
      but it's a good start.
      
      * use prometheus properly
      
      Prometheus lets us register simple counters, which aren't very
      interesting. It also allows us to register CounterVecs, which are.
      With a CounterVec, you can provide a set of labels, which can
      later be used to filter the counts.
      
      We were using them wrong, though. This pattern was repeated in a
      variety of places in the code:
      
      ```rust
      // panics with an cardinality mismatch
      let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?;
      my_counter.with_label_values(&["succeeded"]).inc()
      ```
      
      The problem is that the labels provided in the constructor are not
      the set of legal values which can be annotated, but a set of individual
      label names which can have individual, arbitrary values.
      
      This commit fixes that.
      
      * get av-store subsystem to actually run properly and not die on first signal
      
      * typo fix: incomming -> incoming
      
      * don't disable authority discovery in test nodes
      
      * Fix rococo-v1 missing session keys
      
      * Update node/core/av-store/Cargo.toml
      
      * try dummying out av-store on non-full-nodes
      
      * overseer and subsystems are required only for full nodes
      
      * Reduce the amount of warnings on browser target
      
      * Fix two more warnings
      
      * InclusionInherent should actually have an Inherent module on rococo
      
      * Ancestry: don't return genesis' parent hash
      
      * Update Cargo.lock
      
      * fix broken test
      
      * update test script: specify chainspec as script argument
      
      * Apply suggestions from code review
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      
      * Update node/service/src/lib.rs
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      
      * node/service/src/lib: Return error via ? operator
      
      * post-merge blues
      
      * add is_collator flag
      
      * prevent occasional av-store test panic
      
      * simplify fix; expand application
      
      * run authority_discovery in Role::Discover when collating
      
      * distinguish between proposer closed channel errors
      
      * add IsCollator enum, remove is_collator CLI flag
      
      * improve formatting
      
      * remove nop loop
      
      * Fix some stuff
      
      * Adds test parachain adder collator
      
      * Add sudo to Rococo, change session length to 30 seconds and some renaming
      
      * Update to the latest changes on master
      
      * Some fixes
      
      * Fix compilation
      
      * Update parachain/test-parachains/adder/collator/src/lib.rs
      
      Co-authored-by: default avatarSergei Shulepov <[email protected]>
      
      * Review comments
      
      * Downgrade transaction version
      
      * Fixes
      
      * MOARE
      
      * Register notification protocols
      
      * utils: remove unused error
      
      * av-store: more resilient to some errors
      
      * address review nits
      
      * address more review nits
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      Co-authored-by: default avatarAndronik Ordian <[email protected]>
      Co-authored-by: default avatarFedor Sakharov <[email protected]>
      Co-authored-by: default avatarRobert Habermeier <[email protected]>
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      Co-authored-by: default avatarMax Inden <[email protected]>
      Co-authored-by: default avatarSergey Shulepov <[email protected]>
      Co-authored-by: default avatarSergei Shulepov <[email protected]>
      8cadebb7
  5. Oct 28, 2020
    • Fedor Sakharov's avatar
      Change SpawnedSubsystem type to log subsystem errors (#1878) · d34585dd
      Fedor Sakharov authored
      * Change SpawnedSubsystem type to log subsystem errors
      
      * Remove clone
      d34585dd
    • Peter Goodspeed-Niklaus's avatar
      start working on building the real overseer (#1795) · 798f781f
      Peter Goodspeed-Niklaus authored
      
      
      * start working on building the real overseer
      
      Unfortunately, this fails to compile right now due to an upstream
      failure to compile which is probably brought on by a recent upgrade
      to rustc v1.47.
      
      * fill in AllSubsystems internal constructors
      
      * replace fn make_metrics with Metrics::attempt_to_register
      
      * update to account for #1740
      
      * remove Metrics::register, rename Metrics::attempt_to_register
      
      * add 'static bounds to real_overseer type params
      
      * pass authority_discovery and network_service to real_overseer
      
      It's not straightforwardly obvious that this is the best way to handle
      the case when there is no authority discovery service, but it seems
      to be the best option available at the moment.
      
      * select a proper database configuration for the availability store db
      
      * use subdirectory for av-store database path
      
      * apply Basti's patch which avoids needing to parameterize everything on Block
      
      * simplify path extraction
      
      * get all tests to compile
      
      * Fix Prometheus double-registry error
      
      for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476:
      
      ```rust
      Some(registry) => Self::try_register(registry).map_err(|err| {
      	eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err);
      	err
      }),
      ```
      
      That pointed out where the registration was failing, which led to
      this fix. The test still doesn't pass, but it now fails in a new
      and different way!
      
      * authorities must have authority discovery, but not necessarily overseer handlers
      
      * fix broken SpawnedSubsystem impls
      
      detailed logging determined that using the `Box::new` style of
      future generation, the `self.run` method was never being called,
      leading to dropped receivers / closed senders for those subsystems,
      causing the overseer to shut down immediately.
      
      This is not the final fix needed to get things working properly,
      but it's a good start.
      
      * use prometheus properly
      
      Prometheus lets us register simple counters, which aren't very
      interesting. It also allows us to register CounterVecs, which are.
      With a CounterVec, you can provide a set of labels, which can
      later be used to filter the counts.
      
      We were using them wrong, though. This pattern was repeated in a
      variety of places in the code:
      
      ```rust
      // panics with an cardinality mismatch
      let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?;
      my_counter.with_label_values(&["succeeded"]).inc()
      ```
      
      The problem is that the labels provided in the constructor are not
      the set of legal values which can be annotated, but a set of individual
      label names which can have individual, arbitrary values.
      
      This commit fixes that.
      
      * get av-store subsystem to actually run properly and not die on first signal
      
      * typo fix: incomming -> incoming
      
      * don't disable authority discovery in test nodes
      
      * Fix rococo-v1 missing session keys
      
      * Update node/core/av-store/Cargo.toml
      
      * try dummying out av-store on non-full-nodes
      
      * overseer and subsystems are required only for full nodes
      
      * Reduce the amount of warnings on browser target
      
      * Fix two more warnings
      
      * InclusionInherent should actually have an Inherent module on rococo
      
      * Ancestry: don't return genesis' parent hash
      
      * Update Cargo.lock
      
      * fix broken test
      
      * update test script: specify chainspec as script argument
      
      * Apply suggestions from code review
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      
      * Update node/service/src/lib.rs
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      
      * node/service/src/lib: Return error via ? operator
      
      * post-merge blues
      
      * add is_collator flag
      
      * prevent occasional av-store test panic
      
      * simplify fix; expand application
      
      * run authority_discovery in Role::Discover when collating
      
      * distinguish between proposer closed channel errors
      
      * add IsCollator enum, remove is_collator CLI flag
      
      * improve formatting
      
      * remove nop loop
      
      * Fix some stuff
      
      Co-authored-by: default avatarAndronik Ordian <[email protected]>
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      Co-authored-by: default avatarFedor Sakharov <[email protected]>
      Co-authored-by: default avatarRobert Habermeier <[email protected]>
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      Co-authored-by: default avatarMax Inden <[email protected]>
      798f781f
  6. Oct 27, 2020
  7. Oct 07, 2020
  8. Oct 06, 2020
    • ordian's avatar
      collator: fix a typo (#1788) · 7ac23ed3
      ordian authored
      * collator: fix a typo
      
      * collator: fix more typos
      
      * collator: fix even more typos
      7ac23ed3
    • ordian's avatar
      NetworkBridge: validator (authorities) discovery api (#1699) · b84f3c03
      ordian authored
      
      
      * stupid, but it compiles
      
      * redo
      
      * cleanup
      
      * add ValidatorDiscovery to msgs
      
      * sketch network bridge code
      
      * ConnectToAuthorities instead of validators
      
      * more stuff
      
      * cleanup
      
      * more stuff
      
      * complete ConnectToAuthoritiesState
      
      * Update node/network/bridge/src/lib.rs
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      
      * Collator protocol subsystem (#1659)
      
      * WIP
      
      * The initial implementation of the collator side.
      
      * Improve comments
      
      * Multiple collation requests
      
      * Add more tests and comments to validator side
      
      * Add comments, remove dead code
      
      * Apply suggestions from code review
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      
      * Fix build after suggested changes
      
      * Also connect to the next validator group
      
      * Remove a Future impl and move TimeoutExt to util
      
      * Minor nits
      
      * Fix build
      
      * Change FetchCollations back to FetchCollation
      
      * Try this
      
      * Final fixes
      
      * Fix build
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      
      * handle multiple in-flight connection requests
      
      * handle cancelled requests
      
      * Update node/core/runtime-api/src/lib.rs
      
      Co-authored-by: default avatarBernhard Schuster <[email protected]>
      
      * redo it again
      
      * more stuff
      
      * redo it again
      
      * update comments
      
      * workaround Future is not Send
      
      * fix trailing spaces
      
      * clarify comments
      
      * bridge: fix compilation in tests
      
      * update more comments
      
      * small fixes
      
      * port collator protocol to new validator discovery api
      
      * collator tests compile
      
      * collator tests pass
      
      * do not revoke a request when the stream receiver is closed
      
      * make revoking opt-in
      
      * fix is_fulfilled
      
      * handle request revokation in collator
      
      * tests
      
      * wait for validator connections asyncronously
      
      * fix compilation
      
      * relabel my todos
      
      * apply Fedor's patch
      
      * resolve reconnection TODO
      
      * resolve revoking TODO
      
      * resolve channel capacity TODO
      
      * resolve peer cloning TODO
      
      * resolve peer disconnected TODO
      
      * resolve PeerSet TODO
      
      * wip tests
      
      * more tests
      
      * resolve Arc TODO
      
      * rename pending to non_revoked
      
      * one more test
      
      * extract utility function into util crate
      
      * fix compilation in tests
      
      * Apply suggestions from code review
      
      Co-authored-by: default avatarFedor Sakharov <[email protected]>
      
      * revert pin_project removal
      
      * fix while let loop
      
      * Revert "revert pin_project removal"
      
      This reverts commit ae7f529d
      
      .
      
      * fix compilation
      
      * Update node/subsystem/src/messages.rs
      
      * docs on pub items
      
      * guide updates
      
      * remove a TODO
      
      * small guide update
      
      * fix a typo
      
      * link to the issue
      
      * validator discovery: on_request docs
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      Co-authored-by: default avatarFedor Sakharov <[email protected]>
      Co-authored-by: default avatarBernhard Schuster <[email protected]>
      b84f3c03
    • Bastian Köcher's avatar
      Make collation an optional return (#1787) · ce0fa3ad
      Bastian Köcher authored
      This pr changes the collator interface function to return an optional
      collation instead of a collation. This is required as the parachain
      itself can fail to generate a valid collation for various reason. Now if
      the collation fails it will return `None`.
      
      Besides that the pr adds some `RuntimeDebug` derive for `ValidationData`
      and removes some whitespaces.
      ce0fa3ad
  9. Sep 29, 2020
  10. Sep 28, 2020
  11. Sep 10, 2020
    • Fedor Sakharov's avatar
      Collator protocol subsystem (#1659) · b32dcc44
      Fedor Sakharov authored
      
      
      * WIP
      
      * The initial implementation of the collator side.
      
      * Improve comments
      
      * Multiple collation requests
      
      * Add more tests and comments to validator side
      
      * Add comments, remove dead code
      
      * Apply suggestions from code review
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      
      * Fix build after suggested changes
      
      * Also connect to the next validator group
      
      * Remove a Future impl and move TimeoutExt to util
      
      * Minor nits
      
      * Fix build
      
      * Change FetchCollations back to FetchCollation
      
      * Try this
      
      * Final fixes
      
      * Fix build
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      b32dcc44