1. 28 Oct, 2020 2 commits
    • Fedor Sakharov's avatar
      Change SpawnedSubsystem type to log subsystem errors (#1878) · d34585dd
      Fedor Sakharov authored
      * Change SpawnedSubsystem type to log subsystem errors
      
      * Remove clone
      d34585dd
    • Peter Goodspeed-Niklaus's avatar
      start working on building the real overseer (#1795) · 798f781f
      Peter Goodspeed-Niklaus authored
      
      
      * start working on building the real overseer
      
      Unfortunately, this fails to compile right now due to an upstream
      failure to compile which is probably brought on by a recent upgrade
      to rustc v1.47.
      
      * fill in AllSubsystems internal constructors
      
      * replace fn make_metrics with Metrics::attempt_to_register
      
      * update to account for #1740
      
      * remove Metrics::register, rename Metrics::attempt_to_register
      
      * add 'static bounds to real_overseer type params
      
      * pass authority_discovery and network_service to real_overseer
      
      It's not straightforwardly obvious that this is the best way to handle
      the case when there is no authority discovery service, but it seems
      to be the best option available at the moment.
      
      * select a proper database configuration for the availability store db
      
      * use subdirectory for av-store database path
      
      * apply Basti's patch which avoids needing to parameterize everything on Block
      
      * simplify path extraction
      
      * get all tests to compile
      
      * Fix Prometheus double-registry error
      
      for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476:
      
      ```rust
      Some(registry) => Self::try_register(registry).map_err(|err| {
      	eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err);
      	err
      }),
      ```
      
      That pointed out where the registration was failing, which led to
      this fix. The test still doesn't pass, but it now fails in a new
      and different way!
      
      * authorities must have authority discovery, but not necessarily overseer handlers
      
      * fix broken SpawnedSubsystem impls
      
      detailed logging determined that using the `Box::new` style of
      future generation, the `self.run` method was never being called,
      leading to dropped receivers / closed senders for those subsystems,
      causing the overseer to shut down immediately.
      
      This is not the final fix needed to get things working properly,
      but it's a good start.
      
      * use prometheus properly
      
      Prometheus lets us register simple counters, which aren't very
      interesting. It also allows us to register CounterVecs, which are.
      With a CounterVec, you can provide a set of labels, which can
      later be used to filter the counts.
      
      We were using them wrong, though. This pattern was repeated in a
      variety of places in the code:
      
      ```rust
      // panics with an cardinality mismatch
      let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?;
      my_counter.with_label_values(&["succeeded"]).inc()
      ```
      
      The problem is that the labels provided in the constructor are not
      the set of legal values which can be annotated, but a set of individual
      label names which can have individual, arbitrary values.
      
      This commit fixes that.
      
      * get av-store subsystem to actually run properly and not die on first signal
      
      * typo fix: incomming -> incoming
      
      * don't disable authority discovery in test nodes
      
      * Fix rococo-v1 missing session keys
      
      * Update node/core/av-store/Cargo.toml
      
      * try dummying out av-store on non-full-nodes
      
      * overseer and subsystems are required only for full nodes
      
      * Reduce the amount of warnings on browser target
      
      * Fix two more warnings
      
      * InclusionInherent should actually have an Inherent module on rococo
      
      * Ancestry: don't return genesis' parent hash
      
      * Update Cargo.lock
      
      * fix broken test
      
      * update test script: specify chainspec as script argument
      
      * Apply suggestions from code review
      
      Co-authored-by: default avatarBastian Köcher <bkchr@users.noreply.github.com>
      
      * Update node/service/src/lib.rs
      
      Co-authored-by: default avatarBastian Köcher <bkchr@users.noreply.github.com>
      
      * node/service/src/lib: Return error via ? operator
      
      * post-merge blues
      
      * add is_collator flag
      
      * prevent occasional av-store test panic
      
      * simplify fix; expand application
      
      * run authority_discovery in Role::Discover when collating
      
      * distinguish between proposer closed channel errors
      
      * add IsCollator enum, remove is_collator CLI flag
      
      * improve formatting
      
      * remove nop loop
      
      * Fix some stuff
      
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      Co-authored-by: Bastian Köcher's avatarBastian Köcher <git@kchr.de>
      Co-authored-by: Fedor Sakharov's avatarFedor Sakharov <fedor.sakharov@gmail.com>
      Co-authored-by: default avatarRobert Habermeier <robert@Roberts-MBP.lan1>
      Co-authored-by: default avatarBastian Köcher <bkchr@users.noreply.github.com>
      Co-authored-by: default avatarMax Inden <mail@max-inden.de>
      798f781f
  2. 27 Oct, 2020 1 commit
  3. 08 Oct, 2020 1 commit
  4. 07 Oct, 2020 1 commit
  5. 06 Oct, 2020 2 commits
    • Bastian Köcher's avatar
      Pass relay parent as argument when collating (#1789) · 923c97a4
      Bastian Köcher authored
      This pr changes the collation function to also pass the current relay
      parent the parachain block should be build on.
      923c97a4
    • Bastian Köcher's avatar
      Make collation an optional return (#1787) · ce0fa3ad
      Bastian Köcher authored
      This pr changes the collator interface function to return an optional
      collation instead of a collation. This is required as the parachain
      itself can fail to generate a valid collation for various reason. Now if
      the collation fails it will return `None`.
      
      Besides that the pr adds some `RuntimeDebug` derive for `ValidationData`
      and removes some whitespaces.
      ce0fa3ad
  6. 01 Oct, 2020 2 commits
  7. 29 Sep, 2020 1 commit
  8. 20 Aug, 2020 1 commit
    • Andronik Ordian's avatar
      proposer: wait for a hash to be in the active-leaves set (#1616) · 08f4be76
      Andronik Ordian authored
      * overseer: add ExternalRequest to Event
      
      * proposer: wait for the hash to be activated
      
      * update comments
      
      * overseer: handle unbounded growth of listeners map
      
      * overseer: fix compilation
      
      * overseer: clean up dead listeners
      
      * overseer: cosmetic changes
      
      * overseer: cosmetic changes t.2
      
      * overseer: add debug_assertions
      
      * overseer: fix formatting
      08f4be76
  9. 18 Aug, 2020 2 commits
    • asynchronous rob's avatar
      Implement validation data refactor (#1585) · 8c881e45
      asynchronous rob authored
      
      
      * update primitives
      
      * correct parent_head field
      
      * make hrmp field pub
      
      * refactor validation data: runtime
      
      * refactor validation data: messages
      
      * add arguments to full_validation_data runtime API
      
      * port runtime API
      
      * mostly port over candidate validation
      
      * remove some parameters from ValidationParams
      
      * guide: update candidate validation
      
      * update candidate outputs
      
      * update ValidationOutputs in primitives
      
      * port over candidate validation
      
      * add a new test for no-transient behavior
      
      * update util runtime API wrappers
      
      * candidate backing
      
      * fix missing imports
      
      * change some fields of validation data around
      
      * runtime API impl
      
      * update candidate validation
      
      * fix backing tests
      
      * grumbles from review
      
      * fix av-store tests
      
      * fix some more crates
      
      * fix provisioner tests
      
      * fix availability distribution tests
      
      * port collation-generation to new validation data
      
      * fix overseer tests
      
      * Update roadmap/implementers-guide/src/node/utility/candidate-validation.md
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
      8c881e45
    • Andronik Ordian's avatar
      initial prometheus metrics (#1536) · 804958ab
      Andronik Ordian authored
      * service-new: cosmetic changes
      
      * overseer: draft of prometheus metrics
      
      * metrics: update active_leaves metrics
      
      * metrics: extract into functions
      
      * metrics: resolve XXX
      
      * metrics: it's ugly, but it works
      
      * Bump Substrate
      
      * metrics: move a bunch of code around
      
      * Bumb substrate again
      
      * metrics: fix a warning
      
      * fix a warning in runtime
      
      * metrics: statements signed
      
      * metrics: statements impl RegisterMetrics
      
      * metrics: refactor Metrics trait
      
      * metrics: add Metrics assoc type to JobTrait
      
      * metrics: move Metrics trait to util
      
      * metrics: fix overseer
      
      * metrics: fix backing
      
      * metrics: fix candidate validation
      
      * metrics: derive Default
      
      * metrics: docs
      
      * metrics: add stubs for other subsystems
      
      * metrics: add more stubs and fix compilation
      
      * metrics: fix doctest
      
      * metrics: move to subsystem
      
      * metrics: fix candidate validation
      
      * metrics: bitfield signing
      
      * metrics: av store
      
      * metrics: chain API
      
      * metrics: runtime API
      
      * metrics: stub for avad
      
      * metrics: candidates seconded
      
      * metrics: ok I gave up
      
      * metrics: provisioner
      
      * metrics: remove a clone by requiring Metrics: Sync
      
      * metrics: YAGNI
      
      * metrics: remove another TODO
      
      * metrics: for later
      
      * metrics: add parachain_ prefix
      
      * metrics: s/signed_statement/signed_statements
      
      * utils: add a comment for job metrics
      
      * metrics: address review comments
      
      * metrics: oops
      
      * metrics: make sure to save files before commit 😅
      
      
      
      * use _total suffix for requests metrics
      
      Co-authored-by: default avatarMax Inden <mail@max-inden.de>
      
      * metrics: add tests for overseer
      
      * update Cargo.lock
      
      * overseer: add a test for CollationGeneration
      
      * collation-generation: impl metrics
      
      * collation-generation: use kebab-case for name
      
      * collation-generation: add a constructor
      
      Co-authored-by: default avatarGav Wood <gavin@parity.io>
      Co-authored-by: default avatarAshley Ruglys <ashley.ruglys@gmail.com>
      Co-authored-by: default avatarMax Inden <mail@max-inden.de>
      804958ab
  10. 17 Aug, 2020 3 commits
    • Andronik Ordian's avatar
      overseer: fix build (#1596) · c61d9076
      Andronik Ordian authored
      * overseer: add a test for CollationGeneration
      
      * fix build
      c61d9076
    • Peter Goodspeed-Niklaus's avatar
      implement collation generation subsystem (#1557) · e750a825
      Peter Goodspeed-Niklaus authored
      * start sketching out a collation generation subsystem
      
      * invent a basic strategy for double initialization
      
      * clean up warnings
      
      * impl util requests from runtime assuming a context instead of a FromJob sender
      
      * implement collation generation algorithm from guide
      
      * update AllMessages in tests
      
      * fix trivial review comments
      
      * remove another redundant declaration from merge
      
      * filter availability cores by para_id
      
      * handle new activations each in their own async task
      
      * update guide according to the actual current implementation
      
      * add initialization to guide
      
      * add general-purpose subsystem_test_harness helper
      
      * write first handle_new_activations test
      
      * add test that handle_new_activations filters local_validation_data requests
      
      * add (failing) test of collation distribution message sending
      
      * rustfmt
      
      * broken: work on fixing sender test
      
      Unfortunately, for reasons that are not yet clear, despite the public key
      and checked data being identical, the signer is not producing an identical
      signature. This commit produces this output (among more):
      
      signing with  Public(c4733ab0bbe3ba4c096685d1737a7f498cdbdd167a767d04a21dc7df12b8c858 (5GWHUNm5...))
      checking with Public(c4733ab0bbe3ba4c096685d1737a7f498cdbdd167a767d04a21dc7df12b8c858 (5GWHUNm5...))
      signed payload:  [4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 10, 0, 0, 0, c7, e5, c0, 64, 7a, db, fe, 44, 81, e5, 51, 11, 79, 9f, a5, 63, 93, 94, 3c, c4, 36, c6, 30, 36, c2, c5, 44, a2, 1b, db, b7, 82, 3, 17, a, 2e, 75, 97, b7, b7, e3, d8, 4c, 5, 39, 1d, 13, 9a, 62, b1, 57, e7, 87, 86, d8, c0, 82, f2, 9d, cf, 4c, 11, 13, 14]
      checked payload: [4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 10, 0, 0, 0, c7, e5, c0, 64, 7a, db, fe, 44, 81, e5, 51, 11, 79, 9f, a5, 63, 93, 94, 3c, c4, 36, c6, 30, 36, c2, c5, 44, a2, 1b, db, b7, 82, 3, 17, a, 2e, 75, 97, b7, b7, e3, d8, 4c, 5, 39, 1d, 13, 9a, 62, b1, 57, e7, 87, 86, d8, c0, 82, f2, 9d, cf, 4c, 11, 13, 14]
      
      * fix broken test
      
      * collation function returns commitments hash
      
      It doesn't look like we use the actual commitments data anywhere, and
      it's not obvious if there are any fields of `CandidateCommitments`
      not available to the collator, so this commit just assigns them the
      entire responsibility of generating the hash.
      
      * add missing overseer impls
      
      * calculating erasure coding is polkadot's responsibility, not cumulus
      
      * concurrentize per-relay_parent requests
      e750a825
    • Andronik Ordian's avatar
      Add spawn_blocking to SubsystemContext (#1570) · 474b72a5
      Andronik Ordian authored
      * subsystem: add spawn_blocking to SubsystemContext
      
      * candidate-validation: use spawn_blocking for exhaustive tasks
      474b72a5
  11. 14 Aug, 2020 1 commit
  12. 12 Aug, 2020 1 commit
    • asynchronous rob's avatar
      Network bridge refactoring impl (#1537) · 1e9c0540
      asynchronous rob authored
      * update networking types
      
      * port over overseer-protocol message types
      
      * Add the collation protocol to network bridge
      
      * message sending
      
      * stub for ConnectToValidators
      
      * add some helper traits and methods to protocol types
      
      * add collator protocol message
      
      * leaves-updating
      
      * peer connection and disconnection
      
      * add utilities for dispatching multiple events
      
      * implement message handling
      
      * add an observedrole enum with equality and no sentry nodes
      
      * derive partial-eq on network bridge event
      
      * add PartialEq impls for network message types
      
      * add Into implementation for observedrole
      
      * port over existing network bridge tests
      
      * add some more tests
      
      * port bitfield distribution
      
      * port over bitfield distribution tests
      
      * add codec indices
      
      * port PoV distribution
      
      * port over PoV distribution tests
      
      * port over statement distribution
      
      * port over statement distribution tests
      
      * update overseer and service-new
      
      * address review comments
      
      * port availability distribution
      
      * port over availability distribution tests
      1e9c0540
  13. 04 Aug, 2020 1 commit
  14. 27 Jul, 2020 2 commits
    • Fedor Sakharov's avatar
      Availability store subsystem (#1404) · b838b386
      Fedor Sakharov authored
      * Initial commit
      
      * WIP
      
      * Make atomic transactions
      
      * Remove pruning code
      
      * Fix build and add a Nop to bridge
      
      * Fixes from review
      
      * Move config struct around for clarity
      
      * Rename constructor and warn on missing docs
      
      * Fix a test and rename a message
      
      * Fix some more reviews
      
      * Obviously failed to rebase cleanly
      b838b386
    • Peter Goodspeed-Niklaus's avatar
      add ActiveLeavesUpdate, remove StartWork, StopWork (#1458) · 12a02556
      Peter Goodspeed-Niklaus authored
      * add ActiveLeavesUpdate, remove StartWork, StopWork
      
      * replace StartWork, StopWork in subsystem crate tests
      
      * mechanically update OverseerSignal in other modules
      
      * convert overseer to take advantage of new multi-hash update abilities
      
      Note: this does not yet convert the tests; some of the tests now freeze:
      
      test tests::overseer_start_stop_works ... test tests::overseer_start_stop_works has been running for over 60 seconds
      test tests::overseer_finalize_works ... test tests::overseer_finalize_works has been running for over 60 seconds
      
      * fix broken overseer tests
      
      * manually impl PartialEq for ActiveLeavesUpdate, rm trait Equivalent
      
      This cleans up the code a bit and makes it easier in the future to
      do the right thing when comparing ALUs.
      
      * use target in all network bridge logging
      
      * reduce spamming of  and
      12a02556
  15. 26 Jul, 2020 1 commit
  16. 23 Jul, 2020 1 commit
    • Peter Goodspeed-Niklaus's avatar
      implement bitfield signing subsystem (#1364) · 8217ca67
      Peter Goodspeed-Niklaus authored
      
      
      * update guide to reduce confusion and TODOs
      
      * work from previous bitfield signing effort
      
      There were large merge issues with the old bitfield signing PR, so
      we're just copying all the work from that onto this and restarting.
      
      Much of the existing work will be discarded because we now have better
      tools available, but that's fine.
      
      * start rewriting bitfield signing in terms of the util module
      
      * implement construct_availability_bitvec
      
      It's not an ideal implementation--we can make it much more concurrent--
      but at least it compiles.
      
      * implement the unimplemented portions of bitfield signing
      
      * get core availability concurrently, not sequentially
      
      * use sp-std instead of std for a parachain item
      
      * resolve type inference failure caused by multiple From impls
      
      * handle bitfield signing subsystem & Allmessages variant in overseer
      
      * fix more multi-From inference issues
      
      * more concisely handle overflow
      
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      
      * Revert "resolve type inference failure caused by multiple From impls"
      
      This reverts commit 7fc77805.
      
      * Revert "fix more multi-From inference issues"
      
      This reverts commit f14ffe58
      
      .
      
      * impl From<i32> for ParaId
      
      * handle another instance of AllSubsystems
      
      * improve consistency when returning existing options
      
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      8217ca67
  17. 17 Jul, 2020 1 commit
  18. 10 Jul, 2020 1 commit
    • asynchronous rob's avatar
      Refactor primitives (#1383) · 96af6ead
      asynchronous rob authored
      * create a v1 primitives module
      
      * Improve guide on availability types
      
      * punctuate
      
      * new parachains runtime uses new primitives
      
      * tests of new runtime now use new primitives
      
      * add ErasureChunk to guide
      
      * export erasure chunk from v1 primitives
      
      * subsystem crate uses v1 primitives
      
      * node-primitives uses new v1 primitives
      
      * port overseer to new primitives
      
      * new-proposer uses v1 primitives (no ParachainHost anymore)
      
      * fix no-std compilation for primitives
      
      * service-new uses v1 primitives
      
      * network-bridge uses new primitives
      
      * statement distribution uses v1 primitives
      
      * PoV distribution uses v1 primitives; add PoV::hash fn
      
      * move parachain to v0
      
      * remove inclusion_inherent module and place into v1
      
      * remove everything from primitives crate root
      
      * remove some unused old types from v0 primitives
      
      * point everything else at primitives::v0
      
      * squanch some warns up
      
      * add RuntimeDebug import to no-std as well
      
      * port over statement-table and validation
      
      * fix final errors in validation and node-primitives
      
      * add dummy Ord impl to committed candidate receipt
      
      * guide: update CandidateValidationMessage
      
      * add primitive for validationoutputs
      
      * expand CandidateValidationMessage further
      
      * bikeshed
      
      * add some impls to omitted-validation-data and available-data
      
      * expand CandidateValidationMessage
      
      * make erasure-coding generic over v1/v0
      
      * update usages of erasure-coding
      
      * implement commitments.hash()
      
      * use Arc<Pov> for CandidateValidation
      
      * improve new erasure-coding method names
      
      * fix up candidate backing
      
      * update docs a bit
      
      * fix most tests and add short-circuiting to make_pov_available
      
      * fix remainder of candidate backing tests
      
      * squanching warns
      
      * squanch it up
      
      * some fallout
      
      * overseer fallout
      
      * free from polkadot-test-service hell
      96af6ead
  19. 09 Jul, 2020 2 commits
    • Andronik Ordian's avatar
      Integrate all (dummy) subsystems with the Overseer (#1374) · 8845df22
      Andronik Ordian authored
      * overseer: introduce a utility typemap
      
      * it's ugly but it compiles
      
      * move DummySubsystem to subsystem crate
      
      * fix tests fallout
      
      * use a struct for all subsystems
      
      * more tests fallout
      
      * add missing pov_distribution subsystem
      
      * remove unused imports and bounds
      
      * fix minimal-example
      8845df22
    • Fedor Sakharov's avatar
      CandidateBackingSubsystem (#1312) · 6aa5bfba
      Fedor Sakharov authored
      * Updates guide for CandidateBacking
      
      * Move assignment types to primitives
      
      * Initial implementation.
      
      * More functionality
      
      * use assert_matches
      
      * Changes to report misbehaviors
      
      * Some fixes after a review
      
      * Remove a blank line
      
      * Update guide and some types
      
      * Adds run_job function
      
      * Some comments and refactorings
      
      * Fix review
      
      * Remove warnings
      
      * Use summary in kicking off validation
      
      * Parallelize requests
      
      * Validation provides local and global validation params
      
      * Test issued validity tracking
      
      * Nits from review
      6aa5bfba
  20. 07 Jul, 2020 1 commit
  21. 30 Jun, 2020 2 commits
    • asynchronous rob's avatar
      Subsystem::start takes self by-value (#1325) · 17650cef
      asynchronous rob authored
      * Subsystem::start takes self by-value
      
      * fix doc-test compilation
      17650cef
    • asynchronous rob's avatar
      Implement Network Bridge (#1280) · eaae8653
      asynchronous rob authored
      
      
      * network bridge skeleton
      
      * move some primitives around and add debug impls
      
      * protocol registration glue & abstract network interface
      
      * add send_msgs to subsystemctx
      
      * select logic
      
      * transform different events into actions and handle
      
      * implement remaining network bridge state machine
      
      * start test skeleton
      
      * make network methods asynchronous
      
      * extract subsystem out to subsystem crate
      
      * port over overseer to subsystem context trait
      
      * fix minimal example
      
      * fix overseer doc test
      
      * update network-bridge crate
      
      * write a subsystem test-helpers crate
      
      * write a network test helper for network-bridge
      
      * set up (broken) view test
      
      * Revamp network to be more async-friendly and not require Sync
      
      * fix spacing
      
      * fix test compilation
      
      * insert side-channel for actions
      
      * Add some more message types to AllMessages
      
      * introduce a test harness
      
      * add some tests
      
      * ensure service compiles and passes tests
      
      * fix typo
      
      * fix service-new compilation
      
      * Subsystem test helpers send messages synchronously
      
      * remove smelly action inspector
      
      * remove superfluous let binding
      
      * fix warnings
      
      * Update node/network/bridge/src/lib.rs
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
      
      * fix compilation
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
      eaae8653
  22. 17 Jun, 2020 1 commit
  23. 11 Jun, 2020 1 commit
  24. 05 Jun, 2020 1 commit
  25. 03 Jun, 2020 2 commits
  26. 02 Jun, 2020 1 commit
    • Fedor Sakharov's avatar
      Overseer (#1152) · 162486cd
      Fedor Sakharov authored
      
      
      * Initial commit
      
      * Licenses, spaces, docs
      
      * Add a spawner
      
      * Watch spawned subsystems with a FuturesUnordered
      
      * Move the types around a bit
      
      * Suggested fixes by Max
      
      * Add a handler to talk to the Overseer
      
      * FromOverseer and ToOverseer msgs and stopping
      
      * Docs and return errors
      
      * Dont broadcast, have add a from field to messages
      
      * Allow communication between subsystems and outside world
      
      * A message with a oneshot to send result example
      
      * Remove leftover can_recv_msg
      
      * Remove from field from messages
      
      * Dont be generic over stuff
      
      * Gather messages with StreamUnordered
      
      * Fix comments and formatting
      
      * More docs fixes and an example
      
      * Apply suggestions from code review
      
      Co-authored-by: asynchronous rob's avatarRobert Habermeier <rphmeier@gmail.com>
      
      * Fixes from review
      
      Move function from impl block.
      Do not panic but resolve with errors if spawner fails or subsystem
      resolves.
      
      * Dropping a handler results in a flaky test
      
      Co-authored-by: asynchronous rob's avatarRobert Habermeier <rphmeier@gmail.com>
      162486cd