1. Dec 10, 2020
    • Peter Goodspeed-Niklaus's avatar
      the `inclusion` inherent call is Operational, not Mandatory · e58858d1
      Peter Goodspeed-Niklaus authored
      This resolves a lot of the trickiness about this issue, because
      we no longer need to override or supplant any existing proposer
      logic; the existing logic should exhibit these behaviors:
      
      - the `inclusion` inherent is prioritized over standard transactions
      - but if it's too heavy, i.e. in case of runtime upgrade, it'll be
        dropped in favor of that.
      
      It is my belief that allowing the proposer to just not include
      this data won't have any adverse effects: it's equivalent to replacing
      them with empty versions of themselves, which the `ProvideInherent`
      impl already does.
      e58858d1
  2. Dec 07, 2020
  3. Dec 02, 2020
  4. Nov 30, 2020
  5. Nov 27, 2020
  6. Nov 24, 2020
  7. Nov 23, 2020
  8. Nov 20, 2020
    • Peter Goodspeed-Niklaus's avatar
      Add tracing support to node (#1940) · f7ea3d07
      Peter Goodspeed-Niklaus authored
      * drop in tracing to replace log
      
      * add structured logging to trace messages
      
      * add structured logging to debug messages
      
      * add structured logging to info messages
      
      * add structured logging to warn messages
      
      * add structured logging to error messages
      
      * normalize spacing and Display vs Debug
      
      * add instrumentation to the various 'fn run'
      
      * use explicit tracing module throughout
      
      * fix availability distribution test
      
      * don't double-print errors
      
      * remove further redundancy from logs
      
      * fix test errors
      
      * fix more test errors
      
      * remove unused kv_log_macro
      
      * fix unused variable
      
      * add tracing spans to collation generation
      
      * add tracing spans to av-store
      
      * add tracing spans to backing
      
      * add tracing spans to bitfield-signing
      
      * add tracing spans to candidate-selection
      
      * add tracing spans to candidate-validation
      
      * add tracing spans to chain-api
      
      * add tracing spans to provisioner
      
      * add tracing spans to runtime-api
      
      * add tracing spans to availability-distribution
      
      * add tracing spans to bitfield-distribution
      
      * add tracing spans to network-bridge
      
      * add tracing spans to collator-protocol
      
      * add tracing spans to pov-distribution
      
      * add tracing spans to statement-distribution
      
      * add tracing spans to overseer
      
      * cleanup
      f7ea3d07
  9. Nov 17, 2020
    • ordian's avatar
      update most of the dependencies (#1946) · 31fb8fed
      ordian authored
      * update tiny-keccak to 0.2
      
      * update deps except bitvec and shared_memory
      
      * fix some warning after futures upgrade
      
      * remove useless package rename caused by bug in cargo-upgrade
      
      * revert parity-util-mem *
      
      * remove unused import
      
      * cargo update
      
      * remove all renames on parity-scale-codec
      
      * remove the leftovers
      
      * remove unused dep
      31fb8fed
  10. Nov 03, 2020
  11. Oct 28, 2020
    • Peter Goodspeed-Niklaus's avatar
      start working on building the real overseer (#1795) · 798f781f
      Peter Goodspeed-Niklaus authored
      
      
      * start working on building the real overseer
      
      Unfortunately, this fails to compile right now due to an upstream
      failure to compile which is probably brought on by a recent upgrade
      to rustc v1.47.
      
      * fill in AllSubsystems internal constructors
      
      * replace fn make_metrics with Metrics::attempt_to_register
      
      * update to account for #1740
      
      * remove Metrics::register, rename Metrics::attempt_to_register
      
      * add 'static bounds to real_overseer type params
      
      * pass authority_discovery and network_service to real_overseer
      
      It's not straightforwardly obvious that this is the best way to handle
      the case when there is no authority discovery service, but it seems
      to be the best option available at the moment.
      
      * select a proper database configuration for the availability store db
      
      * use subdirectory for av-store database path
      
      * apply Basti's patch which avoids needing to parameterize everything on Block
      
      * simplify path extraction
      
      * get all tests to compile
      
      * Fix Prometheus double-registry error
      
      for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476:
      
      ```rust
      Some(registry) => Self::try_register(registry).map_err(|err| {
      	eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err);
      	err
      }),
      ```
      
      That pointed out where the registration was failing, which led to
      this fix. The test still doesn't pass, but it now fails in a new
      and different way!
      
      * authorities must have authority discovery, but not necessarily overseer handlers
      
      * fix broken SpawnedSubsystem impls
      
      detailed logging determined that using the `Box::new` style of
      future generation, the `self.run` method was never being called,
      leading to dropped receivers / closed senders for those subsystems,
      causing the overseer to shut down immediately.
      
      This is not the final fix needed to get things working properly,
      but it's a good start.
      
      * use prometheus properly
      
      Prometheus lets us register simple counters, which aren't very
      interesting. It also allows us to register CounterVecs, which are.
      With a CounterVec, you can provide a set of labels, which can
      later be used to filter the counts.
      
      We were using them wrong, though. This pattern was repeated in a
      variety of places in the code:
      
      ```rust
      // panics with an cardinality mismatch
      let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?;
      my_counter.with_label_values(&["succeeded"]).inc()
      ```
      
      The problem is that the labels provided in the constructor are not
      the set of legal values which can be annotated, but a set of individual
      label names which can have individual, arbitrary values.
      
      This commit fixes that.
      
      * get av-store subsystem to actually run properly and not die on first signal
      
      * typo fix: incomming -> incoming
      
      * don't disable authority discovery in test nodes
      
      * Fix rococo-v1 missing session keys
      
      * Update node/core/av-store/Cargo.toml
      
      * try dummying out av-store on non-full-nodes
      
      * overseer and subsystems are required only for full nodes
      
      * Reduce the amount of warnings on browser target
      
      * Fix two more warnings
      
      * InclusionInherent should actually have an Inherent module on rococo
      
      * Ancestry: don't return genesis' parent hash
      
      * Update Cargo.lock
      
      * fix broken test
      
      * update test script: specify chainspec as script argument
      
      * Apply suggestions from code review
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      
      * Update node/service/src/lib.rs
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      
      * node/service/src/lib: Return error via ? operator
      
      * post-merge blues
      
      * add is_collator flag
      
      * prevent occasional av-store test panic
      
      * simplify fix; expand application
      
      * run authority_discovery in Role::Discover when collating
      
      * distinguish between proposer closed channel errors
      
      * add IsCollator enum, remove is_collator CLI flag
      
      * improve formatting
      
      * remove nop loop
      
      * Fix some stuff
      
      Co-authored-by: default avatarAndronik Ordian <[email protected]>
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      Co-authored-by: default avatarFedor Sakharov <[email protected]>
      Co-authored-by: default avatarRobert Habermeier <[email protected]>
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      Co-authored-by: default avatarMax Inden <[email protected]>
      798f781f
  12. Oct 27, 2020
  13. Oct 21, 2020
  14. Aug 20, 2020
    • ordian's avatar
      proposer: wait for a hash to be in the active-leaves set (#1616) · 08f4be76
      ordian authored
      * overseer: add ExternalRequest to Event
      
      * proposer: wait for the hash to be activated
      
      * update comments
      
      * overseer: handle unbounded growth of listeners map
      
      * overseer: fix compilation
      
      * overseer: clean up dead listeners
      
      * overseer: cosmetic changes
      
      * overseer: cosmetic changes t.2
      
      * overseer: add debug_assertions
      
      * overseer: fix formatting
      08f4be76
  15. Jul 28, 2020
  16. Jul 10, 2020
    • asynchronous rob's avatar
      Refactor primitives (#1383) · 96af6ead
      asynchronous rob authored
      * create a v1 primitives module
      
      * Improve guide on availability types
      
      * punctuate
      
      * new parachains runtime uses new primitives
      
      * tests of new runtime now use new primitives
      
      * add ErasureChunk to guide
      
      * export erasure chunk from v1 primitives
      
      * subsystem crate uses v1 primitives
      
      * node-primitives uses new v1 primitives
      
      * port overseer to new primitives
      
      * new-proposer uses v1 primitives (no ParachainHost anymore)
      
      * fix no-std compilation for primitives
      
      * service-new uses v1 primitives
      
      * network-bridge uses new primitives
      
      * statement distribution uses v1 primitives
      
      * PoV distribution uses v1 primitives; add PoV::hash fn
      
      * move parachain to v0
      
      * remove inclusion_inherent module and place into v1
      
      * remove everything from primitives crate root
      
      * remove some unused old types from v0 primitives
      
      * point everything else at primitives::v0
      
      * squanch some warns up
      
      * add RuntimeDebug import to no-std as well
      
      * port over statement-table and validation
      
      * fix final errors in validation and node-primitives
      
      * add dummy Ord impl to committed candidate receipt
      
      * guide: update CandidateValidationMessage
      
      * add primitive for validationoutputs
      
      * expand CandidateValidationMessage further
      
      * bikeshed
      
      * add some impls to omitted-validation-data and available-data
      
      * expand CandidateValidationMessage
      
      * make erasure-coding generic over v1/v0
      
      * update usages of erasure-coding
      
      * implement commitments.hash()
      
      * use Arc<Pov> for CandidateValidation
      
      * improve new erasure-coding method names
      
      * fix up candidate backing
      
      * update docs a bit
      
      * fix most tests and add short-circuiting to make_pov_available
      
      * fix remainder of candidate backing tests
      
      * squanching warns
      
      * squanch it up
      
      * some fallout
      
      * overseer fallout
      
      * free from polkadot-test-service hell
      96af6ead
  17. Jul 05, 2020
    • Peter Goodspeed-Niklaus's avatar
      implement custom proposer (#1320) · 69ce9ff3
      Peter Goodspeed-Niklaus authored
      
      
      * network bridge skeleton
      
      * move some primitives around and add debug impls
      
      * protocol registration glue & abstract network interface
      
      * add send_msgs to subsystemctx
      
      * select logic
      
      * transform different events into actions and handle
      
      * implement remaining network bridge state machine
      
      * start test skeleton
      
      * make network methods asynchronous
      
      * extract subsystem out to subsystem crate
      
      * port over overseer to subsystem context trait
      
      * fix minimal example
      
      * fix overseer doc test
      
      * update network-bridge crate
      
      * write a subsystem test-helpers crate
      
      * write a network test helper for network-bridge
      
      * set up (broken) view test
      
      * Revamp network to be more async-friendly and not require Sync
      
      * fix spacing
      
      * fix test compilation
      
      * insert side-channel for actions
      
      * Add some more message types to AllMessages
      
      * introduce a test harness
      
      * impl ProvideInherent for InclusionInherent
      
      * reduce import churn; correct expect message
      
      * move inclusion inherent identifier into primitives
      
      It's not clear precisely why this is desired, but it's a pattern
      I've seen in several places, so I'm going this to be on the
      safe side. Worst case, we can revert this commit pretty easily.
      
      * bump kusama spec_version to placate CI
      
      * copy sc_basic_authorship::{ProposerFactory, Proposer}
      
      We have from the problem description:
      
      > This Proposer will require an OverseerHandle to make requests via.
      
      That's next on the plate.
      
      * use polkadot custom proposer instead of basic-authorship one
      
      * add some tests
      
      * ensure service compiles and passes tests
      
      * fix typo
      
      * fix service-new compilation
      
      * Subsystem test helpers send messages synchronously
      
      * remove smelly action inspector
      
      * remove superfluous let binding
      
      * fix warnings
      
      * add license header
      
      * empty commit; maybe github will notice the one with changes
      
      * Update node/network/bridge/src/lib.rs
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      
      * add sanity check to only include valid inherents
      
      * stub: encapsulate block production mechanics instead of copying them
      
      The goal is to end up with something like what's in
      validation::block_production::*, which encapsulates
      basic block production mechanics. This is a better idea than
      just straight-up copying those mechanics.
      
      * partial implementation of propose fn
      
      Doesn't actually compile yet; need to bring in some other
      commits to ensure ProvisionerMessage is a thing, and also
      figure out how to get the block hash given the current
      context.
      
      * fix compilation
      
      * clear a few more compile errors
      
      * finish fn propose
      
      * broken: add timeout to proposal
      
      * add timeout to proposal
      
      * guide: provisioner is responsible for selecting parachain candidates
      
      * implement ProvisionerMessage::RequestInherentData & update fn propose
      
      * impl CreateProposer::init; clean up
      
      * impl std::error::Error for Error
      
      * document error-handling rationale
      
      * cause polkadot-service-new to compile correctly
      
      * Move potentially-blocking call from fn init -> fn propose
      
      This means that we can wrap the delayed call into the same
      timeout check used elsewhere.
      
      * document struct Proposer
      
      * extract provisioner data fetch
      
      This satisfies two requirements:
      
      - only applies the timeout to actually fetching the provisioner data,
        not to constructing the block after
      - simplifies the problem of injecting default data if we could not
        get the real provisioner data in time.
      
      Co-authored-by: default avatarRobert Habermeier <[email protected]>
      Co-authored-by: default avatarGavin Wood <[email protected]>
      69ce9ff3