1. 23 Feb, 2021 1 commit
  2. 19 Feb, 2021 1 commit
    • Bernhard Schuster's avatar
      feat/jaeger: more spans, more stages (#2477) · f9186eb2
      Bernhard Schuster authored
      * feat/jaeger: more spans, more stages
      
      Stage numbers are still arbitrarily picked.
      
      * feat/jaeger: additional spans
      
      * chore/spellcheck: improve the dictionary
      
      * fix/jaeger JaegerSpan -> jaeger::Span
      f9186eb2
  3. 17 Feb, 2021 1 commit
  4. 14 Feb, 2021 1 commit
    • Bastian Köcher's avatar
      Notify collators about seconded collation (#2430) · ac5ef00e
      Bastian Köcher authored
      * Notify collators about seconded collation
      
      This pr adds functionality to inform a collator that its collation was
      seconded by a parachain validator. Before this signed statement was only
      gossiped over the validation substream. Now, we explicitly send the
      seconded statement to the collator after it was validated successfully.
      
      Besides that it changes the `CollatorFn` to return an optional result
      sender that is informed when the build collation was seconded by a
      parachain validator.
      
      * Add test
      
      * Make sure we only send `Seconded` statements
      
      * Make sure we only receive valid statements
      
      * Review feedback
      ac5ef00e
  5. 09 Feb, 2021 1 commit
  6. 08 Feb, 2021 1 commit
    • Andronik Ordian's avatar
      validator_discovery: pass PeerSet to the request (#2372) · 57eb9d10
      Andronik Ordian authored
      
      
      * validator_discovery: pass PeerSet to the request
      
      * validator_discovery: track PeerSet of connected peers
      
      * validator_discovery: fix tests
      
      * validator_discovery: fix long line
      
      * some fixes
      
      * some validator_discovery logs
      
      * log validator discovery request
      
      * Also connect to validators on `DistributePoV`.
      
      * validator_discovery: store the whole state per peer_set
      
      * bump spec versions in kusama, polkadot and westend
      
      * Correcting doc.
      
      * validator_discovery: bump channel capacity
      
      * pov-distribution: some cleanup
      
      * this should fix the test, but it does not
      
      * I just got some brain damage while fixing this
      
      Why are you even reading this???
      
      * wrap long line
      
      * address some review nits
      
      Co-authored-by: default avatarRobert Klotzner <robert.klotzner@gmx.at>
      57eb9d10
  7. 03 Feb, 2021 1 commit
    • Robert Klotzner's avatar
      Generic request/response infrastructure for Polkadot (#2352) · ecc3772d
      Robert Klotzner authored
      * Move NetworkBridgeEvent to subsystem::messages.
      
      It is not protocol related at all, it is in fact only part of the
      subsystem communication as it gets wrapped into messages of each
      subsystem.
      
      * Request/response infrastructure is taking shape.
      
      WIP: Does not compile.
      
      * Multiplexer variant not supported by Rusts type system.
      
      * request_response::request type checks.
      
      * Cleanup.
      
      * Minor fixes for request_response.
      
      * Implement request sending + move multiplexer.
      
      Request multiplexer is moved to bridge, as there the implementation is
      more straight forward as we can specialize on `AllMessages` for the
      multiplexing target.
      
      Sending of requests is mostly complete, apart from a few `From`
      instances. Receiving is also almost done, initializtion needs to be
      fixed and the multiplexer needs to be invoked.
      
      * Remove obsolete multiplexer.
      
      * Initialize bridge with multiplexer.
      
      * Finish generic request sending/receiving.
      
      Subsystems are now able to receive and send requests and responses via
      the overseer.
      
      * Doc update.
      
      * Fixes.
      
      * Link issue for not yet implemented code.
      
      * Fixes suggested by @ordian
      
       - thanks!
      
      - start encoding at 0
      - don't crash on zero protocols
      - don't panic on not yet implemented request handling
      
      * Update node/network/protocol/src/request_response/v1.rs
      
      Use index 0 instead of 1.
      
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      
      * Update node/network/protocol/src/request_response.rs
      
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      
      * Fix existing tests.
      
      * Better avoidance of division by zoro errors.
      
      * Doc fixes.
      
      * send_request -> start_request.
      
      * Fix missing renamings.
      
      * Update substrate.
      
      * Pass TryConnect instead of true.
      
      * Actually import `IfDisconnected`.
      
      * Fix wrong import.
      
      * Update node/network/bridge/src/lib.rs
      
      typo
      
      Co-authored-by: Pierre Krieger's avatarPierre Krieger <pierre.krieger1708@gmail.com>
      
      * Update node/network/bridge/src/multiplexer.rs
      
      Remove redundant import.
      
      Co-authored-by: Pierre Krieger's avatarPierre Krieger <pierre.krieger1708@gmail.com>
      
      * Stop doing tracing from within `From` instance.
      
      Thanks for the catch @tomaka
      
      !
      
      * Get rid of redundant import.
      
      * Formatting cleanup.
      
      * Fix tests.
      
      * Add link to issue.
      
      * Clarify comments some more.
      
      * Fix tests.
      
      * Formatting fix.
      
      * tabs
      
      * Fix link
      
      Co-authored-by: default avatarBernhard Schuster <bernhard@ahoi.io>
      
      * Use map_err.
      
      Co-authored-by: default avatarBernhard Schuster <bernhard@ahoi.io>
      
      * Improvements inspired by suggestions by @drahnr.
      
      - Channel size is now determined by function.
      - Explicitely scope NetworkService::start_request.
      
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      Co-authored-by: Pierre Krieger's avatarPierre Krieger <pierre.krieger1708@gmail.com>
      Co-authored-by: default avatarBernhard Schuster <bernhard@ahoi.io>
      ecc3772d
  8. 21 Jan, 2021 1 commit
  9. 05 Jan, 2021 1 commit
  10. 17 Dec, 2020 1 commit
    • Andronik Ordian's avatar
      refactor View to include finalized_number (#2128) · c429e15c
      Andronik Ordian authored
      * refactor View to include finalized_number
      
      * guide: update the NetworkBridge on BlockFinalized
      
      * av-store: fix the tests
      
      * actually fix tests
      
      * grumbles
      
      * ignore macro doctest
      
      * use Hash::repeat_bytes more consistently
      
      * broadcast empty leaves updates as well
      
      * fix issuing view updates on empty leaves updates
      c429e15c
  11. 11 Dec, 2020 1 commit
  12. 07 Dec, 2020 1 commit
    • Bastian Köcher's avatar
      Rework `ConnectionsRequests` (#2081) · b40c3566
      Bastian Köcher authored
      * Rework `ConnectionsRequests`
      
      Instead of implementing the `Stream` trait, this struct now provides a
      function `next()`. This enables us to encode into the type system that
      it will always return a value or block indefinitely.
      
      * Review feedback
      b40c3566
  13. 02 Dec, 2020 2 commits
  14. 26 Nov, 2020 1 commit
    • Andronik Ordian's avatar
      past-session validator discovery APIs (#2009) · 734eda87
      Andronik Ordian authored
      
      
      * guide: fix formatting for SessionInfo module
      
      * primitives: SessionInfo type
      
      * punt on approval keys
      
      * ah, revert the type alias
      
      * session info runtime module skeleton
      
      * update the guide
      
      * runtime/configuration: sync with the guide
      
      * runtime/configuration: setters for newly added fields
      
      * runtime/configuration: set codec indexes
      
      * runtime/configuration: update test
      
      * primitives: fix SessionInfo definition
      
      * runtime/session_info: initial impl
      
      * runtime/session_info: use initializer for session handling (wip)
      
      * runtime/session_info: mock authority discovery trait
      
      * guide: update the initializer's order
      
      * runtime/session_info: tests skeleton
      
      * runtime/session_info: store n_delay_tranches in Configuration
      
      * runtime/session_info: punt on approval keys
      
      * runtime/session_info: add some basic tests
      
      * Update primitives/src/v1.rs
      
      * small fixes
      
      * remove codec index annotation on structs
      
      * fix off-by-one error
      
      * validator_discovery: accept a session index
      
      * runtime: replace validator_discovery api with session_info
      
      * Update runtime/parachains/src/session_info.rs
      
      Co-authored-by: Sergey Pepyakin's avatarSergei Shulepov <sergei@parity.io>
      
      * runtime/session_info: add a comment about missing entries
      
      * runtime/session_info: define the keys
      
      * util: expose connect_to_past_session_validators
      
      * util: allow session_info requests for jobs
      
      * runtime-api: add mock test for session_info
      
      * collator-protocol: add session_index to test state
      
      * util: fix error message for runtime error
      
      * fix compilation
      
      * fix tests after merge with master
      
      Co-authored-by: Sergey Pepyakin's avatarSergei Shulepov <sergei@parity.io>
      734eda87
  15. 23 Nov, 2020 1 commit
  16. 22 Nov, 2020 1 commit
    • Bastian Köcher's avatar
      Improve collator side of the collator-protocol (#1955) · ba74791b
      Bastian Köcher authored
      * Improve collator side of the collator-protocol
      
      This pr improves the collator-protocol implementation of the collator
      side. Besides cleaning up code and rewriting it, the following changed:
      
      - Before on `PeerViewChange` we send an advertisment to every peer, now
      this only happens for validators.
      - It also adds a check that we send an advertisment message only once
      for a connected peer.
      - If the same validator was part of the current and next group, we
      requested to be connected to this validator two times. This is also
      fixed now.
      - Instead of having only one connection request, we now are being able
      to store multiple of them. This is required as we can have multiple
      active leafs at any point of time.
      
      * Switch to common `ConnectionRequests`
      
      * Update node/network/collator-protocol/src/collator_side.rs
      ba74791b
  17. 20 Nov, 2020 3 commits
    • Andronik Ordian's avatar
      cleanup validator discovery (#1992) · f4897f74
      Andronik Ordian authored
      * use snake_case for log targets
      
      * remove unused continue
      
      * validator_discovery: when disconnecting, use all addresses
      
      * validator_discovery: simplify request revokation
      
      * fix a typo
      f4897f74
    • Peter Goodspeed-Niklaus's avatar
      Add Prometheus timers to the subsystems (#1923) · e655654e
      Peter Goodspeed-Niklaus authored
      * reexport prometheus-super for ease of use of other subsystems
      
      * add some prometheus timers for collation generation subsystem
      
      * add timing metrics to av-store
      
      * add metrics to candidate backing
      
      * add timing metric to bitfield signing
      
      * add timing metrics to candidate selection
      
      * add timing metrics to candidate-validation
      
      * add timing metrics to chain-api
      
      * add timing metrics to provisioner
      
      * add timing metrics to runtime-api
      
      * add timing metrics to availability-distribution
      
      * add timing metrics to bitfield-distribution
      
      * add timing metrics to collator protocol: collator side
      
      * add timing metrics to collator protocol: validator side
      
      * fix candidate validation test failures
      
      * add timing metrics to pov distribution
      
      * add timing metrics to statement-distribution
      
      * use substrate_prometheus_endpoint prometheus reexport instead of prometheus_super
      
      * don't include JOB_DELAY in bitfield-signing metrics
      
      * give adder-collator ability to easily export its genesis-state and validation code
      
      * wip: adder-collator pushbutton script
      
      * don't attempt to register the adder-collator automatically
      
      Instead, get these values with
      
      ```sh
      target/release/adder-collator export-genesis-state
      target/release/adder-collator export-genesis-wasm
      ```
      
      And then register the parachain on https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A9944#/explorer
      
      To collect prometheus data, after running the script, create `prometheus.yml` per the instructions
      at https://www.notion.so/paritytechnologies/Setting-up-Prometheus-locally-835cb3a9df7541a781c381006252b5ff
      
      
      and then run:
      
      ```sh
      docker run -v `pwd`/prometheus.yml:/etc/prometheus/prometheus.yml:z --network host prom/prometheus
      ```
      
      Demonstrates that data makes it across to prometheus, though it is likely to be useful in the future
      to tweak the buckets.
      
      * Update parachain/test-parachains/adder/collator/src/cli.rs
      
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      
      * use the grandpa-pause parameter
      
      * skip metrics in tracing instrumentation
      
      * remove unnecessary grandpa_pause cli param
      
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      e655654e
    • Peter Goodspeed-Niklaus's avatar
      Add tracing support to node (#1940) · f7ea3d07
      Peter Goodspeed-Niklaus authored
      * drop in tracing to replace log
      
      * add structured logging to trace messages
      
      * add structured logging to debug messages
      
      * add structured logging to info messages
      
      * add structured logging to warn messages
      
      * add structured logging to error messages
      
      * normalize spacing and Display vs Debug
      
      * add instrumentation to the various 'fn run'
      
      * use explicit tracing module throughout
      
      * fix availability distribution test
      
      * don't double-print errors
      
      * remove further redundancy from logs
      
      * fix test errors
      
      * fix more test errors
      
      * remove unused kv_log_macro
      
      * fix unused variable
      
      * add tracing spans to collation generation
      
      * add tracing spans to av-store
      
      * add tracing spans to backing
      
      * add tracing spans to bitfield-signing
      
      * add tracing spans to candidate-selection
      
      * add tracing spans to candidate-validation
      
      * add tracing spans to chain-api
      
      * add tracing spans to provisioner
      
      * add tracing spans to runtime-api
      
      * add tracing spans to availability-distribution
      
      * add tracing spans to bitfield-distribution
      
      * add tracing spans to network-bridge
      
      * add tracing spans to collator-protocol
      
      * add tracing spans to pov-distribution
      
      * add tracing spans to statement-distribution
      
      * add tracing spans to overseer
      
      * cleanup
      f7ea3d07
  18. 05 Nov, 2020 1 commit
    • Andronik Ordian's avatar
      more resilient subsystems (#1908) · 9758a180
      Andronik Ordian authored
      * backing: extract log target
      
      * bitfield-signing: extract log target
      
      * utils: fix a typo
      
      * provisioner: extract log target
      
      * candidate selection: remove unused error variant
      
      * bitfield-distribution: change the return type of run
      
      * pov-distribution: extract log target
      
      * collator-protocol: simplify runtime request
      
      * collation-generation: do not exit early on error
      
      * collation-generation: do not exit on double init
      
      * collator-protocol: do not exit on errors and rename LOG_TARGET
      
      * collator-protocol: a workaround for ununused imports warning
      
      * Update node/network/bitfield-distribution/src/lib.rs
      
      * collation-generation: elevate warn! to error!
      
      * collator-protocol: fix imports
      
      * post merge fix
      
      * fix compilation
      9758a180
  19. 03 Nov, 2020 1 commit
  20. 01 Nov, 2020 1 commit
  21. 28 Oct, 2020 1 commit
    • Peter Goodspeed-Niklaus's avatar
      start working on building the real overseer (#1795) · 798f781f
      Peter Goodspeed-Niklaus authored
      
      
      * start working on building the real overseer
      
      Unfortunately, this fails to compile right now due to an upstream
      failure to compile which is probably brought on by a recent upgrade
      to rustc v1.47.
      
      * fill in AllSubsystems internal constructors
      
      * replace fn make_metrics with Metrics::attempt_to_register
      
      * update to account for #1740
      
      * remove Metrics::register, rename Metrics::attempt_to_register
      
      * add 'static bounds to real_overseer type params
      
      * pass authority_discovery and network_service to real_overseer
      
      It's not straightforwardly obvious that this is the best way to handle
      the case when there is no authority discovery service, but it seems
      to be the best option available at the moment.
      
      * select a proper database configuration for the availability store db
      
      * use subdirectory for av-store database path
      
      * apply Basti's patch which avoids needing to parameterize everything on Block
      
      * simplify path extraction
      
      * get all tests to compile
      
      * Fix Prometheus double-registry error
      
      for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476:
      
      ```rust
      Some(registry) => Self::try_register(registry).map_err(|err| {
      	eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err);
      	err
      }),
      ```
      
      That pointed out where the registration was failing, which led to
      this fix. The test still doesn't pass, but it now fails in a new
      and different way!
      
      * authorities must have authority discovery, but not necessarily overseer handlers
      
      * fix broken SpawnedSubsystem impls
      
      detailed logging determined that using the `Box::new` style of
      future generation, the `self.run` method was never being called,
      leading to dropped receivers / closed senders for those subsystems,
      causing the overseer to shut down immediately.
      
      This is not the final fix needed to get things working properly,
      but it's a good start.
      
      * use prometheus properly
      
      Prometheus lets us register simple counters, which aren't very
      interesting. It also allows us to register CounterVecs, which are.
      With a CounterVec, you can provide a set of labels, which can
      later be used to filter the counts.
      
      We were using them wrong, though. This pattern was repeated in a
      variety of places in the code:
      
      ```rust
      // panics with an cardinality mismatch
      let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?;
      my_counter.with_label_values(&["succeeded"]).inc()
      ```
      
      The problem is that the labels provided in the constructor are not
      the set of legal values which can be annotated, but a set of individual
      label names which can have individual, arbitrary values.
      
      This commit fixes that.
      
      * get av-store subsystem to actually run properly and not die on first signal
      
      * typo fix: incomming -> incoming
      
      * don't disable authority discovery in test nodes
      
      * Fix rococo-v1 missing session keys
      
      * Update node/core/av-store/Cargo.toml
      
      * try dummying out av-store on non-full-nodes
      
      * overseer and subsystems are required only for full nodes
      
      * Reduce the amount of warnings on browser target
      
      * Fix two more warnings
      
      * InclusionInherent should actually have an Inherent module on rococo
      
      * Ancestry: don't return genesis' parent hash
      
      * Update Cargo.lock
      
      * fix broken test
      
      * update test script: specify chainspec as script argument
      
      * Apply suggestions from code review
      
      Co-authored-by: default avatarBastian Köcher <bkchr@users.noreply.github.com>
      
      * Update node/service/src/lib.rs
      
      Co-authored-by: default avatarBastian Köcher <bkchr@users.noreply.github.com>
      
      * node/service/src/lib: Return error via ? operator
      
      * post-merge blues
      
      * add is_collator flag
      
      * prevent occasional av-store test panic
      
      * simplify fix; expand application
      
      * run authority_discovery in Role::Discover when collating
      
      * distinguish between proposer closed channel errors
      
      * add IsCollator enum, remove is_collator CLI flag
      
      * improve formatting
      
      * remove nop loop
      
      * Fix some stuff
      
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      Co-authored-by: Bastian Köcher's avatarBastian Köcher <git@kchr.de>
      Co-authored-by: Fedor Sakharov's avatarFedor Sakharov <fedor.sakharov@gmail.com>
      Co-authored-by: default avatarRobert Habermeier <robert@Roberts-MBP.lan1>
      Co-authored-by: default avatarBastian Köcher <bkchr@users.noreply.github.com>
      Co-authored-by: default avatarMax Inden <mail@max-inden.de>
      798f781f
  22. 07 Oct, 2020 1 commit
  23. 06 Oct, 2020 3 commits
    • Andronik Ordian's avatar
      collator: fix a typo (#1788) · 7ac23ed3
      Andronik Ordian authored
      * collator: fix a typo
      
      * collator: fix more typos
      
      * collator: fix even more typos
      7ac23ed3
    • Andronik Ordian's avatar
      NetworkBridge: validator (authorities) discovery api (#1699) · b84f3c03
      Andronik Ordian authored
      
      
      * stupid, but it compiles
      
      * redo
      
      * cleanup
      
      * add ValidatorDiscovery to msgs
      
      * sketch network bridge code
      
      * ConnectToAuthorities instead of validators
      
      * more stuff
      
      * cleanup
      
      * more stuff
      
      * complete ConnectToAuthoritiesState
      
      * Update node/network/bridge/src/lib.rs
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
      
      * Collator protocol subsystem (#1659)
      
      * WIP
      
      * The initial implementation of the collator side.
      
      * Improve comments
      
      * Multiple collation requests
      
      * Add more tests and comments to validator side
      
      * Add comments, remove dead code
      
      * Apply suggestions from code review
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
      
      * Fix build after suggested changes
      
      * Also connect to the next validator group
      
      * Remove a Future impl and move TimeoutExt to util
      
      * Minor nits
      
      * Fix build
      
      * Change FetchCollations back to FetchCollation
      
      * Try this
      
      * Final fixes
      
      * Fix build
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
      
      * handle multiple in-flight connection requests
      
      * handle cancelled requests
      
      * Update node/core/runtime-api/src/lib.rs
      
      Co-authored-by: default avatarBernhard Schuster <bernhard@ahoi.io>
      
      * redo it again
      
      * more stuff
      
      * redo it again
      
      * update comments
      
      * workaround Future is not Send
      
      * fix trailing spaces
      
      * clarify comments
      
      * bridge: fix compilation in tests
      
      * update more comments
      
      * small fixes
      
      * port collator protocol to new validator discovery api
      
      * collator tests compile
      
      * collator tests pass
      
      * do not revoke a request when the stream receiver is closed
      
      * make revoking opt-in
      
      * fix is_fulfilled
      
      * handle request revokation in collator
      
      * tests
      
      * wait for validator connections asyncronously
      
      * fix compilation
      
      * relabel my todos
      
      * apply Fedor's patch
      
      * resolve reconnection TODO
      
      * resolve revoking TODO
      
      * resolve channel capacity TODO
      
      * resolve peer cloning TODO
      
      * resolve peer disconnected TODO
      
      * resolve PeerSet TODO
      
      * wip tests
      
      * more tests
      
      * resolve Arc TODO
      
      * rename pending to non_revoked
      
      * one more test
      
      * extract utility function into util crate
      
      * fix compilation in tests
      
      * Apply suggestions from code review
      
      Co-authored-by: Fedor Sakharov's avatarFedor Sakharov <fedor.sakharov@gmail.com>
      
      * revert pin_project removal
      
      * fix while let loop
      
      * Revert "revert pin_project removal"
      
      This reverts commit ae7f529d
      
      .
      
      * fix compilation
      
      * Update node/subsystem/src/messages.rs
      
      * docs on pub items
      
      * guide updates
      
      * remove a TODO
      
      * small guide update
      
      * fix a typo
      
      * link to the issue
      
      * validator discovery: on_request docs
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
      Co-authored-by: Fedor Sakharov's avatarFedor Sakharov <fedor.sakharov@gmail.com>
      Co-authored-by: default avatarBernhard Schuster <bernhard@ahoi.io>
      b84f3c03
    • Bastian Köcher's avatar
      Make collation an optional return (#1787) · ce0fa3ad
      Bastian Köcher authored
      This pr changes the collator interface function to return an optional
      collation instead of a collation. This is required as the parachain
      itself can fail to generate a valid collation for various reason. Now if
      the collation fails it will return `None`.
      
      Besides that the pr adds some `RuntimeDebug` derive for `ValidationData`
      and removes some whitespaces.
      ce0fa3ad
  24. 29 Sep, 2020 1 commit
  25. 28 Sep, 2020 1 commit
  26. 10 Sep, 2020 1 commit