1. 18 May, 2021 1 commit
  2. 03 May, 2021 1 commit
  3. 29 Apr, 2021 1 commit
  4. 09 Apr, 2021 1 commit
    • Robert Klotzner's avatar
      Req/res optimization for statement distribution (#2803) · 48d5b143
      Robert Klotzner authored
      * Wip
      * Increase proposer timeout.
      * WIP.
      * Better timeout values now that we are going to be connected to all nodes. (#2778)
      * Better timeout values.
      * Fix typo.
      * Fix validator bandwidth.
      * Fix compilation.
      * Better and more consistent sizes.
      Most importantly code size is now 5 Meg, which is the limit we currently
      want to support in statement distribution.
      * Introduce statement fetching request.
      * WIP
      * Statement cache retrieval logic.
      * Review remarks by @rphmeier
      * Fixes.
      * Better requester logic.
      * WIP: Handle requester messages.
      * Missing dep.
      * Fix request launching logic.
      * Finish fetching logic.
      * Sending logic.
      * Redo code size calculations.
      Now that max code size is compressed size.
      * Update Cargo.lock (new dep)
      * Get request receiver to statement distribution.
      * Expose new functionality for responding to requests.
      * Cleanup.
      * Responder logic.
      * Fixes + Cleanup.
      * Cargo.lock
      * Whitespace.
      * Add lost copyright.
      * Launch responder task.
      * Typo.
      * info -> warn
      * Typo.
      * Fix.
      * Fix.
      * Update comment.
      * Doc fix.
      * Better large statement heuristics.
      * Fix tests.
      * Fix network bridge tests.
      * Add test for size estimate.
      * Very simple tests that checks we get LargeStatement.
      * Basic check, that fetching of large candidates is performed.
      * More tests.
      * Basic metrics for responder.
      * More metrics.
      * Use Encode::encoded_size().
      * Some useful spans.
      * Get rid of redundant metrics.
      * Don't add peer on duplicate.
      * Properly check hash
      instead of relying on signatures alone.
      * Preserve ordering + better flood protection.
      * Get rid of redundant clone.
      * Don't shutdown responder on failed query.
      And add test for this.
      * Smaller fixes.
      * Quotes.
      * Better queue size calculation.
      * A bit saner response sizes.
      * Fixes.
  5. 08 Apr, 2021 1 commit
    • asynchronous rob's avatar
      Remove real-overseer 🎉 (#2834) · e4f5ade5
      asynchronous rob authored
      * remove real-overseer
      * overseer: only activate leaves which support parachains
      * integrate HeadSupportsParachains into service
      * remove unneeded line
  6. 29 Mar, 2021 2 commits
  7. 28 Mar, 2021 2 commits
    • asynchronous rob's avatar
      Overseer: subsystems communicate directly (#2227) · 322ccd0d
      asynchronous rob authored
      * overseer: pass messages directly between subsystems
      * test that message is held on to
      * Update node/overseer/src/lib.rs
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
      * give every subsystem an unbounded sender too
      * remove metered_channel::name
      1. we don't provide good names
      2. these names are never used anywhere
      * unused mut
      * remove unnecessary &mut
      * subsystem unbounded_send
      * remove unused MaybeTimer
      We have channel size metrics that serve the same purpose better now and the implementation of message timing was pretty ugly.
      * remove comment
      * split up senders and receivers
      * update metrics
      * fix tests
      * fix test subsystem context
      * fix flaky test
      * fix docs
      * doc
      * use select_biased to favor signals
      * Update node/subsystem/src/lib.rs
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
    • Robert Klotzner's avatar
      Request based PoV distribution (#2640) · 39124b0a
      Robert Klotzner authored
      * Indentation fix.
      * Prepare request-response for PoV fetching.
      * Drop old PoV distribution.
      * WIP: Fetch PoV directly from backing.
      * Backing compiles.
      * Runtime access and connection management for PoV distribution.
      * Get rid of seemingly dead code.
      * Implement PoV fetching.
      Backing does not yet use it.
      * Don't send `ConnectToValidators` for empty list.
      * Even better - no need to check over and over again.
      * PoV fetching implemented.
      + Typechecks
      + Should work
      - Guide
      - Tests
      - Do fallback fetching in case fetching from seconding validator fails.
      * Check PoV hash upon reception.
      * Implement retry of PoV fetching in backing.
      * Avoid pointless validation spawning.
      * Add jaeger span to pov requesting.
      * Add back tracing.
      * Review remarks.
      * Whitespace.
      * Whitespace again.
      * Cleanup + fix tests.
      * Log to log target in overseer.
      * Fix more tests.
      * Don't fail if group cannot be found.
      * Simple test for PoV fetcher.
      * Handle missing group membership better.
      * Add test for retry functionality.
      * Fix flaky test.
      * Spaces again.
      * Guide updates.
      * Spaces.
  8. 26 Mar, 2021 2 commits
  9. 25 Mar, 2021 1 commit
  10. 19 Mar, 2021 1 commit
  11. 10 Mar, 2021 1 commit
  12. 02 Mar, 2021 1 commit
  13. 26 Feb, 2021 2 commits
    • Robert Klotzner's avatar
      Request based availability distribution (#2423) · 950447e1
      Robert Klotzner authored
      * WIP
      * availability distribution, still very wip.
      Work on the requesting side of things.
      * Some docs on what I intend to do.
      * Checkpoint of session cache implementation
      as I will likely replace it with something smarter.
      * More work, mostly on cache
      and getting things to type check.
      * Only derive MallocSizeOf and Debug for std.
      * availability-distribution: Cache feature complete.
      * Sketch out logic in `FetchTask` for actual fetching.
      - Compile fixes.
      - Cleanup.
      * Format cleanup.
      * More format fixes.
      * Almost feature complete `fetch_task`.
      - Check for cancel
      - Actual querying of peer ids.
      * Finish FetchTask so far.
      * Directly use AuthorityDiscoveryId in protocol and cache.
      * Resolve `AuthorityDiscoveryId` on sending requests.
      * Rework fetch_task
      - also make it impossible to check the wrong chunk index.
      - Export needed function in validator_discovery.
      * From<u32> implementation for `ValidatorIndex`.
      * Fixes and more integration work.
      * Make session cache proper lru cache.
      * Use proper lru cache.
      * Requester finished.
      * ProtocolState -> Requester
      Also make sure to not fetch our own chunk.
      * Cleanup + fixes.
      * Remove unused functions
      - FetchTask::is_finished
      - SessionCache::fetch_session_info
      * availability-distribution responding side.
      * Cleanup + Fixes.
      * More fixes.
      * More fixes.
      adder-collator is running!
      * Some docs.
      * Docs.
      * Fix reporting of bad guys.
      * Fix tests
      * Make all tests compile.
      * Fix test.
      * Cleanup + get rid of some warnings.
      * state -> requester
      * Mostly doc fixes.
      * Fix test suite.
      * Get rid of now redundant message types.
      * WIP
      * Rob's review remarks.
      * Fix test suite.
      * core.relay_parent -> leaf for session request.
      * Style fix.
      * Decrease request timeout.
      * Cleanup obsolete errors.
      * Metrics + don't fail on non fatal errors.
      * requester.rs -> requester/mod.rs
      * Panic on invalid BadValidator report.
      * Fix indentation.
      * Use typed default timeout constant.
      * Make channel size 0, as each sender gets one slot anyways.
      * Fix incorrect metrics initialization.
      * Fix build after merge.
      * More fixes.
      * Hopefully valid metrics names.
      * Better metrics names.
      * Some tests that already work.
      * Slightly better docs.
      * Some more tests.
      * Fix network bridge test.
    • Andronik Ordian's avatar
  14. 25 Feb, 2021 1 commit
  15. 24 Feb, 2021 1 commit
  16. 19 Feb, 2021 1 commit
    • Bernhard Schuster's avatar
      feat/jaeger: more spans, more stages (#2477) · f9186eb2
      Bernhard Schuster authored
      * feat/jaeger: more spans, more stages
      Stage numbers are still arbitrarily picked.
      * feat/jaeger: additional spans
      * chore/spellcheck: improve the dictionary
      * fix/jaeger JaegerSpan -> jaeger::Span
  17. 17 Feb, 2021 2 commits
    • asynchronous rob's avatar
      A fast-path for requesting `AvailableData` from backing validators (#2453) · f778e527
      asynchronous rob authored
      * guide changes for a fast-path requesting from backing validators
      * add backing group to availability recovery message
      * add new phase to interaction
      * typos
      * add full data messages
      * handle new network messages
      * dispatch full data requests
      * cleanup
      * check chunk index
      * test for invalid recovery
      * tests
      * Typos.
      * fix some grumbles
      * be more explicit about error handling and control flow
      * fast-path param
      * use with_chunks_only in Service
      Co-authored-by: default avatarRobert Klotzner <robert.klotzner@gmx.at>
    • Bernhard Schuster's avatar
      refactor/reputation: unify the values used (#2462) · 8c1b2e87
      Bernhard Schuster authored
      * refactor/reputation: unify the values used
      * chore/rep: rename Annoy* to Cost*, make duplicate message Cost*Repeated
      * fix/reputation: lost and found, convert at the boundary to substrate
      * refactor/rep: move conversion to base reputation one level down, left conversions
      * fix/rep: order of magnitude adjustments
      Thanks pierre!
      * remove spaces
      * chore/rep: give rationale for order of magnitude
      * refactor/rep: move UnifiedReputationChange to separate file
      * fix/rep: order of magnitudes correction
  18. 16 Feb, 2021 1 commit
    • asynchronous rob's avatar
      Integrate Approval Voting into Overseer / Service / GRANDPA (#2412) · 7a1e581c
      asynchronous rob authored
      * integrate approval voting into overseer
      * expose public API and make keystore arc
      * integrate overseer in service
      * guide: `ApprovedAncestor` returns block number
      * return block number along with hash from ApprovedAncestor
      * introduce a voting rule for reporting on approval checking
      * integrate the delay voting rule
      * Rococo configuration
      * fix compilation and add slack
      * fix web-wasm build
      * tweak parameterization
      * migrate voting rules to asycn
      * remove hack comment
  19. 14 Feb, 2021 1 commit
    • Bastian Köcher's avatar
      Notify collators about seconded collation (#2430) · ac5ef00e
      Bastian Köcher authored
      * Notify collators about seconded collation
      This pr adds functionality to inform a collator that its collation was
      seconded by a parachain validator. Before this signed statement was only
      gossiped over the validation substream. Now, we explicitly send the
      seconded statement to the collator after it was validated successfully.
      Besides that it changes the `CollatorFn` to return an optional result
      sender that is informed when the build collation was seconded by a
      parachain validator.
      * Add test
      * Make sure we only send `Seconded` statements
      * Make sure we only receive valid statements
      * Review feedback
  20. 11 Feb, 2021 1 commit
    • asynchronous rob's avatar
      Implement Approval Voting Subsystem (#2112) · 35ea1c4b
      asynchronous rob authored
      * skeleton
      * skeleton aux-schema module
      * start approval types
      * start aux schema with aux store
      * doc
      * finish basic types
      * start approval types
      * doc
      * finish basic types
      * write out schema types
      * add debug and codec impls to approval types
      * add debug and codec impls to approval types
      also add some key computation
      * add debug and codec impls to approval types
      * getters for block and candidate entries
      * grumbles
      * remove unused AssignmentId
      * load_decode utility
      * implement DB clearing
      * function for adding new block entry to aux store
      * start `canonicalize` implementation
      * more skeleton
      * finish implementing canonicalize
      * tag TODO
      * implement a test AuxStore
      * add allow(unused)
      * basic loading and deleting test
      * block_entry test function
      * add a test for `add_block_entry`
      * ensure range is exclusive at end
      * test clear()
      * test that add_block sets children
      * add a test for canonicalize
      * extract Pre-digest from header
      * utilities for extracting RelayVRFStory from the header-chain
      * add approval voting message types
      * approval distribution message type
      * subsystem skeleton
      * state struct
      * add futures-timer
      * prepare service for babe slot duration
      * more skeleton
      * better integrate AuxStore
      * RelayVRF -> RelayVRFStory
      * canonicalize
      * implement some tick functionality
      * guide: tweaks
      * check_approval
      * more tweaks and helpers
      * guide: add core index to candidate event
      * primitives: add core index to candidate event
      * runtime: add core index to candidate events
      * head handling (session window)
      * implement `determine_new_blocks`
      * add TODO
      * change error type on functions
      * compute RelayVRFModulo assignments
      * compute RelayVRFDelay assignments
      * fix delay tranche calc
      * assignment checking
      * pluralize
      * some dummy code for fetching assignments
      * guide: add babe epoch runtime API
      * implement a current_epoch() runtime API
      * compute assignments
      * candidate events get backing group
      * import blocks and assignments into DB
      * push block approval meta
      * add message types, no overseer integration yet
      * notify approval distribution of new blocks
      * refactor import into separate functions
      * impl tranches_to_approve
      * guide: improve function signatures
      * guide: remove Tick from ApprovalEntry
      * trigger and broadcast assignment
      * most of approval launching
      * remove byteorder crate
      * load blocks back to finality, except on startup
      * check unchecked assignments
      * add claimed core to approval voting message
      * fix checks
      * assign only to backing group
      * remove import_checked_assignment from guide
      * newline
      * import assignments
      * abstract out a bit
      * check and import approvals
      * check full approvals from assignment import too
      * comment
      * create a Transaction utility
      * must_use
      * use transaction in `check_full_approvals`
      * wire up wakeups
      * add Ord to CandidateHash
      * wakeup refactoring
      * return candidate info from add_block_entry
      * schedule wakeups
      * background task: do candidate validation
      * forward candidate validation requests
      * issue approval votes when requested
      * clean up a couple TODOs
      * fix up session caching
      * clean up last unimplemented!() items
      * fix remaining warnings
      * remove TODO
      * implement handle_approved_ancestor
      * update Cargo.lock
      * fix runtime API tests
      * guide: cleanup assignment checking
      * use claimed candidate index instead of core
      * extract time to a trait
      * tests module
      * write a mock clock for testing
      * allow swapping out the clock
      * make abstract over assignment criteria
      * add some skeleton tests and simplify params
      * fix backing group check
      * do backing group check inside check_assignment_cert
      * write some empty test functions to implement
      * add a test for non-backing
      * test that produced checks pass
      * some empty test ideas
      * runtime/inclusion: remove outdated TODO
      * fix compilation
      * av-store: fix tests
      * dummy cert
      * criteria tests
      * move `TestStore` to main tests file
      * fix unused warning
      * test harness beginnings
      * resolve slots renaming fallout
      * more compilation fixes
      * wip: extract pure data into a separate module
      * wip: extract pure data into a separate module
      * move types completely to v1
      * add persisted_entries
      * add conversion trait impls
      * clean up some warnings
      * extract import logic to own module
      * schedule wakeups
      * experiment with Actions
      * uncomment approval-checking
      * separate module for approval checking utilities
      * port more code to use actions
      * get approval pipeline using actions
      * all logic is uncommented
      * main loop processes actions
      * all loop logic uncommented
      * separate function for handling actions
      * remove last unimplemented item
      * clean up warnings
      * State gives read-only access to underlying DB
      * tests for approval checking
      * tests for approval criteria
      * skeleton test module for import
      * list of import tests to do
      * some test glue code
      * test reject bad assignment
      * test slot too far in future
      * test reject assignment with unknown candidate
      * remove loads_blocks tests
      * determine_new_blocks back to finalized & harness
      * more coverage for determining new blocks
      * make `imported_block_info` have less reliance on State
      * candidate_info tests
      * tests for session caching
      * remove println
      * extricate DB and main TestStores
      * rewrite approval checking logic to counteract early delays
      * move state out of function
      * update approval-checking tests
      * tweak wakeups & scheduling logic
      * rename check_full_approvals
      * test that assignment import updates candidate
      * some approval import tests
      * some tests for check_and_apply_approval
      * add 'full' qualifier to avoid confusion
      * extract should-trigger logic to separate function
      * some tests for all triggering
      * tests for when we trigger assignments
      * test wakeups
      * add block utilities for testing
      * some more tests for approval updates
      * approved_ancestor tests
      * new action type for launch approval
      * process-wakeup tests
      * clean up some warnings
      * fix in_future test
      * approval checking tests
      * tighten up too-far-in-future
      * special-case genesis when caching sessions
      * fix bitfield len
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
  21. 03 Feb, 2021 1 commit
    • Robert Klotzner's avatar
      Generic request/response infrastructure for Polkadot (#2352) · ecc3772d
      Robert Klotzner authored
      * Move NetworkBridgeEvent to subsystem::messages.
      It is not protocol related at all, it is in fact only part of the
      subsystem communication as it gets wrapped into messages of each
      * Request/response infrastructure is taking shape.
      WIP: Does not compile.
      * Multiplexer variant not supported by Rusts type system.
      * request_response::request type checks.
      * Cleanup.
      * Minor fixes for request_response.
      * Implement request sending + move multiplexer.
      Request multiplexer is moved to bridge, as there the implementation is
      more straight forward as we can specialize on `AllMessages` for the
      multiplexing target.
      Sending of requests is mostly complete, apart from a few `From`
      instances. Receiving is also almost done, initializtion needs to be
      fixed and the multiplexer needs to be invoked.
      * Remove obsolete multiplexer.
      * Initialize bridge with multiplexer.
      * Finish generic request sending/receiving.
      Subsystems are now able to receive and send requests and responses via
      the overseer.
      * Doc update.
      * Fixes.
      * Link issue for not yet implemented code.
      * Fixes suggested by @ordian
       - thanks!
      - start encoding at 0
      - don't crash on zero protocols
      - don't panic on not yet implemented request handling
      * Update node/network/protocol/src/request_response/v1.rs
      Use index 0 instead of 1.
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      * Update node/network/protocol/src/request_response.rs
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      * Fix existing tests.
      * Better avoidance of division by zoro errors.
      * Doc fixes.
      * send_request -> start_request.
      * Fix missing renamings.
      * Update substrate.
      * Pass TryConnect instead of true.
      * Actually import `IfDisconnected`.
      * Fix wrong import.
      * Update node/network/bridge/src/lib.rs
      Co-authored-by: Pierre Krieger's avatarPierre Krieger <pierre.krieger1708@gmail.com>
      * Update node/network/bridge/src/multiplexer.rs
      Remove redundant import.
      Co-authored-by: Pierre Krieger's avatarPierre Krieger <pierre.krieger1708@gmail.com>
      * Stop doing tracing from within `From` instance.
      Thanks for the catch @tomaka
      * Get rid of redundant import.
      * Formatting cleanup.
      * Fix tests.
      * Add link to issue.
      * Clarify comments some more.
      * Fix tests.
      * Formatting fix.
      * tabs
      * Fix link
      Co-authored-by: default avatarBernhard Schuster <bernhard@ahoi.io>
      * Use map_err.
      Co-authored-by: default avatarBernhard Schuster <bernhard@ahoi.io>
      * Improvements inspired by suggestions by @drahnr.
      - Channel size is now determined by function.
      - Explicitely scope NetworkService::start_request.
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      Co-authored-by: Pierre Krieger's avatarPierre Krieger <pierre.krieger1708@gmail.com>
      Co-authored-by: default avatarBernhard Schuster <bernhard@ahoi.io>
  22. 25 Jan, 2021 1 commit
    • Andronik Ordian's avatar
      impl approval distribution (#2160) · ab606e14
      Andronik Ordian authored
      * initial impl approval distribution
      * initial tests and fixes
      * batching seems difficult: different peers have different needs
      * bridge: fix test after merge
      * some guide updates
      * only send assignments to peers who know about the block
      * fix a test, add approvals test
      * simplify
      * do not send assignment to peers for finalized blocks
      * guide: protocol input and output
      * one more test
      * more comments, logs, initial metrics
      * fix a typo
      * one more thing: early return when reimporting a thing locally
  23. 15 Jan, 2021 2 commits
    • Bernhard Schuster's avatar
    • Fedor Sakharov's avatar
      Availability recovery subsystem (#2122) · 030502a0
      Fedor Sakharov authored
      * Adds message types
      * Add code skeleton
      * Adds subsystem code.
      * Adds a first test
      * Adds interaction result to availability_lru
      * Use LruCache instead of a HashMap
      * Whitespaces to tabs
      * Do not ignore errors
      * Change error type
      * Add a timeout to chunk requests
      * Add custom errors and log them
      * Adds replace_availability_recovery method
      * recovery_threshold computed by erasure crate
      * change core to std
      * adds docs to error type
      * Adds a test for invalid reconstruction
      * refactors interaction run into multiple methods
      * Cleanup AwaitedChunks
      * Even more fixes
      * Test that recovery with wrong root is an error
      * Break to launch another requests
      * Styling fixes
      * Add SessionIndex to API
      * Proper relay parents for MakeRequest
      * Remove validator_discovery and use message
      * Remove a stream on exhaustion
      * On cleanup free the request streams
      * Fix merge and refactor
  24. 14 Jan, 2021 1 commit
  25. 13 Jan, 2021 1 commit
  26. 08 Jan, 2021 2 commits
  27. 07 Jan, 2021 1 commit
  28. 06 Jan, 2021 1 commit
    • Peter Goodspeed-Niklaus's avatar
      Add metrics timing message passing from OverseerSubsystemContext to Overseer::route_message (#2201) · 004dc50d
      Peter Goodspeed-Niklaus authored
      * add timing setup to OverseerSubsystemContext
      * figure out how to initialize the rng
      * attach a timer to a portion of the messages traveling to the Overseer
      This timer only exists / logs a fraction of the time (configurable
      by `MESSAGE_TIMER_METRIC_CAPTURE_RATE`). When it exists, it tracks
      the span between the `OverSubsystemContext` receiving the message
      and its receipt in `Overseer::run`.
      * propagate message timing to the start of route_message
      This should be more accurate; it ensures that the timer runs
      at least as long as that function. As `route_message` is async,
      it may not actually run for some time after it is called (or ever).
      * fix failing test
      * rand_chacha apparently implicitly has getrandom feature
      * change rng initialization
      The previous impl using `from_entropy` depends on the `getrandom`
      crate, which uses the system entropy source, and which does not
      work on `wasm32-unknown-unknown` because it wants to fall back to
      a JS implementation which we can't assume exists.
      This impl depends only on `rand::thread_rng`, which has no documentation
      stating that it's similarly limited.
      * remove randomness in favor of a simpler 1 of N procedure
      This deserves a bit of explanation, as the motivating issue explicitly
      requested randomness. In short, it's hard to get randomness to compile
      for `wasm32-unknown-unknown` because that is explicitly intended to be
      as deterministic as practical. Additionally, even though it would never
      be used for consensus purposes, it still felt offputting to intentionally
      introduce randomness into a node's operations. Except, it wasn't really
      random, either: it was a deterministic PRNG varying only in its state,
      and getting the state to work right for that target would have required
      initializing from a constant.
      Given that it was a deterministic sequence anyway, it seemed much simpler
      and more explicit to simply select one of each N messages instead of
      attempting any kind of realistic randomness.
      * reinstate randomness for better statistical properties
      This partially reverts commit 0ab8594c.
      `oorandom` is much lighter than the previous `rand`-based implementation,
      which makes this easier to work with.
      This implementation gives each subsystem and each child RNG a distinct
      increment, which should ensure they produce distinct streams of values.
  29. 05 Jan, 2021 1 commit
  30. 20 Dec, 2020 1 commit
  31. 18 Dec, 2020 1 commit
  32. 17 Dec, 2020 1 commit
    • Andronik Ordian's avatar
      refactor View to include finalized_number (#2128) · c429e15c
      Andronik Ordian authored
      * refactor View to include finalized_number
      * guide: update the NetworkBridge on BlockFinalized
      * av-store: fix the tests
      * actually fix tests
      * grumbles
      * ignore macro doctest
      * use Hash::repeat_bytes more consistently
      * broadcast empty leaves updates as well
      * fix issuing view updates on empty leaves updates
  33. 07 Dec, 2020 1 commit