1. 22 May, 2021 1 commit
  2. 03 May, 2021 1 commit
    • Robert Klotzner's avatar
      More secure `Signed` implementation (#2963) · e9a29ecc
      Robert Klotzner authored
      * Remove signature verification in backing.
      `SignedFullStatement` now signals that the signature has already been
      * Remove unused check_payload function.
      * Introduced unchecked signed variants.
      * Fix inclusion to use unchecked variant.
      * More unchecked variants.
      * Use unchecked variants in protocols.
      * Start fixing statement-distribution.
      * Fixup statement distribution.
      * Fix inclusion.
      * Fix warning.
      * Fix backing properly.
      * Fix bitfield distribution.
      * Make crypto store optional for `RuntimeInfo`.
      * Factor out utility functions.
      * get_group_rotation_info
      * WIP: Collator cleanup + check signatures.
      * Convenience signature checking functions.
      * Check signature on collator-side.
      * Fix warnings.
      * Fix collator side tests.
      * Get rid of warnings.
      * Better Signed/UncheckedSigned implementation.
      Also get rid of Encode/Decode for Signed! *party*
      * Get rid of dead code.
      * Move Signed in its own module.
      * into_checked -> try_into_checked
      * Fix merge.
  3. 09 Apr, 2021 1 commit
    • Robert Klotzner's avatar
      Req/res optimization for statement distribution (#2803) · 48d5b143
      Robert Klotzner authored
      * Wip
      * Increase proposer timeout.
      * WIP.
      * Better timeout values now that we are going to be connected to all nodes. (#2778)
      * Better timeout values.
      * Fix typo.
      * Fix validator bandwidth.
      * Fix compilation.
      * Better and more consistent sizes.
      Most importantly code size is now 5 Meg, which is the limit we currently
      want to support in statement distribution.
      * Introduce statement fetching request.
      * WIP
      * Statement cache retrieval logic.
      * Review remarks by @rphmeier
      * Fixes.
      * Better requester logic.
      * WIP: Handle requester messages.
      * Missing dep.
      * Fix request launching logic.
      * Finish fetching logic.
      * Sending logic.
      * Redo code size calculations.
      Now that max code size is compressed size.
      * Update Cargo.lock (new dep)
      * Get request receiver to statement distribution.
      * Expose new functionality for responding to requests.
      * Cleanup.
      * Responder logic.
      * Fixes + Cleanup.
      * Cargo.lock
      * Whitespace.
      * Add lost copyright.
      * Launch responder task.
      * Typo.
      * info -> warn
      * Typo.
      * Fix.
      * Fix.
      * Update comment.
      * Doc fix.
      * Better large statement heuristics.
      * Fix tests.
      * Fix network bridge tests.
      * Add test for size estimate.
      * Very simple tests that checks we get LargeStatement.
      * Basic check, that fetching of large candidates is performed.
      * More tests.
      * Basic metrics for responder.
      * More metrics.
      * Use Encode::encoded_size().
      * Some useful spans.
      * Get rid of redundant metrics.
      * Don't add peer on duplicate.
      * Properly check hash
      instead of relying on signatures alone.
      * Preserve ordering + better flood protection.
      * Get rid of redundant clone.
      * Don't shutdown responder on failed query.
      And add test for this.
      * Smaller fixes.
      * Quotes.
      * Better queue size calculation.
      * A bit saner response sizes.
      * Fixes.
  4. 03 Apr, 2021 1 commit
    • asynchronous rob's avatar
      Collation protocol: stricter validators (#2810) · 54e18e65
      asynchronous rob authored
      * guide: declare one para as a collator
      * add ParaId to Declare messages and clean up
      * fix build
      * fix the testerinos
      * begin adding keystore to collator-protocol
      * remove request_x_ctx
      * add core_for_group
      * add bump_rotation
      * add some more helpers to subsystem-util
      * change signing_key API to take ref
      * determine current and next para assignments
      * disconnect collators who are not on current or next para
      * add collator peer count metric
      * notes for later
      * some fixes
      * add data & keystore to test state
      * add a test utility for answering runtime API requests
      * fix existing collator tests
      * add new tests
      * remove sc_keystore
      * update cargo lock
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
  5. 30 Mar, 2021 1 commit
  6. 28 Mar, 2021 1 commit
    • Robert Klotzner's avatar
      Request based PoV distribution (#2640) · 39124b0a
      Robert Klotzner authored
      * Indentation fix.
      * Prepare request-response for PoV fetching.
      * Drop old PoV distribution.
      * WIP: Fetch PoV directly from backing.
      * Backing compiles.
      * Runtime access and connection management for PoV distribution.
      * Get rid of seemingly dead code.
      * Implement PoV fetching.
      Backing does not yet use it.
      * Don't send `ConnectToValidators` for empty list.
      * Even better - no need to check over and over again.
      * PoV fetching implemented.
      + Typechecks
      + Should work
      - Guide
      - Tests
      - Do fallback fetching in case fetching from seconding validator fails.
      * Check PoV hash upon reception.
      * Implement retry of PoV fetching in backing.
      * Avoid pointless validation spawning.
      * Add jaeger span to pov requesting.
      * Add back tracing.
      * Review remarks.
      * Whitespace.
      * Whitespace again.
      * Cleanup + fix tests.
      * Log to log target in overseer.
      * Fix more tests.
      * Don't fail if group cannot be found.
      * Simple test for PoV fetcher.
      * Handle missing group membership better.
      * Add test for retry functionality.
      * Fix flaky test.
      * Spaces again.
      * Guide updates.
      * Spaces.
  7. 25 Mar, 2021 1 commit
  8. 24 Mar, 2021 1 commit
    • André Silva's avatar
      collator-protocol: add message authentication (#2635) · c836fb85
      André Silva authored
      * collator: authenticate collator protocol messages
      * fix tests compilation
      * node: verify collator protocol signatures in tests
      * collator: fix tests
      * implementers-guide: update CollatorProtocol messages
      * collator: add test for verification of collator protocol signatures
      * node: remove fixmes
      * node: remove signature from advertisecollation message
      * node: add magic constant to Declare message signature payload
  9. 18 Mar, 2021 1 commit
    • Robert Klotzner's avatar
      Request based collation fetching (#2621) · 134090c2
      Robert Klotzner authored
      * Introduce collation fetching protocol
      also move to mod.rs
      * Allow `PeerId`s in requests to network bridge.
      * Fix availability distribution tests.
      * Move CompressedPoV to primitives.
      * Request based collator protocol: validator side
      - Missing: tests
      - Collator side
      - don't connect, if not connected
      * Fixes.
      * Basic request based collator side.
      * Minor fix on collator side.
      * Don't connect in requests in collation protocol.
      Also some cleanup.
      * Fix PoV distribution
      * Bump substrate
      * Add back metrics + whitespace fixes.
      * Add back missing spans.
      * More cleanup.
      * Guide update.
      * Fix tests
      * Handle results in tests.
      * Fix weird compilation issue.
      * Add missing )
      * Get rid of dead code.
      * Get rid of redundant import.
      * Fix runtime build.
      * Cleanup.
      * Fix wasm build.
      * Format fixes.
      Thanks @andronik !
  10. 26 Feb, 2021 1 commit
    • Robert Klotzner's avatar
      Request based availability distribution (#2423) · 950447e1
      Robert Klotzner authored
      * WIP
      * availability distribution, still very wip.
      Work on the requesting side of things.
      * Some docs on what I intend to do.
      * Checkpoint of session cache implementation
      as I will likely replace it with something smarter.
      * More work, mostly on cache
      and getting things to type check.
      * Only derive MallocSizeOf and Debug for std.
      * availability-distribution: Cache feature complete.
      * Sketch out logic in `FetchTask` for actual fetching.
      - Compile fixes.
      - Cleanup.
      * Format cleanup.
      * More format fixes.
      * Almost feature complete `fetch_task`.
      - Check for cancel
      - Actual querying of peer ids.
      * Finish FetchTask so far.
      * Directly use AuthorityDiscoveryId in protocol and cache.
      * Resolve `AuthorityDiscoveryId` on sending requests.
      * Rework fetch_task
      - also make it impossible to check the wrong chunk index.
      - Export needed function in validator_discovery.
      * From<u32> implementation for `ValidatorIndex`.
      * Fixes and more integration work.
      * Make session cache proper lru cache.
      * Use proper lru cache.
      * Requester finished.
      * ProtocolState -> Requester
      Also make sure to not fetch our own chunk.
      * Cleanup + fixes.
      * Remove unused functions
      - FetchTask::is_finished
      - SessionCache::fetch_session_info
      * availability-distribution responding side.
      * Cleanup + Fixes.
      * More fixes.
      * More fixes.
      adder-collator is running!
      * Some docs.
      * Docs.
      * Fix reporting of bad guys.
      * Fix tests
      * Make all tests compile.
      * Fix test.
      * Cleanup + get rid of some warnings.
      * state -> requester
      * Mostly doc fixes.
      * Fix test suite.
      * Get rid of now redundant message types.
      * WIP
      * Rob's review remarks.
      * Fix test suite.
      * core.relay_parent -> leaf for session request.
      * Style fix.
      * Decrease request timeout.
      * Cleanup obsolete errors.
      * Metrics + don't fail on non fatal errors.
      * requester.rs -> requester/mod.rs
      * Panic on invalid BadValidator report.
      * Fix indentation.
      * Use typed default timeout constant.
      * Make channel size 0, as each sender gets one slot anyways.
      * Fix incorrect metrics initialization.
      * Fix build after merge.
      * More fixes.
      * Hopefully valid metrics names.
      * Better metrics names.
      * Some tests that already work.
      * Slightly better docs.
      * Some more tests.
      * Fix network bridge test.
  11. 23 Feb, 2021 2 commits
    • asynchronous rob's avatar
      Approval Checking Improvements Omnibus (#2480) · 1e2b8ae5
      asynchronous rob authored
      * add tracing to approval voting
      * notify if session info is not working
      * add dispute period to chain specs
      * propagate genesis session to parachains runtime
      * use `on_genesis_session`
      * protect against zero cores in computation
      * tweak voting rule to be based off of best and add logs
      * genesis configuration should use VRF slots only
      * swallow more keystore errors
      * add some docs
      * make validation-worker args non-optional and update clap
      * better tracing for bitfield signing and provisioner
      * pass amount of bits in bitfields to inclusion instead of recomputing
      * debug -> warn for some logs
      * better tracing for availability recovery
      * a little av-store tracing
      * bridge: forward availability recovery messages
      * add missing try_from impl
      * some more tracing
      * improve approval distribution tracing
      * guide: hold onto pending approval messages until NewBlocks
      * Hold onto pending approval messages until NewBlocks
      * guide: adjust comment
      * process all actions for one wakeup at a time
      * vec
      * fix network bridge test
      * replace randomness-collective-flip with Babe
      * remove PairNotFound
    • Bernhard Schuster's avatar
      feat/view: assure heads in a view are sorted (#2493) · 302d6582
      Bernhard Schuster authored
      * feat/view: assure heads in a view are sorted
      Allows O(n) comparisons, adds an alternate equiv relation
      which takes O(n^2) for integrity verification.
      Ref #2133
      * revert: remove custom PartialEq impl, there are no duplicates
      * fix: do not sort the live_heads, that alters the local view
      * refactor/view: heads should not be public
      * chore/spellcheck: add unfinalized
      * fix/view: add missing len() and is_empty() fns
      * quirk
      * vec is not view
      * Update node/network/approval-distribution/src/tests.rs
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      * Update node/network/bridge/src/lib.rs
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      * Update node/network/protocol/src/lib.rs
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      * fixup comment
      * fix botched test
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
  12. 19 Feb, 2021 1 commit
    • Bernhard Schuster's avatar
      feat/jaeger: more spans, more stages (#2477) · f9186eb2
      Bernhard Schuster authored
      * feat/jaeger: more spans, more stages
      Stage numbers are still arbitrarily picked.
      * feat/jaeger: additional spans
      * chore/spellcheck: improve the dictionary
      * fix/jaeger JaegerSpan -> jaeger::Span
  13. 17 Feb, 2021 2 commits
    • asynchronous rob's avatar
      Full chunk messages for availability-recovery (#2466) · b0ff3d49
      asynchronous rob authored
      * add full data messages
      * handle new messages
      * flip condition
    • Bernhard Schuster's avatar
      refactor/reputation: unify the values used (#2462) · 8c1b2e87
      Bernhard Schuster authored
      * refactor/reputation: unify the values used
      * chore/rep: rename Annoy* to Cost*, make duplicate message Cost*Repeated
      * fix/reputation: lost and found, convert at the boundary to substrate
      * refactor/rep: move conversion to base reputation one level down, left conversions
      * fix/rep: order of magnitude adjustments
      Thanks pierre!
      * remove spaces
      * chore/rep: give rationale for order of magnitude
      * refactor/rep: move UnifiedReputationChange to separate file
      * fix/rep: order of magnitudes correction
  14. 15 Feb, 2021 1 commit
  15. 14 Feb, 2021 1 commit
    • Bastian Köcher's avatar
      Notify collators about seconded collation (#2430) · ac5ef00e
      Bastian Köcher authored
      * Notify collators about seconded collation
      This pr adds functionality to inform a collator that its collation was
      seconded by a parachain validator. Before this signed statement was only
      gossiped over the validation substream. Now, we explicitly send the
      seconded statement to the collator after it was validated successfully.
      Besides that it changes the `CollatorFn` to return an optional result
      sender that is informed when the build collation was seconded by a
      parachain validator.
      * Add test
      * Make sure we only send `Seconded` statements
      * Make sure we only receive valid statements
      * Review feedback
  16. 04 Feb, 2021 1 commit
    • Sergey Pepyakin's avatar
      Diagnostics quality of life improvements (#2375) · 4f9e845d
      Sergey Pepyakin authored
      * Implement `Debug` manually for CandidateHash
      This will make candidate hashes printed consistently without the
      `CandidateHash(` and `)` decorations.
      * Do not print CompressedPov's guts
      It can be overwhelming. Better just use the size.
      * Log when candidate is generated
      * Print para_id and candidate_hash upon receiving a collation
  17. 03 Feb, 2021 1 commit
    • Robert Klotzner's avatar
      Generic request/response infrastructure for Polkadot (#2352) · ecc3772d
      Robert Klotzner authored
      * Move NetworkBridgeEvent to subsystem::messages.
      It is not protocol related at all, it is in fact only part of the
      subsystem communication as it gets wrapped into messages of each
      * Request/response infrastructure is taking shape.
      WIP: Does not compile.
      * Multiplexer variant not supported by Rusts type system.
      * request_response::request type checks.
      * Cleanup.
      * Minor fixes for request_response.
      * Implement request sending + move multiplexer.
      Request multiplexer is moved to bridge, as there the implementation is
      more straight forward as we can specialize on `AllMessages` for the
      multiplexing target.
      Sending of requests is mostly complete, apart from a few `From`
      instances. Receiving is also almost done, initializtion needs to be
      fixed and the multiplexer needs to be invoked.
      * Remove obsolete multiplexer.
      * Initialize bridge with multiplexer.
      * Finish generic request sending/receiving.
      Subsystems are now able to receive and send requests and responses via
      the overseer.
      * Doc update.
      * Fixes.
      * Link issue for not yet implemented code.
      * Fixes suggested by @ordian
       - thanks!
      - start encoding at 0
      - don't crash on zero protocols
      - don't panic on not yet implemented request handling
      * Update node/network/protocol/src/request_response/v1.rs
      Use index 0 instead of 1.
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      * Update node/network/protocol/src/request_response.rs
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      * Fix existing tests.
      * Better avoidance of division by zoro errors.
      * Doc fixes.
      * send_request -> start_request.
      * Fix missing renamings.
      * Update substrate.
      * Pass TryConnect instead of true.
      * Actually import `IfDisconnected`.
      * Fix wrong import.
      * Update node/network/bridge/src/lib.rs
      Co-authored-by: Pierre Krieger's avatarPierre Krieger <pierre.krieger1708@gmail.com>
      * Update node/network/bridge/src/multiplexer.rs
      Remove redundant import.
      Co-authored-by: Pierre Krieger's avatarPierre Krieger <pierre.krieger1708@gmail.com>
      * Stop doing tracing from within `From` instance.
      Thanks for the catch @tomaka
      * Get rid of redundant import.
      * Formatting cleanup.
      * Fix tests.
      * Add link to issue.
      * Clarify comments some more.
      * Fix tests.
      * Formatting fix.
      * tabs
      * Fix link
      Co-authored-by: default avatarBernhard Schuster <bernhard@ahoi.io>
      * Use map_err.
      Co-authored-by: default avatarBernhard Schuster <bernhard@ahoi.io>
      * Improvements inspired by suggestions by @drahnr.
      - Channel size is now determined by function.
      - Explicitely scope NetworkService::start_request.
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      Co-authored-by: Pierre Krieger's avatarPierre Krieger <pierre.krieger1708@gmail.com>
      Co-authored-by: default avatarBernhard Schuster <bernhard@ahoi.io>
  18. 29 Jan, 2021 1 commit
  19. 25 Jan, 2021 1 commit
    • Andronik Ordian's avatar
      impl approval distribution (#2160) · ab606e14
      Andronik Ordian authored
      * initial impl approval distribution
      * initial tests and fixes
      * batching seems difficult: different peers have different needs
      * bridge: fix test after merge
      * some guide updates
      * only send assignments to peers who know about the block
      * fix a test, add approvals test
      * simplify
      * do not send assignment to peers for finalized blocks
      * guide: protocol input and output
      * one more test
      * more comments, logs, initial metrics
      * fix a typo
      * one more thing: early return when reimporting a thing locally
  20. 21 Jan, 2021 1 commit
  21. 15 Jan, 2021 1 commit
    • Fedor Sakharov's avatar
      Availability recovery subsystem (#2122) · 030502a0
      Fedor Sakharov authored
      * Adds message types
      * Add code skeleton
      * Adds subsystem code.
      * Adds a first test
      * Adds interaction result to availability_lru
      * Use LruCache instead of a HashMap
      * Whitespaces to tabs
      * Do not ignore errors
      * Change error type
      * Add a timeout to chunk requests
      * Add custom errors and log them
      * Adds replace_availability_recovery method
      * recovery_threshold computed by erasure crate
      * change core to std
      * adds docs to error type
      * Adds a test for invalid reconstruction
      * refactors interaction run into multiple methods
      * Cleanup AwaitedChunks
      * Even more fixes
      * Test that recovery with wrong root is an error
      * Break to launch another requests
      * Styling fixes
      * Add SessionIndex to API
      * Proper relay parents for MakeRequest
      * Remove validator_discovery and use message
      * Remove a stream on exhaustion
      * On cleanup free the request streams
      * Fix merge and refactor
  22. 14 Jan, 2021 1 commit
    • Robert Klotzner's avatar
      Some refactoring in network-bridge in the course of dealing with #2177 (#2263) · 490cbd72
      Robert Klotzner authored
      * More doc fixes.
      * Minor refactorings in the process of #2177
      By having everything peer set related depend directly on the enum the
      code becomes more clear and it is also straight forward to add more
      peersets/protocols as the compiler will complain if you forget to
      implement parts of it.
      * Add peer set infos on startup properly
      For feature real_overseer.
      + Fixes from review. Thanks @coriolinus and @ordian
      * More structure in network-bridge
      Some changes, which would have helped me in groking the code faster.
      Entry points/public types more to the top. Factored out implementation
      in their own files, to clear up the top-level view.
      * Get rid of local ProtocolName type definition.
      Does not add much at this level.
      * Fix tests + import cleanup.
      * Make spaces tabs.
      * Clarify what correct parameters to send_message are
      * Be more less vague in docs of send_message.
      * Apply suggestions from code review
      Extend copyright on new files to 2021 as well.
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
  23. 05 Jan, 2021 1 commit
  24. 17 Dec, 2020 1 commit
    • Andronik Ordian's avatar
      refactor View to include finalized_number (#2128) · c429e15c
      Andronik Ordian authored
      * refactor View to include finalized_number
      * guide: update the NetworkBridge on BlockFinalized
      * av-store: fix the tests
      * actually fix tests
      * grumbles
      * ignore macro doctest
      * use Hash::repeat_bytes more consistently
      * broadcast empty leaves updates as well
      * fix issuing view updates on empty leaves updates
  25. 24 Nov, 2020 1 commit
    • Bastian Köcher's avatar
      Do not send messages twice in bitfield distribution (#2005) · bb91bedf
      Bastian Köcher authored
      * Do not send messages twice in bitfield distribution
      This removes a bug which resulted in sending bitfield messages multiple
      times by not checking if we already relayed them. Besides that it also
      adds an optimization to not relay a message to a peer that send us
      this message.
      * Review comments
      * Break some lines
  26. 05 Nov, 2020 1 commit
  27. 27 Oct, 2020 1 commit
  28. 12 Aug, 2020 1 commit
    • asynchronous rob's avatar
      Network bridge refactoring impl (#1537) · 1e9c0540
      asynchronous rob authored
      * update networking types
      * port over overseer-protocol message types
      * Add the collation protocol to network bridge
      * message sending
      * stub for ConnectToValidators
      * add some helper traits and methods to protocol types
      * add collator protocol message
      * leaves-updating
      * peer connection and disconnection
      * add utilities for dispatching multiple events
      * implement message handling
      * add an observedrole enum with equality and no sentry nodes
      * derive partial-eq on network bridge event
      * add PartialEq impls for network message types
      * add Into implementation for observedrole
      * port over existing network bridge tests
      * add some more tests
      * port bitfield distribution
      * port over bitfield distribution tests
      * add codec indices
      * port PoV distribution
      * port over PoV distribution tests
      * port over statement distribution
      * port over statement distribution tests
      * update overseer and service-new
      * address review comments
      * port availability distribution
      * port over availability distribution tests