1. 10 Mar, 2021 1 commit
  2. 14 Feb, 2021 1 commit
    • Bastian Köcher's avatar
      Notify collators about seconded collation (#2430) · ac5ef00e
      Bastian Köcher authored
      * Notify collators about seconded collation
      
      This pr adds functionality to inform a collator that its collation was
      seconded by a parachain validator. Before this signed statement was only
      gossiped over the validation substream. Now, we explicitly send the
      seconded statement to the collator after it was validated successfully.
      
      Besides that it changes the `CollatorFn` to return an optional result
      sender that is informed when the build collation was seconded by a
      parachain validator.
      
      * Add test
      
      * Make sure we only send `Seconded` statements
      
      * Make sure we only receive valid statements
      
      * Review feedback
      ac5ef00e
  3. 04 Feb, 2021 1 commit
    • Sergey Pepyakin's avatar
      Diagnostics quality of life improvements (#2375) · 4f9e845d
      Sergey Pepyakin authored
      * Implement `Debug` manually for CandidateHash
      
      This will make candidate hashes printed consistently without the
      `CandidateHash(` and `)` decorations.
      
      * Do not print CompressedPov's guts
      
      It can be overwhelming. Better just use the size.
      
      * Log when candidate is generated
      
      * Print para_id and candidate_hash upon receiving a collation
      4f9e845d
  4. 25 Jan, 2021 1 commit
  5. 18 Jan, 2021 1 commit
    • Sergey Pepyakin's avatar
      Remove TransientValidationData (#2272) · 44f01a39
      Sergey Pepyakin authored
      * collation-generation: use persisted validation data
      
      * node: remote FullValidationData API
      
      * runtime: remove FullValidationData API
      
      * backing tests: use persisted validation data
      
      * FullCandidateReceipt: use persisted validation data
      
      This is not a big change since this type is not used anywhere
      
      * Remove ValidationData and TransientValidationData
      
      Also update the guide
      44f01a39
  6. 05 Jan, 2021 1 commit
  7. 17 Dec, 2020 1 commit
    • Andronik Ordian's avatar
      refactor View to include finalized_number (#2128) · c429e15c
      Andronik Ordian authored
      * refactor View to include finalized_number
      
      * guide: update the NetworkBridge on BlockFinalized
      
      * av-store: fix the tests
      
      * actually fix tests
      
      * grumbles
      
      * ignore macro doctest
      
      * use Hash::repeat_bytes more consistently
      
      * broadcast empty leaves updates as well
      
      * fix issuing view updates on empty leaves updates
      c429e15c
  8. 16 Dec, 2020 1 commit
    • Sergey Pepyakin's avatar
      Add logging to collation-generation (#2121) · 782cf447
      Sergey Pepyakin authored
      Right now if the collation is not happening one will have to sprinkle
      log statements and then recompile the code. It's doubly annoying if that
      happens when working with Cumulus: that means one has to resort to
      .cargo/config's `paths` or `diener`, which both are not ideal.
      
      This just adds some verbose logging to save the investigators some time
      when looking why the collations are not happening
      782cf447
  9. 02 Dec, 2020 1 commit
  10. 27 Nov, 2020 1 commit
    • asynchronous rob's avatar
      Move erasure root out of candidate commitments and into descriptor (#2010) · fff46359
      asynchronous rob authored
      
      
      * guide: move erasure-root to candidate descriptor
      
      * primitives: move erasure root to descriptor
      
      * guide: unify candidate commitments and validation outputs
      
      * primitives: unify validation outputs and candidate commitments
      
      * parachains-runtime: fix fallout
      
      * runtimes: fix fallout
      
      * collation generation: fix fallout
      
      * fix stray reference in primitives
      
      * fix fallout in node-primitives
      
      * fix remaining fallout in collation generation
      
      * fix fallout in candidate validation
      
      * fix fallout in runtime API subsystem
      
      * fix fallout in subsystem messages
      
      * fix fallout in candidate backing
      
      * fix fallout in availability distribution
      
      * don't clone
      
      * clone
      
      Co-authored-by: Sergey Pepyakin's avatarSergei Shulepov <sergei@parity.io>
      fff46359
  11. 23 Nov, 2020 1 commit
  12. 20 Nov, 2020 2 commits
    • Peter Goodspeed-Niklaus's avatar
      Add Prometheus timers to the subsystems (#1923) · e655654e
      Peter Goodspeed-Niklaus authored
      * reexport prometheus-super for ease of use of other subsystems
      
      * add some prometheus timers for collation generation subsystem
      
      * add timing metrics to av-store
      
      * add metrics to candidate backing
      
      * add timing metric to bitfield signing
      
      * add timing metrics to candidate selection
      
      * add timing metrics to candidate-validation
      
      * add timing metrics to chain-api
      
      * add timing metrics to provisioner
      
      * add timing metrics to runtime-api
      
      * add timing metrics to availability-distribution
      
      * add timing metrics to bitfield-distribution
      
      * add timing metrics to collator protocol: collator side
      
      * add timing metrics to collator protocol: validator side
      
      * fix candidate validation test failures
      
      * add timing metrics to pov distribution
      
      * add timing metrics to statement-distribution
      
      * use substrate_prometheus_endpoint prometheus reexport instead of prometheus_super
      
      * don't include JOB_DELAY in bitfield-signing metrics
      
      * give adder-collator ability to easily export its genesis-state and validation code
      
      * wip: adder-collator pushbutton script
      
      * don't attempt to register the adder-collator automatically
      
      Instead, get these values with
      
      ```sh
      target/release/adder-collator export-genesis-state
      target/release/adder-collator export-genesis-wasm
      ```
      
      And then register the parachain on https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A9944#/explorer
      
      To collect prometheus data, after running the script, create `prometheus.yml` per the instructions
      at https://www.notion.so/paritytechnologies/Setting-up-Prometheus-locally-835cb3a9df7541a781c381006252b5ff
      
      
      and then run:
      
      ```sh
      docker run -v `pwd`/prometheus.yml:/etc/prometheus/prometheus.yml:z --network host prom/prometheus
      ```
      
      Demonstrates that data makes it across to prometheus, though it is likely to be useful in the future
      to tweak the buckets.
      
      * Update parachain/test-parachains/adder/collator/src/cli.rs
      
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      
      * use the grandpa-pause parameter
      
      * skip metrics in tracing instrumentation
      
      * remove unnecessary grandpa_pause cli param
      
      Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
      e655654e
    • Peter Goodspeed-Niklaus's avatar
      Add tracing support to node (#1940) · f7ea3d07
      Peter Goodspeed-Niklaus authored
      * drop in tracing to replace log
      
      * add structured logging to trace messages
      
      * add structured logging to debug messages
      
      * add structured logging to info messages
      
      * add structured logging to warn messages
      
      * add structured logging to error messages
      
      * normalize spacing and Display vs Debug
      
      * add instrumentation to the various 'fn run'
      
      * use explicit tracing module throughout
      
      * fix availability distribution test
      
      * don't double-print errors
      
      * remove further redundancy from logs
      
      * fix test errors
      
      * fix more test errors
      
      * remove unused kv_log_macro
      
      * fix unused variable
      
      * add tracing spans to collation generation
      
      * add tracing spans to av-store
      
      * add tracing spans to backing
      
      * add tracing spans to bitfield-signing
      
      * add tracing spans to candidate-selection
      
      * add tracing spans to candidate-validation
      
      * add tracing spans to chain-api
      
      * add tracing spans to provisioner
      
      * add tracing spans to runtime-api
      
      * add tracing spans to availability-distribution
      
      * add tracing spans to bitfield-distribution
      
      * add tracing spans to network-bridge
      
      * add tracing spans to collator-protocol
      
      * add tracing spans to pov-distribution
      
      * add tracing spans to statement-distribution
      
      * add tracing spans to overseer
      
      * cleanup
      f7ea3d07
  13. 06 Nov, 2020 2 commits
    • Sergey Pepyakin's avatar
      Implement HRMP (#1900) · 73553661
      Sergey Pepyakin authored
      * HRMP: Update the impl guide
      
      * HRMP: Incorporate the channel notifications into the guide
      
      * HRMP: Renaming in the impl guide
      
      * HRMP: Constrain the maximum number of HRMP messages per candidate
      
      This commit addresses the HRMP part of https://github.com/paritytech/polkadot/issues/1869
      
      
      
      * XCM: Introduce HRMP related message types
      
      * HRMP: Data structures and plumbing
      
      * HRMP: Configuration
      
      * HRMP: Data layout
      
      * HRMP: Acceptance & Enactment
      
      * HRMP: Test base logic
      
      * Update adder collator
      
      * HRMP: Runtime API for accessing inbound messages
      
      Also, removing some redundant fully-qualified names.
      
      * HRMP: Add diagnostic logging in acceptance criteria
      
      * HRMP: Additional tests
      
      * Self-review fixes
      
      * save test refactorings for the next time
      
      * Missed a return statement.
      
      * a formatting blip
      
      * Add missing logic for appending HRMP digests
      
      * Remove the channel contents vectors which became empty
      
      * Tighten HRMP channel digests invariants.
      
      * Apply suggestions from code review
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
      
      * Remove a note about sorting for channel id
      
      * Add missing rustdocs to the configuration
      
      * Clarify and update the invariant for HrmpChannelDigests
      
      * Make the onboarding invariant less sloppy
      
      Namely, introduce `Paras::is_valid_para` (in fact, it already is present
      in the implementation) and hook up the invariant to that.
      
      Note that this says "within a session" because I don't want to make it
      super strict on the session boundary. The logic on the session boundary
      should be extremely careful.
      
      * Make `CandidateCheckContext` use T::BlockNumber for hrmp_watermark
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
      73553661
    • Bastian Köcher's avatar
      Distribute a PoV after seconding it (#1924) · 3a3aace3
      Bastian Köcher authored
      We need to distribute the PoV after we have seconded it. Other nodes
      that will receive our `Secondded` statement and want to validate the
      candidate another time will request this PoV from us.
      3a3aace3
  14. 05 Nov, 2020 1 commit
    • Andronik Ordian's avatar
      more resilient subsystems (#1908) · 9758a180
      Andronik Ordian authored
      * backing: extract log target
      
      * bitfield-signing: extract log target
      
      * utils: fix a typo
      
      * provisioner: extract log target
      
      * candidate selection: remove unused error variant
      
      * bitfield-distribution: change the return type of run
      
      * pov-distribution: extract log target
      
      * collator-protocol: simplify runtime request
      
      * collation-generation: do not exit early on error
      
      * collation-generation: do not exit on double init
      
      * collator-protocol: do not exit on errors and rename LOG_TARGET
      
      * collator-protocol: a workaround for ununused imports warning
      
      * Update node/network/bitfield-distribution/src/lib.rs
      
      * collation-generation: elevate warn! to error!
      
      * collator-protocol: fix imports
      
      * post merge fix
      
      * fix compilation
      9758a180
  15. 28 Oct, 2020 3 commits
  16. 07 Oct, 2020 1 commit
  17. 06 Oct, 2020 2 commits
    • Bastian Köcher's avatar
      Pass relay parent as argument when collating (#1789) · 923c97a4
      Bastian Köcher authored
      This pr changes the collation function to also pass the current relay
      parent the parachain block should be build on.
      923c97a4
    • Bastian Köcher's avatar
      Make collation an optional return (#1787) · ce0fa3ad
      Bastian Köcher authored
      This pr changes the collator interface function to return an optional
      collation instead of a collation. This is required as the parachain
      itself can fail to generate a valid collation for various reason. Now if
      the collation fails it will return `None`.
      
      Besides that the pr adds some `RuntimeDebug` derive for `ValidationData`
      and removes some whitespaces.
      ce0fa3ad
  18. 29 Sep, 2020 1 commit
  19. 25 Sep, 2020 1 commit
    • Sergey Pepyakin's avatar
      collation-generation: guide and tidying (#1753) · 507b9cbc
      Sergey Pepyakin authored
      * Guide: Change for {Global,Local}ValidationData to ValidationData
      
      Seems like that was missed during the transition from notions of global and local validation data to persistent and transient
      
      * collation-generation: a small drive-by fix of line length
      
      * collation-generation: extract an error module
      
      this allows us to avoid piling Error and Result types from different contexts, specifically std vs. module-local
      
      * collation-generation: extract LOG_TARGET from log statements
      507b9cbc
  20. 18 Aug, 2020 2 commits
    • asynchronous rob's avatar
      Implement validation data refactor (#1585) · 8c881e45
      asynchronous rob authored
      
      
      * update primitives
      
      * correct parent_head field
      
      * make hrmp field pub
      
      * refactor validation data: runtime
      
      * refactor validation data: messages
      
      * add arguments to full_validation_data runtime API
      
      * port runtime API
      
      * mostly port over candidate validation
      
      * remove some parameters from ValidationParams
      
      * guide: update candidate validation
      
      * update candidate outputs
      
      * update ValidationOutputs in primitives
      
      * port over candidate validation
      
      * add a new test for no-transient behavior
      
      * update util runtime API wrappers
      
      * candidate backing
      
      * fix missing imports
      
      * change some fields of validation data around
      
      * runtime API impl
      
      * update candidate validation
      
      * fix backing tests
      
      * grumbles from review
      
      * fix av-store tests
      
      * fix some more crates
      
      * fix provisioner tests
      
      * fix availability distribution tests
      
      * port collation-generation to new validation data
      
      * fix overseer tests
      
      * Update roadmap/implementers-guide/src/node/utility/candidate-validation.md
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <coriolinus@users.noreply.github.com>
      8c881e45
    • Andronik Ordian's avatar
      initial prometheus metrics (#1536) · 804958ab
      Andronik Ordian authored
      * service-new: cosmetic changes
      
      * overseer: draft of prometheus metrics
      
      * metrics: update active_leaves metrics
      
      * metrics: extract into functions
      
      * metrics: resolve XXX
      
      * metrics: it's ugly, but it works
      
      * Bump Substrate
      
      * metrics: move a bunch of code around
      
      * Bumb substrate again
      
      * metrics: fix a warning
      
      * fix a warning in runtime
      
      * metrics: statements signed
      
      * metrics: statements impl RegisterMetrics
      
      * metrics: refactor Metrics trait
      
      * metrics: add Metrics assoc type to JobTrait
      
      * metrics: move Metrics trait to util
      
      * metrics: fix overseer
      
      * metrics: fix backing
      
      * metrics: fix candidate validation
      
      * metrics: derive Default
      
      * metrics: docs
      
      * metrics: add stubs for other subsystems
      
      * metrics: add more stubs and fix compilation
      
      * metrics: fix doctest
      
      * metrics: move to subsystem
      
      * metrics: fix candidate validation
      
      * metrics: bitfield signing
      
      * metrics: av store
      
      * metrics: chain API
      
      * metrics: runtime API
      
      * metrics: stub for avad
      
      * metrics: candidates seconded
      
      * metrics: ok I gave up
      
      * metrics: provisioner
      
      * metrics: remove a clone by requiring Metrics: Sync
      
      * metrics: YAGNI
      
      * metrics: remove another TODO
      
      * metrics: for later
      
      * metrics: add parachain_ prefix
      
      * metrics: s/signed_statement/signed_statements
      
      * utils: add a comment for job metrics
      
      * metrics: address review comments
      
      * metrics: oops
      
      * metrics: make sure to save files before commit 😅
      
      
      
      * use _total suffix for requests metrics
      
      Co-authored-by: default avatarMax Inden <mail@max-inden.de>
      
      * metrics: add tests for overseer
      
      * update Cargo.lock
      
      * overseer: add a test for CollationGeneration
      
      * collation-generation: impl metrics
      
      * collation-generation: use kebab-case for name
      
      * collation-generation: add a constructor
      
      Co-authored-by: default avatarGav Wood <gavin@parity.io>
      Co-authored-by: default avatarAshley Ruglys <ashley.ruglys@gmail.com>
      Co-authored-by: default avatarMax Inden <mail@max-inden.de>
      804958ab
  21. 17 Aug, 2020 1 commit
    • Peter Goodspeed-Niklaus's avatar
      implement collation generation subsystem (#1557) · e750a825
      Peter Goodspeed-Niklaus authored
      * start sketching out a collation generation subsystem
      
      * invent a basic strategy for double initialization
      
      * clean up warnings
      
      * impl util requests from runtime assuming a context instead of a FromJob sender
      
      * implement collation generation algorithm from guide
      
      * update AllMessages in tests
      
      * fix trivial review comments
      
      * remove another redundant declaration from merge
      
      * filter availability cores by para_id
      
      * handle new activations each in their own async task
      
      * update guide according to the actual current implementation
      
      * add initialization to guide
      
      * add general-purpose subsystem_test_harness helper
      
      * write first handle_new_activations test
      
      * add test that handle_new_activations filters local_validation_data requests
      
      * add (failing) test of collation distribution message sending
      
      * rustfmt
      
      * broken: work on fixing sender test
      
      Unfortunately, for reasons that are not yet clear, despite the public key
      and checked data being identical, the signer is not producing an identical
      signature. This commit produces this output (among more):
      
      signing with  Public(c4733ab0bbe3ba4c096685d1737a7f498cdbdd167a767d04a21dc7df12b8c858 (5GWHUNm5...))
      checking with Public(c4733ab0bbe3ba4c096685d1737a7f498cdbdd167a767d04a21dc7df12b8c858 (5GWHUNm5...))
      signed payload:  [4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 10, 0, 0, 0, c7, e5, c0, 64, 7a, db, fe, 44, 81, e5, 51, 11, 79, 9f, a5, 63, 93, 94, 3c, c4, 36, c6, 30, 36, c2, c5, 44, a2, 1b, db, b7, 82, 3, 17, a, 2e, 75, 97, b7, b7, e3, d8, 4c, 5, 39, 1d, 13, 9a, 62, b1, 57, e7, 87, 86, d8, c0, 82, f2, 9d, cf, 4c, 11, 13, 14]
      checked payload: [4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 10, 0, 0, 0, c7, e5, c0, 64, 7a, db, fe, 44, 81, e5, 51, 11, 79, 9f, a5, 63, 93, 94, 3c, c4, 36, c6, 30, 36, c2, c5, 44, a2, 1b, db, b7, 82, 3, 17, a, 2e, 75, 97, b7, b7, e3, d8, 4c, 5, 39, 1d, 13, 9a, 62, b1, 57, e7, 87, 86, d8, c0, 82, f2, 9d, cf, 4c, 11, 13, 14]
      
      * fix broken test
      
      * collation function returns commitments hash
      
      It doesn't look like we use the actual commitments data anywhere, and
      it's not obvious if there are any fields of `CandidateCommitments`
      not available to the collator, so this commit just assigns them the
      entire responsibility of generating the hash.
      
      * add missing overseer impls
      
      * calculating erasure coding is polkadot's responsibility, not cumulus
      
      * concurrentize per-relay_parent requests
      e750a825