Skip to content
  1. Nov 23, 2020
  3. Nov 22, 2020
    • Bastian Köcher's avatar
      Improve collator side of the collator-protocol (#1955) · b9a9e836
      Bastian Köcher authored 
      * Improve collator side of the collator-protocol

This pr improves the collator-protocol implementation of the collator
side. Besides cleaning up code and rewriting it, the following changed:

- Before on `PeerViewChange` we send an advertisment to every peer, now
this only happens for validators.
- It also adds a check that we send an advertisment message only once
for a connected peer.
- If the same validator was part of the current and next group, we
requested to be connected to this validator two times. This is also
fixed now.
- Instead of having only one connection request, we now are being able
to store multiple of them. This is required as we can have multiple
active leafs at any point of time.

* Switch to common `ConnectionRequests`

* Update node/network/collator-protocol/src/collator_side.rs
      b9a9e836
  5. Nov 20, 2020
    • Sergey Pepyakin's avatar
      sane messaging defaults (#1994) · 56c07e8c
      Sergey Pepyakin authored
      56c07e8c
    • ordian's avatar
      cleanup validator discovery (#1992) · 97f5bd90
      ordian authored 
      * use snake_case for log targets

* remove unused continue

* validator_discovery: when disconnecting, use all addresses

* validator_discovery: simplify request revokation

* fix a typo
      97f5bd90
    • Peter Goodspeed-Niklaus's avatar
      Add Prometheus timers to the subsystems (#1923) · 0a5bc825
      Peter Goodspeed-Niklaus authored 
      

* reexport prometheus-super for ease of use of other subsystems

* add some prometheus timers for collation generation subsystem

* add timing metrics to av-store

* add metrics to candidate backing

* add timing metric to bitfield signing

* add timing metrics to candidate selection

* add timing metrics to candidate-validation

* add timing metrics to chain-api

* add timing metrics to provisioner

* add timing metrics to runtime-api

* add timing metrics to availability-distribution

* add timing metrics to bitfield-distribution

* add timing metrics to collator protocol: collator side

* add timing metrics to collator protocol: validator side

* fix candidate validation test failures

* add timing metrics to pov distribution

* add timing metrics to statement-distribution

* use substrate_prometheus_endpoint prometheus reexport instead of prometheus_super

* don't include JOB_DELAY in bitfield-signing metrics

* give adder-collator ability to easily export its genesis-state and validation code

* wip: adder-collator pushbutton script

* don't attempt to register the adder-collator automatically

Instead, get these values with

```sh
target/release/adder-collator export-genesis-state
target/release/adder-collator export-genesis-wasm
```

And then register the parachain on https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A9944#/explorer

To collect prometheus data, after running the script, create `prometheus.yml` per the instructions
at https://www.notion.so/paritytechnologies/Setting-up-Prometheus-locally-835cb3a9df7541a781c381006252b5ff
and then run:

```sh
docker run -v `pwd`/prometheus.yml:/etc/prometheus/prometheus.yml:z --network host prom/prometheus
```

Demonstrates that data makes it across to prometheus, though it is likely to be useful in the future
to tweak the buckets.

* Update parachain/test-parachains/adder/collator/src/cli.rs

Co-authored-by: default avatarAndronik Ordian <[email protected]>

* use the grandpa-pause parameter

* skip metrics in tracing instrumentation

* remove unnecessary grandpa_pause cli param

Co-authored-by: default avatarAndronik Ordian <[email protected]>
      0a5bc825
    • Peter Goodspeed-Niklaus's avatar
      Add tracing support to node (#1940) · e4998997
      Peter Goodspeed-Niklaus authored 
      * drop in tracing to replace log

* add structured logging to trace messages

* add structured logging to debug messages

* add structured logging to info messages

* add structured logging to warn messages

* add structured logging to error messages

* normalize spacing and Display vs Debug

* add instrumentation to the various 'fn run'

* use explicit tracing module throughout

* fix availability distribution test

* don't double-print errors

* remove further redundancy from logs

* fix test errors

* fix more test errors

* remove unused kv_log_macro

* fix unused variable

* add tracing spans to collation generation

* add tracing spans to av-store

* add tracing spans to backing

* add tracing spans to bitfield-signing

* add tracing spans to candidate-selection

* add tracing spans to candidate-validation

* add tracing spans to chain-api

* add tracing spans to provisioner

* add tracing spans to runtime-api

* add tracing spans to availability-distribution

* add tracing spans to bitfield-distribution

* add tracing spans to network-bridge

* add tracing spans to collator-protocol

* add tracing spans to pov-distribution

* add tracing spans to statement-distribution

* add tracing spans to overseer

* cleanup
      e4998997
  7. Nov 19, 2020
  9. Nov 18, 2020
  11. Nov 17, 2020
    • ordian's avatar
      update most of the dependencies (#1946) · 0a8a607a
      ordian authored 
      * update tiny-keccak to 0.2

* update deps except bitvec and shared_memory

* fix some warning after futures upgrade

* remove useless package rename caused by bug in cargo-upgrade

* revert parity-util-mem *

* remove unused import

* cargo update

* remove all renames on parity-scale-codec

* remove the leftovers

* remove unused dep
      0a8a607a
  13. Nov 11, 2020
    • Jon Häggblad's avatar
      Companion PR for Substrate #7186 (#1816) · 4b082997
      Jon Häggblad authored 
      * babe: backoff authoring blocks when finality lags

* service: use default constructor for backoff authoring strat

* slots: update to use renamed backoff strategy name

* "Update Substrate"

Co-authored-by: parity-processbot <>
      4b082997
  15. Nov 10, 2020
  17. Nov 09, 2020
  19. Nov 06, 2020
    • Sergey Pepyakin's avatar
      Implement HRMP (#1900) · c96f8cfc
      Sergey Pepyakin authored 
      

* HRMP: Update the impl guide

* HRMP: Incorporate the channel notifications into the guide

* HRMP: Renaming in the impl guide

* HRMP: Constrain the maximum number of HRMP messages per candidate

This commit addresses the HRMP part of https://github.com/paritytech/polkadot/issues/1869

* XCM: Introduce HRMP related message types

* HRMP: Data structures and plumbing

* HRMP: Configuration

* HRMP: Data layout

* HRMP: Acceptance & Enactment

* HRMP: Test base logic

* Update adder collator

* HRMP: Runtime API for accessing inbound messages

Also, removing some redundant fully-qualified names.

* HRMP: Add diagnostic logging in acceptance criteria

* HRMP: Additional tests

* Self-review fixes

* save test refactorings for the next time

* Missed a return statement.

* a formatting blip

* Add missing logic for appending HRMP digests

* Remove the channel contents vectors which became empty

* Tighten HRMP channel digests invariants.

* Apply suggestions from code review

Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>

* Remove a note about sorting for channel id

* Add missing rustdocs to the configuration

* Clarify and update the invariant for HrmpChannelDigests

* Make the onboarding invariant less sloppy

Namely, introduce `Paras::is_valid_para` (in fact, it already is present
in the implementation) and hook up the invariant to that.

Note that this says "within a session" because I don't want to make it
super strict on the session boundary. The logic on the session boundary
should be extremely careful.

* Make `CandidateCheckContext` use T::BlockNumber for hrmp_watermark

Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      c96f8cfc
    • Bastian Köcher's avatar
      Distribute a PoV after seconding it (#1924) · 8a2911b8
      Bastian Köcher authored 
      We need to distribute the PoV after we have seconded it. Other nodes
that will receive our `Secondded` statement and want to validate the
candidate another time will request this PoV from us.
      8a2911b8
  21. Nov 05, 2020
    • Bastian Köcher's avatar
      Adds a comment as requested (#1922) · 2cdc1c04
      Bastian Köcher authored 
      

* Adds a comment as requested

* Update node/core/bitfield-signing/src/lib.rs

Co-authored-by: default avatarFedor Sakharov <[email protected]>

Co-authored-by: default avatarFedor Sakharov <[email protected]>
      2cdc1c04
    • Bastian Köcher's avatar
      Make `CandidateHash` a real type (#1916) · 640264f3
      Bastian Köcher authored 
      

* Make `CandidateHash` a real type

This pr adds a new type `CandidateHash` that is used instead of the
opaque `Hash` type. This helps to ensure on the type system level that
we are passing the correct types.

This pr also fixes wrong usage of `relay_parent` as `candidate_hash`
when communicating with the av storage.

* Update core-primitives/src/lib.rs

Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>

* Wrap the lines

Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      640264f3
    • ordian's avatar
      more resilient subsystems (#1908) · 2cde7732
      ordian authored 
      * backing: extract log target

* bitfield-signing: extract log target

* utils: fix a typo

* provisioner: extract log target

* candidate selection: remove unused error variant

* bitfield-distribution: change the return type of run

* pov-distribution: extract log target

* collator-protocol: simplify runtime request

* collation-generation: do not exit early on error

* collation-generation: do not exit on double init

* collator-protocol: do not exit on errors and rename LOG_TARGET

* collator-protocol: a workaround for ununused imports warning

* Update node/network/bitfield-distribution/src/lib.rs

* collation-generation: elevate warn! to error!

* collator-protocol: fix imports

* post merge fix

* fix compilation
      2cde7732
    • Bastian Köcher's avatar
      Simplify the bitfield signing job (#1920) · c418758e
      Bastian Köcher authored 
      Besides that the pr also adds a simple test.
      c418758e
    • Bastian Köcher's avatar
      Do not validate a candidate in candidate selection (#1912) · f2b286d6
      Bastian Köcher authored 
      * Do not validate a candidate in candidate selection

The candidate selection subsystem should not validate a candidate, as
this is done by the backing subsystem on a `Second` request. Otherwise
we validate one candidate twice.

* Update candidate-selection.md
      f2b286d6
  23. Nov 04, 2020
  25. Nov 03, 2020
  27. Nov 02, 2020
  29. Nov 01, 2020
  31. Oct 31, 2020
    • asynchronous rob's avatar
      fix service build: enable notifications protocol only under real overseer (#1894) · ad9e1fdf
      asynchronous rob authored 
      

Co-authored-by: default avatarRobert Habermeier <[email protected]>
      ad9e1fdf
    • Bastian Köcher's avatar
      Adds test parachain adder collator (#1864) · f82de7b9
      Bastian Köcher authored 
      

* start working on building the real overseer

Unfortunately, this fails to compile right now due to an upstream
failure to compile which is probably brought on by a recent upgrade
to rustc v1.47.

* fill in AllSubsystems internal constructors

* replace fn make_metrics with Metrics::attempt_to_register

* update to account for #1740

* remove Metrics::register, rename Metrics::attempt_to_register

* add 'static bounds to real_overseer type params

* pass authority_discovery and network_service to real_overseer

It's not straightforwardly obvious that this is the best way to handle
the case when there is no authority discovery service, but it seems
to be the best option available at the moment.

* select a proper database configuration for the availability store db

* use subdirectory for av-store database path

* apply Basti's patch which avoids needing to parameterize everything on Block

* simplify path extraction

* get all tests to compile

* Fix Prometheus double-registry error

for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476:

```rust
Some(registry) => Self::try_register(registry).map_err(|err| {
	eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err);
	err
}),
```

That pointed out where the registration was failing, which led to
this fix. The test still doesn't pass, but it now fails in a new
and different way!

* authorities must have authority discovery, but not necessarily overseer handlers

* fix broken SpawnedSubsystem impls

detailed logging determined that using the `Box::new` style of
future generation, the `self.run` method was never being called,
leading to dropped receivers / closed senders for those subsystems,
causing the overseer to shut down immediately.

This is not the final fix needed to get things working properly,
but it's a good start.

* use prometheus properly

Prometheus lets us register simple counters, which aren't very
interesting. It also allows us to register CounterVecs, which are.
With a CounterVec, you can provide a set of labels, which can
later be used to filter the counts.

We were using them wrong, though. This pattern was repeated in a
variety of places in the code:

```rust
// panics with an cardinality mismatch
let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?;
my_counter.with_label_values(&["succeeded"]).inc()
```

The problem is that the labels provided in the constructor are not
the set of legal values which can be annotated, but a set of individual
label names which can have individual, arbitrary values.

This commit fixes that.

* get av-store subsystem to actually run properly and not die on first signal

* typo fix: incomming -> incoming

* don't disable authority discovery in test nodes

* Fix rococo-v1 missing session keys

* Update node/core/av-store/Cargo.toml

* try dummying out av-store on non-full-nodes

* overseer and subsystems are required only for full nodes

* Reduce the amount of warnings on browser target

* Fix two more warnings

* InclusionInherent should actually have an Inherent module on rococo

* Ancestry: don't return genesis' parent hash

* Update Cargo.lock

* fix broken test

* update test script: specify chainspec as script argument

* Apply suggestions from code review

Co-authored-by: default avatarBastian Köcher <[email protected]>

* Update node/service/src/lib.rs

Co-authored-by: default avatarBastian Köcher <[email protected]>

* node/service/src/lib: Return error via ? operator

* post-merge blues

* add is_collator flag

* prevent occasional av-store test panic

* simplify fix; expand application

* run authority_discovery in Role::Discover when collating

* distinguish between proposer closed channel errors

* add IsCollator enum, remove is_collator CLI flag

* improve formatting

* remove nop loop

* Fix some stuff

* Adds test parachain adder collator

* Add sudo to Rococo, change session length to 30 seconds and some renaming

* Update to the latest changes on master

* Some fixes

* Fix compilation

* Update parachain/test-parachains/adder/collator/src/lib.rs

Co-authored-by: default avatarSergei Shulepov <[email protected]>

* Review comments

* Downgrade transaction version

* Fixes

* MOARE

* Register notification protocols

* utils: remove unused error

* av-store: more resilient to some errors

* address review nits

* address more review nits

Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
Co-authored-by: default avatarAndronik Ordian <[email protected]>
Co-authored-by: default avatarFedor Sakharov <[email protected]>
Co-authored-by: default avatarRobert Habermeier <[email protected]>
Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
Co-authored-by: default avatarMax Inden <[email protected]>
Co-authored-by: default avatarSergey Shulepov <[email protected]>
Co-authored-by: default avatarSergei Shulepov <[email protected]>
      f82de7b9
    • Fedor Sakharov's avatar
      A real overseer feature (#1892) · 16f8da1d
      Fedor Sakharov authored 
      

* A real overseer feature

* Fix build without feature

* Update node/service/src/lib.rs

Co-authored-by: default avatarBastian Köcher <[email protected]>

Co-authored-by: default avatarRobert Habermeier <[email protected]>
Co-authored-by: default avatarBastian Köcher <[email protected]>
      16f8da1d
  33. Oct 29, 2020
    • Bastian Köcher's avatar
      Make sure validator discovery works with a delayed peer to validator mapping (#1886) · a4b92cd3
      Bastian Köcher authored 
      

* Make sure validator discovery works with a delayed peer to validator mapping

Currently the implementation checks on connect of a peer if this peer is
a validator by asking the authority discovery. It can now happen that
the authority discovery is not yet aware that a given peer is an
authority. This can for example happen on start up of the node.

This pr changes the behavior, to make it possible to later associate a
peer to a validator id. Instead of just storing the connected
validators, we now store all connected peers with a vector of associated
validator ids. When we get a request to connect to a given given set of
validators, we start by checking the connected peers. If we didn't find
a validator id in the connected peers, we ask the authority discovery
for the peerid of a given authority id. When the returned peerid is part
of our connected peers set, we cache and return the authority id.

* Update node/network/bridge/Cargo.toml

Co-authored-by: default avatarPierre Krieger <[email protected]>

* Update node/network/bridge/src/validator_discovery.rs

Co-authored-by: default avatarPierre Krieger <[email protected]>

* Update `Cargo.lock`

Co-authored-by: default avatarPierre Krieger <[email protected]>
      a4b92cd3