1. Nov 03, 2020
  2. Nov 02, 2020
  3. Oct 31, 2020
    • Bastian Köcher's avatar
      Adds test parachain adder collator (#1864) · 8cadebb7
      Bastian Köcher authored
      
      
      * start working on building the real overseer
      
      Unfortunately, this fails to compile right now due to an upstream
      failure to compile which is probably brought on by a recent upgrade
      to rustc v1.47.
      
      * fill in AllSubsystems internal constructors
      
      * replace fn make_metrics with Metrics::attempt_to_register
      
      * update to account for #1740
      
      * remove Metrics::register, rename Metrics::attempt_to_register
      
      * add 'static bounds to real_overseer type params
      
      * pass authority_discovery and network_service to real_overseer
      
      It's not straightforwardly obvious that this is the best way to handle
      the case when there is no authority discovery service, but it seems
      to be the best option available at the moment.
      
      * select a proper database configuration for the availability store db
      
      * use subdirectory for av-store database path
      
      * apply Basti's patch which avoids needing to parameterize everything on Block
      
      * simplify path extraction
      
      * get all tests to compile
      
      * Fix Prometheus double-registry error
      
      for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476:
      
      ```rust
      Some(registry) => Self::try_register(registry).map_err(|err| {
      	eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err);
      	err
      }),
      ```
      
      That pointed out where the registration was failing, which led to
      this fix. The test still doesn't pass, but it now fails in a new
      and different way!
      
      * authorities must have authority discovery, but not necessarily overseer handlers
      
      * fix broken SpawnedSubsystem impls
      
      detailed logging determined that using the `Box::new` style of
      future generation, the `self.run` method was never being called,
      leading to dropped receivers / closed senders for those subsystems,
      causing the overseer to shut down immediately.
      
      This is not the final fix needed to get things working properly,
      but it's a good start.
      
      * use prometheus properly
      
      Prometheus lets us register simple counters, which aren't very
      interesting. It also allows us to register CounterVecs, which are.
      With a CounterVec, you can provide a set of labels, which can
      later be used to filter the counts.
      
      We were using them wrong, though. This pattern was repeated in a
      variety of places in the code:
      
      ```rust
      // panics with an cardinality mismatch
      let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?;
      my_counter.with_label_values(&["succeeded"]).inc()
      ```
      
      The problem is that the labels provided in the constructor are not
      the set of legal values which can be annotated, but a set of individual
      label names which can have individual, arbitrary values.
      
      This commit fixes that.
      
      * get av-store subsystem to actually run properly and not die on first signal
      
      * typo fix: incomming -> incoming
      
      * don't disable authority discovery in test nodes
      
      * Fix rococo-v1 missing session keys
      
      * Update node/core/av-store/Cargo.toml
      
      * try dummying out av-store on non-full-nodes
      
      * overseer and subsystems are required only for full nodes
      
      * Reduce the amount of warnings on browser target
      
      * Fix two more warnings
      
      * InclusionInherent should actually have an Inherent module on rococo
      
      * Ancestry: don't return genesis' parent hash
      
      * Update Cargo.lock
      
      * fix broken test
      
      * update test script: specify chainspec as script argument
      
      * Apply suggestions from code review
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      
      * Update node/service/src/lib.rs
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      
      * node/service/src/lib: Return error via ? operator
      
      * post-merge blues
      
      * add is_collator flag
      
      * prevent occasional av-store test panic
      
      * simplify fix; expand application
      
      * run authority_discovery in Role::Discover when collating
      
      * distinguish between proposer closed channel errors
      
      * add IsCollator enum, remove is_collator CLI flag
      
      * improve formatting
      
      * remove nop loop
      
      * Fix some stuff
      
      * Adds test parachain adder collator
      
      * Add sudo to Rococo, change session length to 30 seconds and some renaming
      
      * Update to the latest changes on master
      
      * Some fixes
      
      * Fix compilation
      
      * Update parachain/test-parachains/adder/collator/src/lib.rs
      
      Co-authored-by: default avatarSergei Shulepov <[email protected]>
      
      * Review comments
      
      * Downgrade transaction version
      
      * Fixes
      
      * MOARE
      
      * Register notification protocols
      
      * utils: remove unused error
      
      * av-store: more resilient to some errors
      
      * address review nits
      
      * address more review nits
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      Co-authored-by: default avatarAndronik Ordian <[email protected]>
      Co-authored-by: default avatarFedor Sakharov <[email protected]>
      Co-authored-by: default avatarRobert Habermeier <[email protected]>
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      Co-authored-by: default avatarMax Inden <[email protected]>
      Co-authored-by: default avatarSergey Shulepov <[email protected]>
      Co-authored-by: default avatarSergei Shulepov <[email protected]>
      8cadebb7
  4. Oct 30, 2020
  5. Oct 28, 2020
    • Sergey Pepyakin's avatar
      Downward Message Processing implementation (#1859) · ed759c75
      Sergey Pepyakin authored
      * DMP: data structures and plumbing
      
      * DMP: Implement DMP logic in the router module
      
      DMP: Integrate DMP parts into the inclusion module
      
      * DMP: Introduce the max size limit for the size of a downward message
      
      * DMP: Runtime API for accessing inbound messages
      
      * OCD
      
      Small clean ups
      
      * DMP: fix the naming of the error
      
      * DMP: add caution about a non-existent recipient
      ed759c75
    • Peter Goodspeed-Niklaus's avatar
      start working on building the real overseer (#1795) · 798f781f
      Peter Goodspeed-Niklaus authored
      
      
      * start working on building the real overseer
      
      Unfortunately, this fails to compile right now due to an upstream
      failure to compile which is probably brought on by a recent upgrade
      to rustc v1.47.
      
      * fill in AllSubsystems internal constructors
      
      * replace fn make_metrics with Metrics::attempt_to_register
      
      * update to account for #1740
      
      * remove Metrics::register, rename Metrics::attempt_to_register
      
      * add 'static bounds to real_overseer type params
      
      * pass authority_discovery and network_service to real_overseer
      
      It's not straightforwardly obvious that this is the best way to handle
      the case when there is no authority discovery service, but it seems
      to be the best option available at the moment.
      
      * select a proper database configuration for the availability store db
      
      * use subdirectory for av-store database path
      
      * apply Basti's patch which avoids needing to parameterize everything on Block
      
      * simplify path extraction
      
      * get all tests to compile
      
      * Fix Prometheus double-registry error
      
      for debugging purposes, added this to node/subsystem-util/src/lib.rs:472-476:
      
      ```rust
      Some(registry) => Self::try_register(registry).map_err(|err| {
      	eprintln!("PrometheusError calling {}::register: {:?}", std::any::type_name::<Self>(), err);
      	err
      }),
      ```
      
      That pointed out where the registration was failing, which led to
      this fix. The test still doesn't pass, but it now fails in a new
      and different way!
      
      * authorities must have authority discovery, but not necessarily overseer handlers
      
      * fix broken SpawnedSubsystem impls
      
      detailed logging determined that using the `Box::new` style of
      future generation, the `self.run` method was never being called,
      leading to dropped receivers / closed senders for those subsystems,
      causing the overseer to shut down immediately.
      
      This is not the final fix needed to get things working properly,
      but it's a good start.
      
      * use prometheus properly
      
      Prometheus lets us register simple counters, which aren't very
      interesting. It also allows us to register CounterVecs, which are.
      With a CounterVec, you can provide a set of labels, which can
      later be used to filter the counts.
      
      We were using them wrong, though. This pattern was repeated in a
      variety of places in the code:
      
      ```rust
      // panics with an cardinality mismatch
      let my_counter = register(CounterVec::new(opts, &["succeeded", "failed"])?, registry)?;
      my_counter.with_label_values(&["succeeded"]).inc()
      ```
      
      The problem is that the labels provided in the constructor are not
      the set of legal values which can be annotated, but a set of individual
      label names which can have individual, arbitrary values.
      
      This commit fixes that.
      
      * get av-store subsystem to actually run properly and not die on first signal
      
      * typo fix: incomming -> incoming
      
      * don't disable authority discovery in test nodes
      
      * Fix rococo-v1 missing session keys
      
      * Update node/core/av-store/Cargo.toml
      
      * try dummying out av-store on non-full-nodes
      
      * overseer and subsystems are required only for full nodes
      
      * Reduce the amount of warnings on browser target
      
      * Fix two more warnings
      
      * InclusionInherent should actually have an Inherent module on rococo
      
      * Ancestry: don't return genesis' parent hash
      
      * Update Cargo.lock
      
      * fix broken test
      
      * update test script: specify chainspec as script argument
      
      * Apply suggestions from code review
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      
      * Update node/service/src/lib.rs
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      
      * node/service/src/lib: Return error via ? operator
      
      * post-merge blues
      
      * add is_collator flag
      
      * prevent occasional av-store test panic
      
      * simplify fix; expand application
      
      * run authority_discovery in Role::Discover when collating
      
      * distinguish between proposer closed channel errors
      
      * add IsCollator enum, remove is_collator CLI flag
      
      * improve formatting
      
      * remove nop loop
      
      * Fix some stuff
      
      Co-authored-by: default avatarAndronik Ordian <[email protected]>
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      Co-authored-by: default avatarFedor Sakharov <[email protected]>
      Co-authored-by: default avatarRobert Habermeier <[email protected]>
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      Co-authored-by: default avatarMax Inden <[email protected]>
      798f781f
  6. Oct 24, 2020
    • Sergey Pepyakin's avatar
      Runtime API for checking validation outputs (#1842) · 4a17a2bc
      Sergey Pepyakin authored
      * annoying whitespaces
      
      * update guide
      
      Add `CheckValidationOutputs` runtime api and also change the
      candidate-validation stuff
      
      * promote ValidationOutputs to global primitives
      
      i.e. move it from node specific primitives to global v1 primitives. This
      will be needed when we share it later in the runtime inclusion module
      
      * refactor acceptance checks in the inclusion module
      
      factor out the common code to share it during the block inclusion and
      for the forthcoming CheckValidationOutputs runtime api.
      
      Also note that the acceptance criteria was updated to incorporate checks
      that exist now in candidate-validation
      
      * plumb the runtime api outside
      
      * extract validation_data from ValidationOutputs
      
      * use runtime-api to check validation outputs
      
      apart from that refactor, update docs and tidy a bit
      
      * Update the maxium code size
      
      This is to fix a test that performs an upgrade.
      4a17a2bc
  7. Oct 16, 2020
  8. Oct 08, 2020
  9. Oct 06, 2020
    • ordian's avatar
      NetworkBridge: validator (authorities) discovery api (#1699) · b84f3c03
      ordian authored
      
      
      * stupid, but it compiles
      
      * redo
      
      * cleanup
      
      * add ValidatorDiscovery to msgs
      
      * sketch network bridge code
      
      * ConnectToAuthorities instead of validators
      
      * more stuff
      
      * cleanup
      
      * more stuff
      
      * complete ConnectToAuthoritiesState
      
      * Update node/network/bridge/src/lib.rs
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      
      * Collator protocol subsystem (#1659)
      
      * WIP
      
      * The initial implementation of the collator side.
      
      * Improve comments
      
      * Multiple collation requests
      
      * Add more tests and comments to validator side
      
      * Add comments, remove dead code
      
      * Apply suggestions from code review
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      
      * Fix build after suggested changes
      
      * Also connect to the next validator group
      
      * Remove a Future impl and move TimeoutExt to util
      
      * Minor nits
      
      * Fix build
      
      * Change FetchCollations back to FetchCollation
      
      * Try this
      
      * Final fixes
      
      * Fix build
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      
      * handle multiple in-flight connection requests
      
      * handle cancelled requests
      
      * Update node/core/runtime-api/src/lib.rs
      
      Co-authored-by: default avatarBernhard Schuster <[email protected]>
      
      * redo it again
      
      * more stuff
      
      * redo it again
      
      * update comments
      
      * workaround Future is not Send
      
      * fix trailing spaces
      
      * clarify comments
      
      * bridge: fix compilation in tests
      
      * update more comments
      
      * small fixes
      
      * port collator protocol to new validator discovery api
      
      * collator tests compile
      
      * collator tests pass
      
      * do not revoke a request when the stream receiver is closed
      
      * make revoking opt-in
      
      * fix is_fulfilled
      
      * handle request revokation in collator
      
      * tests
      
      * wait for validator connections asyncronously
      
      * fix compilation
      
      * relabel my todos
      
      * apply Fedor's patch
      
      * resolve reconnection TODO
      
      * resolve revoking TODO
      
      * resolve channel capacity TODO
      
      * resolve peer cloning TODO
      
      * resolve peer disconnected TODO
      
      * resolve PeerSet TODO
      
      * wip tests
      
      * more tests
      
      * resolve Arc TODO
      
      * rename pending to non_revoked
      
      * one more test
      
      * extract utility function into util crate
      
      * fix compilation in tests
      
      * Apply suggestions from code review
      
      Co-authored-by: default avatarFedor Sakharov <[email protected]>
      
      * revert pin_project removal
      
      * fix while let loop
      
      * Revert "revert pin_project removal"
      
      This reverts commit ae7f529d
      
      .
      
      * fix compilation
      
      * Update node/subsystem/src/messages.rs
      
      * docs on pub items
      
      * guide updates
      
      * remove a TODO
      
      * small guide update
      
      * fix a typo
      
      * link to the issue
      
      * validator discovery: on_request docs
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      Co-authored-by: default avatarFedor Sakharov <[email protected]>
      Co-authored-by: default avatarBernhard Schuster <[email protected]>
      b84f3c03
  10. Oct 04, 2020
    • Kian Paimani's avatar
      Companion for substrate/pull/7215 (#1768) · 6d126150
      Kian Paimani authored
      * Companion for substrate/pull/7215
      
      * More fixes
      
      * Fix build
      
      * fix
      
      * Fix again with normal
      
      * Fix build
      
      * Latest changes
      
      * "Update Substrate"
      
      Co-authored-by: parity-processbot <>
      6d126150
  11. Oct 01, 2020
  12. Sep 28, 2020
  13. Sep 24, 2020
  14. Sep 22, 2020
  15. Sep 21, 2020
  16. Sep 18, 2020
    • Peter Goodspeed-Niklaus's avatar
      c68aee35
    • Peter Goodspeed-Niklaus's avatar
      Remove service, migrate all to service-new (#1630) · af14ea54
      Peter Goodspeed-Niklaus authored
      * import rococo into chain-spec
      
      * make a few stabs at moving forward
      
      * wip: rococo readme
      
      * remove /service crate
      
      - Move the chain-spec files to node-service
      - update sufficient cargo files that polkadot-service-new builds
      - not everything else builds yet
      
      * wip: chase down some build errors in polkadot-cli
      
      There's a lot more to go, but some progress has happened.
      
      * make more progress getting polkadot-cli to build
      
      * don't ignore polkadot.json within the res directory
      
      * don't recreate pathbufs
      
      * Prepare Polkadot to be used by Cumulus
      
      This begins to make Polkadot usable from Cumulus.
      
      * Remove old test
      
      * migrate new_chain_ops fix from /service
      
      * partially remove node/test-service
      
      * Reset some changes
      
      * Revert "partially remove node/test-service"
      
      This reverts commit 7b8f9ba5.
      
      * WIP: replace v0 ParachainHost impl with v1 for test runtime
      
      This is necessary because one of the current errors when building
      the test service boils down to:
      
      the trait bound `polkadot_test_runtime::RuntimeApiImpl<...>`:
        `polkadot_primitives::v1::ParachainHost<...>` is not satisfied
      
      This is WIP because it appears to be causing some std leakage into
      the wasm environment, or something; the compiler is currently
      complaining about duplicate definitions of `panic_handler` and `oom`.
      Presumably I have to identify all std types (Vec etc) and replace
      them with sp_std equivalents.
      
      * fix test runtime build
      
      it wasn't std leakage, after all
      
      * bump westend spec version
      
      * use service-new as service within cli
      
      * to revert: demo that forwarding the test runtime to the real impl blows up
      
      * Revert "to revert: demo that forwarding the test runtime to the real impl blows up"
      
      This reverts commit 68d2f385.
      
      * Revert "Revert "to revert: demo that forwarding the test runtime to the real impl blows up""
      
      This reverts commit 04cb1cbf.
      
      Might have just forgotten to disable default features
      
      * More reverts
      
      * MOARE
      
      * plug in the runtime as the generic instantiation
      
      This feels closer to a solution, but it still has problems: in particular,
      it's assumed that Runtime implements all appropriate Trait traits,
      which this one apparently does not.
      
      * implement necessary traits to get the test runtime compiling
      
      This is almost certainly not correct in some way; it really
      looks like I need to mess with the construct_runtime! macro
      somehow, to inject the inclusion trait's event type as a Event
      variant. Still, better lock down this changeset while it all
      compiles.
      
      * add inclusion::Event as variant into Event enum
      
      * implement unimplemented bits in kusama
      
      * implement unimplemented bits in polkadot runtime
      
      * implement unimplemented bits in westend runtime
      
      * migrate client upgrades from master
      
      * update test service with new node changes
      
      * package metadata--that wasn't intended to be removed
      
      * add parachains v1 modules to each runtime
      
      It's not clear what precisely this does, but it's probably the right
      thing to do.
      
      * enable cli to opt out of full node features
      
      * adjust rococo chainspec per example
      
      https://github.com/paritytech/polkadot/blob/26f1fa47f7836ab4bee5d4aad127ebce748320dd/service/src/chain_spec.rs#L362
      
      
      
      * try to fix Cargo.lock
      
      Co-authored-by: default avatarBastian Köcher <[email protected]>
      Co-authored-by: default avatarAndronik Ordian <[email protected]>
      af14ea54
  17. Sep 17, 2020
  18. Sep 16, 2020
  19. Aug 18, 2020
    • asynchronous rob's avatar
      Implement validation data refactor (#1585) · 8c881e45
      asynchronous rob authored
      
      
      * update primitives
      
      * correct parent_head field
      
      * make hrmp field pub
      
      * refactor validation data: runtime
      
      * refactor validation data: messages
      
      * add arguments to full_validation_data runtime API
      
      * port runtime API
      
      * mostly port over candidate validation
      
      * remove some parameters from ValidationParams
      
      * guide: update candidate validation
      
      * update candidate outputs
      
      * update ValidationOutputs in primitives
      
      * port over candidate validation
      
      * add a new test for no-transient behavior
      
      * update util runtime API wrappers
      
      * candidate backing
      
      * fix missing imports
      
      * change some fields of validation data around
      
      * runtime API impl
      
      * update candidate validation
      
      * fix backing tests
      
      * grumbles from review
      
      * fix av-store tests
      
      * fix some more crates
      
      * fix provisioner tests
      
      * fix availability distribution tests
      
      * port collation-generation to new validation data
      
      * fix overseer tests
      
      * Update roadmap/implementers-guide/src/node/utility/candidate-validation.md
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      
      Co-authored-by: default avatarPeter Goodspeed-Niklaus <[email protected]>
      8c881e45
  20. Aug 04, 2020
  21. Aug 03, 2020
  22. Aug 02, 2020
  23. Jul 31, 2020
    • Fedor Sakharov's avatar
      Parachains-runtime (#1505) · eca01a8c
      Fedor Sakharov authored
      * Initial commit
      
      * v0 to v1 primitives and remove attestations
      
      * Review fixes
      
      * implement candidate_events
      
      * remove dead code
      eca01a8c
  24. Jul 24, 2020
  25. Jul 23, 2020
    • asynchronous rob's avatar
      Include a reference to the validation data in the candidate descriptor (#1442) · cce0a950
      asynchronous rob authored
      * rename GlobalValidationSchedule to GlobalValidationData
      
      * guide: update candidate descriptor to contain validation data hash
      
      * guide: add note in inclusion module about checking validation data hash
      
      * primitives: update CandidateDescriptor to contain new hash
      
      * fix payload computation
      
      * add helpers for computing validation data to runtime modules
      
      * guide: note routines
      
      * inclusion: check validation data hash and fix local_validation_data bug
      
      * add a case to candidate_checks and improve that test substantially
      
      * bump versions
      
      * address review comments
      
      * add a test for including code upgrade
      
      * bump kusama version
      
      * bump westend & polkadot versions
      cce0a950
  26. Jul 17, 2020
  27. Jul 10, 2020
    • asynchronous rob's avatar
      Refactor primitives (#1383) · 96af6ead
      asynchronous rob authored
      * create a v1 primitives module
      
      * Improve guide on availability types
      
      * punctuate
      
      * new parachains runtime uses new primitives
      
      * tests of new runtime now use new primitives
      
      * add ErasureChunk to guide
      
      * export erasure chunk from v1 primitives
      
      * subsystem crate uses v1 primitives
      
      * node-primitives uses new v1 primitives
      
      * port overseer to new primitives
      
      * new-proposer uses v1 primitives (no ParachainHost anymore)
      
      * fix no-std compilation for primitives
      
      * service-new uses v1 primitives
      
      * network-bridge uses new primitives
      
      * statement distribution uses v1 primitives
      
      * PoV distribution uses v1 primitives; add PoV::hash fn
      
      * move parachain to v0
      
      * remove inclusion_inherent module and place into v1
      
      * remove everything from primitives crate root
      
      * remove some unused old types from v0 primitives
      
      * point everything else at primitives::v0
      
      * squanch some warns up
      
      * add RuntimeDebug import to no-std as well
      
      * port over statement-table and validation
      
      * fix final errors in validation and node-primitives
      
      * add dummy Ord impl to committed candidate receipt
      
      * guide: update CandidateValidationMessage
      
      * add primitive for validationoutputs
      
      * expand CandidateValidationMessage further
      
      * bikeshed
      
      * add some impls to omitted-validation-data and available-data
      
      * expand CandidateValidationMessage
      
      * make erasure-coding generic over v1/v0
      
      * update usages of erasure-coding
      
      * implement commitments.hash()
      
      * use Arc<Pov> for CandidateValidation
      
      * improve new erasure-coding method names
      
      * fix up candidate backing
      
      * update docs a bit
      
      * fix most tests and add short-circuiting to make_pov_available
      
      * fix remainder of candidate backing tests
      
      * squanching warns
      
      * squanch it up
      
      * some fallout
      
      * overseer fallout
      
      * free from polkadot-test-service hell
      96af6ead
  28. Jul 09, 2020
  29. Jul 08, 2020
  30. Jul 06, 2020
  31. Jul 04, 2020