* Extract CLI to separate module in node/cli

* Make node/cli compile for WASM

* More work on node/cli browser

* More work on browser node

* More work

* More work

* Purge a bit the CI script

* More clean up

* Remove substrate-finality-grandpa from the CI

Its tests use tokio, which fails to compile.

* Address review

* Add rocksdb feature to the service

* Fix substrate-service WASM CI

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <[email protected]>

* Don't WASM-compile substrate-service altogether

* Put the DB configuration in an enum

* Allow passing a custom database to client-db

* Clean-ups in client-db

* Fix client tests

* Fix service tests

* Hopefully fix tests for good this time 😩

* Address review

* node: add sentry mode flag

* cli: extend docs on validator and sentry modes

* service: add missing field in test Configuration

* node: Display instead of Debug when printing node role

* Rename NewService to Service

* Move new_impl! macro to builder module

* Inline new_impl!

* Minor cleanup

* Inline the offchain_workers() function

* Fix indentation level

* Inline start_rpc

* Remove RpcBuilder trait

* Split the telemetry net status report in two

* Update core/service/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Remove clone()

* Move code to status_sinks.rs instead

* Add basic usage for status_sinks

* Update core/service/src/status_sinks.rs

Co-Authored-By: Bastian Köcher <[email protected]>

Fix some Clippy issues

specified

* Add a Node Role RPC call

* Formatting

* Fix tests

* Change tests to use NodeRole::Authority so I don't forget to update the test

* Improve role checking

* return a vec instead

* fix tests

* executor: Move definitions of externals out of wasm_executor module.

* executor: Create WasmRuntime trait.

This will be used to decouple the runtime cache from wasmi execution.

* executor: Remove WasmExecutor and move methods to wasmi_execution.

These will now be crate-internal functions and there is no need
for the struct.

* executor: Set default default_heap_pages in NativeExecutor.

* cli: CLI configuration for Wasm execution method.

* executor: Remove wasmi-specific code from wasm_runtime.

* Respond to review comments.

* Update all dependencies

* Upgrade dependencies whenever “easy”

“easy” means that there are no major changes required.

* Fix build and bump paste dependency to 0.1.6

* Remove dead code

* Re-add = dependency for futures-preview

* Add missing std features for runtime-io

* Remove git dependencies

as updated versions have been published to crates.io

* try to debug bug

* For sr-io, "std" should imply "no_oom" and "no_panic_handler".

Otherwise, rustc complains (correctly) about duplicate lang items.

* Add missing "runtime-io/std" features

* Fix compilation errors

* Prevent duplicate lang items

Rust does not allow duplicate lang items.  When compiled without the
`std` feature, `sr-io` defines two lang items.  Therefore, `sr-io`
compiled without `feature = "std"` must not be linked with `std`.

However, `pwasm-utils` and `wasmi-validation` both bring in `std` unless
compiled with `default-features = "false"`.  This caused a duplicate
lang item error.  Building both with `default-features = "false"`
prevents this error.  When building with `feature = "std"`, they should
both be built with the `std` feature, so this feature needs to be
explicitly depended on.

* Bump `impl_version`

* Make tests pass

Three tests used 1 less gas than they had previously.

* Try to un-break build

* Add a Cargo.lock file

* Revert offchain code

* Revert "Revert offchain code"

This reverts commit d216d08c

.

* Don’t try to send a body with a GET request

without adding a Transfer-Encoding or Content-Length header.

This has always been wrong, but hyperium/hyper#1925 hid the bug until
hyper was upgraded to 0.12.35.

* Change some more GET requests to POST requests

* Fix excess line width and remove an `extern crate`

* Delete commented-out extern crate

Co-Authored-By: Sergei Pepyakin <[email protected]>

* Fix regression in Cargo.toml files

dev-dependencies need `default-features = false`, too.

* Bump parity-wasm dependency

* Bump `futures-preview`

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <[email protected]>

* Update Cargo.lock files

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <[email protected]>

* Update core/service/src/chain_ops.rs

Co-Authored-By: Sergei Pepyakin <[email protected]>

* async txpool API

* Update core/rpc/src/author/mod.rs

Co-Authored-By: Tomasz Drwięga <[email protected]>

* Update core/transaction-pool/graph/src/pool.rs

Co-Authored-By: Tomasz Drwięga <[email protected]>

* Pool -> Pool + ValidatedPool

* removed lost block_on when importing xt from network

* fix grumbles

* alias for future::Executor in rpc

* removed executor from Author RPCs

* Pool + SharedValidatedPool -> Pool

* fix compilation after merge

* another fix

* another fix

* Add some chainspec tests and make sure we validate it.

* Manual implementation of Extension + Forks definitions.

* Move chain spec to separate crate.

* Allow using ChainSpec with extensions.

* Renames.

* Implement Extension derive.

* Implement Extension for Forks.

* Support specifying fork blocks.

* make for_blocks work

* Support forks correctly.

* Add a bunch of docs.

* Make fork blocks optional.

* Add missing docs.

* Fix build.

* Use struct for check_block params.

* Fix tests?

* Clean up.

* Add `BlockAnnounceValidator` trait.

* Add associated data to block announcement.

* Make tests compile.

* Move validator into `sync.rs`.

* Smaller changes.

* Update core/network/src/protocol.rs

Co-Authored-By: Kian Paimani <[email protected]>

* Update core/network/src/protocol.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update core/network/src/test/sync.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Formatting.

* Remove assoc. data from `BlockImportNotification`.

* Use `Option<Vec<u8>>` for associated data.

* Update core/network/src/protocol/sync.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Fix type error.

* always fetch epoch from runtime

* node integration tests don't test light nodes

* give stand-in full node a FULL role

* rejig babe APIs

* introduce next-epoch-descriptor type

* overhaul srml-BABE epoch logic

* ensure VRF outputs end up in the right epoch-randomness

* rewrite `do_initialize` to remove unnecessary loop

* begin accounting for next epoch in epoch function

* slots passes header to epoch_data

* pass slot_number to SlotWorker::epoch_data

* begin extracting epoch-change logic into its own module

* aux methods for block weight

* aux methods for genesis configuration

* comment-out most, refactor header-check pipeline

* mostly flesh out verifier again

* reinstantiate babe BlockImport implementation

* reinstate import-queue instantiation

* reintroduce slot-worker implementation

* reinstate pretty much all the rest

* move fork-choice logic to BlockImport

* fix some, but not all errors

* patch test-runtime

* make is_descendent of slightly more generic

* get skeleton compiling when passing is_descendent_of

* make descendent-of-builder more succinct

* restore ordering of authority_index / slot_number

* start fiddling with tests

* fix warnings

* improve initialization architecture and handle genesis

* tests use correct block-import

* fix BABE tests

* fix some compiler errors

* fix node-cli compilation

* all crates compile

* bump runtime versions and fix some warnings

* tweak fork-tree search implementation

* do backtracking search in fork-tree

* node-cli integration tests now work

* fix broken assumption in test_connectivity

* babe tests fail for the right reasons.

* test genesis epoch logic for epoch_changes

* test that epochs can change between blocks

* First BABE SRML test

* Testing infrastructure for BABE

Also includes a trivial additional test.

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <[email protected]>

* A little more test progress

* More work on BABE testing

* Try to get the tests working

* Implement `UintAuthorityId`-based test mocks

* Fix compilation errors

* Adjust to upstream changes

* Block numbers are ignored in BABE epoch calculation

* authority_index() should ignore invalid authorities

* Fix compile error

* Add tests that session transitions happen

* Check if BABE produces logs

It currently does not.

* Fix test suite

This was really nasty, due to a type confusion that showed up as an
off-by-1 buffer error.

* Add additional tests

Most of these were derived from the current output, so they are only
useful to guard against regressions.

* Make the tests more readable

Also bump impl_version.

* Fix excessive line width

* Remove unused imports

* Update srml/babe/src/lib.rs

Co-Authored-By: André Silva <[email protected]>

* try to fix imports

* Fix build errors in test suite

* tests did not pass

* Try to get at least one digest to be output

Currently, the code emits either no digests (if I don’t call
`Session::rotate_session()` or two digests (if I do), which is wrong.

* More tests

They still don’t work, but this should help debugging.

* fix silly error

* Don’t even try to compile a broken test

* remove broken check_epoch test and add one for genesis epoch

* Check that the length of the pre-digests is correct

* Bump `impl_version`

* use epoch_for_descendent_of even for genesis

* account for competing block 1s

* finish srml-babe docs

Co-Authored-By: André Silva <[email protected]>

* address grumbles

* Only submit signed extrinsics back to pool when retracting block

* Fix import

* Send block status with announcement

* Fixed tests

* Whitespace

Co-Authored-By: Gavin Wood <[email protected]>

* Additional comment

* Update comment

Co-Authored-By: André Silva <[email protected]>

* Add meta rpc_methods call.

* Sort methods.

* Bump runtime.

* Change format a bit to support versioning.

* removing fetcher dependency from light backend

* fix compilation

* Augment every task spawned by Service with `on_exit`

* Add CI test that the node exits

* Add transactions from retracted blocks back to the pool

* Line width

* Reverse retracted

With the *authority-discovery* module an authoritative node makes itself
discoverable and is able to discover other authorities. Once discovered, a node
can directly connect to other authorities instead of multi-hop gossiping
information.

1. **Making itself discoverable**

    1. Retrieve its external addresses

    2. Adds its network peer id to the addresses

    3. Sign the above

    4. Put the signature and the addresses on the libp2p Kademlia DHT

2. **Discovering other authorities**

    1. Retrieve the current set of authorities

    2. Start DHT queries for the ids of the authorities

    3. Validate the signatures of the retrieved key value pairs

    4. Add the retrieved external addresses as ~reserved~ priority nodes to the
       peerset


* node/runtime: Add authority-discovery as session handler

The srml/authority-discovery module implements the OneSessionHandler in
order to keep its authority set in sync. This commit adds the module to
the set of session handlers.

* core/network: Make network worker return Dht events on poll

Instead of network worker implement the Future trait, have it implement
the Stream interface returning Dht events.

For now these events are ignored in build_network_future but will be
used by the core/authority-discovery module in subsequent commits.

* *: Add scaffolding and integration for core/authority-discovery module

* core/authority-discovery: Implement module logic itself

* chain+state RPCs are async now

* wrapped too long lines

* create full/light RPC impls from service

* use ordering

* post-merge fix

* generalize tree_root to remove client.backend dependency

* replace client.backend.blockchain.header with client.header

* move used_state_cache_size into client info

* Create intermediate Setup State. Fixes #1134

* remove client.backend from finality proof

* update node-template

* move memory backend into test helper mode

* move test helper into client

* starting the big refactor, remove unused functions

* apply_finality

* apply_finality

* replacing more .backend from environment with client directly

* remove .backend from grandpa by using traits

* remove .backend from babe

* remove .backend from tests where it is not needed

* remove .backend from tests

* fixing tests

* fixing tests

* fixing more tests

* fixing tests

* fix all forks test

* fix style

* fixing unnecessary allocation

* remove old test.

* fix service docs

* apply suggestion

* minor clean ups

* turns out the test-helper features actually is being used!

* fixing line length.

* fix line length

* minor cleaning

* Apply suggestions from code review

thanks, @Basti

Co-Authored-By: Bastian Köcher <[email protected]>

* address grumbles

* simplify finalize block on client

* move block back into inner function

* Apply suggestions from code review

Co-Authored-By: DemiMarie-parity <[email protected]>

* use as.ref instead of match

* Update core/client/src/backend.rs

Co-Authored-By: DemiMarie-parity <[email protected]>

Tomasz Drwięga authored Aug 29, 2019 and Svyatoslav Nikolsky committed Aug 29, 2019

* Extract author API from the substrate-rpc crate.

* Split out API from RPC.

* Clean up naming.

* Fix tests.

* Shorten error translations.

* Update Cargo.lock

* Move Service::new to a macro

* Move function calls to macros

* Extract offchain_workers and start_rpc in separate function

In follow-up commits, we want to be able to directly call maintain_transaction_pool, offchain_workers, and start_rpc, without having to implement the Components trait.
This commit is a preliminary step: we extract the code to freestanding functions.

* Introduce an AbstractService trait

* Introduce NewService as an implementation detail of Service

* Implement traits on NewService instead

Instead of implementing AbstractService, Future, and Executor on Service, we implement them on NewService instead.

The implementations of AbstractService, Future, and Executor on Service still exist, but they just wrap to the respective implementations for NewService.

* Move components creation back to macro invocation

Instead of having multiple $build_ parameters passed to the macro, let's group them all into one.

This change is necessary for the follow-up commits, because we are going to call new_impl! only after all the components have already been built.

* Add a $block parameter to new_impl

This makes it possible to be explicit as what the generic parameter of the NewServiceis, without relying on type inference.

* Introduce the ServiceBuilder struct

Introduces a new builder-like ServiceBuilder struct that creates a NewService.

* Macro-ify import_blocks, export_blocks and revert_chain

Similar to the introduction of new_impl!, we extract the actual code into a macro, letting us get rid of the Components and Factory traits

* Add export_blocks, import_blocks and revert_chain methods on ServiceBuilder

Can be used as a replacement for the chain_ops::* methods

* Add run_with_builder

Instead of just run, adds run_with_builder to ParseAndPrepareExport/Import/Revert. This lets you run these operations with a ServiceBuilder instead of a ServiceFactory.

* Transition node and node-template to ServiceBuilder

* Transition transaction-factory to the new service factory

This is technically a breaking change, but the transaction-factory crate is only ever used from within substrate-node, which this commit updates as well.

* Remove old service factory

* Adjust the AbstractService trait to be more usable

We slightly change the trait bounds in order to make all the methods usable.

* Make substrate-service-test compile

* Fix the node-cli tests

* Remove the old API

* Remove the components module

* Fix indentation on chain_ops

* Line widths

* Fix bad line widths commit

* Line widths again 🤦



* Fix the sync test

* Apply suggestions from code review

Co-Authored-By: Gavin Wood <[email protected]>

* Address some concerns

* Remove TelemetryOnConnect

* Remove informant::start

* Update jsonrpc

* Rename factory to builder

* Line widths 😩

* Allow RPCs to be customized.

* Implement node-rpc extensions.

* Working on a test.

* Add node-testing crate.

* Fix genesis test config

* Fix nonce lookups.

* Clean up.

* Fix expected block type.

* Make the RPC extension function optional.

* Fix service doc test.

* Bump jsonrpc.

* Bump client version.

* Update Cargo.lock

* Update jsonrpc.

* Fix build.

* Remove unused imports.

* Fix signed extra.

* Post merge clean up.

* Fix tests.

* Patch hashmap-core.

* Fix build.

* Fix build.

* Remove hashmap_core patches.

* value ranges in consensus cache

* skip values in cache

* read epoch0 + epoch1 data from genesis in babe

* sync authorities + session validators at genesis

* removed some debug printlns

* fixed cache encoding

* Revert "skip values in cache"

This reverts commit ce451c32.

* Revert "value ranges in consensus cache"

This reverts commit 9062f943

.

* get rid of cache::AUTHORITIES in Babe

* cleaning up

* cleaning up

* update spec version

* lost changes

* fixed tests

* Update node/runtime/src/lib.rs

Co-Authored-By: DemiMarie-parity <[email protected]>

* fix once-per-block condition

* fix standalone babe + temp_storage in BuildGenesis

* fix benhes compilation

* fixed comment

* re-added light nodes to integration tests

* finalize_with_ancestors from extra_requests

* post-merge fix

* aaand removed debug code

* (another one)

* fix warn in logs (do not call ForkTree::finalize twice for the same block)

* sync digest.next_authorities with actual next authorities

* more docs

* reverting all commits affecting storage

* also remove keys from babe trait

* fixed warnings

* post-merge fixes

* reverted some redundant changes

* reverted more changes

* Transition the transaction-pool to new futures

* Fix tests

* Fix tests again?!

* Add transaction pool to babe import queue

* Add transaction pool to Babe check header

* Fix tests

* Add tx pool to Aura import_queue

* Fix tests, node-template

* Add comments regarding unused _transaction_pool

* Make tx pool optional in check_header

André Silva authored Aug 12, 2019 and asynchronous rob committed Aug 12, 2019

* node: exit on GRANDPA voter or BABE authoring error

* node: exit process with non-zero return code when service fails

* service: rename infallible task to essential task

* service: revert field name changes

* core: fix service testnet

* document that session key generation stores keys

* final typo fix

* Implement `is_validator` for offchain-workers

* Introduce `--no-validator` flag

* Don't run babe/grandpa/im-online when `--no-validator` is given

* Fixes compilation

* Bump spec version

* Improve error handling in executor

* Add missing extern function

* Revert making error public

* Remove `--no-validator` CLI

* Remove generic from sign()

* Remove mandatory RuntimeGenesis trait req

* Remove requirement from Configuration

* Relax trait bounds of core/cli

* Move method

* The config field is no longer public

* Remove Components from bounds of functions

* Implement DerefMut for LightComponents

* Implement Executor for Full/LightComponents

* Fix bad merge

* Fix forgotten config()

* Fix build

* Using child storage, (srml-support only), test failing .

* fix simple tests.

* Enumerable by requiring owned struct (previous form only allow
&'static).
Broken tests are from genesis init.

* implement for_child_keys_with_prefix

* indent

* clear_child_prefix fix.

* clear_child_prefix fix 2.

* fix for storage_impl, if/when allowing child and not child this could be
reverted.

* Fix lot of urlinked child genesis, still need to look upon actual
genesis srml module code.
Probably still a lot of broken code needing debugging.

* switch well_known_key to their associated module child trie.
Fix a genesis init (balance).
Complete some testing.
Comment some tests before using.

* fixing test runtime child keys

* latest commit fix broken genesis init

* fix system balances child name.

* Important fix: storage_root from test externalities need children (it is
already the case for ext).

* executive root with child calculation

* Avoid empty trie on test ext.

* Symetric removal of key for system.

* commenting changes related tests.

* Remove child module specifics.

* fix issues.

* fix some formatting

* fix bench and bump runtime

* Remove extend_storage_overlays, assimilate_storage do the same as is
proper considering srml macro.

* Fix warning for assimilate.

* Removing kill as they do not impact any test cases.

* Use tuple of storage map instead of two parameters. This changes the
behavior of decl_storage genesis build closure (breaking api).

* Do not use build storage before assimilate.

* fix error

* Update core/state-machine/src/backend.rs

* Adds `author_rotateKeys` RPC

`author_rotateKeys` will rotate all session keys, store them into the
keystore and returns their encoded public keys as result. These encoded
public keys can be used directly to send the transaction to the chain.

* Review comments

* Switch offchain workers to new futures

* Fix tests

* Cache idle state

* Display import error by default