This exposes memory statistics from the state-db.

* Start removing last few instances of futures01

* Use to_poll on wasm

* Revert "Use to_poll on wasm"

This reverts commit 1c61728f

.

* Fix fg test

* Upgrade network test futures

* Update offchain hyper version

* Update service test

* bump tokio to 0.2.10

* Removed some unneeded tokios

* fixes

* fix run_until_all_full

* Make service test debuggable

* Update client/offchain/src/api/http.rs

Co-Authored-By: Demi Obenour <[email protected]>

* Add service_test to test-int output

* nitpicking

* Finally fix test

* Give up and revert client/serviec/test

* Revert gitlab ci too

Co-authored-by: Demi Obenour <[email protected]>

* Propagate DispatchError for benchmarks.

* Apply review suggestions.

* Use RuntimeString.

* fix expect

Co-Authored-By: Bastian Köcher <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>

* Bump to alpha.3

* update gitlab-ci

This patch renames the crate for the following two reasons:

1. The prometheus-exporter crate introduces native in-process Prometheus
style instrumentation to the Substrate project. Within the Prometheus
ecosystem the term "exporter" is used for external processes exposing
metrics for e.g. the Linux Kernel. In-process exposition would be
described via the term "endpoint".

2. "prometheus-exporter" is generic and ignores the fact that it is only
usable within the context of Substrate. In addition the name
"prometheus-exporter" is already taken on crates.io.

* removes use of sc-client from sc-basic-authorship

* refactor use of ProposerFactory

* correct dep path

* setting first batch of descriptions

* fix what I just broke

* next batch

* and pallets, too

* last batch

* set cargo.lock

* keep'em dev-deps

* bump version to alpha.2

* repro ui bug

* fix the tests

* test with the new image

* test without CARGO_HOME

* test without fixes

* test again

* fix trybuild old versions

* bump CArgo.lock

* fix trybuild newest versions

* bump Cargo.lock

* trying on the latest image

* bump Cargo.lock

* run with the old image

* ci will be green on the image from 2020-02-19 [skip ci]

* bump Cargo.lock

bump version to -alpha.1

* Remove indices from node-template

* Use identity lookup instead

* Bump impl

* clean cargo.toml

* adding unleash to ci

* fixing formatting

* with a dot please

* alpha.3 now

* do not publish testing helpers

* remove old test-helpers cruft

* fix cargo.lock

* with alpha 4

* do not publish runtime-interface-test either

* disable more test crates from publishing

* switch to alpha.5

* replace tempdir with tempfile

* update lru

* switch to bytes 0.5

* release script fixes

* switch on and to latest alpha

* BUT THE SPACES

* setting versions to development pre-release

fixing version in dependencies

* unset already released wasm-builder

* do not publish test crates

* adding licenses

* setting homepage metadata

* set repository url

Related to #4776 
Related to https://github.com/paritytech/polkadot/pull/832

To summarize the changes:
1. I did not manage to validate with types the service's Configuration. But I did reduce the possibility of errors by moving all the "fill" functions to their respective structopts
2. I split params.rs to multiple modules: one module params for just CLI parameters and one module commands for CLI subcommands (and RunCmd). Every command and params are in their own file so things are grouped better together and easier to remove
3. I removed the run and run_subcommand helpers as they are not helping much anymore. Running a command is always a set of 3 commands: 1. init 2. update config 3. run. This still allow the user to change the config before arguments get parsed or right after.
4. I added tests for all subcommands.
5. [deleted]

Overall the aim is to improve the situation with the Configuration and the optional parameters, add tests, make the API more consistent and simpler.

* Open one substream for each notifications protocol

* Fix WASM build

* Apply suggestions from code review

Co-Authored-By: Toralf Wittner <[email protected]>

* Address concerns

* Use unsigned-varint to read the varint

* Use unsigned-varint

* Forgot Cargo.lock

Co-authored-by: Toralf Wittner <[email protected]>

* MAcro benchamrks

* Iterative macro

* Tidying it up.

* Macro improvements

* Bits..

* Last benchmaks.

* Repo benchmark macro

* Add the possibility of evaluating arbitrary expressions in a
benchmaark

* Better syntax and docs

* Update `BenchmarkParameter`

* Add `ignore` to sudo-code in docs

* First try of timestamp implementation.

* Fix macro docs, remove warnings.

* Use macro in balances pallet.

* Make some space in frame benchmarking.

* Remove _benchmarks_seed variable.

* Bump impl_version.

Co-authored-by: Shawn Tabrizi <[email protected]>
Co-authored-by: Marcio Diaz <[email protected]>

* Make unsigned also note weight

* Bump.

* Fix tests

* Better fee test

* Fix another test

* Update lock file

* Example of offchain worker pallet.

* Fix compilation issues.

* Use serde_json to parse JSON.

* Add some basic tests.

* Working on docs.

* Fix compilation

* Finish docs for signed.

* Work on unsigned send.

* Add some tests and missing docs.

* Add example of StorageValueRef

* Add weight.

* Extra \n

* Fix im-online test.

* Bump runtime.

* Fix tests.

* Apply suggestions from code review

Co-Authored-By: Joshy Orndorff <[email protected]>
Co-Authored-By: Gavin Wood <[email protected]>

* Address review comments.

Co-authored-by: Joshy Orndorff <[email protected]>
Co-authored-by: Gavin Wood <[email protected]>

* Refactor rebase master prometheus_v0.3

* Milestone1: Final Version of v0.3

* no-std or warm compatibility issues, grapana-data -source code reference and correction,applicable

* Cargo.lock paritytech/master rebase

* prometheus networking.rs del, grafana-data-source networking.rs pub edit and note

* chore: reflect various feedback

* Spaces to tabs.

* Replace grafana and tidy

* Add generics

* Add photo back

* Re-fix spaces in primitives/consensus/babe/src/inherents.rs

* Refactor rebase master prometheus_v0.3

* Milestone1: Final Version of v0.3

* no-std or warm compatibility issues, grapana-data -source code reference and correction,applicable

* prometheus networking.rs del, grafana-data-source networking.rs pub edit and note

* chore: reflect various feedback

* Replace grafana and tidy

* Add generics

* Add photo back

* Re-fix spaces in primitives/consensus/babe/src/inherents.rs

* chore: revert this file back to paritytech/master inherents.rs.

* Add newline at EOF

* Tidy

* Use local registry

* fix typo

Co-Authored-By: Max Inden <[email protected]>

* chore:  Apply review feedback

* endpoint -> exporter

* fix readme

* Remove lazy_static, use ServiceMetrics struct instead

* Switch to using GaugeVecs

* chore: without nightly , edit README

* block_height -> block_height_number

* Switch to a ready_transactions_number gauge

* Update utils/prometheus/src/lib.rs

Co-Authored-By: Max Inden <[email protected]>

* no-prometheus flag add

* /metrics url Input check

* remove prometheus in Tracing

* remove prometheus in Tracing

* chore: master code rebase edit

* gitlab-check-web-wasm edit code

* From:from and cargo.lock update

* with_prometheus_registry add background_tasks

* utils/prometheus/src/lib.rs: Restructure #[cfg] for wasm without hyper

Given that Hyper is not compatible with WASM targets it needs to be
excluded from WASM builds. Instead of introducing #[cfg] lines
throughout the crate, this patch splits the crate into two: known_os and
unknown_os (WASM).

* utils/prometheus/src/lib.rs: Feature gate known_os module

* client/cli/src/lib.rs: Re-add newline at end of file

Co-authored-by: JeseonLEE <[email protected]>
Co-authored-by: Gavin Wood <[email protected]>
Co-authored-by: Ashley <[email protected]>
Co-authored-by: Hyungsuk Kang <[email protected]>

This prs cleans up some of the frame benchmarking stuff:
- Move CLI into `frame-benchmarking-cli`. No frame related CLI should
exists in the default Substrate CLI.
- Move all traits and types related to frame benchmarking into the
`frame-benchmarking` trait. Frame types should be isolated in Frame.

* Initial inspect.

* WiP

* Add parsing tests.

* Finalize CLI.

* Update to latest substrate.

* Remove unused imports.

* Support ImportParams as well, to get the right pruning setting.

* Mention in docs that hash is no 0x.

* Move bytes above extrinsics.

* Switch to fill helper from sc_cli.

* Remove overwrite.

* Fix error.

* Fix error message.

* Remove extra allow.

* init_config

libp2p-0.16 allows configuring yamux through libp2p-yamux, so the
direct dependency is no longer needed.

While at it we also update to the latest versions of yamux and
nohash-hasher, though the code changes do not depend on it.

* move to node-testing/bench

* to tabs

* service: Don't depend on tokio-executor

Seems to be a leftover dependency that's not used anymore.

* offchain: Upgrade hyper to 0.13, which uses tokio 0.2

* offchain: Adapt HTTP tests to Tokio 0.2

* network: Don't transitively include tokio 0.2 in WASM

1) We don't specifically depend on Tokio codec impls
2) Conflating features in Cargo means that enabling Tokio runtime
in the native environment will also do so in WASM, where it's
obviously not implemented and causes a compilation error.

* grafana-data-source: Pull hyper/tokio only in native environment

* babe_epochAuthorship
remove test-helpers from sp-keyring, bump spec_version, impl_version

* bump Cargo.lock

* add BabeRPC to node-rpc

* rename to BabeApi, remove err_derive

* pass &ServiceBuilder to with_rpc_extensions callback

* sc-consensus-babe-rpc

* Update client/consensus/babe/src/lib.rs

Co-Authored-By: Tomasz Drwięga <[email protected]>

* Better docs, code style chanegs

Co-Authored-By: André Silva <[email protected]>

* new line at the end of Cargo.toml

Co-authored-by: Tomasz Drwięga <[email protected]>
Co-authored-by: André Silva <[email protected]>

* Revalidation queeue.

* add docs and license

* move test

* refactor worker to async/await

* address review

* fix warnings

* update Cargo.lock

* move background task to service

* use tomusdrw loop

* naming

* return From::from

* add doc comment

* add more doc comments

* fix merge bug

* add doc comment for test function

* Update client/transaction-pool/src/testing/pool.rs

Co-Authored-By: Tomasz Drwięga <[email protected]>

* more review fixes

* refactor to allow service keep background tasks from isntantiated subsystems

* use const delay

* fix fallout

* remove fallout

* remove already moved test

* fix doc test

* add valid_at helper

Co-authored-by: Tomasz Drwięga <[email protected]>

* Update libp2p to 0.16

* Actually update to libp2p 0.16 🤦

* Fix missed updates

* Fix peerset tests

* adds finalization support to sc-transaction-pool using MaintainedTransactionPool for finalization events

* adds TransactionStatus::Retracted, notify watchers of retracted blocks, finalized now finalizes, transactions for current finalized -> last finalized block

* adds last_finalized to ChainApi, use generic BlockT for ChainEvent

* fix tests

* Apply suggestions from code review

Co-Authored-By: Tomasz Drwięga <[email protected]>

* tests

* fix tests, docs, lazily dedupe pruned hashes

* fix tests, Cargo.lock

* Apply suggestions from code review

Co-Authored-By: Tomasz Drwięga <[email protected]>

* remove tree_route, last_finalized from ChainApi, add block hash to Finalization and Retracted events

* prune finality watchers

* fix tests

* remove HeaderBackend bound from FullChainApi

* code style nits, terminate stream in finality_timeout

Co-authored-by: Tomasz Drwięga <[email protected]>

* Basic account composition.

* Add try_mutate_exists

* De-duplicate

* Refactor away the UpdateBalanceOutcome

* Expunge final UpdateBalanceOutcome refs

* Refactor transfer

* Refactor reservable currency stuff.

* Test with the alternative setup.

* Fixes

* Test with both setups.

* Fixes

* Fix

* Fix macros

* Make indices opt-in

* Remove CreationFee, and make indices opt-in.

* Fix construct_runtime

* Fix last few bits

* Fix tests

* Update trait impls

* Don't hardcode the system event

* Make tests build and fix some stuff.

* Pointlessly bump runtime version

* Fix benchmark

* Another fix

* Whitespace

* Make indices module economically safe

* Migrations for indices.

* Fix

* Whilespace

* Trim defunct migrations

* Remove unused storage item

* More contains_key fixes

* Docs.

* Bump runtime

* Remove unneeded code

* Fix test

* Fix test

* Update frame/balances/src/lib.rs

Co-Authored-By: Shawn Tabrizi <[email protected]>

* Fix ED logic

* Repatriate reserved logic

* Typo

* Fix typo

* Update frame/system/src/lib.rs

Co-Authored-By: Shawn Tabrizi <[email protected]>

* Update frame/system/src/lib.rs

Co-Authored-By: Shawn Tabrizi <[email protected]>

* Last few fixes

* Another fix

* Build fix

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Jaco Greeff <[email protected]>
Co-authored-by: Shawn Tabrizi <[email protected]>

* pallet-evm: add support for transaction-level create2

* Bump runtime version

* Switch to FunctionOf for weights

* Migrate wasmtime backend to wasmtime-api

* Port to a newer version of wasmtime

* Update to the latest changes.

* Rejig the sandbox module a bit

* Materialze

* Fixes.

* executor wasm_runtime fix

* Refactor everything

* More refactoring

* Even more refactorings

* More cleaning.

* Update to the latest wasmtime

* Reformat

* Renames

* Refactoring and comments.

* Docs

* Rename FunctionExecutor to host.

* Imrpove docs.

* fmt

* Remove panic

* Assert the number of arguments are equal between wasmtime and hostfunc.

* Comment a possible panic if there is no corresponding value variant.

* Check signature of the entrypoint.

* Use git version of wasmtime

* Refine and doc the sandbox code.

* Comment RefCells.

* Update wasmtime to the latest-ish master.

This may solve a problem with segfaults.

* Apply suggestions from code review

Co-Authored-By: Tomasz Drwięga <[email protected]>

* Use full SHA1 hash of wasmtime commit.

* Add a panic message.

* Add some documentation

* Update wasmtime version to include SIGSEGV fix

* Update to crates.io version of wasmtime

* Make it work.

* Move the creation of memory into `InstanceWrapper::new`

* Make `InstanceWrapper` !Send & !Sync

* Avoid using `take_mut`

* Update client/executor/wasmtime/Cargo.toml

Co-Authored-By: Bastian Köcher <[email protected]>

* Limit maximum size of memory.

* Rename `init_state` to `with_initialized_state`

Co-authored-by: Tomasz Drwięga <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* full block import benchmark

* try rocksdb cache

* add profiling helper

* use random keyring instead of zero caching

* update docs

* add more io stats

* remove last sentence

* add ci job to see

* Update primitives/keyring/src/sr25519.rs

Co-Authored-By: Marcio Diaz <[email protected]>

* switch to 100tx-block

* remove ci script

Co-authored-by: Marcio Diaz <[email protected]>

* network: Use "one shot" protocol handler.

Add two new `NetworkBehaviour`s, one handling remote block requests
and another one to handle light client requests (both local and from
remote). The change is motivated by the desire to use multiple
substreams of a single connection for different protocols. To achieve
this, libp2p's `OneShotHandler` is used as a protocol handler in each
behaviour. It will open a fresh substream for the duration of the
request and close it afterwards. For block requests, we currently only
handle incoming requests from remote and tests are missing. For light
client handling we support incoming requests from remote and also
ported a substantial amount of functionality over from
`light_dispatch.rs` (including several tests). However the result lacks
in at least two aspects:

(1) We require external updates w.r.t. the best block per peer and
currently nothing updates this information.
(2) We carry a lot of peer-related state around.

Both aspects could be simplified by externalising peer selection and
just requiring a specific peer ID where the request should be sent to.
We still have to maintain some peer related state due to the way
libp2p's swarm and network behaviour work (e.g. we must make sure to
always issue `NetworkBehaviourAction::SendEvent`s to peers we are
connected to, otherwise the actions die a silent death.

Another change implemented here is the use of protocol buffers as the
encoding for network messages. Certain individual fields of messages
are still SCALE encoded. There has been some discussion about this
in another PR (https://github.com/paritytech/substrate/pull/3452), so
far without resolution.

* Uncomment `Behaviour::light_client_request`.

* Add license headers.

* Add command-line flag to enable yamux flow control.

We never enabled proper flow-control for yamux streams which may cause
stream buffers to exceed their configured limit when the stream
producer outpaces the stream consumer. By switching the window update
mode to on-read, producers will only receive more sending credit when
all data has been consumed from the stream buffer. Using this option
creates backpressure on producers. However depending on the protocol
there is a risk of deadlock, if both endpoints concurrently attempt to
send more data than they have credit for and neither side reads before
finishing their writes. To facilitate proper testing, this PR adds a
command-line flag `use-yamux-flow-control`.

* Replace comment with generic message.

* Add a sub command to generate a node key file in the format required by a substrate node

* Update lock file

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <[email protected]>

* Updates as per code review

Co-authored-by: Bastian Köcher <[email protected]>