* WIP: read child trie and write child trie

* add test

* refactor a bit + improve log

* better naming

* trigger CI

* Revert "trigger CI"

This reverts commit d0aadaeb.

* Generate CHT roots on a full client

* add changes_trie_root function

* Add a test

* Line widths

* Fix sc-service-test

* Clarify comments

* Revert comments

* Reset code, almost ready for PR

* Improved build_hardcoded_spec

* Fix line widths

* Fix tests

* Fix sc-service-test

* Suggestions from code review

* Rename to LightSyncState

* It's not syncing :^(

* It syncs!

* Remove rpc call

* Convert spaces to tabs

* Moved sc-service things to export_sync_state.rs

* Fix tests

* Wait for syncing with network_status_sinks

* Remove sc-network from node-template

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* Various changes, split the flag up into 2 pieces to make testing easier.

* Update client/cli/src/commands/build_spec_cmd.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Revert a lot of changes

Co-authored-by: Bastian Köcher <[email protected]>

* Add `get_whitelist` api

* add whitelisted caller

* Whitelist caller

* remove caller 0

* initial piping of origin (not actual value yet)

* remove attempt to pass origin around

* Add whitelist for `DidUpdate` storage on `pallet_timestamp`

* fix traits

* only add to whitelist if !contains

* PassBy not implemented error

* Whitelist read/writes explicitly per key

* update docs

* reduce trait constraint

* copy pasta

* Apply suggestions from code review

Co-authored-by: Guillaume Thiolliere <[email protected]>
Co-authored-by: Alexander Popiak <[email protected]>

* rename functions @apopiak

* missed some renaming

* enable doc tests

* Update docs

Co-authored-by: Guillaume Thiolliere <[email protected]>
Co-authored-by: Alexander Popiak <[email protected]>

* Allow blacklisting blocks from being finalized again after block revert

* Use BlockRules for storing unfinalized and add have_state_at in revert

* Move finalization_check in finalize_block upward

* Directly mark finalization blacklist as badblocks

* Remove obselete comment

* Improve overall performance

* Clean up code

Co-authored-by: Bastian Köcher <[email protected]>

* Remove needless ::

Co-authored-by: Bastian Köcher <[email protected]>

* Remove needless ::

Co-authored-by: Kian Paimani <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>

Fixes #6305

* primitives: use generic Header in testing runtime

* frame: remove unused imports

* Remove warning

Co-authored-by: Bastian Köcher <[email protected]>

* initial mockup

* add and wipe

* track writes

* start to add to pipeline

* return all reads/writes

* Log reads and writes from bench db

* causes panic

* Allow multiple commits

* commit before ending benchmark

* doesn't work???

* fix

* Update lib.rs

* switch to struct for `BenchmarkResults`

* add to output

* fix test

* line width

* @Kianenigma

 review

* Add Whitelist to DB Tracking in Benchmarks Pipeline (#6405)

* hardcoded whitelist

* Add whitelist to pipeline

* Remove whitelist pipeline from CLI, add to runtime

* clean-up unused db initialized whitelist

* Add regression analysis to DB Tracking (#6475)

* Add selector

* add tests

* debug formatter for easy formula

* Update client/db/src/bench.rs

Co-authored-by: Kian Paimani <[email protected]>

Co-authored-by: arkpar <[email protected]>
Co-authored-by: Kian Paimani <[email protected]>

* Initial commit

Forked at: d735e4d0


No parent branch.

* Make sc_service::Configuration derive Debug

* Replace task_executor fn's input by proper TaskExecutor type (cleaner)

* impl From<Fn> for TaskExecutor

* Update client/cli/src/runner.rs

* Add some doc, examples and tests

* Replace Deref by fn spawn as suggested

Co-authored-by: Bastian Köcher <[email protected]>

* draft

* steps

* chore: fmt

* step by step

* more details

* make test public

* refactor: split into on and offchain

* test stab

* tabs my friend

* offchain overlay: split key into prefix and true key

Simplifies inspection and makes key actually unique.

* test: share state

* fix & test

* docs improv

* address review comments

* cleanup test chore

* refactor, abbrev link text

* chore: linewidth

* fix prefix key split fallout

* minor fallout

* minor changes

* addresses review comments

* rename historical.rs -> historical/mod.rs

* avoid shared::* wildcard import

* fix: add missing call to store_session_validator_set_to_offchain

* fix/compile: missing shared:: prefix

* fix/test: flow

* fix/review: Apply suggestions from code review

Co-authored-by: Tomasz Drwięga <[email protected]>

* fix/review: more review comment fixes

* fix/review: make ValidatorSet private

* fix/include: core -> sp_core

* fix/review: fallout

* fix/visbility: make them public API

Ref #6358

* fix/review: review changes fallout - again

Co-authored-by: Bernhard Schuster <[email protected]>
Co-authored-by: Tomasz Drwięga <[email protected]>

* make all features explicit

* Change to -feature suffix to with- prefix

* Add newline at the end of the Cargo.toml file

* Remove rhd feature

* Remove some features from Cargo.toml

* Remove test-helpers feature in tx pool

* Return db_open_error("with-"..

Co-authored-by: Bastian Köcher <[email protected]>

* Rename subdb feature to with-subdb

Co-authored-by: Bastian Köcher <[email protected]>

* Remove 'strict' feature and cfg_attr

* Check for with-subdb feature instead of subdb

Co-authored-by: Bastian Köcher <[email protected]>

* feat/offchain/storage: add remove interface method

* feat/offchain/storeage: add remote to StorageValueRef

* feat/offchain/storage: add storage lock

* fix/review: Apply suggestions from code review

Co-authored-by: Tomasz Drwięga <[email protected]>
Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* refactor/offchain/storage/lock: introduce `Lockable` trait part 1 of 2

* chore/offchain/rename: _remove -> clean

* feat/offchain/storage/lock: add TimeAndBlock based part 2 of 2

* fix/offchain/storage/lock: block and time expiry must be && not ||

* chore/offchain/storage: minor fmt doc comments

* doc/comment: prefer markdown emphasis over CAPS

* doc/comment: rewrap multiline module level docs

* doc/comment: rephrase

* impl sleep_until and use the actual time for the test env

* feat/test: add more tests, ignore some sample impl doctests

* fix/review: Apply suggestions from code review

Co-authored-by: Nikolay Volf <[email protected]>

* doc/comment: better description

* fix/review: Apply suggestions from code review

Co-authored-by: Nikolay Volf <[email protected]>

* chore/storage: lifetime cleanup

* fix/cleanup: trait bounds, cargo-spellcheck + extra explanations

* fix/doc: periods +-

* fix/review: Apply suggestions from code review

Co-authored-by: Tomasz Drwięga <[email protected]>

* cleanup: remove explicit lifetime bound, copy -> clone

* fix/review: make trait Lockable contain only static, try_lock should not return Err(Option<L>),

* chore/lifetimes: remove a couple of lifetime bounds which the compiler can figure out

* refactor: migrate to an instant based

* fix/feedback: fix, reduce, rename, docs update pending

* docs/reword: adjust to changed code

* fix/offchain/testing: timestamp and sleep_until shall not block

* chore/lines: lines must < 100 chars

* fix/docs: add missing pub field doc comments

* refactor/x: try_lock does not need to return an Option<_>

* refactor/simplify: a better way of waiting for a lock to resolve

* docs: consistency

* fix/line: < 100

* fix/doctest/use: avoid crate::

* fix/doctest: *

* fix/review: remove unused trait bound

* fix/review: pretty by const fn

* fix/review: reduce default timeout to 20s

* docs: grammar

* fix/review: add with_block_deadline

* doc: revamp BlockNumberProvider documentation to be less frame centric

* chore: fmt

* docs: add missing doc comment

Co-authored-by: Bernhard Schuster <[email protected]>
Co-authored-by: Tomasz Drwięga <[email protected]>
Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>
Co-authored-by: Nikolay Volf <[email protected]>

* Don't clone values when calculating storage root

Instead of cloning all the keys and values of the overlay when
calculating the storage root, we pass all the values by reference. This
should probably bring some performance improvements when calculating the
storage root.

* no cow version (#6113)

Co-authored-by: cheme <[email protected]>

* Use header cache for state_at

* Fixed warnings

* Update client/db/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>

* purge-chain accepts --db option

* print DB info on startup

* Small refactoring

* Added back &self

* Add DatabaseParams for PurgeChain, ImportParams and ExportBlocks

* Don't force default value

* Remove unused fields

* Update client/cli/src/commands/export_blocks_cmd.rs

* Fix stuff

Co-authored-by: Cecile Tonglet <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* Drop client from sc-network and sc-client-db, move LongestChain to sc-client-api

* move leaves, cht, in_mem to sc-client-api, drop client from sc-finality-grandpa

* drop sc-service from sc-rpc

* drop sc-service from sc-consensus-aura

* drop sc-client from manual-seal and babe

* drop sc-client from utils/frame/rpc/system and utils/frame/benchmarking-cli

* drop sc-client from bin/node and bin/node-template

* drop sc-client

* fix tests

* remove check -p sc-client from gitlab.yml

* fix warnings

* fixes ui test

* fix light client tests

* adds associated Client type to AbstractService

* adds UsageProvider to Client

* fixed ui test, again

* tried and failed to get node-cli to compile for wasm

* thanks to tomaka for helping me get node-cli to compile for wasmm

* ui test pls pas 🙏🏾



* all tests passing 🪄

* no_run documentation code

* rm -f documentation code

* ClientProvider

* fix mega trait

* move LongestChain to sc-consensus, use adds minimal bounds to AbstractService::Client

* adds license to sc-consensus

Co-authored-by: Benjamin Kampmann <[email protected]>

* Optimize pinning

* Ref counting state backend

* Style

Co-Authored-By: Wei Tang <[email protected]>

* Update Cargo.lock

* Handle empty node

Co-authored-by: Wei Tang <[email protected]>

Co-Authored-By: Kian Paimani <[email protected]>
Co-Authored-By: Tomasz Drwięga <[email protected]>

Co-Authored-By: thiolliere <[email protected]>

* Introduce trait

* The trait

* Generic

* Basic impls.

* Remove unneeded bounds

* Minor changes

* Switch over to the new DB trait

* Integrated parity-db and added CLI for db selection

* Default impl.

* Fix logs.

* Started integrating subdb

* Apply suggestions from code review

Co-Authored-By: Cecile Tonglet <[email protected]>

* Apply suggestions from code review

Co-Authored-By: Nikolay Volf <[email protected]>

* Enable subdb

* Bump parity-db

* Fixed CLI macro

* Fixed browser build

* Fixed features

* Sort out features

* Use parity-db from crates.io

* Typo

Co-authored-by: arkpar <[email protected]>
Co-authored-by: Cecile Tonglet <[email protected]>
Co-authored-by: Nikolay Volf <[email protected]>

* in-mem state for benchmarks

* Use caching state

* Update Cargo.lock

Co-authored-by: Shawn Tabrizi <[email protected]>

* 128 -> 1024

* add trace and set = 512

* fix doc comment

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Rename IntoConfiguration to CliConfiguration

* Renamed into_configuration to create_configuration

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Move keystore params to its own module

* Use in-memory keystore even for build-spec

* Enforce proper value for node name

* dev_key_seed

* Telemetry endpoints

* rustfmt

* Converted all RunCmd

* rustfmt

* Added export-blocks

* Missed something

* Removed config_path in NetworkConfiguration (not used)

* Fixed warnings

* public_addresses is used but never set, keeping it

* Merge Configuration.node and NetworkConfiguration.node_name

...because they are the same thing

* Added: import-blocks

* Adding a proc_macro to help impl SubstrateCli

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Re-export spec_factory from sc_cli

* Re-added all the commands

* Refactored node_key_params

* Fixed previous refucktoring

* Clean-up and removed full_version()

* Renamed get_is_dev to not confuse with Configuration field

* Fixed sc-cli-derive example

* Fixing tests

* Fixing tests and removing some (will re-add later)

* Fixing more tests

* Removes the need of type parameter

* Converting bin/node and simplifying API

* Converting more

* Converting last command

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Fixing tests and added default for WasmExecutionMethod

* Fixing stuff

* Fixed something I broke oops

* Update Cargo.lock

* Moving things around

* Convert everything to Result

* Added new macros to simplify the impl of CliConfiguration

* Added a macro to generate CliConfiguration automatically for subcommands

* Revert... too many macros (this one is not really useful)

This reverts commit 9c516dd3

.

* Renamed is_dev to get_is_dev

Good enough for now

* Fixed name roles (this is plural, not singular)

* Clean-up

* Re-export NodeKeyConfig and TelemetryEndpoints from sc_service

* Improve styling/formatting

* Added copyrights

* Added doc and fixed warnings

* Added myself to code owners

* Yes it is needed according to the history

* Revert formatting

* Fixing conflict

* Updated build.rs

* Cargo.lock

* Clean-up

* Update client/cli-derive/Cargo.toml

Co-Authored-By: Seun Lanlege <[email protected]>

* Fail if using proc_macro and build.rs is not set properly

* Dropped all get_ in front of methods

* Clean-up

* Fixing proc macro missing env var

* Get the configuration inside the Runtime (needed for polkadot)

* Clean-up

* Get is_dev from argument like the others

* Get chain ID instead of chain spec from shared params

* &self is passed to spec_factory/load_spec

* Wrong text

* Fix example

* Officialize macro and made a cool doc

* Renamed spec_factory to load_spec (substrate_cli_configuration)

* Removed not so useful ChainSpec

* Renamed SubstrateCLI to SubstrateCli

* Added changelog for impl_version being full now

* Renamed Runtime to Runner

* Update changelog to show example

* Removed option on database cache size

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Fix on removal of option

* typo

* Clean-up imports

* Added info in Cargo.toml

* typo

* remarks

* Moved function for build.rs to substrate-build-script-utils

* Fixed example & test of cli-derive

* Moved function for build.rs to substrate-build-script-utils

* Renamed substrate_cli_configuration to substrate_cli oops

It implements SubstrateCli not CliConfiguration!

* Added documentation and wrapper macro

* Removed option on database cache size

* Removed option on database cache size

* Clean-up

* Reduce risk of errors due to typos

* Removed option on database cache size

* Added NOTE as suggested

* Added doc as suggested

* Fixed test

* typo

* renamed runtime to runner

* Fixed weird argument

* More commas

* Moved client/cli-derive to client/cli/derive

* Added 7 tests for the macros

* Improve error message

* Upgrade assert_cmd

* Fixing missing stuff

* Fixed unused import

* Improve SubstrateCli doc

* Applied suggestions

* Fix and clean-up imports

* Started replacing macros WIP

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* Started removing substrate_cli

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81
Parent branch: origin/master

* WIP

Forked at: 2afecf81


Parent branch: origin/master

* fixed bug introduced while refactoring

* Renamed NetworkConfigurationParams to NetworkParams for consistency sake

* Fixed test

* Update client/cli/src/commands/runcmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/runcmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/check_block_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update bin/node/cli/src/command.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update bin/node/cli/src/command.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Revert "Update client/cli/src/commands/export_blocks_cmd.rs"

This reverts commit 59067769.

* Revert "Update client/cli/src/commands/check_block_cmd.rs"

This reverts commit f705f42b.

* Revert "Update client/cli/src/commands/export_blocks_cmd.rs"

This reverts commit 8d57c055.

* Revert "Update client/cli/src/commands/runcmd.rs"

This reverts commit 93e74cf5.

* Revert "Update client/cli/src/commands/runcmd.rs"

This reverts commit 11d527ba

.

* Update client/cli/src/commands/export_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/import_blocks_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Changed ::sc_cli to $crate in the macro

* fixed tests

* fixed conflicts

* Fixing test

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/params/pruning_params.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Remove comment as suggested

* Apply suggestion

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/purge_chain_cmd.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update utils/frame/benchmarking-cli/src/command.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/runner.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/params/pruning_params.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/params/node_key_params.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/params/network_params.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/cli/src/config.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Added doc

* Fixed error introduced after applying suggestion

* Revert "Update client/cli/src/params/pruning_params.rs"

This reverts commit 0574d06a

.

* Print error

* Apply suggestions from code review

* Remove useless Results

* Fixed CI failing on polkadot approval

Co-authored-by: Seun Lanlege <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

This PR refactors the metrics measuring and Prometheus exposing entity in sc-service into its own submodule and extends the parameters it exposes by:

- system load average (over one, five and 15min)
- the TCP connection state of the process (lsof), refs #5304
- number of tokio threads
- number of known forks
- counter for items in each unbounded queue (with internal unbounded channels)
- number of file descriptors opened by this process (*nix only at this point)
- number of system threads (*nix only at this point)

refs #4679

Co-authored-by: Max Inden <[email protected]>
Co-authored-by: Ashley <[email protected]>

* Registering state from overlay.

* fix

* fix2

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* Reset by removing pos-genesis changes

* CLI option for DB cache size

* Update Cargo.lock

Co-authored-by: Shawn Tabrizi <[email protected]>

* ChainSpec trait

* Apply suggestions from code review

Co-Authored-By: Bastian Köcher <[email protected]>

* Added docs

* Fixed build

* Fixed build

Co-authored-by: Bastian Köcher <[email protected]>

* Remove `Backend::destroy_state`

This removes the `destroy_state` function of `Backend` and instead moves
the functionality into the `Drop` implementation of the state. This
makes it much easier to work with the state, as the user no longer needs
to call` destroy_state` manually. However, it requires that we switch
from `RwLock` to `ReentrantMutex` as while importing a block we maybe
need to lock again in `drop`.

* Bring back the `RwLock` and some other clean ups

* Fix compilation

* Add a few metrics to Client

* Improve PrometheusConfig

* Fix client docs