Improve Benchmark Writer: Remove Unused Components, Remove Multiply by Zero, Files Split by Pallet (#6785)

* initial improvements

* better file management, ignore unused components

* Output warning when components unused

* update comment

* Write even when base weight is zero

* remove unwrap where possible

* Dont sort components to dedup

* undo delete

* improve clarity of unused components

* remove unused dep

* Update Process.json

* Allow blacklisting blocks from being finalized again after block revert

* Use BlockRules for storing unfinalized and add have_state_at in revert

* Move finalization_check in finalize_block upward

* Directly mark finalization blacklist as badblocks

* Remove obselete comment

* BabeWorker -> BabeSlotWorker

* SlotWorker::notify_slot: similar to claim_slot, but called no matter authoring

* Wrap the future with a new struct BabeWorker

* Add type definition slot_notification_sinks

* Function slot_notification_streams for the receiver side

* Get a handle of slot_notification_sinks in BabeSlotWorker

* Implement notify_slot

* Switch to use bounded mpsc

* Do not drop the sink when channel is full

Only skip sending the message and emit a warning, because it is recoverable.

* Fix future type bounds

* Add must_use and sink type alias

* pallet-evm: add builtin support for the four basic Ethereum precompiles

* linear_cost -> ensure_linear_cost to directly return OutOfGas error

* Add DefaultQueue

* Add DefaultImportQueue to the top level of sp-consensus

* Add memory tracker feature to sp-trie to fix wasm panic

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>

* seal: Fix syntax that confuses rust-analyzer

* seal: Add benchmarks for Dispatchables

These are only the benchmarks for the dispatchables of
the pallet. Those are not listed in the Schedule because
we do not want to pull the Schedule from storage before
dispatching.

This OK because those costs are not related to actual contract
execution. Those costs (instruction costs, ext_* costs) will
be benchmarked seperatly and entered into the default Schedule.

* seal: Add a maximum code size

* Fix comments from review

* Removed SEED constant

This updates parity-scale-codec{-derive} to prepare for a rustc release
that would otherwise break the derive implementation:
https://github.com/rust-lang/rust/pull/73084

* Remove unused node template deps

Backport changes made by @c410-f3r
https://github.com/substrate-developer-hub/substrate-node-template/pull/66

* Enhancements to README

* Revert change to serde per @thiolliere

* Remove service components and add build_network, build_offchain_workers etc

* Improve transaction pool api

* Remove commented out line

* Add PartialComponents

* Add BuildNetworkParams, documentation

* Remove unused imports in tests

* Apply suggestions from code review

Co-authored-by: Nikolay Volf <[email protected]>

* Remove unused imports in node-bench

Co-authored-by: Nikolay Volf <[email protected]>

* pallet-evm: add support for tuple-based precompile declarations

* Add missing license header

* Switch to use impl_for_tuples

* Remove unnecessary impl for ()

* Remove any implementation of `Spawn` or `Executor` from our task executors

* Fix compilation

* Rename `SpawnBlockingExecutor`

* Update primitives/core/src/traits.rs

Co-authored-by: Kian Paimani <[email protected]>

* Fix tests

Co-authored-by: Kian Paimani <[email protected]>

* Switching from git back to released versions for wasmtime

* filter out cratelift_codegen messages-a

Co-authored-by: NikVolf <[email protected]>

* Better handling of stable-only build

* Fix node template build

* Fix wasm builder node-template version mismatch

* Fix load_spec error

* Add , in parameter

* Add descrptive panic messages in tests

* Add descriptive tests in node/executor benches

* Fix missing compact_code_unwrap

* Add missing wasm_binary_unwrap function for executor integration test

* Only define import_sp_io in no_std

* Small Cargo.toml styling fix

* Bump wasm-builder to 2.0.0

* Fix all `with_wasm_builder_from_crates` version in Substrate

* Use `with_wasm_builder_from_crates` for node-template

Co-authored-by: Gavin Wood <[email protected]>

* :)

* Slight tidy

* Remove ServiceBuilderCommand

* Remove whitespace

* Keep task manager alive for check_block/import_blocks

* Pass task_manager to run_until_exit

* WIP

* WIP

* Get rid of the macros

* Simplify a few chain components creation APIs related to the service

* Fix basic-authorship doc tests

* Remove DefaultQueue

* Update client/service/src/builder.rs

Co-authored-by: André Silva <[email protected]>

* Move ExecutionExtensions comment around

* Remove unused BlakeTwo256

* Add sc-prelude

* Rename sc-prelude to sc-service-prelude

* Rename to sc-service-types

* Improve service types

* Fix line widths

* Remove sc-service-types and move type definitions to crates

* Update bin/node-template/node/src/service.rs

Co-authored-by: Seun Lanlege <[email protected]>

* Add TLightClientWithHash

* Rework types

Co-authored-by: André Silva <[email protected]>
Co-authored-by: Seun Lanlege <[email protected]>

* update wasmtime to (almost) lastest master

* bump lock

* bump once more

* Ensure we close old file handles hold by sysinfo

* Dropping is needed unfortunately

* enable process refreshing, ignore result from refresh_process

* jumping to proposed patch

* switch to latest sysinfo

* Initial commit

Forked at: 60e3a693


Parent branch: origin/master

* Move task_manager.rs to mod.rs

* Graceful shutdown for the task manager

* Await all background task JoinHandle at the same time

* Add tests

* Make future() wait also for exit signal + fix essential task failed

Probably related to https://github.com/paritytech/cumulus/issues/111

* add comments for non-obvious code

* Use clean_shutdown() in sc-cli

* Adapt code and upgrade tokio in sc-cli

* cleanup spacing in doc

* Add license

* I guess actually running the clean shutdown would be a good idea

* fix tests

* Update client/cli/src/runner.rs

Co-authored-by: Benjamin Kampmann <[email protected]>

* Improve error logging

* disable other tests (can't reproduce on my machine)

* Revert "disable other tests (can't reproduce on my machine)"

This reverts commit c133c590.

* It is possible that the tasks are ended first

* Revert "It is possible that the tasks are ended first"

This reverts commit 502aba4a.

* Use single threaded scheduler for more predictability

* enable_time

* Revert "enable_time"

This reverts commit 4e152140.

* Revert "Use single threaded scheduler for more predictability"

This reverts commit ee5e13c5.

* Revert "Revert "It is possible that the tasks are ended first""

This reverts commit 1b91a8ca

.

* This cannot be verified either with a threaded pool

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Benjamin Kampmann <[email protected]>
Co-authored-by: Benjamin Kampmann <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* Bumping to rc5

* Revert "Bumping to rc5"

This reverts commit 3c6767db.

* Bumping for rc5, correctly this time

* Update libp2p to 0.22.

Primarily to begin the libp2p-noise upgrade process.

See https://github.com/libp2p/rust-libp2p/issues/1631.

* CI

* Fix compilation problems due to ed25519-dalek 1.0.0-pre.4

* Update libp2p-core

* Revert rand downgrade

* txpool benchmarks

* updated api

* Update bin/node/bench/src/txpool.rs

Co-authored-by: Kian Paimani <[email protected]>

Co-authored-by: Kian Paimani <[email protected]>

* grandpa: use unsigned extrinsics for equivocation reporting

* grandpa: allow signed reports as well

* grandpa: change runtime api for submitting unsigned extrinsics

* grandpa: fix tests

* grandpa: add test for unsigned validation

* grandpa: add benchmark for equivocation proof checking

* offences: fix grandpa benchmark

* grandpa: add proper weight for equivocation reporting extrinsic

* grandpa: fix weight unit

* Switch to bounded mpsc for txpool import notification stream

* Update client/transaction-pool/graph/src/validated_pool.rs

Co-authored-by: Nikolay Volf <[email protected]>

Co-authored-by: Nikolay Volf <[email protected]>

* sc-offchain: upgrade hyper-rustls to 0.18

* Cargo.lock: revert rand downgrade due to rust-lang/cargo/issues/5702

* pallet-evm: return Ok(()) when EVM execution fails

* Bump spec version

* Implement apply_state flag and allow fetching return data and used gas

* Update evm version

This reverts commit 802a0d0b.

* Simplify a few chain components creation APIs related to the service

* Fix basic-authorship doc tests

* Remove DefaultQueue

* Update client/service/src/builder.rs

Co-authored-by: André Silva <[email protected]>

* Move ExecutionExtensions comment around

* Remove unused BlakeTwo256

Co-authored-by: André Silva <[email protected]>

* Transition getter functions to not use scratch buffer

* Remove scratch buffer from ext_get_storage

* Remove scratch buffer from ext_call

* Remove scratch buffer from ext_instantiate

* Add ext_input and remove scratch buffer

* Rework error handling (changes RPC exposed data)

* ext_return passes a flags field instead of a return code
	* Flags is only for seal and not for the caller
	* flags: u32 replaced status_code: u8 in RPC exposed type
* API functions use a unified error type (ReturnCode)
* ext_transfer now traps on error to be consistent with call and instantiate

* Remove the no longer used `Dispatched` event

* Updated inline documentation

* Prevent skipping of copying the output for getter API

* Return gas_consumed from the RPC contracts call interface

* Updated COMPLEXTITY.md

* Rename ext_gas_price to ext_weight_to_fee

* Align comments with spaces

* Removed no longer used `ExecError`

* Remove possible panic in `from_typed_value`

* Use a struct as associated data for SpecialTrap::Return

* Fix nits in COMPLEXITY.md

* Renamed SpecialTrap to TrapReason

* Fix test

* Finish renaming special_trap -> trap_reason

* Remove no longer used get_runtime_storage

* fixup! Remove no longer used get_runtime_storage

* Removed tabs for comment aligment

Fixes #6305

* *Cargo.toml: Update versions

* client/network/src/discovery: Adjust to Kademlia  API changes

* client/network: Adjust to one_shot.rs changes

* client/network/discovery: Log address list on trace level

* client/network/discovery: Ignore RoutablePeer and PendingRoutablePeer

* Commit Cargo.lock

* Finish update

Co-authored-by: Pierre Krieger <[email protected]>

* Use flexi_logger; Add log rotation

* Add default rotation; Add FlexiLogger error

* Fix compilation error

* Remove logging to stdout if it's not a tty

* Fix formatting

Co-authored-by: Kian Paimani <[email protected]>

* Remove needless debug statement

* Default to unlimited size for log rotation

* Add more comments about log-age option

* Remove unused variable

* Fix typo in comment

Co-authored-by: Kian Paimani <[email protected]>

* Upgrade `kvdb-*`, `trie-db` and `memory-db`

The updates of `trie-db` and `memory-db` are important, as they fix the
non-deterministic build of Polkadot/Substrate.

* Change `trie-db` version

* Update test-utils/runtime/Cargo.toml

Co-authored-by: Andronik Ordian <[email protected]>

* Update primitives/trie/Cargo.toml

Co-authored-by: Andronik Ordian <[email protected]>

* Update `Cargo.lock` and `trie-bench`

* Fix UI tests

* Switch to fixed version of memory-db

Co-authored-by: Andronik Ordian <[email protected]>

* Use async/await in build_network_future

* Address concerns

* Fix test