* forks support in changes trie storage

* moved convert_hash to primitives

* optimize ChangesTrieRootsStorage::root when anchor is canonicalized

docs: Extends #999. Remove disused package readmes. Rename to sr-metadata. Fix Readme Rustdoc names (#1016)

* docs: Remove safe-mix since is no longer local dependency

* docs: Fix existing packages link. Still only includes Substrate Core

* docs: Remove empty package Readmes. Remove old docs links. Rename to Readme

* docs: Remove link to old docs

* misc: Merge latest from master. Renaem to sr-metadata. Fix list of RustDoc package names

* Rename substrate-metadata to sr-metadata, since it is in the "srml" subdirectory

* Change example to use package name that opens all packages in RustDocs

* Fix list of packages names that are available to open in RustDocs

* fix typo

* fix ordering of CLI options and add missing backslash

* Remove superfluous serde requirements.

* Try to ensure hash is serde

* Fixups

* Building again

* Attempt to reenable Block (doesn't build)

* Fixes compilation for node cli

* Fixes test compilation

* Fix wasm

* Fix tests

* Remove unneeded changes

* Fix up comments

* Reenable some code

* Compile error when origin misused.

* Remove unnecessary includes of `serde_derive`

* Cleanups

* [core/primitives] Update crunchy 0.1 => 0.2

* [core/primitives] Update fixed-hash 0.2 => 0.3.0-beta

* [core/primitives] Update `uint` 0.4 => 0.5.0-beta

* [core/state-machine] Fix errors emerged by fixed-hash update

* [core/state-db] Fix errors that emerged from fixed-hash update

* [core/sr-io] Fix errors that emerged from the fixed-hash update

* [core/trie] Fix errors emerged from fixed-hash updates

* [core/trie] Make use of new Hash::as_fixed_bytes_mut method in tests

* [core/sr-primitives] Fix errors emerged from updating fixed-hash

* [core/executor] Fix errors that emerged from fixed-hash update

* [core/test-runtime] Fix errors that emerged from updating fixed-hash

* [core/test-runtime] Fix an error that emerged from fixed-hash update

* [core/transaction-pool] Fix errors that emerged from updating fixed-hash

* Add From<u64> impl for hash types defined in core/primitives

* [core/client/db] Fix errors that emerged from fixed-hash update

* [core/{client/network/rpc}]: Fix errors emerged by fixed-hash update

* [node/{cli/executor/runtime}]: Fix errors emerged by updating fixed-hash

* [core/network]: Fix bug in parsing constant str

* Update Cargo.lock

- Add crunchy 0.2.1
- fixed-hash 0.2.2 => 0.3.0-beta.3
- Add static-assertions 0.2.5
- uint 0.4.1 => 0.5.0-beta.1

* [core/primitives]: Add fixed-hash/rustc-hex feature for FromStr impl

* [core/primitives] No longer provide From<u64> impl for hash types if byteorder support is not enabled

* [core/primitives] Revert to using From impl again in primitives tests

* [core/..] Fix some bugs that emerged by recent fixed-hash updates

* Update a bunch of Cargo.lock files

* [core/state-db] Re-add whitespace between attr and extern crate/mod

* [core/primitives] Fix bug in From<u64> impl for hash types using the wrong feature guard

* Support multi trie in genesis generation

* Fix merge issues

* Generalize BlockImport

 - move ImportBlock, BlockOrigin, ImportResult into shared sr-primitives
 - let Consensus provide  and  traits again
 - update consensus traits to latest development
 - implement traits on client::Client, test_client::TestClient
 - update RHD to use the new import_block API

* Move ImportBlock into consensus-common
* Send import notification in aura tests
* Integrating aura into service
* Make Signatures more generic
* Aura Block Production with the given key
* run aura on the thread pool
* start at exact step start in aura
* Add needed wasm blob, in leiu of better solutions.
* Make API ids consistent with traits and bring upstream for sharing.
* Add decrease_free_balance to Balances module
* Encode `Metadata` once instead of two times
* Bitops include xor
* Upgrade key module.
* Default pages to somewhat bigger.
* Introduce upgrade key into node
* Add `Created` event

* docs: Remove safe-mix since is no longer local dependency

* docs: Fix existing packages link. Still only includes Substrate Core

* Implement a non generic version of child delta trie

* Use delta_trie_root in state_machine

* Expand InMemory backend to support multi-storage

* Create Consolidate trait

* Fix all crate compile and remove unused OverlayedChanges::drain

* Implement child storage root support and overlay changes

* Add child storage reader

* Add child storage writer

* Implement child storage cleaning

* Fix light backend compile

* Add all required ext functions for wasm executor

* Add ext def to io

* Add all io functions

* Fix nostd compile

* Add simple test

* Remove unnecessary vec copy in child_storage_root_transaction

* Use values_mut/for_each to make it shorter

* Use extend to shorter a for loop

* Move record_all_keys to trie so it's easier to generic them later

* space -> tab

* Remove to_owned in debug format

* Clean out all to_owned

* Break debug_trace to multiple lines

* Remove 0..

* UserError copy/paste typo

* Replace Vec::from_raw_parts by slice::from_raw_parts

* Use iter::empty()

* Wrap some long lines

* Wrap a missing line

* Remove unnecessary map

https://github.com/paritytech/substrate/pull/856#discussion_r226222663

* Call ext_free after from_raw_parts

* Fix tests in other crates

* Adds new `ProvideInherent` trait

Also implements the new trait for `srml/consensus` and `srml/timestamp`.

* Adds `impl_outer_inherent!` macro

* Reexport macros from `alloc`

* Introduce `RuntimeString` and fix `ProvideInherent` on `no_std`

* Replace `VersionString` with `RuntimeString`

* Improvements for `impl_outer_inherent!`

* Make `construct_runtime!` support `impl_outer_inherent!`

* Fixes after rebase

* Whitespace

* Revert "Revert runtime changes."

This reverts commit 01a7d1aa.

* Parallel queue reads.

* Avoid recursion in best iterator.

* Try to make everything compact

* Make `Clone` work

* Fix

* Move contracts and balances over to compact encoding

* Session and timestamp are compact

* Sataking uses compact numbers

* Treasury now compact

* Compact Democracy

* Council is compact

* Fix

* Adds new `check_inherents` function to the `BlockBuilder` API

* Switch to `check_inherents` in `node-consensus`

* Remove `CheckedBlock`, because it is not required anymore

* Fixes after rebase

* Fixes compilation on stable

* Split out Consensus
* Supply ImportQueue through network-service
  - simplify ImportQueue.import_blocks
  - remove Deadlock on import_block
  - Adding Verifier-Trait
  - Implement import_queue provisioning in service; allow cli to import
* Allow to actually customize import queue
* Consensus Gossip: Cache Message hash per Topic

* fix ignored test

* move CI to --release per Gav

* Make the graph generic.

* Adapting pool API for the graph.

* Merge pool & graph.

* Restructure.

* Fix test of transaction pool.

* Get rid of node/transaction-pool.

* Compilation fixes.

* Test7

* Fix compilation of tests.

* Revert runtime changes.

* Add validate_transaction to test-runtime.

* Fix RPC tests.

* Add clearing of the old transactions.

* Trigger pool events.

* Use new queue API.

* Fix wasm build, re-export Hasher.

* No warning if validate transaction fails.

* Get rid of Into<u64> and use As

* Adds a script for automatically update all copyrights

* Ran the `update-copyright` script

* Rename `NewTxQueue` to `TaggedTransactionQueue`

* Remove `BlockBuilder` API and support adding documentation to the API functions

* Adds new `BlockBuilder` API

* Fixes compilation with new `BlockBuilder` API

* Some more cleanup

* Cargo.lock update

* Try to fix on stable

* Add missing `As` imports.

* Adds new API traits that will be used by the client and runtime

* Switch consensus to new API's

* Switches transaction-pool to new API's

* Move runtime api stuff into its own crate

* Adds `impl_apis!` macro for implementing the new API traits

* Make `metadata` return directly a blob

* Runtime replace `impl_stubs!` with `impl_apis!`

* Switches to none feature based approach for declaring the different API traits

* Fixes compilation error

* Fixes errors

* Make the `decl_apis!` trait usable from the outside

* Make the `test-client` use the new API traits

* Remove last `impl_stubs!` bits and move some of them into wasm executor for tests

* A little bit more documentation

* Digest::log fn

* change DigestItem::log impl

* Auxiliary data storage in client backend

* Runtime error handling

* Consensus cleanup

* Fixed DB initialization issue

* Spacing

* changes_trie

* changs_trie: continue

* changes_trie: adding tests

* fixed TODO

* removed obsolete ExtrinsicChanges

* encodable ChangesTrieConfiguration

* removed polkadot fle

* fixed grumbles

* ext_storage_changes_root returns u32

* moved changes trie root to digest

* removed commented code

* read storage values from native code

* fixed grumbles

* fixed grumbles

* missing comma

* key changes proof generation + query

* fix grumbles

* check that changes trie config is not changed by block.finalize()

* fixed changes trie config check

* Started substrate tests

* Sync test

* Test updates

* Improved tests

* Use on-chain block delay

* Parallel test execution

* Otimized tests

* Logging

* Fixed racing test

* Fixed compilation

* Fixed timestamp test

* Removed rlp dependency

* Minor fixes

* Fixed tests

* Removed best_block_id and resolved fdlimit issue

* Whitespace

* Use keyring

* Style

* Added API execution setting

* Removed stale import

* Removes `rlp` dependency

* Enable warnings about unused crates in the CI

* Remove all the unused crates

* Make tests working again

snd authored Sep 26, 2018 and asynchronous rob committed Sep 26, 2018

* add stub for Client.best_chain_containing_block_hash

* add fn blockchain::Backend::leaf_hashes

* fix typo

* sketch out Client.best_chain_containing_block_hash

* fix indent

* Blockchain.leaf_hashes -> Blockchain.leaves

* add unimplemented! stub impls for Blockchain.leaves

* start impl of Blockchain.leaves for in-memory client db

* Client.best_chain_containing...: check canonical first and make compile

* first rough attempt at maintaining leaf list in in-memory db

* fix tab indent

* add test best_chain_containing_single_block

* add failing test best_chain_containing_with_fork

* pub use client::blockchain; in test-client to prevent circular dep in client tests

* best_chain_containing_with_single_block: improve and test leaves

* far improve in-memory Backend::leaves impl

* test blockchain::Backend::leaves more thoroughly

* handle more edge cases in blockchain::Backend::leaves impl for in memory

* fix test best_chain_containing_with_fork (two distinct test blocks had same hash)

* make best_chain_containing_block_hash pass existing tests

* improve docstring for Blockchain::leaves

* Client.best_chain_containing: some cleanup. support max_block_number

* best_chain_containing: remove broken outcommented fast check for best = canonical

* remove blank line

* best_block_containing: return None if target_hash not found

* best_chain_containing: add unreachable! at end of function

* improve tests for best_chain_containing

* renames

* more elaborate test scenario for best_containing

* best_containing: fix restriction of search through maybe_max_number

* best_containing: tests for restriction of search

* get rid of unnecessary clones

* replace Client::new_in_mem by new_with_backend which is useful for testing backends

* add test_client::new_with_backend for testing different backend impls

* add test for in_mem::Backend::leaves

* remove unused imports

* in_mem test_leaves: simplify

* flesh out tests for in_mem leaves impl

* remove tests for leaves from client which are now covered in implementing module

* improve comment

* add Client.new_in_mem again

* unwrap in test_client::new_with_backend

* make test_client::BlockBuilderExt work not just with in-mem backend

* make test client ext not just work with in mem backend

* add failing Backend.leaves test for client-db

* update Cargo.lock

* replace KeccakHasher with Blake2Hasher

* refactor

address grumble https://github.com/paritytech/substrate/pull/740#discussion_r217822862

* refactor using NumberFor

address grumble https://github.com/paritytech/substrate/pull/740#discussion_r217823341

* add test that exposes possible problem

* update docstring for Client.best_containing

* extract test for Backend.leaves for reuse

* improve test blockchain_header_and_hash_return_blocks_from_canonical_chain_given_block_numbers

* extract test_blockchain_query_by_number_gets_canonical to easily test multiple impls

* remove whitespace

* remove todo

* Client.best_containing: pre-empt search loop when target in canonical

* best_containing: prevent race condition by holding import lock

* add todo

* extract leaf list update code into function

* add comment

* client-db: use in-memory-kvdb for tests

* use BTreeSet to store leaves for in-mem which is faster and simpler

* add docstring

* add comments and fix formatting

* add initial raw version of LeafSet

* remove Client::update_leaves which has been superceded by LeafSet

* use LeafSet in in-mem backend

* keccak -> blake2

* don't reexport codec traits in primitives

addresses https://github.com/paritytech/substrate/pull/740#discussion_r219538185

* fix rebase mistake

* improve LeafSet and use it in state-db

* correct Transfer nonces to fix ApplyExtinsicFailed(Stale)

* use given backend in canoncal test

* kill dead tree-route code in util

* fix warnings

* tests for leafset

* reorganizations in in_mem backend

* fix reorganization canon block logic

* DB commit and safe reversion on write error

* fix style nits

* state-db: remove the assertion and replace it with Result<>

* state-db: unit test fixes

* comment fixes

* typo fix

Gav Wood authored Sep 25, 2018 and Arkadiy Paronyan committed Sep 25, 2018

* Bump codec

* Fix tests

* Patricia trie builds

* Introduce trie

* Some yak shaving.

* Some fixes

* Remove RLP ref

* Fixes

* It builds!

* Some tests fixed

* Another test fix

* Rejig more hashes

* substrate-trie::iterator_works test

* Update lock

* Polish

* Docs

* Undo incorrect "fix" for tests

* Fix nits

asynchronous rob authored Sep 24, 2018 and Arkadiy Paronyan committed Sep 24, 2018

* finalization for in_mem

* fetch last finalized block

* pruning: use canonical term instead of final

* finalize blocks in full node

* begin to port light client DB

* add tree-route

* keep number index consistent in full nodes

* fix tests

* disable cache and finish porting light client

* add AsMut to system module

* final leaf is always best

* fix all tests

* Fix comment and trace

* removed unused Into call

* add comment on behavior of `finalize_block`

* move `tree_route` to client common

* tree_route tests

* return slices in TreeRoute

* apply finality up to parent

* add `finalize_block` call

* adjust formatting

* finality notifications and add last finalized block to chain info

* exhaustive match and comments

* fix sync tests by using non-instant finality

* finalization for in_mem

* fetch last finalized block

* pruning: use canonical term instead of final

* finalize blocks in full node

* begin to port light client DB

* add tree-route

* keep number index consistent in full nodes

* fix tests

* disable cache and finish porting light client

* add AsMut to system module

* final leaf is always best

* fix all tests

* Fix comment and trace

* removed unused Into call

* add comment on behavior of `finalize_block`

* Address grumbles

* Fix hash

* Initial groundwork

* A mess.

* Integrate

* Fix tests

* Unit tests

* Tests for unchecked_extrisnic

* fix tab

* Improve binary format.

* fix tests

* Rename extrinsic-pool -> transaction-pool

Closes #770

* Implement unimplemented.

* typo

* Move module metadata from json string to custom metadata

* Revisit the metadata structures

1. Move the structures into the metadata crate.
2. Switch to using Cow/MaybeOwnedArray to support Encode/Decode

* Adapt to new metadata structures

* Convert event json metadata to new metadata structures

* Convert storage json metadata to new metadata structures

* Convert runtime metadata from json to new metadata structs

* Implements new metadata structures in client and runtime

* Fixes errors on `no_std`

* Fixes errors after rebasing master

* Do not use `Cow` anymore in metadata

Also replace `String` with our own type definition `StringBuf`.
This fixes compilation on `no_std`.

* Wrap `RuntimeMetadata` in `RuntimeMetadataVersioned` to support versioning

* Move metadata into `srml` and make core unaware of the implementation

* Extract well known keys into a single place

* Fixes.