polkadot companion: paritytech/polkadot#2535

* Start

* Finish!!!!

* Update client/basic-authorship/src/basic_authorship.rs

Co-authored-by: André Silva <[email protected]>

* Review comments

Co-authored-by: André Silva <[email protected]>

* Ensure we spawn the block import worker as an essential task

This pr ensures that we spawn the block import worker as an essential
task. This is quite important as we need to bring down the node when the
block import is done. Besides that it adds some debug output to the
block import worker.

* Don't be stupid :D

* Ahh

* Work work work

* Fix all the compilation errors

* Fix test

* More fixes...

* transaction-pool: drop unpropagable txs if local node cant author blocks

* fix test compilation

* transaction-pool: remove unnecessary static bound on CanAuthor

Co-authored-by: Tomasz Drwięga <[email protected]>

* rpc-api: add translation for PoolError::Unactionable

* transaction-pool: add test for rejecting unactionable transactions

* basic-authorship: fix doc test

* transaction-pool: fix benchmark compilation

* transaction-pool: rename CanAuthor to IsValidator

* transaction-pool: nit in error message

Co-authored-by: Tomasz Drwięga <[email protected]>

* Introduce a `Slot` type

Instead of having some type definition that only was used in half of the
code or directly using `u64`, this adds a new unit type wrapper `Slot`.
This makes it especially easy for the outside api to know what type is
expected/returned.

* Change epoch duratioC

* rename all instances of slot number to slot

* Make the constructor private

Co-authored-by: André Silva <[email protected]>

* Happy new year

Updates the copyright years and fixes wrong license headers.

* Fix the template

* Split HEADER into HEADER-APACHE & HEADER-GPL

This prevents nodes from executing the same block 2 times.

* grandpa: remove light-client specific block import

* consensus, network: remove finality proofs

* parent affbc38a


author Seun Lanlege <[email protected]> 1599568164 +0100
committer Seun Lanlege <[email protected]> 1604321289 +0100
gpgsig -----BEGIN PGP SIGNATURE-----

 iQGzBAABCgAdFiEECvQ02MnjnssnSbjr3HzzEhjN254FAl+gAAkACgkQ3HzzEhjN
 254soAv+KO5JA0HXSe0R0XS5TnwA3IxYsW+UvdF5dXFeC3jFdGTMvor818uoBePD
 dxzYEsUK6gjsNcM9+hpFhoy5JnUrUPInd2BZ7pmZiDuXmYJrHi0s7K5qL0EYDoe0
 m1egPNNyRR125ozJ24M+09c3OQsi3bvTx1TJaV9Aov8hK4So8UmlJTHWpkLw97ku
 HuTre2IPSFbV4GwJE40V+KNuDVHxaKL7zrInYScqbr6/hOTqBCvFn4ib3CjpF5HG
 zDAA5S2PrcbL9NQOothVcVB/TZr3IkhglCFqEjVyCX80IL0JkNZkw8jAh0B8uqXx
 Ug/c1/Mssa8F1jLZMmW45Cway60txqVbcWntPJAymGJbrRErOO/++oUrV0u1C65u
 LW7gXAaIJWQTX9KnX0SEyejNod7ubZktBz7n5WfkJAPIzdw5wtJalhLa673YTgQ9
 zyTPKiWjJj2myCq1AYrJvlK8hSsIBqbBFcUf1zX4SzZWKS+5mtp51o4gfVzcCRPd
 z/6/iPbB
 =g5tx
 -----END PGP SIGNATURE-----

BabeConsensusDataProvider works with existing chains

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>

* Initial commit

Forked at: 601e2fa1
Parent branch: origin/master

* WIP

Forked at: 601e2fa1
Parent branch: origin/master

* WIP

Forked at: 601e2fa1
Parent branch: origin/master

* WIP

Forked at: 601e2fa1
Parent branch: origin/master

* WIP

Forked at: 601e2fa1
Parent branch: origin/master

* WIP

Forked at: 601e2fa1
Parent branch: origin/master

* CLEANUP

Forked at: 601e2fa1
Parent branch: origin/master

* Add notes to original source code

* CLEANUP

Forked at: 601e2fa1
Parent branch: origin/master

* CLEANUP

Forked at: 601e2fa1
Parent branch: origin/master

* WIP

Forked at: 601e2fa1
Parent branch: origin/master

* WIP

Forked at: 601e2fa1
Parent branch: origin/master

* WIP

Forked at: 601e2fa1
Parent branch: origin/master

* CLEANUP

Forked at: 601e2fa1
Parent branch: origin/master

* WIP

Forked at: 601e2fa1
Parent branch: origin/master

* Some doc

* Test with trybuild

* Revert "Test with trybuild" (issue with trybuild atm)

This reverts commit 9055ec2206808ba3ddce6e3d87eb358907fa5e42.

https://github.com/dtolnay/trybuild/issues/53

* Apply suggestions

* Rename derive to proc-macro

* Remove "prefix" feature from informant

* Blocking task should use SpawnHandle::spawn_blocking

* Improve doc as suggested

* Fixes

Forked at: 601e2fa1


Parent branch: origin/master

* Apply suggestion

* Update client/cli/proc-macro/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* More suggestions

* CLEANUP

Forked at: 601e2fa1
Parent branch: origin/master

* Improve error message

* CLEANUP

Forked at: 601e2fa1
Parent branch: origin/master

* Fix async issue

* CLEANUP

Forked at: 601e2fa1
Parent branch: origin/master

* CLEANUP

Forked at: 601e2fa1


Parent branch: origin/master

* Add test

* fix doc test

* Update client/cli/src/logging.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/basic-authorship/src/basic_authorship.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/basic-authorship/src/basic_authorship.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Apply suggestions

* Suggestions

* Clarify doc

* WIP

Forked at: 601e2fa1


Parent branch: origin/master

Co-authored-by: Bastian Köcher <[email protected]>

* Asyncify sign_with

* Asyncify generate/get keys

* Complete BareCryptoStore asyncification

* Cleanup

* Rebase

* Add Proxy

* Inject keystore proxy into extensions

* Implement some methods

* Await on send

* Cleanup

* Send result over the oneshot channel sender

* Process one future at a time

* Fix cargo stuff

* Asyncify sr25519_vrf_sign

* Cherry-pick and fix changes

* Introduce SyncCryptoStore

* SQUASH ME WITH THE first commit

* Implement into SyncCryptoStore

* Implement BareCryptoStore for KeystoreProxyAdapter

* authority-discovery

* AURA

* BABE

* finality-grandpa

* offchain-workers

* benchmarking-cli

* sp_io

* test-utils

* application-crypto

* Extensions and RPC

* Client Service

* bin

* Update cargo.lock

* Implement BareCryptoStore on proxy directly

* Simplify proxy setup

* Fix authority-discover

* Pass async keystore to authority-discovery

* Fix tests

* Use async keystore in authority-discovery

* Rename BareCryptoStore to CryptoStore

* WIP

* Remote mutable borrow in CryptoStore trait

* Implement Keystore with backends

* Remove Proxy implementation

* Fix service builder and keystore user-crates

* Fix tests

* Rework authority-discovery after refactoring

* futures::select!

* Fix multiple mut borrows in authority-discovery

* Merge fixes

* Require sync

* Restore Cargo.lock

* PR feedback - round 1

* Remove Keystore and use LocalKeystore directly

Also renamed KeystoreParams to KeystoreContainer

* Join

* Remove sync requirement

* Fix keystore tests

* Fix tests

* client/authority-discovery: Remove event stream dynamic dispatching

With authority-discovery moving from a poll based future to an `async`
future Rust has difficulties propagating the `Sync` trade through the
generated state machine.

Instead of using dynamic dispatching, use a trait parameter to specify
the DHT event stream.

* Make it compile

* Fix submit_transaction

* Fix block_on issue

* Use await in async context

* Fix manual seal keystore

* Fix authoring_blocks test

* fix aura authoring_blocks

* Try to fix tests for auth-discovery

* client/authority-discovery: Fix lookup_throttling test

* client/authority-discovery: Fix triggers_dht_get_query test

* Fix epoch_authorship_works

* client/authority-discovery: Remove timing assumption in unit test

* client/authority-discovery: Revert changes to termination test

* PR feedback

* Remove deadcode and mark test code

* Fix test_sync

* Use the correct keyring type

* Return when from_service stream is closed

* Convert SyncCryptoStore to a trait

* Fix line width

* Fix line width - take 2

* Remove unused import

* Fix keystore instantiation

* PR feedback

* Remove KeystoreContainer

* Revert "Remove KeystoreContainer"

This reverts commit ea4a37c7d74f9772b93d974e05e4498af6192730.

* Take a ref of keystore

* Move keystore to dev-dependencies

* Address some PR feedback

* Missed one

* Pass keystore reference - take 2

* client/finality-grandpa: Use `Arc<dyn CryptoStore>` instead of SyncXXX

Instead of using `SyncCryptoStorePtr` within `client/finality-grandpa`,
which is a type alias for `Arc<dyn SyncCryptoStore>`, use `Arc<dyn
CryptoStore>`. Benefits are:

1. No additional mental overhead of a `SyncCryptoStorePtr`.

2. Ability for new code to use the asynchronous methods of `CryptoStore`
instead of the synchronous `SyncCryptoStore` methods within
`client/finality-granpa` without the need for larger refactorings.

Note: This commit uses `Arc<dyn CryptoStore>` instead of
`CryptoStorePtr`, as I find the type signature more descriptive. This is
subjective and in no way required.

* Remove SyncCryptoStorePtr

* Remove KeystoreContainer & SyncCryptoStorePtr

* PR feedback

* *: Use CryptoStorePtr whereever possible

* *: Define SyncCryptoStore as a pure extension trait of CryptoStore

* Follow up to SyncCryptoStore extension trait

* Adjust docs for SyncCryptoStore as Ben suggested

* Cleanup unnecessary requirements

* sp-keystore

* Use async_std::task::block_on in keystore

* Fix block_on std requirement

* Update primitives/keystore/src/lib.rs

Co-authored-by: Max Inden <[email protected]>

* Fix wasm build

* Remove unused var

* Fix wasm compilation - take 2

* Revert async-std in keystore

* Fix indent

* Fix version and copyright

* Cleanup feature = "std"

* Auth Discovery: Ignore if from_service is cloed

* Max's suggestion

* Revert async-std usage for block_on

* Address PR feedback

* Fix example offchain worker build

* Address PR feedback

* Update Cargo.lock

* Move unused methods to test helper functions

* Restore accidentally deleted cargo.lock files

* Fix unused imports

Co-authored-by: Max Inden <[email protected]>
Co-authored-by: Shawn Tabrizi <[email protected]>

* manual seal is now consensus agnostic

* pr grumbles

* Revalidate transactions only on latest best block

We should revalidate transactions only on the latest best block and not
on any arbitrary block. The revalidation before failed when there were
multiple blocks on the height given to the revalidation function, but no
block was imported as best block.

* Update test-utils/runtime/transaction-pool/src/lib.rs

Co-authored-by: Jaco Greeff <[email protected]>

* Fix tests

* Only process best blocks in the transaction pool

Co-authored-by: Jaco Greeff <[email protected]>

* Remove any implementation of `Spawn` or `Executor` from our task executors

* Fix compilation

* Rename `SpawnBlockingExecutor`

* Update primitives/core/src/traits.rs

Co-authored-by: Kian Paimani <[email protected]>

* Fix tests

Co-authored-by: Kian Paimani <[email protected]>

* Simplify a few chain components creation APIs related to the service

* Fix basic-authorship doc tests

* Remove DefaultQueue

* Update client/service/src/builder.rs

Co-authored-by: André Silva <[email protected]>

* Move ExecutionExtensions comment around

* Remove unused BlakeTwo256

Co-authored-by: André Silva <[email protected]>

* Seperate out the complexity in ServiceBuilder::build_common into seperate functions

* Fix line widths

* Move some functions to their respective crates

The `tree_route` generated by the import notification is only from the
old best block to the new best parent. This means, it does not contain
the new best block in `enacted()`. We need to prune the transactions of
the new best block "manually" to fix this bug.

Besides that, this pr also changed the `id` parameter of the `NewBlock`
chain event to `hash`. The hash of a block is unique in contrast to the
block number. (Block id can either be number or hash)

* Make tx pool aware of retracted fork blocks

* Make it compile

* Update client/transaction-pool/src/lib.rs

Co-authored-by: Nikolay Volf <[email protected]>

* Fix doc test

* Simplify the implementation

* Send tree route as arc to prevent heavy clones

* Switch to use `ExtrinsicHash` to make it more clear

* Fix benchmark

Co-authored-by: Nikolay Volf <[email protected]>

* Make `Proposer` consume its reference on `propose`

A proposer must be created per new round, so it makes sense to have the
proposer consume its own reference.

* Remove `ProposerInner`

* Add basic authorship metrics

* fixes

* no arc

* move to crate

* Update client/proposer-metrics/Cargo.toml

Co-authored-by: Max Inden <[email protected]>

* remove prefix

* use HistogramTimer

* Update client/proposer-metrics/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/basic-authorship/src/basic_authorship.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/basic-authorship/src/basic_authorship.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/basic-authorship/src/basic_authorship.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/basic-authorship/src/basic_authorship.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/basic-authorship/src/basic_authorship.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/proposer-metrics/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/proposer-metrics/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Max Inden <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* WIP

* WIP

* Expose that `BasicQueue` expects blocking spawn

Up to now `BasicQueue` expected a closure that to spawn a `Future`.
This was expected to be a closure that spawns a blocking future.
However, this wasn't documented anywhere. This pr introduces a new trait
`SpawnBlocking` that exposes this requirement to the outside.

* Feedback

* replace replace with take

* Update client/api/src/cht.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/api/src/cht.rs

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>

Co-Authored-By: Pierre Krieger <[email protected]>

* make new contructor

* add metrics to txpool

* fix review

* fix doc comment

* change to counters

* Update client/transaction-pool/src/metrics.rs

Co-Authored-By: Max Inden <[email protected]>

* Update client/transaction-pool/src/metrics.rs

Co-Authored-By: Max Inden <[email protected]>

* Update client/transaction-pool/src/metrics.rs

Co-Authored-By: Max Inden <[email protected]>

* Update client/transaction-pool/src/lib.rs

Co-Authored-By: Max Inden <[email protected]>

* Update client/transaction-pool/src/lib.rs

Co-Authored-By: Max Inden <[email protected]>

* use dedicated wrapper

Co-authored-by: Max Inden <[email protected]>

* manual-seal uses Finalizer trait for finalization instead of Backend

* fix tests

* use Transaction type

* refactor import_queue

* ugh

* line-width

* Update client/consensus/manual-seal/src/lib.rs

Co-Authored-By: Joshy Orndorff <[email protected]>

* fix tests

* update docs

* Update client/consensus/manual-seal/src/seal_new_block.rs

Co-Authored-By: André Silva <[email protected]>

* Don't auto-finalize on verification

* Explicity don't finalize on import.

Co-authored-by: Joshy Orndorff <[email protected]>
Co-authored-by: Joshy Orndorff <[email protected]>
Co-authored-by: André Silva <[email protected]>

* WiP

* Support source in the runtime API.

* Finish implementation in txpool.

* Fix warning.

* Fix tests.

* Apply suggestions from code review

Co-Authored-By: Kian Paimani <[email protected]>
Co-Authored-By: Nikolay Volf <[email protected]>

* Extra changes.

* Fix test and benches.

* fix test

* Fix test & benches again.

* Fix tests.

* Update bumpalo

* Fix doc test.

* Fix doctest.

* Fix doctest.

Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Nikolay Volf <[email protected]>

* make sure return ready iterator once state is updated

* update sc_basic_authorship tests

* update node tests

* fix manual seal

* actually fix service test

* add tests

* Update client/basic-authorship/src/basic_authorship.rs

Co-Authored-By: Tomasz Drwięga <[email protected]>

* helper function

* review suggestions

* warning and continue

* add debug log

* use futures::chennel::oneshot

* use declaration bound

* no option for updated_at

* no allocation

* ready_at / ready

* Update client/transaction-pool/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/transaction-pool/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/transaction-pool/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/transaction-pool/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/transaction-pool/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

* Update client/transaction-pool/src/lib.rs

Co-Authored-By: Bastian Köcher <[email protected]>

Co-authored-by: Tomasz Drwięga <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* removes use of sc-client from sc-basic-authorship

* refactor use of ProposerFactory

* correct dep path

* Revalidation queeue.

* add docs and license

* move test

* refactor worker to async/await

* address review

* fix warnings

* update Cargo.lock

* move background task to service

* use tomusdrw loop

* naming

* return From::from

* add doc comment

* add more doc comments

* fix merge bug

* add doc comment for test function

* Update client/transaction-pool/src/testing/pool.rs

Co-Authored-By: Tomasz Drwięga <[email protected]>

* more review fixes

* refactor to allow service keep background tasks from isntantiated subsystems

* use const delay

* fix fallout

* remove fallout

* remove already moved test

* fix doc test

* add valid_at helper

Co-authored-by: Tomasz Drwięga <[email protected]>

* Refactor BlockImportParams to be non_exhaustive

* Fix cargo check compile

* adds finalization support to sc-transaction-pool using MaintainedTransactionPool for finalization events

* adds TransactionStatus::Retracted, notify watchers of retracted blocks, finalized now finalizes, transactions for current finalized -> last finalized block

* adds last_finalized to ChainApi, use generic BlockT for ChainEvent

* fix tests

* Apply suggestions from code review

Co-Authored-By: Tomasz Drwięga <[email protected]>

* tests

* fix tests, docs, lazily dedupe pruned hashes

* fix tests, Cargo.lock

* Apply suggestions from code review

Co-Authored-By: Tomasz Drwięga <[email protected]>

* remove tree_route, last_finalized from ChainApi, add block hash to Finalization and Retracted events

* prune finality watchers

* fix tests

* remove HeaderBackend bound from FullChainApi

* code style nits, terminate stream in finality_timeout

Co-authored-by: Tomasz Drwięga <[email protected]>

* Cleanup transaction pool deps

* Fix it properly

* Fix doc test

* shuffle tests

* update tests

* inc_nonce -> increment_nonce