* Make grandpa work

* Introduce `SharedData`

* Add test and fix bugs

* Switch to `SharedData`

* Make grandpa tests working

* More Babe work

* Make it async

* Fix fix

* Use `async_trait` in sc-consensus-slots

This makes the code a little bit easier to read and also expresses that
there can always only be one call at a time to `on_slot`.

* Make grandpa tests compile

* More Babe tests work

* Fix network test

* Start fixing service test

* Finish service-test

* Fix sc-consensus-aura

* Fix fix fix

* More fixes

* Make everything compile *yeah*

* Fix build when we have Rust 1.51

* Update client/consensus/common/src/shared_data.rs

Co-authored-by: André Silva <[email protected]>

* Update client/consensus/common/src/shared_data.rs

Co-authored-by: André Silva <[email protected]>

* Update client/consensus/common/src/shared_data.rs

Co-authored-by: André Silva <[email protected]>

* Update client/consensus/common/src/shared_data.rs

Co-authored-by: André Silva <[email protected]>

* Update client/consensus/common/src/shared_data.rs

Co-authored-by: André Silva <[email protected]>

* Update client/consensus/babe/src/tests.rs

Co-authored-by: André Silva <[email protected]>

* Update client/consensus/babe/src/tests.rs

Co-authored-by: André Silva <[email protected]>

* Fix warning

Co-authored-by: André Silva <[email protected]>

This is not really required and having a special `test-helpers` feature
is a bad idea anyway.

* Use `async_trait` in sc-consensus-slots

This makes the code a little bit easier to read and also expresses that
there can always only be one call at a time to `on_slot`.

* slots: remove mutex around BlockImport in SlotWorker

Co-authored-by: André Silva <[email protected]>

* Make slot duration being exposed as `Duration` to the outside

* Some slot info love

* Add `build_aura_worker` utility function

* Copy copy copy

* primitives/runtime: initial changes on supporting multiple Justifications

* primitives/runtime: make Justifications strongly typed

* Encode/decode Justifications

* primitives/runtime: add Justification type

* backend: apply_finality and finalize_block takes a single Justification

* manual-seal: create engine id and let rpc take encoded justification

* backend: skeleton functions for appending justifications

* backend: initial implementation append_justification

Initial implementation of append_justification on the Backend trait, and also remove unused skeleton
functions for append_justificaton on Finaziler trait.
k

* backend: guard against duplicate consensus engine id

* client/db: add check for block finality

* client/api: add append_justification to in_mem db

* client/light: add no-op append_justification

* network: fix decode call for Justification

* network: only send a single Justification in BlockData

* network: minor comment update

* protocol: update field names to distinguish single justification

* client: further field renames to plural

* client: update function names to plural justifications

* client/db: upgrade existing database for new format

* network: remove dependency on grandpa crate

* db: fix check for finalized block

* grandpa: check for multiple grandpa justifications hwne importing

* backend: update Finalizer trait to take multiple Justifications

* db: remove debugging statements in migration code

* manual-seal: update note about engine id

* db: fix check for finalized block

* client: update variable name to reflect it is now plural

* grandpa: fix incorrect empty Justications in test

* primitives: make Justifications opaque to avoid being empty

* network: fix detecting empty Justification

* runtime: doc strings for Justifications functions

* runtime: add into_justifications

* primitives: check for duplicates in when adding to Justifications

* network/test: use real grandpa engine id in test

* client: fix reviewer comments

* primitives: rename Justifications::push to append

* backend: revert changes to Finalizer trait

* backend: revert mark_finalized

* backend: revert changes to finalize_block

* backend: revert finalized_blocks

* db: add a quick early return for performance

* client: minor reviewer comments

* service/test: use local ConsensusEngineId

* network: add link to issue for sending multiple Justifications

* Apply suggestions from code review

Co-authored-by: Pierre Krieger <[email protected]>

* Apply suggestions from code review

Co-authored-by: Pierre Krieger <[email protected]>

* network: tweaks to review suggestions

* network: revert change to BlockData for backwards compatibility

* Apply suggestion from code review

Co-authored-by: Pierre Krieger <[email protected]>

* Apply suggestions from code review

Co-authored-by: André Silva <[email protected]>

* primitives: update doc comment for Justifications

* client/db/upgrade: avoid grandpa crate dependency

* consensus: revert to single Justification for import_justification

* primitives: improve justifications docs

* style cleanups

* use and_then

* client: rename JUSTIFICATIONS db column

* network: revert to using FRNK in network-test

Co-authored-by: Pierre Krieger <[email protected]>
Co-authored-by: André Silva <[email protected]>
Co-authored-by: André Silva <[email protected]>

The timestamp inherent type was up to now just a simple `u64`. This
worked, but doesn't give you that much guarantees at compile time about
the type. This pr changes that by converting this type to a unit type
wrapper, similar to what we have done for `Slot`.

This is required for some future pr that touches quite a lot of the
inherents stuff :)

Besides this unit wrapper type, this pr also moves the `OnTimestampSet`
trait to `frame_support::traits`.

polkadot companion: paritytech/polkadot#2535

* Make changes

* Add serialize/deserialize, copy babe epoch config defaults from node runtime

* Fix line widths and turn default features off for serde

* Remove ser/deser from Epoch, fix node-cli

* Apply suggestions

* Add comment to BABE_GENESIS_EPOCH_CONFIG in bin

* Apply suggestions

* Add a sketchy migration function

* Add a migration test

* Check for PendingEpochConfigChange as well

* Make epoch_config in node-cli

* Move updating EpochConfig out of the if

* Fix executor tests

* Calculate weight for add_epoch_configurations

* Fix babe test

* Apply suggestions from code review

Co-authored-by: André Silva <[email protected]>

* Add more asserts to tests, remove unused changes to primitives/slots

* Allow setting the migration pallet prefix

* Rename to BabePalletPrefix

Co-authored-by: André Silva <[email protected]>

* Introduce new concept of "slot portion for proposing"

Currently when building a block we actually give the proposer all of the
time in the slot, while this is wrong. The slot is actually split in at
least two phases proposing and propagation or in the polkadot case into
three phases validating pov's, proposing and propagation. As we don't
want to bring that much polkadot concepts into Substrate, we only
support splitting the slot into proposing and propagation. The portion
can now be passed as parameter to AuRa and BABE to configure this value.
However, this slot portion for propagation doesn't mean that the
proposer can not go over this limit. When we miss slots we still apply
the lenience factor to increase the proposing time, so that we have
enough time to build a heavy block.

Besides all what was said above, this is especially required for
parachains. Parachains have a much more constraint proposing window.
Currently the slot duration is at minimum 12 seconds, but we only have
around 500ms for proposing. So, this slot portion for proposing is
really required to make it working without hacks.

* Offgit feedback

* Cast cast cast

* AuRa improvements

Hot and fresh AuRa improvements. This pr does the following:

- Move code belonging to the import queue etc to import_queue.rs
- Introduce `ImportQueueParams` and `StartAuraParams` structs to make
it more easier to understand what parameters we pass to AuRa.
- Introduce `CheckForEquivocation` to tell AuRa if it should check for
equivocation on block import. This is required for parachains, because
they are allowed to equivocate when they build two blocks for the same
slot, but for different relay chain parents.

* Update client/consensus/aura/src/import_queue.rs

Co-authored-by: André Silva <[email protected]>

* Fix compilation

* AAA

Co-authored-by: André Silva <[email protected]>

* babe: introduce a request-answering mechanic

* gromble

* send method

* Move AuRa digest from client to primitives

This makes the digest stuff usable from inside the runtime ;)

* Update primitives/runtime/src/generic/digest.rs

Co-authored-by: André Silva <[email protected]>

* Review feedback

* Make BABE use the new functionality

Co-authored-by: André Silva <[email protected]>

* Start

* Finish!!!!

* Update client/basic-authorship/src/basic_authorship.rs

Co-authored-by: André Silva <[email protected]>

* Review comments

Co-authored-by: André Silva <[email protected]>

* Make `on_slot` return the block with the post header

Before this pr `on_slot` returned the pre block. However this is wrong,
because adding some post digest changes the hash of the header. Thus,
we need to make sure to return the correct block that uses the post
header.

* Update primitives/consensus/common/src/block_import.rs

Co-authored-by: André Silva <[email protected]>

Co-authored-by: André Silva <[email protected]>

* Make keystore return `None` when a key doesn't exist

* Fixes

* More fixes

* Update comment

* Update primitives/keystore/src/lib.rs

Co-authored-by: André Silva <[email protected]>

* Update client/keystore/src/local.rs

Co-authored-by: André Silva <[email protected]>

* Address comments

* Update client/keystore/src/local.rs

Co-authored-by: André Silva <[email protected]>

Co-authored-by: André Silva <[email protected]>

* Ensure we spawn the block import worker as an essential task

This pr ensures that we spawn the block import worker as an essential
task. This is quite important as we need to bring down the node when the
block import is done. Besides that it adds some debug output to the
block import worker.

* Don't be stupid :D

Currently we always use a boxed future everywhere anyway. This also
enables us to use a boxed `SlotWorker` (which is required for Cumulus).

* Ahh

* Work work work

* Fix all the compilation errors

* Fix test

* More fixes...

* bumping version for next release
* add changelog
* add guide

* transaction-pool: drop unpropagable txs if local node cant author blocks

* fix test compilation

* transaction-pool: remove unnecessary static bound on CanAuthor

Co-authored-by: Tomasz Drwięga <[email protected]>

* rpc-api: add translation for PoolError::Unactionable

* transaction-pool: add test for rejecting unactionable transactions

* basic-authorship: fix doc test

* transaction-pool: fix benchmark compilation

* transaction-pool: rename CanAuthor to IsValidator

* transaction-pool: nit in error message

Co-authored-by: Tomasz Drwięga <[email protected]>

* update cargo.toml

* use 2.0 in mmmr

* Introduce a `Slot` type

Instead of having some type definition that only was used in half of the
code or directly using `u64`, this adds a new unit type wrapper `Slot`.
This makes it especially easy for the outside api to know what type is
expected/returned.

* Change epoch duratioC

* rename all instances of slot number to slot

* Make the constructor private

Co-authored-by: André Silva <[email protected]>

* babe: log block and slot number on verification

* babe: debug log formatting

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>

Bumps [retain_mut](https://github.com/upsuper/retain_mut) from 0.1.1 to 0.1.2.
- [Release notes](https://github.com/upsuper/retain_mut/releases)
- [Commits](https://github.com/upsuper/retain_mut/compare/v0.1.1...v0.1.2)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Backport paritytech/substrate#7381

* Bring back genesis storage build in aura/timestamp

To not change spec version, see
https://github.com/paritytech/substrate/pull/7686#discussion_r540032743

* Backport paritytech/substrate#7238

* Backport paritytech/substrate#7395

* Bump impl_version

* Fix UI tests and bump trybuild dep

See https://github.com/rust-lang/rust/pull/73996

Backports:
https://github.com/paritytech/substrate/pull/7764
https://github.com/paritytech/substrate/pull/7656

* Partially backport paritytech/substrate#7838

* Release frame-support with a dep compilation fix

* Bump patch level for remaining crates

This is done because at the time of writing cargo-unleash does not fully
support partial workspace publishing and mixes both local and crates.io
versions of the packages, leading to errors in the release check workflow.

* Backport paritytech/substrate#7854

...to fix compilation error when using futures-* v0.3.9.

* Adding Changelog  entry for patch release

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Benjamin Kampmann <[email protected]>

* upgrade a few dependencies

* make it compile at the expense of duplicate deps

* fix web-wasm and a warning

* introduce activate-wasm-bindgen-features crate

* Revert "introduce activate-wasm-bindgen-features crate"

This reverts commit 5a6e41e683f8a4844c0a735dcd08caabb2313f11.

* add getrandom feature to sc-consensus-aura

* Happy new year

Updates the copyright years and fixes wrong license headers.

* Fix the template

* Split HEADER into HEADER-APACHE & HEADER-GPL

* handle executor should_panic test better

* Revert "reduce should panic, due to extended error messages"

This reverts commit c0805940184a62cd9302603ad911c3591e70a60c.

* remove excessive constraints

* remove duplicate documentation messages for error variants

* reduce T: constraints to the abs minimum

* whoops

* fewer bounds again

Co-authored-by: Bernhard Schuster <[email protected]>

* chore/error: remove from str conversion and add deprecation notifications

* fixup changes

* fix test looking for gone ::Msg variant

* another test fix

* one is duplicate, the other is not, so duplicates reported are n-1

* darn spaces

Co-authored-by: Andronik Ordian <[email protected]>

* remove pointless doc comments of error variants without any value

* low hanging fruits (for a tall person)

* moar error type variants

* avoid the storage modules for now

They are in need of a refactor, and the pain is rather large
removing all String error and DefaultError occurences.

* chore remove pointless error generic

* fix test for mocks, add a bunch of non_exhaustive

* max line width

* test fixes due to error changes

* fin

* error outputs... again

* undo stderr adjustments

* Update client/consensus/slots/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* remove closure clutter

Co-authored-by: Bastian Köcher <[email protected]>

* more error types

* introduce ApiError

* extract Mock error

* ApiError refactor

* even more error types

* the last for now

* chore unused deps

* another extraction

* reduce should panic, due to extended error messages

* error test happiness

* shift error lines by one

* doc tests

* white space

Co-authored-by: Bastian Köcher <[email protected]>

* Into -> From

Co-authored-by: Bastian Köcher <[email protected]>

* remove pointless codec

Co-authored-by: Bastian Köcher <[email protected]>

* avoid pointless self import

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Bernhard Schuster <[email protected]>
Co-authored-by: Andronik Ordian <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

This prevents nodes from executing the same block 2 times.

* grandpa: remove light-client specific block import

* consensus, network: remove finality proofs

* babe: backoff authoring blocks when finality lags

* babe: move backoff authoring params to default constructor

* babe: deduplicate the test a bit

* babe: set backoff constants in service

* babe: use better names for backoff authoring block parameters

* babe: remove last unwrap

* babe: slight style tweak

* babe: fix comment

* slots: move backoff block authorship logic to SimpleSlotWorker

* aura: append SlotInfo in on_slot

* slots: use the correct types for parameters

* slots: fix review comments

* aura: add missing backoff authoring blocks parameters

* slots: add comments for default values

* slots: add additional checks in test

* slots: update implementation for new master

* slots: revert the change to SlotInfo

* Fix review comments

* slots: rework unit tests for backing off claiming slots

* slots: add test for asymptotic behaviour for slot claims

* slots: address review comments

* slots: add test for max_interval

* slots: add assertion for intervals between between claimed slots

* slots: remove rustfmt directive

* slots: another attempt at explaining authoring_rate

* slots: up unfinalized_slack to 50 by default

* slots: add tests for time to reach max_interval

* slots: fix typo in comments

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* slots: additional tweaks to comments and info calls

* slots: rename to BackoffAuthoringOnFinalizedHeadLagging

* slots: make the backing off strategy generic

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* slots: implement backoff trait for () for simplicity

* slots: move logging inside backing off function to make it more specific

* aura: add missing function parameter

Co-authored-by: Bastian Köcher <[email protected]>

The genesis header doesn't have the BABE pre-digest and we insert `0` as
slot number. The slot lenience calculation will return the maximum in
this situation. Besides returning the maximum which is not bad at all,
it also prints some a debug message that can be confusing in the first
moment. To prevent printing this debug message, we now just return early
when we see that the parent block is the genesis block.

* change (CI): markdown link checker

* Fix some invalid doc links (re-run of cargo-unleash gen-readme w/ fixes).

* Fix some invalid doc links

* Fix some invalid doc links

* Fix some links

* Fix some links

* Apply @bkchr

 suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* Fix more links

* Fix more links

* typo

* Fix more links

* Fix more links

* Ignore valid link .. check wrongly sees it as invalid

* Fix style issue

* Fix style issue

* change (CI): update style guide link

* change (lib): suggestions

Co-authored-by: Dan Forbes <[email protected]>
Co-authored-by: Steve Degosserie <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>