* primitives/runtime: initial changes on supporting multiple Justifications

* primitives/runtime: make Justifications strongly typed

* Encode/decode Justifications

* primitives/runtime: add Justification type

* backend: apply_finality and finalize_block takes a single Justification

* manual-seal: create engine id and let rpc take encoded justification

* backend: skeleton functions for appending justifications

* backend: initial implementation append_justification

Initial implementation of append_justification on the Backend trait, and also remove unused skeleton
functions for append_justificaton on Finaziler trait.
k

* backend: guard against duplicate consensus engine id

* client/db: add check for block finality

* client/api: add append_justification to in_mem db

* client/light: add no-op append_justification

* network: fix decode call for Justification

* network: only send a single Justification in BlockData

* network: minor comment update

* protocol: update field names to distinguish single justification

* client: further field renames to plural

* client: update function names to plural justifications

* client/db: upgrade existing database for new format

* network: remove dependency on grandpa crate

* db: fix check for finalized block

* grandpa: check for multiple grandpa justifications hwne importing

* backend: update Finalizer trait to take multiple Justifications

* db: remove debugging statements in migration code

* manual-seal: update note about engine id

* db: fix check for finalized block

* client: update variable name to reflect it is now plural

* grandpa: fix incorrect empty Justications in test

* primitives: make Justifications opaque to avoid being empty

* network: fix detecting empty Justification

* runtime: doc strings for Justifications functions

* runtime: add into_justifications

* primitives: check for duplicates in when adding to Justifications

* network/test: use real grandpa engine id in test

* client: fix reviewer comments

* primitives: rename Justifications::push to append

* backend: revert changes to Finalizer trait

* backend: revert mark_finalized

* backend: revert changes to finalize_block

* backend: revert finalized_blocks

* db: add a quick early return for performance

* client: minor reviewer comments

* service/test: use local ConsensusEngineId

* network: add link to issue for sending multiple Justifications

* Apply suggestions from code review

Co-authored-by: Pierre Krieger <[email protected]>

* Apply suggestions from code review

Co-authored-by: Pierre Krieger <[email protected]>

* network: tweaks to review suggestions

* network: revert change to BlockData for backwards compatibility

* Apply suggestion from code review

Co-authored-by: Pierre Krieger <[email protected]>

* Apply suggestions from code review

Co-authored-by: André Silva <[email protected]>

* primitives: update doc comment for Justifications

* client/db/upgrade: avoid grandpa crate dependency

* consensus: revert to single Justification for import_justification

* primitives: improve justifications docs

* style cleanups

* use and_then

* client: rename JUSTIFICATIONS db column

* network: revert to using FRNK in network-test

Co-authored-by: Pierre Krieger <[email protected]>
Co-authored-by: André Silva <[email protected]>
Co-authored-by: André Silva <[email protected]>

We introduced banning of peers who spam us with the same request (more
than 2 times). However, we missed that it is completely legal to send
the same request multiple times as long as we did not provide any
answer. An example for that is the justification request. This request
is send multiple times until we could fetch the justification from one
of our peers. So, the solution to this problem is to tag requests as
fulfilled and to start counting these fulfilled requests. If the number
is higher than what we allow, the peer should be banned.

The timestamp inherent type was up to now just a simple `u64`. This
worked, but doesn't give you that much guarantees at compile time about
the type. This pr changes that by converting this type to a unit type
wrapper, similar to what we have done for `Slot`.

This is required for some future pr that touches quite a lot of the
inherents stuff :)

Besides this unit wrapper type, this pr also moves the `OnTimestampSet`
trait to `frame_support::traits`.

Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.22 to 0.1.25.
- [Release notes](https://github.com/tokio-rs/tracing/releases)
- [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.22...tracing-0.1.25)

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

polkadot companion: paritytech/polkadot#2535

* Add MMR custom RPC.

* Change RuntimeApi to avoid hardcoding leaf type.

* Properly implement the new RuntimeAPI and wire up RPC.

* Extract Offchain DB as separate execution extension.

* Enable offchain DB access for offchain calls.

* Fix offchain_election tests.

* Skip block initialisation for proof generation.

* Fix integration test setup.

* Fix offchain tests. Not sure how I missed them earlier 🤷.

* Fix long line.

* One more test missing.

* Update mock for multi-phase.

* Address review grumbbles.

* Address review grumbles.

* Fix line width of a comment

* Make changes

* Add serialize/deserialize, copy babe epoch config defaults from node runtime

* Fix line widths and turn default features off for serde

* Remove ser/deser from Epoch, fix node-cli

* Apply suggestions

* Add comment to BABE_GENESIS_EPOCH_CONFIG in bin

* Apply suggestions

* Add a sketchy migration function

* Add a migration test

* Check for PendingEpochConfigChange as well

* Make epoch_config in node-cli

* Move updating EpochConfig out of the if

* Fix executor tests

* Calculate weight for add_epoch_configurations

* Fix babe test

* Apply suggestions from code review

Co-authored-by: André Silva <[email protected]>

* Add more asserts to tests, remove unused changes to primitives/slots

* Allow setting the migration pallet prefix

* Rename to BabePalletPrefix

Co-authored-by: André Silva <[email protected]>

* Introduce new concept of "slot portion for proposing"

Currently when building a block we actually give the proposer all of the
time in the slot, while this is wrong. The slot is actually split in at
least two phases proposing and propagation or in the polkadot case into
three phases validating pov's, proposing and propagation. As we don't
want to bring that much polkadot concepts into Substrate, we only
support splitting the slot into proposing and propagation. The portion
can now be passed as parameter to AuRa and BABE to configure this value.
However, this slot portion for propagation doesn't mean that the
proposer can not go over this limit. When we miss slots we still apply
the lenience factor to increase the proposing time, so that we have
enough time to build a heavy block.

Besides all what was said above, this is especially required for
parachains. Parachains have a much more constraint proposing window.
Currently the slot duration is at minimum 12 seconds, but we only have
around 500ms for proposing. So, this slot portion for proposing is
really required to make it working without hacks.

* Offgit feedback

* Cast cast cast

* Decrease the peer reputation on invalid block requests

This pr changes the block request handler to decrease the reputation of
peers when they send the same request multiple times or they send us an
invalid block request.

* Review feedback

* Change log target

* Remove unused code

* AuRa improvements

Hot and fresh AuRa improvements. This pr does the following:

- Move code belonging to the import queue etc to import_queue.rs
- Introduce `ImportQueueParams` and `StartAuraParams` structs to make
it more easier to understand what parameters we pass to AuRa.
- Introduce `CheckForEquivocation` to tell AuRa if it should check for
equivocation on block import. This is required for parachains, because
they are allowed to equivocate when they build two blocks for the same
slot, but for different relay chain parents.

* Update client/consensus/aura/src/import_queue.rs

Co-authored-by: André Silva <[email protected]>

* Fix compilation

* AAA

Co-authored-by: André Silva <[email protected]>

* implement

* make default pre/post_upgrade

* simplify Cargo.toml

* revert removal of outdated/private links

* link in pallet-mmr

We were using the wrong syntax and that will be dropped with Rust 2021.
The compiler already starts to hint the wrong syntax with warnings. So,
we fix this here.

* babe: introduce a request-answering mechanic

* gromble

* send method

* Move AuRa digest from client to primitives

This makes the digest stuff usable from inside the runtime ;)

* Update primitives/runtime/src/generic/digest.rs

Co-authored-by: André Silva <[email protected]>

* Review feedback

* Make BABE use the new functionality

Co-authored-by: André Silva <[email protected]>

* grandpa: maintain invariants when evaluating aggregated voting rules

* grandpa: update comment on VotingRules::restrict_vote

* grandpa: simplify comment

* Fix networking debug_asserts

* Fix comment

* grandpa: use AuthoritySetChanges to generate warp sync proof

* node: init grandpa warp sync protocol

* grandpa: iterator for AuthoritySetChanges

* grandpa: rewrite warp sync proof generation

* grandpa: remove old code for warp sync generation

* grandpa: fix indentation

* grandpa: fix off by one

* grandpa: use binary search to find start idx when generating warp sync proof

* grandpa: add method to verify warp sync proofs

* grandpa: remove unnecessary code to skip authority set changes

* grandpa: add test for warp sync proof generation and verification

* grandpa: add missing docs

* grandpa: remove trailing comma

* Start

* Finish!!!!

* Update client/basic-authorship/src/basic_authorship.rs

Co-authored-by: André Silva <[email protected]>

* Review comments

Co-authored-by: André Silva <[email protected]>

* Make `on_slot` return the block with the post header

Before this pr `on_slot` returned the pre block. However this is wrong,
because adding some post digest changes the hash of the header. Thus,
we need to make sure to return the correct block that uses the post
header.

* Update primitives/consensus/common/src/block_import.rs

Co-authored-by: André Silva <[email protected]>

Co-authored-by: André Silva <[email protected]>

* Make keystore return `None` when a key doesn't exist

* Fixes

* More fixes

* Update comment

* Update primitives/keystore/src/lib.rs

Co-authored-by: André Silva <[email protected]>

* Update client/keystore/src/local.rs

Co-authored-by: André Silva <[email protected]>

* Address comments

* Update client/keystore/src/local.rs

Co-authored-by: André Silva <[email protected]>

Co-authored-by: André Silva <[email protected]>

* Ensure we spawn the block import worker as an essential task

This pr ensures that we spawn the block import worker as an essential
task. This is quite important as we need to bring down the node when the
block import is done. Besides that it adds some debug output to the
block import worker.

* Don't be stupid :D

* A clean new attempt

* Checkpoint to move remote.

* A lot of dependency wiring to make it feature gated.

* bad macro, bad macro.

* Undo the DB mess.

* Update frame/support/src/traits.rs

Co-authored-by: Alexander Popiak <[email protected]>

* Apply suggestions from code review

Co-authored-by: Alexander Popiak <[email protected]>

* unbreak the build

* Update frame/try-runtime/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update utils/frame/try-runtime/cli/Cargo.toml

Co-authored-by: Shawn Tabrizi <[email protected]>

* Update frame/try-runtime/Cargo.toml

Co-authored-by: Shawn Tabrizi <[email protected]>

* Address most review grumbles.

* Fix build

* Add some comments

* Remove allowing one pallet at a time.

* More grumbles.

* relocate remote-ext

* Fix build

Co-authored-by: Alexander Popiak <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Shawn Tabrizi <[email protected]>

* Fix warning in rustdoc job

* More fixes

* Remove `build-rust-doc` job

Remove this job until upstream is fixed:
https://github.com/rust-lang/rust/issues/82284

* CI: temp. remove of the publishing job, no use of it w/o build

Co-authored-by: Denis P <[email protected]>

Currently we always use a boxed future everywhere anyway. This also
enables us to use a boxed `SlotWorker` (which is required for Cumulus).

* Extract transactions handling from protocol.rs

* Oops, boolean

* Do this better

* Update client/network/src/transactions.rs

Co-authored-by: Nikolay Volf <[email protected]>

* [WIP] Fix handshake

* Finish handshake change

* Bugfix

Co-authored-by: Nikolay Volf <[email protected]>

* Make it clear in CLI that paritydb is experimental

Sadly this is a breaking change for the CLI.

* Update client/cli/src/params/database_params.rs

Co-authored-by: André Silva <[email protected]>

Co-authored-by: André Silva <[email protected]>

* Fix tracing tests (#8022)

* Fix tracing tests

The tests were not working properly.

1. Some test was setting a global subscriber, this could lead to racy
conditions with other tests.

2. A logging test called `process::exit` which is completly wrong.

* Update client/tracing/src/lib.rs

Co-authored-by: David <[email protected]>

* Review comments

Co-authored-by: David <[email protected]>

* Fix tracing spans are not being forwarded to spawned task (#8009)

* Fix tracing spans are not being forwarded to spawned task

There is a bug that tracing spans are not forwarded to spawned task. The
problem was that only the telemetry span was forwarded. The solution to
this is to use the tracing provided `in_current_span` to capture the
current active span and pass the telemetry span explictely. We will now
always enter the span when the future is polled. This is essentially the
same strategy as tracing is doing with its `Instrumented`, but now
extended for our use case with having multiple spans active.

* More tests

* Proper test for telemetry and prefix span

* WIP

* Fix test (need to create & enter the span at the same time)

* WIP

* Remove telemtry_span from sc_service config

* CLEANUP

* Update comment

* Incorrect indent

* More meaningful name

* Dedent

* Naming XD

* Attempt to make a more complete test

* lint

* Missing licenses

* Remove user data

* CLEANUP

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* CLEANUP

* Apply suggestion

* Update bin/node/cli/tests/telemetry.rs

Co-authored-by: David <[email protected]>

* Wrapping lines

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: David <[email protected]>

* grandpa: make the VotingRule api async

* grandpa: add docs to VotingRuleResult

* grandpa: formatting

* grandpa: use async blocks

Co-authored-by: Bastian Köcher <[email protected]>

* grandpa: expose VotingRuleResult

* grandpa: revert some broken changes to async syntax

* grandpa: use finality-grandpa v0.14.0

* grandpa: bump impl_version

Co-authored-by: Bastian Köcher <[email protected]>

* Ahh

* Work work work

* Fix all the compilation errors

* Fix test

* More fixes...