Companion PR for `Fix the browser node and ensure it doesn't colour the informant output` #6457 (#1307)

* Companion for Substrate #6269 (nested storage tx)

* Adjust to PR changes

* Bump to newest substrate

* Bump spec

* Switch branch

* Remove redundant informant_prefix

* Fix collator tests

* Update branch

* Revert "Switch branch"

This reverts commit bcce560c51fbc9db604cf3b0f615bfd21419f87d.

Co-authored-by: Alexander Theißen <[email protected]>

* node: spawn babe and grandpa as blocking tasks

* update to latest substrate

* introduce polkadot-node-primitives

* guide: change statement distribution message types

* guide: remove variant from `CandidateSelectionMessage`

* add a few more message types

* add TODOs

* Almost all messages

* NewBackedCandidate notification

* Formatting

* Use AttestedCandidate as BackedCandidate

* Update node/primitives/src/lib.rs

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* Fix the tests

* Bring in types from #1242

* Adds network bridge messages

* More message types from doc

* use fn pointer type

* Fixes from the review

* Add missing Runtime subsystem message

* rename to CandidateValidationMessage and fix tests

Co-authored-by: Fedor Sakharov <[email protected]>
Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* New service initial commit

* More separation of the new and old services

* Fix review comments

* Adds polkadot.json

* Fix browser build

* Remove unused import

* Update node/service/src/lib.rs

Co-authored-by: Fedor Sakharov <[email protected]>

* Remove duplicate json files

Co-authored-by: Robert Habermeier <[email protected]>

* service: update to latest ServiceBuilder changes

* Update Cargo.lock

Co-authored-by: Bastian Köcher <[email protected]>

* Initial commit

Forked at: 261d701c
Parent branch: origin/master

* Add the ability to change the informant's prefix

* Update Cargo.lock

* Update Cargo.lock

* fixed tests

* Rename prefix to informant prefix

* Cast keystore pointer

* Update cargo.lock

* Companion for Substrate #6123

* Update Substrate

* Use Substrate block builder.

* Clean up metrics.

* Lock.

* Lock.

* Switch to newest basic authorship interface

* Update Substrate reference and polkadot spec_version

* Let's improve

Co-authored-by: Bastian Köcher <[email protected]>

* Add new trait to offences

* companion for substrate#6069

* update to latest changes

* Update Cargo.lock

Co-authored-by: André Silva <[email protected]>
Co-authored-by: Gav Wood <[email protected]>

* add metrics

* reset & update

* update

Enable babe_epochAuthorship that was added in substrate#4729 for
querying information about slots that can be claimed in the current
epoch.

* grandpa: report equivocations on all runtimes

* runtime: fix test-runtime compilation

* runtime: enable historical session manager on all runtimes

* runtime: fix indentation

* runtime: use strong key types in KeyOwnerProofSystem definitions

* update substrate

* bump spec_version of runtimes

* Companion PR for Substrate #5375

* fix compilation

* Update rpc/Cargo.toml

* update substrate

Co-authored-by: André Silva <[email protected]>
Co-authored-by: Benjamin Kampmann <[email protected]>

* Companion for Substrate#5860

* Update Substrate

* Some fixes to compile for Android

* Revert change to cli

* WIP

* collator

* fix missing reference

* Make collator builder more generic

* all clear

* revert start colator fn

* fix return type of start_collator

* sc-consensus

* bump back to master

* Update collator/src/lib.rs

Co-Authored-By: Benjamin Kampmann <[email protected]>

* Update collator/src/lib.rs

Co-Authored-By: Pierre Krieger <[email protected]>

* uncomment type MaxIterations

* reactivate append storage feature

Co-authored-by: Benjamin Kampmann <[email protected]>
Co-authored-by: Pierre Krieger <[email protected]>

Collators need to join the validation network to tell its connected
relay chain peers the leaf they listen on. This is required to make the
Parachain validator send the signed statement to the collators as well.

* prometheus registry to txpool

* Fix network tests

* Use None as the network config path

* update substrate

Co-authored-by: Ashley Ruglys <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* service/src/lib: Enable authority discovery on sentry nodes

When run as a sentry node, the authority discovery module does not
publish any addresses to the dht, but still discovers validators and
sentry nodes of validators.

* bin/node/cli/src/service: Use expressions instead of statements

* Cargo.lock: Run `cargo update`

* service/src/lib: Fix compile error

* Updated for the new db

* Bumped version

* Sort out features

* Update to substrate master

* Companion PR to channel names

* Err... fix missing event stream name

* Update Substrate

* Initial draft

* More work

* Build

* Docs

* Insert westend keys

* Add badBlock to fork from old chain

* Updated spec to reset westend

* Use raw spec

* Fix spec format and use westend2 for both id's

* Correct public key for bootnode 3

* Build

* Extra space

* Fix build

* Lock

* Update lock

* Fixes

* Fix for he startup text

* Bump

Co-authored-by: Gav Wood <[email protected]>

* Initial commit

Forked at: 9283855d
Parent branch: origin/master

* Switch substrate to branch cecton-the-revenge-of-the-cli

* Adapting code

* Update Cargo.lock

* Adapting code

* Adapt more code

* Implement force_kusama parameter

* Revert dependency update

* Adapt code to use ref to SubstrateCli object

* Updated to latest version

* Updated with latest changes

* Bump spec vesion

* Fixed tests

* WIP

Forked at: 9283855d
Parent branch: origin/master

* More fixes

* Cargo.lock

* Updated code

* Fixed and adapt

* Fixed dependency issue with wasm

* Adapted code

* Revert branch change

* Cargo.lock

* Cargo.lock

* Adapt code

* Clean-up

* More clean-up

* Cargo.lock

* Companion PR to splitting Roles

* Fix network tests

* Fix service build

* Even more fixing

* Oops, quick fix

* use is_network_authority in grandpa service config

Co-authored-by: André Silva <[email protected]>

Before the validator only send the keys if it was updated and thus the
collators would "never" be informed about the key of the validator.

* hardfork the GRANDPA authority set in kusama to fix migration issue

* move grandpa integration code

* remove unnecessary grandpa hard fork

* update to latest substrate polkadot-master

* grandpa: add voting rule to pause new votes for a period

* grandpa: increase delay

* grandpa: parse custom pause delay from cli

* grandpa: log scheduled pause on startup

* grandpa: rename parameter to grandpa_pause

* grandpa: make pause voting rule generic on block

* grandpa: add test for pause voting rule

* grandpa: add hardcoded pause

* collator: fix test compilation

* Flag to force kusama runtime

* Chainspecs for kusama

* Polkadot config for westend

Co-Authored-By: Bastian Köcher <[email protected]>

* network/src/legacy/gossip: Wrap GossipEngine in Arc Mutex & lock it on use

`GossipEngine` in itself has no need to be Send and Sync, given that it
does not rely on separately spawned background tasks anymore.
`RegisteredMessageValidator` needs to be `Send` and `Sync` due to the
inherited trait bounds from implementing `GossipService`. In addition
`RegisteredMessageValidator` derives `Clone`. Thereby `GossipEngine`
needs to be wrapped in an `Arc` and `Mutex` to keep the status quo.

* Needed fixes.

* Fixes

* Fixed build

* Fixed build w benchmarking CLI

* Fixed building tests

* Added --dev shortcut

Co-authored-by: arkpar <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Max Inden <[email protected]>

* fix

* Starting to add benchmarks

* make compile

* add benchmarks

* Make work with Substrate master

* Bench validate unsigned

* back to polkadot master

* starting to add cli with feature flag

* more stuff

* Add to kusama

* Update Cargo.lock

* fix dev dep

* bump wasm builder

* Remove encode from keccak benchmark

* bump spec

* Add weight documentation

* Update Cargo.lock

* Update check_runtime.sh

* Update publish_draft_release.sh

* Update Cargo.lock

Co-authored-by: thiolliere <[email protected]>

* Fix browser node

* Use feature instead

* expunge legacy code from polkadot-network

* mostly rip out old legacy protocol from service

* ensure validation work is spawned by incoming messages

* decouple availabliity store from network logic; clean up data flow

* av_store: test helpers and use futures-abort

* update polkadot-validation to pass n_validators when submitting chunks

* fallible erasure-chunk fetching

* implement `ErasureNetworking` for new network prot

* API for registering availability store in network

* fully integrate new network service into service

* fix validation tests

* scaffolding for porting collator over to new network

* track connected validators' peer IDs and distribute collators' collations

* helper in network for fetching all checked statements

* fix adder-collator

* actually register notifications protocol

* Update service/src/lib.rs

* Make needed changes to service

* Merge two companion PRs.

- #880
- #881

* Some e...

* expunge legacy code from polkadot-network

* mostly rip out old legacy protocol from service

* ensure validation work is spawned by incoming messages

* decouple availabliity store from network logic; clean up data flow

* av_store: test helpers and use futures-abort

* update polkadot-validation to pass n_validators when submitting chunks

* fallible erasure-chunk fetching

* implement `ErasureNetworking` for new network prot

* API for registering availability store in network

* fully integrate new network service into service

* fix validation tests

* scaffolding for porting collator over to new network

* track connected validators' peer IDs and distribute collators' collations

* helper in network for fetching all checked statements

* fix adder-collator

* actually register notifications protocol

* Update service/src/lib.rs

* merge with master

* Switch branch

* Small changes

* Update substrate branch

* Switch

* Revert "Switch branch"

This reverts commit b9d48b2ce8f5cbfa379dd385e817e80870391d9d.

* fix

* add `wipe` and `commit`

* Remove deprecated_host_interface

* Switch branch

* HasherFor -> HashFor

* More HasherFor changes

* Final touches

* Revert "Switch branch"

This reverts commit d0da27313839559de01f59690f3826fe587becb8.

Co-authored-by: thiolliere <[email protected]>
Co-authored-by: Shawn Tabrizi <[email protected]>