* Companion for substrate#9878

Signed-off-by: koushiro <[email protected]>

* update substrate

Co-authored-by: thiolliere <[email protected]>

* move things around, add filter methods

* validator keys, modify availability bitfields according to disputes

* simplify, keep the filter -> sanitize generic for both usecases

* minor

* assure tests still work, reduce changeset

* integration

* start entropy passing

* fixins

* compile, 1 failing test

* filter with coverage

* fixins

* Update runtime/parachains/src/paras_inherent.rs

Co-authored-by: Robert Habermeier <[email protected]>

* slip of the pen

* improve test cases

* misc

* fix

* fixins

* test avoid extra into() calls in assert_noop!

* chores

* ff

* test fixup superfluous into call

* chore: pfmt

* improve apply_block_weight_limit to try to maximize the number of sufficiently backed

blocks and add extra bitfields in a round-robin fashion

* new code treats the lack of backed candidates as ok

* Use vrf based entropy

* fixup vrf random

* add warn

* slip of the pen

* fixup

* assure ordering

* rethink apply_weights

* mock

* use a closure as predicate check

* extract and use DisputedBitfield

* chore: simplify

* remove stray dbg

* chore: fmt

* address feedback

* fix test, halfway there

* stage1

* dbg stuff

* make group selection align

* fix session index

* fix wrongly returned candidates

* cleanup

* chore fmt

* fix ensure check

* make good case test work

* more tests for bitfields

* create sanitize_backed_candidates

* fixup tests

* update guide

* add check referenced in the guide

* improve weights code

* fmt

* fixins

* Update roadmap/implementers-guide/src/runtime/inclusion.md

Co-authored-by: Zeke Mostov <[email protected]>

* compiling + address review

* add comments

* fix weight calc

* address review comments and test failure

* fix

* fix: condition

* Fix random_sel function

* Fix overlength block check

* Zeke + Ladi commit for disputes filtering + integration test builder + runtime benchmarks + integration tests

* Add benchmarks for code upgrades

* Code upgrade bench; Feature gate TestWeightInfo

* Try and make CI happier

* Feature gate enter test to not(benchmarks)

* Make sure no unused imports/fn

* refactor, re-use, the beginning

* Fix issue with frame benchmarking dep compilation

* More precise feature gating for some derives

* integrate piece-wise

* foo

* fixins

* chore fmt

* fixins

* rename const generic

* Update runtime/parachains/src/paras_inherent.rs

Co-authored-by: Zeke Mostov <[email protected]>

* Fix compilation

* limit to test

* remove unused spam slots

* spellcheck

* remove a tick, fix a typo

* Add Code upgrade weights

* comment improvements + >=

Co-authored-by: Zeke Mostov <[email protected]>

* remove another tick

* Update runtime/parachains/src/paras_inherent/benchmarking.rs

Co-authored-by: Zeke Mostov <[email protected]>

* saturating fixins + some spaces

* fix

* benchmarking - preliminary results

* Add training wheels

* Refactor some early exit logic for enter

* Gracefully handle filtering bitfields & candidates (#4280)

This updates the logic for sanitize_bitfields and sanitize_backed_candidates to never error when there is an issue, but instead to simply skip the problematic items.

* Refactor inherent data weight limiting logic (#4287)

* Apply suggestions from code review

* Update runtime/parachains/src/builder.rs

Co-authored-by: Zeke Mostov <[email protected]>

* Update runtime/parachains/src/builder.rs

* Update runtime/parachains/src/paras_inherent.rs

* final pass

* Run cargo +nightly-2021-10-29 fmt

* Update implementors guide with `sanitize_*` & `enter` (#4294)

* Make spell check happier

* Make wasm runtimes compile with benchmarks enabled (#4303)

* comment stuff out, use old toml

* Seems to be working?

* Remove feature gating from builder

* Remove commented out stuff

* Remove generic from digest

* Update weight files for runtime

Co-authored-by: Robert Habermeier <[email protected]>
Co-authored-by: Zeke Mostov <[email protected]>
Co-authored-by: Lldenaurois <[email protected]>
Co-authored-by: Zeke Mostov <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.69 to 1.0.70.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.69...v1.0.70

)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [hex-literal](https://github.com/RustCrypto/utils) from 0.3.3 to 0.3.4.
- [Release notes](https://github.com/RustCrypto/utils/releases)
- [Commits](https://github.com/RustCrypto/utils/compare/hex-literal-v0.3.3...hex-literal-v0.3.4

)

---
updated-dependencies:
- dependency-name: hex-literal
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Remove light client companion

* Update substrate

* cargo fmt

* Fixed benches

* fmt

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.68 to 1.0.69.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.68...v1.0.69

)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* support dispatch_as

* add weight

* Update Cargo.lock

Co-authored-by: Shawn Tabrizi <[email protected]>

* bump rococo spec_version

* replace the spec file

* bump rococo chain id

* use rococo-staging

closes https://github.com/paritytech/polkadot/issues/4133

* Introduce `OriginPrivilegeCmp`

Make use of the new `OriginPrivilegeCmp` feature of pallet scheduler.
The idea is to make sure that a council origin with more yes votes has
higher privileges than a council origin with less yes votes. This solves
a problem that happened recently on Kusama where the council tried to
cancel a scheduled task, but that required that the same council origin
was used while the cancel motion had more yes votes than the origin
motion that scheduled this task. With this origin privilege compare it
should now be solved by checking the yes votes directly.

* Feedback

* update lockfile for substrate

Co-authored-by: parity-processbot <>

Signed-off-by: koushiro <[email protected]>

* fix xcm benchmark

* fix overflow

* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=pallet_xcm_benchmarks::fungible --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --template=./xcm/pallet-xcm-benchmarks/template.hbs --output=./runtime/westend/src/weights/xcm/pallet_xcm_benchmarks_fungible.rs

* fix spelling

* fmt

* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=pallet_xcm_benchmarks::fungible --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --template=./xcm/pallet-xcm-benchmarks/template.hbs --output=./runtime/westend/src/weights/xcm/pallet_xcm_benchmarks_fungible.rs

Co-authored-by: Parity Bot <[email protected]>

* Allow Queries and Subscriptions

* Formatting

* Formatting

* Enable bags-list pallet in polkadot

* add files

* Remove trailing semicolon

* more fixes

* Update runtime/polkadot/src/lib.rs

* add features

* remove par

Co-authored-by: Keith Yeung <[email protected]>
Co-authored-by: Shawn Tabrizi <[email protected]>

* Introduce new Runtime API endpoint

`persisted_validation_data_with_code_hash` that will be used
by the candidate validation subsystem in order to decrease amount
of runtime API requests.

* Node-side part of new runtime API request

* Define code hash getter via macro

* Rename new endpoint to `assumed_validation_data`

* Docs for runtime API impl of new endpoint

* AssumedValidationData specialized request function

* fmt

* export xcm_pallet config

Signed-off-by: Cheng JIANG <[email protected]>

* run format

Signed-off-by: Cheng JIANG <[email protected]>

* fix typo

Signed-off-by: Cheng JIANG <[email protected]>

* add generic parameter to support different runtimes

* Revert "add generic parameter to support different runtimes"

This reverts commit 4405ea9fab24307318ffc20a833010fe864b3be3.

* remove stale migrations

* fix warnings

* Update runtime/westend/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>

* quotes around Job::ToJob
* add new words to dictionary

* Fix typos

* Update ump.rs

* Update ump.rs

* Update ump.rs

* Rename Statemint where appropriate

* Rename to Rockmine

Co-authored-by: Alexander Popiak <[email protected]>

Co-authored-by: Alexander Popiak <[email protected]>

* bump version

* update Cargo.lock

Co-authored-by: Andronik Ordian <[email protected]>

* Move runtime to with_transaction in order to Rollback state changes

* add assert_storage_noop to tests

* found issue

* fix up

* remove printlns

* Dispatchable XCMs should translate to v0

* Formatting

* Update Westend's XCM WrapVersion

* Add para pallets to polkadot benchmarks

* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::configuration --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_configuration.rs

* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::initializer --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_initializer.rs

* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_paras.rs

Co-authored-by: Parity Bot <[email protected]>

* validator-discovery: remove from peer set before inserting

* bump spec versions

* rework into a companion

* fmt

* fix

* fix

* one more time

* one more try

* one more try

* Revert "one more try"

This reverts commit ab6568d3b828a33dc06f5650037597fc88dd06b1.

* one more try

* one more try

* Revert "one more try"

This reverts commit 8d7369f7b78633bd1b1c5ba3e0f2a0544bdd77a5.

* fix a warning

* fix another warn

* correct log

* fix compilation

* ffs

* less cloning

* Apply suggestions from code review

Co-authored-by: Pierre Krieger <[email protected]>

* add comments and a small refactoring

* use set_reserved_peers

* cargo update -p sp-io

* rename added to num_peers

* update Substrate

Co-authored-by: Pierre Krieger <[email protected]>
Co-authored-by: parity-processbot <>

* wip

* Remove unused members

* dummy: impl another runtime API

* query the on chain disputes, and inform self

* make use of the refactor

* minro

* SPLIT ME

* write dispute values

* wip

* impl for all runtimes

* chore: fmt

* [] -> get

* fixup mock runtime

* fixup

* fixup discovery for overseer init

* chore: fmt

* spellcheck

* rename imported_on_chain_disputes -> on_chain_votes

* reduction

* make it mockable

* rename and refactor

* don't query on chain info if it's not needed

* yikes

* fmt

* fix test

* minimal fix for existing tests

* attempt to fetch the session info from the rolling window before falling back

* moved

* comments

* comments

* test for backing votes

* rename

* Update runtime/polkadot/src/lib.rs

* chore: spellcheck + dict

* chore: fmt

* fixup cache size

* add warning

* logging, rationale, less defense

* introduce new unchecked, that still checks in debug builds

* fix

* draft alt approach

* fix unused imports

* include the session

* Update node/core/dispute-coordinator/src/real/mod.rs

Co-authored-by: Robert Habermeier <[email protected]>

* provide where possible

* expand comment

* fixin

* fixup

* ValidityVote <-> ValidityAttestation <-> CompactStatement has a 1:1 representation

* mark TODO

* Update primitives/src/v1/mod.rs

Co-authored-by: Robert Habermeier <[email protected]>

* address review comments

* update docs

Co-authored-by: Robert Habermeier <[email protected]>

* Companion for substrate#9448

* fix on_disabled

* use temporary beefy branch

* revert beefy updates

* update Substrate

Co-authored-by: Andronik Ordian <[email protected]>
Co-authored-by: parity-processbot <>

* Add add call-decoding weight to benchmarks

fixes #3864

* Change order as suggested by @thiolliere

* More reordering as @KiChjang

 suggested

* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_common::claims --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_common_claims.rs

* cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_common::claims --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_common_claims.rs

Co-authored-by: Parity Bot <[email protected]>

* Add benchmarking to rococo; Remove weights from runtime_parachains

* cargo run --quiet --release --features runtime-benchmarks -- benchmark --chain=rococo-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./runtime/rococo/src/weights/runtime_parachains_paras.rs --header=./file_header.txt

* Impl TestWeightInfo for Paras and Configuration

* fmt

* cargo run --quiet --release --features runtime-benchmarks -- benchmark --chain=rococo-dev --steps=50 --repeat=20 --pallet=runtime_parachains::configuration --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./runtime/rococo/src/weights/runtime_parachains_configuration.rs --header=./file_header.txt

* Fix CI complaint: error: unused variable: `c`

* polkadot-runtime-common & polkadot-test-runtime compile

* xcm-simulator compile

* change TestWeightInfo to max_block

* cargo run --quiet --release --features runtime-benchmarks -- benchmark --chain=rococo-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./runtime/rococo/src/weights/runtime_parachains_paras.rs --header=./file_header.txt

* ordering

* cargo run --quiet --release --features runtime-benchmarks -- benchmark --chain=rococo-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./runtime/rococo/src/weights/runtime_parachains_paras.rs --header=./file_header.txt

* Replace max_block with Weight::MAX

Co-authored-by: Parity Bot <[email protected]>
Co-authored-by: Keith Yeung <[email protected]>