* not climate

* explain the intent of the bool in the unsigned phase

* remove glob imports from unsigned.rs

* add OffchainRepeat parameter to ElectionProviderMultiPhase

* migrate core logic from #7976

This is a much smaller diff than that PR contained, but I think
it contains all the essentials.

* improve formatting

* fix test build failures

* cause test to pass

* Apply suggestions from code review

Co-authored-by: Kian Paimani <[email protected]>

* collapse imports

* threshold acquired directly within try_acquire_offchain_lock

* add test of resubmission after interval

* add test that ocw can regenerate a failed cache when resubmitting

* ensure that OCW solutions are of the correct round

This should help prevent stale cached solutions from persisting
past the election for which they are intended.

* add test of pre-dispatch round check

* use `RawSolution.round` instead of redundantly externally

* unpack imports

Co-authored-by: Kian Paimani <[email protected]>

* rename `OFFCHAIN_HEAD_DB` -> `OFFCHAIN_LOCK`

* rename `mine_call` -> `mine_checked_call`

* eliminate extraneous comma

* check cached call is current before submitting

* remove unused consts introduced by bad merge.

Co-authored-by: Guillaume Thiolliere <[email protected]>

* resubmit when our solution beats queued solution

* clear call cache if solution fails to submit

* use local storage; clear on ElectionFinalized

* Revert "use local storage; clear on ElectionFinalized"

This reverts commit 4b46a9388532d0c09b337dc7c7edf76044a6cee8.

* BROKEN: try to filter local events in OCW

* use local storage; simplify score fetching

* fix event filter

* mutate storage instead of setting it

* StorageValueRef::local isn't actually implemented yet

* add logging for some events of interest in OCW miner

* rename kill_solution -> kill_ocw_solution to avoid ambiguity

* defensive err instead of unreachable given unreachable code

* doc punctuation

Co-authored-by: Kian Paimani <[email protected]>

* distinguish miner errors between "out of date" and "call invalid"

* downgrade info logs -> debug

* ensure encoded call decodes as a call

* fix semantics of validation of pre-dispatch failure for wrong round

* move score check within `and_then`

* add test that offchain workers clear their cache after election

* ensure that bad ocw submissions are not retained for resubmission

* simplify fn ocw_solution_exists

* add feasibility check when restoring cached solution

should address https://github.com/paritytech/substrate/pull/8290/files#r617533358

restructures how the checks are sequenced, which simplifies legibility.

* simplify checks again

Co-authored-by: Kian Paimani <[email protected]>
Co-authored-by: Guillaume Thiolliere <[email protected]>

* Removed can_report api from OnOffenceHandler

* Removed DeferredOffences and create a storage migration

* Removed missing comments

* Mock set_deferred_offences and deferred_offences methods

* OnOffenceHandler::on_offence always succeed

* Fix benchmark tests

* Fix runtime-benchmark cfg methods

* Removed 'applied' attribute from Offence event

* refactor deprecated deferred offences getter

* Validate if offences are submited after on_runtime_upgrade

* update changelog

* Remove empty lines

* Fix remove_deferred_storage weights

* Remove Offence::on_runtime_upgrade benchmark

* Revert CHANGELOG.md update

* Deprecate DeferredOffenceOf type

* Update copyright

Co-authored-by: Kian Paimani <[email protected]>

* Add migration logs

Co-authored-by: Kian Paimani <[email protected]>

* Fix migration log

* Remove unused import

* Add migration tests

* rustfmt

* use generate_storage_alias! macro

* Refactor should_resubmit_deferred_offences test

* Replace spaces by tabs

* Refactor should_resubmit_deferred_offences test

* Removed WeightSoftLimit

* Removed WeightSoftLimit from tests and mocks

* Remove unused imports

* Apply suggestions from code review

Co-authored-by: Kian Paimani <[email protected]>

* Refactor election solution trimming for efficiency

The previous version always trimmed the `CompactOf<T>` instance,
which was intrinsically inefficient: that's a packed data structure,
which is naturally expensive to edit. It's much easier to edit
the unpacked data structures: the `voters` and `assignments` lists.

* rework length-trim tests to work with the new interface

Test suite now compiles. Tests still don't pass because the macro
generating the compact structure still generates `unimplemented!()`
for the actual `compact_length_of` implementation.

* simplify

* add a fuzzer which can validate `Compact::encoded_size_for`

The `Compact` solution type is generated distinctly for each runtime,
and has both three type parameters and a built-in limit to the number
of candidates that each voter can vote for. Finally, they have an
optional `#[compact]` attribute which changes the encoding behavior.

The assignment truncation algorithm we're using depends on the ability
to efficiently and accurately determine how much space a `Compact`
solution will take once encoded.

Together, these two facts imply that simple unit tests are not
sufficient to validate the behavior of `Compact::encoded_size_for`.
This commit adds such a fuzzer. It is designed such that it is possible
to add a new fuzzer to the family by simply adjusting the
`generate_solution_type` macro invocation as desired, and making a
few minor documentation edits.

Of course, the fuzzer still fails for now: the generated implementation
for `encoded_size_for` is still `unimplemented!()`. However, once
the macro is updated appropriately, this fuzzer family should allow
us to gain confidence in the correctness of the generated code.

* Revert "add a fuzzer which can validate `Compact::encoded_size_for`"

This reverts commit 916038790887e64217c6a46e9a6d281386762bfb.

The design of `Compact::encoded_size_for` is flawed. When `#[compact]`
mode is enabled, every integer in the dataset is encoded using run-
length encoding. This means that it is impossible to compute the final
length faster than actually encoding the data structure, because the
encoded length of every field varies with the actual value stored.

Given that we won't be adding that method to the trait, we won't be
needing a fuzzer to validate its performance.

* revert changes to `trait CompactSolution`

If `CompactSolution::encoded_size_for` can't be implemented in the
way that we wanted, there's no point in adding it.

* WIP: restructure trim_assignments_length by actually encoding

This is not as efficient as what we'd hoped for, but it should still
be better than what it's replacing. Overall efficiency of
`fn trim_assignments_length` is now `O(edges * lg assignments.len())`.

* fix compiler errors

* don't sort voters, just assignments

Sorting the `voters` list causes lots of problems; an invariant that
we need to maintain is that an index into the voters list has a stable
meaning.

Luckily, it turns out that there is no need for the assignments list
to correspond to the voters list. That isn't an invariant, though previously
I'd thought that it was.

This simplifies things; we can just leave the voters list alone,
and sort the assignments list the way that is convenient.

* WIP: add `IndexAssignment` type to speed up repeatedly creating `Compact`

Next up: `impl<'a, T> From<&'a [IndexAssignmentOf<T>]> for Compact`,
in the proc-macro which makes `Compact`. Should be a pretty straightforward
adaptation of `from_assignment`.

* Add IndexAssignment and conversion method to CompactSolution

This involves a bit of duplication of types from
`election-provider-multi-phase`; we'll clean those up shortly.

I'm not entirely happy that we had to add a `from_index_assignments`
method to `CompactSolution`, but we couldn't define
`trait CompactSolution: TryFrom<&'a [Self::IndexAssignment]` because
that made trait lookup recursive, and I didn't want to propagate
`CompactSolutionOf<T> + TryFrom<&[IndexAssignmentOf<T>]>` everywhere
that compact solutions are specified.

* use `CompactSolution::from_index_assignment` and clean up dead code

* get rid of `from_index_assignments` in favor of `TryFrom`

* cause `pallet-election-provider-multi-phase` tests to compile successfully

Mostly that's just updating the various test functions to keep track of
refactorings elsewhere, though in a few places we needed to refactor some
test-only helpers as well.

* fix infinite binary search loop

Turns out that moving `low` and `high` into an averager function is a
bad idea, because the averager gets copies of those values, which
of course are never updated. Can't use mutable references, because
we want to read them elsewhere in the code. Just compute the average
directly; life is better that way.

* fix a test failure

* fix the rest of test failures

* remove unguarded subtraction

* fix npos-elections tests compilation

* ensure we use sp_std::vec::Vec in assignments

* add IndexAssignmentOf to sp_npos_elections

* move miner types to `unsigned`

* use stable sort

* rewrap some long comments

* use existing cache instead of building a dedicated stake map

* generalize the TryFrom bound on CompactSolution

* undo adding sp-core dependency

* consume assignments to produce index_assignments

* Add a test of Assignment -> IndexAssignment -> Compact

* fix `IndexAssignmentOf` doc

* move compact test from sp-npos-elections-compact to sp-npos-elections

This means that we can put the mocking parts of that into a proper
mock package, put the test into a test package among other tests.

Having the mocking parts in a mock package enables us to create a
benchmark (which is treated as a separate crate) import them.

* rename assignments -> sorted_assignments

* sort after reducing to avoid potential re-sort issues

* add runtime benchmark, fix critical binary search error

"Why don't you add a benchmark?", he said. "It'll be good practice,
and can help demonstrate that this isn't blowing up the runtime."

He was absolutely right.

The biggest discovery is that adding a parametric benchmark means that
you get a bunch of new test cases, for free. This is excellent, because
those test cases uncovered a binary search bug. Fixing that simplified
that part of the code nicely.

The other nice thing you get from a parametric benchmark is data about
what each parameter does. In this case, `f` is the size factor: what
percent of the votes (by size) should be removed. 0 means that we should
keep everything, 95 means that we should trim down to 5% of original size
or less.

```
Median Slopes Analysis
========
-- Extrinsic Time --

Model:
Time ~=     3846
    + v    0.015
    + t        0
    + a    0.192
    + d        0
    + f        0
              µs

Min Squares Analysis
========
-- Extrinsic Time --

Data points distribution:
    v     t     a     d     f   mean µs  sigma µs       %
<snip>
 6000  1600  3000   800     0      4385     75.87    1.7%
 6000  1600  3000   800     9      4089     46.28    1.1%
 6000  1600  3000   800    18      3793     36.45    0.9%
 6000  1600  3000   800    27      3365     41.13    1.2%
 6000  1600  3000   800    36      3096     7.498    0.2%
 6000  1600  3000   800    45      2774     17.96    0.6%
 6000  1600  3000   800    54      2057     37.94    1.8%
 6000  1600  3000   800    63      1885     2.515    0.1%
 6000  1600  3000   800    72      1591     3.203    0.2%
 6000  1600  3000   800    81      1219     25.72    2.1%
 6000  1600  3000   800    90       859     5.295    0.6%
 6000  1600  3000   800    95     684.6     2.969    0.4%

Quality and confidence:
param     error
v         0.008
t         0.029
a         0.008
d         0.044
f         0.185

Model:
Time ~=     3957
    + v    0.009
    + t        0
    + a    0.185
    + d        0
    + f        0
              µs
```

What's nice about this is the clear negative correlation between
amount removed and total time. The more we remove, the less total
time things take.

* first commit

* get to compile

* fix deprecated grandpa

* formatting

* module to pallet

* add authorship pallet to mocks

* Fix upgrade of storage.

Co-authored-by: Xiliang Chen <[email protected]>

* trigger CI

* put back doc

Co-authored-by: Guillaume Thiolliere <[email protected]>
Co-authored-by: Xiliang Chen <[email protected]>

* Add SS58 public key encoding.

* [Companion] Update Cargo.toml subkey version, readme to reflect new output (#8694)

* Update Cargo.toml

* update cargo, readme for subkey

Co-authored-by: Dan Shields <[email protected]>

Co-authored-by: Dan Shields <[email protected]>
Co-authored-by: Dan Shields <[email protected]>

* Use as_deref instead of .map in Node::client_version

* Fix reliance on non-empty NodeInfo::endpoints

* Remove unneeded `Ord` bound from All

* Fixes

* Contains supports tuples

* grandpa: use new latest stored justification in prove_finality

* grandpa: include end in range in FinalityProof::unknown_headers

* grandpa: typo in comment

* grandpa: remove ProvableJustification

* grandpa: revert unnecessary changes

* grandpa: extend AuthoritySetChangeId and cleanup get_set_id

* grandpa: move check_finality_proof to the test module

* grandpa: warn on missing authority set changes data

* grandpa: add missing use statement

* grandpa: simplify finality_proof tests

* grandpa: additional tests for finality_proof

Co-authored-by: André Silva <[email protected]>

* client/network: support sending multiple justifications

* network: flag support for multiple justifications in protobuf request

* Update client/network/src/protocol.rs

Co-authored-by: Pierre Krieger <[email protected]>

* network: update comment on protobuf field

Co-authored-by: Pierre Krieger <[email protected]>

Link was a dead-end

* Introduce macro for building Contains impl based on a match

* Fixes

* Upgrade `cargo_metadata` to 0.13.1

* Update utils/wasm-builder/src/wasm_project.rs

Co-authored-by: Bastian Köcher <[email protected]>

* bounded treasury approvals

* update benchmarks

* update configs

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_treasury --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/treasury/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* fix weight param

Co-authored-by: Parity Benchmarking Bot <[email protected]>

* Simplify some Option / Result / ? operator patterns

When those match a combinator exactly.

Tool-aided by [comby-rust](https://github.com/huitseeker/comby-rust).

* adjust after review

* adjust post-review

* Use host max log level when initializing the `RuntimeLogger`

This should fix performance problems introduced by logging under certain
circumstances. Before we always called into the host and the host was
doing the log filtering, now as the correct max log level is set, we
don't call into the host for every log line to check if it should be
logged. However, we would still call into the host to determine if
something should be logged when `something=trace` is given as we don't
forward the log targets that are enabled.

* Finish the pr

* All done

* Fix benchmarks

* Apply suggestions from code review

Co-authored-by: Guillaume Thiolliere <[email protected]>

* Fix metadata.

* Fix build

* Add migrations

* Fix

* Update frame/elections-phragmen/src/migrations/v4.rs

* Better migeation test

* More test

* Fix warn

* Update frame/elections-phragmen/src/lib.rs

Co-authored-by: Guillaume Thiolliere <[email protected]>

* Fix test

* early exit

* Fix

* Fix build

Co-authored-by: Guillaume Thiolliere <[email protected]>

* impl Zero and One for fixed point numbers

* update librocksdb-sys to support M1

* fix build

* trigger CI

* Pff-chain indexing is independent from worker.

* Remove unauthorized changes.

* 🤦

This reverts commit 6c9c687a

.

Co-authored-by: André Silva <[email protected]>

* Authority-discovery no longer publishes non-global IP addresses

* Cargo.lock

* Update client/authority-discovery/src/lib.rs

Co-authored-by: Andronik Ordian <[email protected]>

Co-authored-by: Andronik Ordian <[email protected]>

* Enable --no-private-ipv4 by default for live chains

* Make the selection a bit easier to read

* Update client/cli/src/params/network_params.rs

Co-authored-by: Andronik Ordian <[email protected]>

Co-authored-by: Andronik Ordian <[email protected]>

* Update parity-wasm

* Cleanup Cargo.toml files

- Sort dependencies
- Remove minor and fix version where they are not necessary

* Update pretty_assertions

* Update rand

* generate accessor to PalletInfo information on pallet placeholder

* remove unused

* use trait, and add tests

* less verbose doc

* add PalletInfoAccess to prelude for ease usage

* Update to libp2p 0.37

* Line widths

* Fix tests

* Add benchmakrs for membership

* Update frame/membership/src/lib.rs

Co-authored-by: Guillaume Thiolliere <[email protected]>

* Make it all work

* Add mock weights

* Update frame/membership/src/lib.rs

* Update frame/membership/src/lib.rs

Co-authored-by: Guillaume Thiolliere <[email protected]>

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_membership --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/membership/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_membership --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/membership/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Co-authored-by: Guillaume Thiolliere <[email protected]>
Co-authored-by: Parity Benchmarking Bot <[email protected]>

* migrate to new macro

* formatting

* Apply suggestions from code review

* Update frame/authority-discovery/src/lib.rs

Co-authored-by: Guillaume Thiolliere <[email protected]>

* CI: opt bench jobs; add bench to triggered job

* CI: no need in manual builds; build=publish

* CI: more logs to the trigger job

* CI: DAGs and louts

* Remove `serde` requirement from FRAME macros

Currently there is some implicit requirement on `serde` being present in
the `Cargo.toml` of a pallet when `GenesisConfig` is used. This pr
removes this requirement by using the serde attribute `serde(crate = "..")`.

* build a unique reexport of serde in impl_opaque_keys, by abusing paste doc concatenation

* Optimize

Co-authored-by: thiolliere <[email protected]>

* Make pallet Assets instantiable

* use instantiable benchmarks

Co-authored-by: Shawn Tabrizi <[email protected]>

* prototype for shawn

* Clean and document it

* Add more docs

* Move imports

* Some changes for easier compat.

* revert exmaple pallet

* rename

* BoundedVec for AccountLocks (#8580)

* Example with balances

* Fix tests

* Make it indexable

* fix

* Fix tests

* fix test

* Fix collective as well

* Fix test

* Update frame/support/src/storage/mod.rs

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>

* Repot and add for value

* Add for map and double map

* Final touches.

* Update frame/support/src/storage/bounded_vec.rs

Co-authored-by: Guillaume Thiolliere <[email protected]>

* Add a few more tests

* Add import

Co-authored-by: Peter Goodspeed-Niklaus <[email protected]>
Co-authored-by: Guillaume Thiolliere <[email protected]>

* Properly close notification substreams

* Some debug asserts

* Fix state inconsistency

* Remove erroneous debug_assert!

* Some comments