* pallet-mmr: fix some typos

* pallet-mmr: make the MMR resilient to chain forks

* pallet-mmr: get hash for block that added node

* beefy-mmr: add debug logging

* add explanatory comment

* account for block offset of pallet activation

* add support for finding all nodes added by leaf

* minor improvements

* add helper to return all nodes added to mmr with a leaf append

* simplify leaf_node_index_to_leaf_index

summing the (shifted) differences in peak positions adds up to the (shifted) final position, so
don't need to fold over positions.

* dead fish: this also doesn't work

The idea was to keep a rolling window of `(parent_hash, pos)` leaf
entries in the offchain db, with the window matching the one
that provides `block_num -> block_hash` mappings in `frame_system`.

Once a leaf exits the window it would be "canonicalized" by switching
its offchain db key from `(parent_hash, pos)` to simple `pos`.

This doesn't work however because there's no way to get leaf contents
from offchain db while in runtime context.. so no way to get+clear+set
leaf to change its key in offchain db.

Ideas:
1. move the "canonicalization" logic to offchain worker
2. enhance IndexingApi with "offchain::move(old_key, new_key)"
   This is weird, but correct, deterministic and safe AFAICT, so
   it could be exposed to runtime.

* simplify rightmost_leaf_node_index_from_pos

* minor fix

* move leaf canonicalization to offchain worker

* move storage related code to storage.rs

* on offchain reads use canonic key for old leaves

* fix offchain worker write using canon key

* fix pallet-mmr tests

* add documentation and fix logging

* add offchain mmr canonicalization test

* test canon + generate + verify

* fix pallet-beefy-mmr tests

* implement review suggestions

* improve test

* pallet-mmr: add offchain pruning of forks

* pallet-mmr: improve offchain pruning

Instead of keeping pruning map as single blob in offchain db,
keep individual parent-hash lists with block-num identifier as part
of the offchain key.

Signed-off-by: acatangiu <[email protected]>

* pallet-mmr: improve MMRStore<OffchainStorage>::get()

Do the math and retrieve node using correct (canon or non-canon)
offchain db key, instead of blindly looking in both canon and non-canon
offchain db locations for each node.

Still fallback on looking at both if for any reason it's not where
expected.

Signed-off-by: acatangiu <[email protected]>

* pallet-mmr: storage: improve logs

* fix tests: correctly persist overlay

runtime indexing API works on overlay, whereas offchain context
bypasses overlay, so for loops > canon-window, canon would fail.

* pallet-mmr: fix numeric typo in test

* add comment around LeafData requirements

Signed-off-by: acatangiu <[email protected]>

Co-authored-by: Robert Hambrock <[email protected]>

* Fixed the bug with transfer value.

* Apply suggestions from code review

Co-authored-by: Alexander Theißen <[email protected]>

* Moved check into `initial_transfer`

* Fmt

Co-authored-by: Alexander Theißen <[email protected]>

* Nomination pool fix

* fmt

* fix

* pushed

* node: fix fee multiplier test

Co-authored-by: André Silva <[email protected]>

* Prevend unsoundness in environments with broken `madvise(MADV_DONTNEED)`

* Add the `std` feature to `rustix` dependency

Apparently not having this breaks compilation on non-nightly toolchains.

* Autodetect the page size when checking whether `madvise` works

* Only make sure that the madvice check doesn't return `Err`

* Democracy.fast_track not allowed with zero voting period

* revert static parameter alter line

* ensure voting period greater zero

* update doc for fast_track

* unit test: instant fast track to the next block referendum is backed

* fix typos in comments

* Separate queueing blocks for import from removal

* Add regression tests

* Remove unnecessary log

* Clear queued blocks when processed

* Move check out of match block

* Track queued block ranges

* Update client/network/sync/src/blocks.rs

* Update client/network/sync/src/blocks.rs

* Update client/network/sync/src/blocks.rs

* Update client/network/sync/src/blocks.rs

* FMT

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>

* Bump `wasmtime` to 0.37.0 and `zstd` to 0.11.2

* Bump `wasmtime` to 0.38.0

* save

* builds and old tests pass

save:  temporary value dropped while borrowed

save: finally builds

test updated but still fails

* type names enhanced

* VarSizedKey bounded to new Config param

* improved wasm runtime updated funcs

* unstable-interface tests fixed

* benchmarks fixed

* Apply suggestions from code review

Co-authored-by: Alexander Theißen <[email protected]>

* fixes on feedback

* fixes on feedback applied + make it build

* benchmarks build but fail (old)

* "Original code too large"

* seal_clear_storage bench fixed (code size workaround hack removal tbd)

* bench_seal_clear_storage pass

* bench_seal_take_storage ... ok

* added new seal_set_storage + updated benchmarks

* added new seal_get_storage + updated benchmarks

* added new seal_contains_storage + updated benchmarks

* added tests for _transparent exec functions

* wasm test for clear_storage

* wasm test for take_storage

* wasm test for new set_storage

* wasm test for new get_storage

* wasm test for new contains_storage

* CI fix

* ci fix

* ci fix

* ci fix

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* fixes according to the review feedback

* tests & benchmarks fixed

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* refactoring

* fix to runtime api

* ci fix

* ctx.get_storage() factored out

* ctx.contains_storage() factored out

* number of batches reduced for transparent hashing storage benchmarks

* contracts RPC & pallet::get_storage to use transparent hashing

* node and rpc updated to use get_storage with VarSizedKey

* refactored (more concize)

* refactored contains_storage (DRYed)

* refactored contains_storage (DRYed)

* fix rpc

* fmt fix

* more fixes in rpc

* rollback `Pallet:get_storage` to Vec<u8> and rpc and node parts related to it

* added `KeyDecodingFailed` error

* Revert weird "fmt fix"

This reverts commit c582cfff4b5cb2c9929fd5e3b45519bb24aeb657.

* node-executor basic test update

* fix node-executor basic test

* benchmarks fix

* more benchmarks fix

* FixedSizedKey is hidden from pub, VarSizedKey is exported as StorageKey

* ci fix

* set_storage benchmark fix

* ci fix

* ci fix

* comments improved

* new error code to rpc: KEY_DECODING_FAILED

* Put `rusty-cachier` before PR merge into `master` for `cargo-check-benches` job

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark pallet --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs

* minor optimization

Co-authored-by: Alexander Theißen <[email protected]>
Co-authored-by: Parity Bot <[email protected]>
Co-authored-by: Vladimir Istyufeev <[email protected]>
Co-authored-by: command-bot <>

* make pallet-tips & pallet-bounties instantiable

* update test

* add default instance

* update

* cargo fmt

* update

* update

* update

* update

* fix merge

* fix tests

* bounties benchmarking instantiable

* fix benchmarks

* make tips benchmarks instantible

Co-authored-by: Shawn Tabrizi <[email protected]>

* pallet-beefy: add Config::OnNewValidatorSet type

Add a hook to pallet-beefy for doing specific work when
BEEFY validator set changes.

For example, this can be used by pallet-beefy-mmr to cache
a lightweight MMR root over validators and make it available
to light clients.

* pallet-beefy-mmr: implement OnNewValidatorSet

Implement pallet-beefy::OnNewValidatorSet to be notified of BEEFY
validator set changes. Use the notifications to compute and cache
a light weight 'BEEFY authority set' which is an MMR root over
BEEFY validator set plus some extra info.

Previously, pallet-beefy-mmr was interogating pallet-beefy about
validator set id on every block to find out when it needs to recompute
the authority set.
By using the event-driven approach in this commit, we also save one
extra state interogation per block.

* pallet-beefy-mmr: add new authority_set() API

Expose current and next BEEFY authority sets through runtime API.
These can be directly used by light clients to avoid having them
compute them themselves based on BEEFY validator sets.

Signed-off-by: acatangiu <[email protected]>

* rename BeefyMmr exposed runtime api

* epochs: don't use gap when there's at least one genesis epoch imported

* epochs: add test for genesis gap fix

* contracts: Reduce size of deletion queue depth

* Remove unused import

* Support offline env variable in wasm builder

* Clean up

* Improve checks

Co-authored-by: Bastian Köcher <[email protected]>

* Update crate docs

* Add docs to `lib.rs` and introduce helper method `offline_build`

Co-authored-by: Bastian Köcher <[email protected]>

This ensures that there isn't any extra data attached that isn't used.

overriden -> overridden

* More robust revert procedure

Return an error if revert is called in a node that is not actively
running grandpa, i.e. grandpa genesis data has not been initialized.
Previous implementation was just firing an `unreachable!` code exception.

Furthermore we skip revert hassle if there is nothing to revert.

* Nit

* Implement Serialize/Deserialize on WeakBoundedVec

* cargo fmt

* Warn when there are too many elements while deserializing WeakBoundedVec

* Pump the gossip engine while waiting for the BEEFY runtime pallet

This fixes a memory leak when the BEEFY gadget is turned on, but
the runtime doesn't actually use BEEFY.

* Implement `FusedFuture` for `GossipEngine`

* Fuse futures outside of loops

the script executed by check-dependent-cumulus only works for PRs, as shown in https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/1630771#L87, which comes from https://github.com/paritytech/pipeline-scripts/blob/3ad10ddc0d985ef5326974a1143229c6429befab/check_dependent_project.sh#L443

* Update syn and fix compilation

* Bump pin-project

Bumps [twox-hash](https://github.com/shepmaster/twox-hash) from 1.6.2 to 1.6.3.
- [Release notes](https://github.com/shepmaster/twox-hash/releases)
- [Commits](https://github.com/shepmaster/twox-hash/compare/v1.6.2...v1.6.3)

---
updated-dependencies:
- dependency-name: twox-hash
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Besides that it also adds some UI tests.

Signed-off-by: Emison Lu <[email protected]>

* chore: reduce uxt encode times

* fmt

* Upgrade to libp2p 0.45.1

* Limit max_negotiating_inbound_streams to 512

* Upgrade prost-build to 0.10

* Set max_negotiating_inbound_streams to 2048

Co-authored-by: Pierre Krieger <[email protected]>

* Fix authority discovery protobuf

* Fix comments in authority-discovery schema

Co-authored-by: Pierre Krieger <[email protected]>

* Add a comment about transport initialization

Co-authored-by: Pierre Krieger <[email protected]>

* Removed multiply_by_rational
Replaced with multiply_by_rational_with_rounding

* fixes

* Test Fixes

* nightly fmt

* Test Fix

* Fixed fuzzer.

* Referenda & CV pallets ready

* Fix build

* Add mel_bound for Voting and Casting types

* Add mel_bound on Tally

* Add mel_bound on another Tally

* Add mel_bound for pallet_collective::RawOrigin

Co-authored-by: Keith Yeung <[email protected]>

* Remove `without_storage_info` for the authorship pallet

* Tweak impl bounds style

* Use `defensive_proof` instead of `expect`

* add Event to Pallet Transaction Payment

* Fix tests in Pallet Balance

* Fix tests in Pallet Balance/Executive/Asset-tx-payment.

* Fix

* fmt

* Fix

* Fix

* Update Cargo.lock

* Fix tests in executor

* Update frame/transaction-payment/src/lib.rs

Co-authored-by: joe petrowski <[email protected]>

* update the name of the event and fmt.

Co-authored-by: Shawn Tabrizi <[email protected]>
Co-authored-by: joe petrowski <[email protected]>

* Initial implementation of mms

* Some more attempts at `mms`

* Functioning `MMS` algorithm implementation.
Adding some tests too

* More tests and typos fixed.

* Adding fuzzer for `mms`
(but could not test it on Mac M1)

* Missing imports

* Fixing rustdoc

* More accurate implementation of `mms`

* Removing the fuzzer `mms` implementation

* Implementing `NposSolver` for `MMS`
had to add the `Clone` trait, maybe I could see if I can get rid of it.

* Fixing rust docs by adding () to resolve ambiguity

* Amending `unwrap` to `expect`
removing unneeded `Clone` trait

* Removing redundant `mms3.rs`

* Implementing `BalancingConfig` and rustdoc changes

* Implementing `weight` for `MMS`

* Implementing `weight` for `MMS`

* Fixing post merge

* Initial implementation of mms

* Some more attempts at `mms`

* Functioning `MMS` algorithm implementation.
Adding some tests too

* More tests and typos fixed.

* Adding fuzzer for `mms`
(but could not test it on Mac M1)

* Missing imports

* Fixing rustdoc

* More accurate implementation of `mms`

* Removing the fuzzer `mms` implementation

* Implementing `NposSolver` for `MMS`
had to add the `Clone` trait, maybe I could see if I can get rid of it.

* Amending `unwrap` to `expect`
removing unneeded `Clone` trait

* Fixing rust docs by adding () to resolve ambiguity

* Removing redundant `mms3.rs`

* Implementing `BalancingConfig` and rustdoc changes

* Implementing `weight` for `MMS`

* Implementing `weight` for `MMS`

* Fixing post merge

* Removing left over from rebase

* Fixing tests

* Removing unneeded import

* Removing unneeded functions

* Removing useless imports

Co-authored-by: kianenigma <[email protected]>

* Introduce BoundedVec

* Fix typos

* Add comments to the bounds

* Remove migration

* Improve bound value access syntax

This makes it possible to run the tests manually, without them expecting to be run in a special
folder etc.

* bump jsonrpsee to fix #11480

In addition it adds WebSocket server-side pings which is configured to
send out pings periodically every 30 seconds.

* use released crates.io version

* Update Cargo.toml

* Make it possible to disable RocksDB completely

* Make ParityDB non-optional

* Address review comments