General Message Queue Pallet (#12485)

* The message queue

* Make fully generic

* Refactor

* Docs

* Refactor

* Use iter not slice

* Per-origin queues

* Multi-queue processing

* Introduce MaxReady

* Remove MaxReady in favour of ready ring

* Cleanups

* ReadyRing and tests

* Stale page reaping

* from_components -> from_parts

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Move WeightCounter to sp_weights

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add MockedWeightInfo

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Deploy to kitchensink

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Use WeightCounter

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Small fixes and logging

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add service_page

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Typo

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Move service_page below service_queue

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add service_message

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Use correct weight function

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Overweight execution

* Refactor

* Missing file

* Fix WeightCounter usage in scheduler

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix peek_index

Take into account that decoding from a mutable slice modifies it.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add tests and bench service_page_item

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add debug_info

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add no-progress check to service_queues

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add more benches

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Bound from_message and try_append_message

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add PageReaped event

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Rename BookStateOf and BookStateFor

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update tests and remove logging

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Remove redundant per-message origins; add footprint() and sweep_queue()

* Move testing stuff to mock.rs

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add integration test

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix no-progress check

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix debug_info

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fixup merge and tests

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix footprint tracking

* Introduce

* Formatting

* OverweightEnqueued event, auto-servicing config item

* Update tests and benchmarks

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Clippy

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add tests

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Provide change handler

* Add missing BookStateFor::insert and call QueueChangeHandler

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Docs

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update benchmarks and weights

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* More tests...

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Use weight metering functions

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* weightInfo::process_message_payload is gone

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add defensive_saturating_accrue

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Rename WeightCounter to WeightMeter

Ctr+Shift+H should do the trick.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Test on_initialize

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add module docs

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Remove origin from MaxMessageLen

The message origin is not encoded into the heap and does
therefore not influence the max message length anymore.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add BoundedVec::as_slice

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Test Page::{from_message, try_append_message}

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fixup docs

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Docs

* Do nothing in sweep_queue if the queue does not exist

... otherwise it inserts default values into the storage.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Test ring (un)knitting

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Upgrade stress-test

Change the test to not assume that all queued messages will be
processed in the next block but split it over multiple.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* More tests...

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Beauty fixes

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* clippy

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Rename BoundedVec::as_slice to as_bounded_slice

Conflicts with deref().as_slice() otherwise.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix imports

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Remove ReadyRing struct

Was used for testing only. Instead use 'fn assert_ring' which also
check the service head and backlinks.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Beauty fixes

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix stale page watermark

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Cleanup

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix test feature and clippy

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* QueueChanged handler is called correctly

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update benches

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Abstract testing functions

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* More tests

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Cleanup

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Clippy

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* fmt

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Simplify tests

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Make stuff compile

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Extend overweight execution benchmark

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Remove TODOs

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Test service queue with faulty MessageProcessor

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* fmt

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Update pallet ui tests to 1.65

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* More docs

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Review doc fixes

Co-authored-by: Robert Klotzner <[email protected]>
Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Add weight_limit to extrinsic weight of execute_overweight

* Correctly return unused weight

* Return actual weight consumed in do_execute_overweight

* Review fixes

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Set version 7.0.0-dev

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Make it compile

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Switch message_size to u64

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Switch message_count to u64

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Fix benchmarks

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Make CI green

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Docs

* Update tests

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* ".git/.scripts/bench-bot.sh" pallet dev pallet_message_queue

* Dont mention README.md in the Cargo.toml

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* Remove reference to readme

Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: parity-processbot <>
Co-authored-by: Robert Klotzner <[email protected]>
Co-authored-by: Keith Yeung <[email protected]>