Skip to content
  • eskimor's avatar
    Parathreads Feature Branch (#6969) · eaf057c5
    eskimor authored
    
    
    * First baby steps
    
    * Split scheduler into several modules
    
    * Towards a more modular approach for scheduling
    
    * move free_cores; IntoInterator -> BTreeMap
    
    * Move clear()
    
    * Move more functions out of scheduler
    
    * Change weight composition
    
    * More abstraction
    
    * Further refactor
    
    * clippy
    
    * fmt
    
    * fix test-runtime
    
    * Add parathreads pallet to construct_runtime!
    
    * Make all runtimes use (Parachains, Parathreads) scheduling
    
    * Delete commented out code
    
    * Remove parathreads scheduler from westend, rococo, and kusama
    
    * fix rococo, westend, and kusama config
    
    * Revert "fix rococo, westend, and kusama config"
    
    This reverts commit 59e4de380d5c7d17eaaba5e2c2b81405de3465e3.
    
    * Revert "Remove parathreads scheduler from westend, rococo, and kusama"
    
    This reverts commit 4c44255296083ac5670560790ed77104917890a4.
    
    * Remove CoreIndex from free_cores
    
    * Remove unnecessary struct for parathreads
    
    * parathreads provider take 1
    
    * Comment out parathread tests
    
    * Pop into lookahead
    
    * fmt
    
    * Fill lookahead with two entries for parachains
    
    * fmt
    
    * Current stage
    
    * Towards ab parathreads
    
    * no AB use
    
    * Make tests typecheck
    
    * quick hack to set scheduling lookahead to 1
    
    * Fix scheduler tests
    
    * fix paras_inherent tests
    
    * misc
    
    * Update more of a test
    
    * cfg(test)
    
    * some cleanup
    
    * Undo paras_inherent changes
    
    * Adjust paras inherent tests
    
    * Undo changes to v2 primitives
    
    * Undo v2 mod changes to tests
    
    * minor
    
    * Remove parathreads assigner and pallet
    
    * minor
    
    * minor
    
    * more cleanup
    
    * fmt
    
    * minor
    
    * minor
    
    * minor
    
    * Remove on_new_session from assignment provider
    
    * Make adder collator integration test pass
    
    * disable failing unit tests
    
    * minor
    
    * minor
    
    * re-enable one unit test
    
    * minor
    
    * handle retries, add concluded para to pop interface
    
    * comment out unused code
    
    * Remove core_para from interface
    
    * Remove first claimqueue element on clear if None instead removing all Nones
    
    * Move claimqueue get out of loop
    
    * Use VecDeque instead of Ved in ClaimQueue
    
    * Make occupied() AB ready(?)
    
    * handle freed disputed in clear_and_fill_claimqueue
    
    * clear_and_fill_claimqueue returns scheduled Vec
    
    * Rename and minor refactor
    
    * return position of assignment taken from claimqueue
    
    * minor
    
    * Fix session boundary parachains number change + extended test
    
    * Fix runtimes
    
    * Fix polkadot runtime
    
    * Remove polkadot pallet from benchmarks
    
    * fix test runtime
    
    * Add storage migration
    
    * Minor refactor
    
    * Minor
    
    * migratin typechecks
    
    * Add migration to runtimes
    
    * Towards modular scheduling II (#6568)
    
    * Add post migration check
    
    * pebkac
    
    * Disable migrations but mine
    
    * Revert "Disable migrations but mine"
    
    This reverts commit 4fa5c5a370c199944a7e0926f50b08626bfbad4c.
    
    * Move scheduler migration
    
    * Revert "Move scheduler migration"
    
    This reverts commit a16b1659a907950bae048a9f7010f2aa76e02b6d.
    
    * Fix migration
    
    * cleanup
    
    * Don't lose retries value anymore
    
    * comment out test function
    
    * Remove retries value from Assignment again
    
    * minor
    
    * Make collator for parathreads optional
    
    * data type refactor
    
    * update scheduler tests
    
    * Change test function cfg
    
    * comment out test function
    
    * Try cfg(test) only
    
    * fix cfg flags
    
    * Add get_max_retries function to provider interface (#7047)
    
    * Fix merge commit
    
    * pebkac
    
    * fix merge
    
    * update cargo.lock
    
    * fix merge
    
    * fix merge
    
    * Use btreemap instead of vec, fix scheduler calls.
    
    * Use imported `ScheduledCore`
    
    * Remove unused import in inclusion tests
    
    * Use keys() instead of mapping over a BTreeMap
    
    * Fix migrations for parachains scheduler
    
    * Use BlockNumberFor<T> everywhere in scheduler
    
    * Add on demand assignment provider pallet (#7110)
    
    * Address some PR comments
    
    * minor
    
    * more cleanup
    
    * find_map and timeout availability fixes
    
    * Change default scheduling_lookahead to 1
    
    * Add on demand assignment provider pallet
    
    * Move test-runtime to new assignment provider
    
    * Run cargo format on scheduler tests
    
    * minor
    
    * Mutate cores in single loop
    
    * timeout predicate simplification
    
    * claimqueue desired size fix
    
    * Replace expect by ok_or
    
    * More improvements
    
    * Fix push back order and next_up_on_timeout
    
    * minor
    
    * session change docs
    
    * Add pre_new_session call to hand pre session updates
    
    * Remove sc_network dependency and PeerId from unnecessary data structures
    
    * Remove unnecessary peer_ids
    
    * Add OnDemandOrdering proxy (#7156)
    
    * Add OnDemandBidding proxy
    
    * Fix names
    
    * OnDemandAssigner for rococo only
    
    * Check PeerId in collator protocol before fetching collation
    
    * On occupied, remove non occupied cores from the claimqueue front and refill
    
    * Add missing docs
    
    * Comment out unused field
    
    * fix ScheduledCore in tests
    
    * Fix the fix
    
    * pebkac
    
    * fmt
    
    * Fix occupied dropping
    
    * Remove double import
    
    * ScheduledCore fixes
    
    * Readd sc-network dep
    
    * pebkac
    
    * OpaquePeerId -> PeerId in can_collate interface
    
    * Cargo.lock update for interface change
    
    * Remove checks not needed anymore?
    
    * Drop occupied core on session change if it would time out after the new session
    
    * Add on demand assignment provider pallet
    
    * Move test-runtime to new assignment provider
    
    * Run cargo format on scheduler tests
    
    * Add OnDemandOrdering proxy (#7156)
    
    * Add OnDemandBidding proxy
    
    * Fix names
    
    * OnDemandAssigner for rococo only
    
    * Remove unneeded config values
    
    * Update comments
    
    * Use and_then for queue position
    
    * Return the max size of the spot queue on error
    
    * Add comments to add_parathread_entry
    
    * Add module comments
    
    * Add log for when can_collate fails
    
    * Change assigner queue type to `Assignment`
    
    * Update assignment provider tests
    
    * More logs
    
    * Remove unused keyring import
    
    * disable can_collate
    
    * comment out can_collate
    
    * Can collate first checks set if empty
    
    * Move can_collate call to collation advertisement
    
    * Fix backing test
    
    * map to loop
    
    * Remove obsolete check
    
    * Move invalid collation test from backing to collator-protocol
    
    * fix unused imports
    
    * fix test
    
    * fix Debug derivation
    
    * Increase time limit on zombienet predicates
    
    * Increase zombienet timeout
    
    * Minor
    
    * Address some PR comments
    
    * Address PR comments
    
    * Comment out failing assert due to on-demand assigner missing
    
    * remove collator_restrictions info from backing
    
    * Move can_collate to ActiveParas
    
    * minor
    
    * minor
    
    * Update weight information for on demand config
    
    * Add ttl to parasentry
    
    * Fix tests missing parasentry ttl
    
    * Adjust scheduler tests to use ttl default values
    
    * Use match instead of if let for ttl drop
    
    * Use RuntimeDebug trait for `ParasEntry` fields
    
    * Add comments to on demand assignment pallet
    
    * Fix spot traffic calculation
    
    * Revert runtimedebug changes to primitives
    
    * Remove runtimedebug derivation from `ParasEntry`
    
    * Mention affinity in pallet level docs
    
    * Use RuntimeDebug trait for ParasEntry child types
    
    * Remove collator restrictions
    
    * Fix primitive versioning and other merge issues
    
    * Fix tests post merge
    
    * Fix node side tests
    
    * Edit parascheduler migration for clarity
    
    * Move parascheduler migration up to next release
    
    * Remove vestiges from merge
    
    * Fix tests
    
    * Refactor ttl handling
    
    * Remove unused things from scheduler tests
    
    * Move on demand assigner to own directory
    
    * Update documentation
    
    * Remove unused sc-network dependency in primitives
    
    Was used for collator restrictions
    
    * Remove unused import
    
    * Reenable scheduler test
    
    * Remove unused storage value
    
    * Enable timeout predicate test and fix fn
    
    Turns out that the issue with the compiler is fixed and we can now
    use impl Trait in the manner used here.
    
    * Remove unused imports
    
    * Add benchmarking entry for perbill in config
    
    * Correct typo
    
    * Address review comments
    
    * Log out errors when calculating spot traffic.
    
    * Change parascheduler's log target name
    
    * Update scheduler_common documentation
    
    * Use mutate for affinity fns, add tests
    
    * Add another on demand affinity test
    
    * Unify parathreads and parachains in HostConfig (take 2) (#7452)
    
    * Unify parathreads and parachains in HostConfig
    
    * Fixed missed occurences
    
    * Remove commented out lines
    
    * `HostConfiguration v7`
    
    * Fix version check
    
    * Add `MigrateToV7` to `Unreleased`
    
    * fmt
    
    * fmt
    
    * Fix compilation errors after the rebase
    
    * Update runtime/parachains/src/scheduler/tests.rs
    
    Co-authored-by: default avatarAnton Vilhelm Ásgeirsson <[email protected]>
    
    * Update runtime/parachains/src/scheduler/tests.rs
    
    Co-authored-by: default avatarAnton Vilhelm Ásgeirsson <[email protected]>
    
    * fmt
    
    * Fix migration test
    
    * Fix tests
    
    * Remove unneeded assert from tests
    
    * parathread_cores -> on_demand_cores; parathread_retries -> on_demand_retries
    
    * Fix a compilation error in tests
    
    * Remove unused `use`
    
    * update colander image version
    
    ---------
    
    Co-authored-by: default avataralexgparity <[email protected]>
    Co-authored-by: default avatarAnton Vilhelm Ásgeirsson <[email protected]>
    Co-authored-by: default avatarJavier Viola <[email protected]>
    
    * Fix branch after merge with master
    
    * Refactor out duplicate checks into a helper fn
    
    * Fix tests post merge
    
    * Rename add_parathread_assignment, add test
    
    * Update docs
    
    * Remove unused on_finalize function
    
    * Add weight info to on demand pallet
    
    * Update runtime/parachains/src/configuration.rs
    
    Co-authored-by: default avatarTsvetomir Dimitrov <[email protected]>
    
    * Update runtime/parachains/src/scheduler_common/mod.rs
    
    Co-authored-by: default avatarTsvetomir Dimitrov <[email protected]>
    
    * Update runtime/parachains/src/assigner_on_demand/mod.rs
    
    Co-authored-by: default avatarTsvetomir Dimitrov <[email protected]>
    
    * Add benchmarking to on demand pallet
    
    * Make place_order test check for success
    
    * Add on demand benchmarks
    
    * Add local test weights to rococo runtime
    
    * Modify TTL drop behaviour to not skip claims
    
    Previous behaviour would jump a new claim from the assignment provider
    ahead in the claimqueue, assuming lookahead is larger than 1.
    
    * Refactor ttl test to test claimqueue order
    
    * Disable place_order ext. when no on_demand cores
    
    * Use default genesis config for benchmark tests
    
    * Refactor config builder param
    
    * Move lifecycle test from scheduler to on demand
    
    * Remove unneeded lifecycle test
    
    Paras module via the parachain assignment provider doesn't provide
    new assignments if a parachain loses it's lease. The on demand
    assignment provider doesn't provide an assignment that is not a
    parathread.
    
    * Re enable validator shuffle test
    
    * More realistic weights for place_order
    
    * Remove redundant import
    
    * Fix backwards compatibility (hopefully)
    
    * ".git/.scripts/commands/bench/bench.sh" --subcommand=runtime --runtime=rococo --target_dir=polkadot --pallet=runtime_parachains::assigner_on_demand
    
    * Fix tests.
    
    * Fix off-by-one.
    
    * Re enable claimqueue fills test
    
    * Re enable schedule_rotates_groups test
    
    * Fix fill_claimqueue_fills test
    
    * Re enable next_up_on_timeout test, move fn
    
    * Do not pop from assignment provider when retrying
    
    * Fix tests missing collator in scheduledcore
    
    * Add comment about timeout predicate.
    
    * Rename parasentry retries to availability timeouts
    
    * Re enable schedule_schedules... test
    
    * Refactor prune retried test to new scheduler
    
    * Have all scheduler tests use genesis_cfg fn
    
    * Update docs
    
    * Update copyright notices on new files
    
    * Rename is_parachain_core to is_bulk_core
    
    * Remove erroneous TODO
    
    * Simplify import
    
    * ".git/.scripts/commands/bench/bench.sh" --subcommand=runtime --runtime=rococo --target_dir=polkadot --pallet=runtime_parachains::configuration
    
    * Revert AdvertiseCollation order shuffle
    
    * Refactor place_order into keepalive and allowdeath
    
    * Revert rename of hrmp max inbound channels
    
    parachain encompasses both on demand and slot auction / bulk.
    
    * Restore availability_timeout_predicate function
    
    * Clean up leftover comments
    
    * Update runtime/parachains/src/scheduler/tests.rs
    
    Co-authored-by: default avatarTsvetomir Dimitrov <[email protected]>
    
    * ".git/.scripts/commands/bench/bench.sh" --subcommand=runtime --runtime=westend --target_dir=polkadot --pallet=runtime_parachains::configuration
    
    ---------
    
    Co-authored-by: default avataralexgparity <[email protected]>
    Co-authored-by: default avataralexgparity <[email protected]>
    Co-authored-by: default avatarTsvetomir Dimitrov <[email protected]>
    Co-authored-by: default avatarJavier Viola <[email protected]>
    Co-authored-by: default avatareskimor <[email protected]>
    Co-authored-by: command-bot <>
    
    * On Demand - update weights and small nits (#7605)
    
    * Remove collator restriction test in inclusion
    
    On demand parachains won't have collator restrictions implemented in
    this way but will instead use a preferred collator registered to a
    `ParaId` in `paras_registrar`.
    
    * Remove redundant config guard for test fns
    
    * Update weights
    
    * Update WeightInfo for on_demand assigner
    
    * Unify assignment provider parameters into one call (#7606)
    
    * Combine assignmentprovider params into one fn call
    
    * Move scheduler_common to a module under scheduler
    
    * Fix ttl handling in benchmark builder
    
    * Run cargo format
    
    * Remove obsolete test.
    
    * Small improvement.
    
    * Use same migration pattern as config module
    
    * Remove old TODO
    
    * Change log target name for assigner on demand
    
    * Fix migration
    
    * Fix clippy warnings
    
    * Add HostConfiguration storage migration to V8
    
    * Add `MigrateToV8` to unreleased migrations for all runtimes
    
    * Fix storage version check for config v8
    
    * Set `StorageVersion` to 8 in `MigrateToV8`
    
    * Remove dups.
    
    * Update primitives/src/v5/mod.rs
    
    Co-authored-by: default avatarBastian Köcher <[email protected]>
    
    ---------
    
    Co-authored-by: default avataralexgparity <[email protected]>
    Co-authored-by: default avataralexgparity <[email protected]>
    Co-authored-by: default avatarantonva <[email protected]>
    Co-authored-by: default avatarTsvetomir Dimitrov <[email protected]>
    Co-authored-by: default avatarAnton Vilhelm Ásgeirsson <[email protected]>
    Co-authored-by: default avatarJavier Viola <[email protected]>
    Co-authored-by: default avatareskimor <[email protected]>
    Co-authored-by: default avatarBastian Köcher <[email protected]>
    eaf057c5