Skip to content
Snippets Groups Projects
  1. Nov 06, 2023
    • Andrei Sandu's avatar
      approval-voting improvement: include all tranche0 assignments in one certificate (#1178) · 0570b6fa
      Andrei Sandu authored
      
      **_PR migrated from https://github.com/paritytech/polkadot/pull/6782_** 
      
      This PR will upgrade the network protocol to version 3 -> VStaging which
      will later be renamed to V3. This version introduces a new kind of
      assignment certificate that will be used for tranche0 assignments.
      Instead of issuing/importing one tranche0 assignment per candidate,
      there will be just one certificate per relay chain block per validator.
      However, we will not be sending out the new assignment certificates,
      yet. So everything should work exactly as before. Once the majority of
      the validators have been upgraded to the new protocol version we will
      enable the new certificates (starting at a specific relay chain block)
      with a new client update.
      
      There are still a few things that need to be done:
      
      - [x] Use bitfield instead of Vec<CandidateIndex>:
      https://github.com/paritytech/polkadot/pull/6802
        - [x] Fix existing approval-distribution and approval-voting tests
        - [x] Fix bitfield-distribution and statement-distribution tests
        - [x] Fix network bridge tests
        - [x] Implement todos in the code
        - [x] Add tests to cover new code
        - [x] Update metrics
        - [x] Remove the approval distribution aggression levels: TBD PR
        - [x] Parachains DB migration 
        - [x] Test network protocol upgrade on Versi
        - [x] Versi Load test
        - [x] Add Zombienet test
        - [x] Documentation updates
      - [x] Fix for sending DistributeAssignment for each candidate claimed by
      a v2 assignment (warning: Importing locally an already known assignment)
       - [x]  Fix AcceptedDuplicate
       - [x] Fix DB migration so that we can still keep old data.
       - [x] Final Versi burn in
      
      ---------
      
      Signed-off-by: default avatarAndrei Sandu <andrei-mihail@parity.io>
      Signed-off-by: default avatarAlexandru Gheorghe <alexandru.gheorghe@parity.io>
      Co-authored-by: default avatarAlexandru Gheorghe <alexandru.gheorghe@parity.io>
  2. Oct 15, 2023
  3. Aug 31, 2023
    • Bastian Köcher's avatar
      Rename `polkadot-parachain` to `polkadot-parachain-primitives` (#1334) · a33d7922
      Bastian Köcher authored
      * Rename `polkadot-parachain` to `polkadot-parachain-primitives`
      
      While doing this it also fixes some last `rustdoc` issues and fixes
      another Cargo warning related to `pallet-paged-list`.
      
      * Fix compilation
      
      * ".git/.scripts/commands/fmt/fmt.sh"
      
      * Fix XCM docs
      
      ---------
      
      Co-authored-by: command-bot <>
    • Alin Dima's avatar
      backing: move the min votes threshold to the runtime (#1200) · d6af073a
      Alin Dima authored
      
      * move min backing votes const to runtime
      
      also cache it per-session in the backing subsystem
      
      Signed-off-by: default avataralindima <alin@parity.io>
      
      * add runtime migration
      
      * introduce api versioning for min_backing votes
      
      also enable it for rococo/versi for testing
      
      * also add min_backing_votes runtime calls to statement-distribution
      
      this dependency has been recently introduced by async backing
      
      * remove explicit version runtime API call
      
      this is not needed, as the RuntimeAPISubsystem already takes care
      of versioning and will return NotSupported if the version is not
      right.
      
      * address review comments
      
      - parametrise backing votes runtime API with session index
      - remove RuntimeInfo usage in backing subsystem, as runtime API
      caches the min backing votes by session index anyway.
      - move the logic for adjusting the configured needed backing votes with the size of the backing group
      to a primitives helper.
      - move the legacy min backing votes value to a primitives helper.
      - mark JoinMultiple error as fatal, since the Canceled (non-multiple) counterpart is also fatal.
      - make backing subsystem handle fatal errors for new leaves update.
      - add HostConfiguration consistency check for zeroed backing votes threshold
      - add cumulus accompanying change
      
      * fix cumulus test compilation
      
      * fix tests
      
      * more small fixes
      
      * fix merge
      
      * bump runtime api version for westend and rollback version for rococo
      
      ---------
      
      Signed-off-by: default avataralindima <alin@parity.io>
      Co-authored-by: default avatarJavier Viola <javier@parity.io>
  4. Aug 29, 2023
  5. Aug 28, 2023
  6. Aug 18, 2023
    • asynchronous rob's avatar
      Asynchronous Backing MegaPR (#5022) · 5174b9d2
      asynchronous rob authored
      
      * inclusion emulator logic for asynchronous backing (#4790)
      
      * initial stab at candidate_context
      
      * fmt
      
      * docs & more TODOs
      
      * some cleanups
      
      * reframe as inclusion_emulator
      
      * documentations yes
      
      * update types
      
      * add constraint modifications
      
      * watermark
      
      * produce modifications
      
      * v2 primitives: re-export all v1 for consistency
      
      * vstaging primitives
      
      * emulator constraints: handle code upgrades
      
      * produce outbound HRMP modifications
      
      * stack.
      
      * method for applying modifications
      
      * method just for sanity-checking modifications
      
      * fragments produce modifications, not prospectives
      
      * make linear
      
      * add some TODOs
      
      * remove stacking; handle code upgrades
      
      * take `fragment` private
      
      * reintroduce stacking.
      
      * fragment constructor
      
      * add TODO
      
      * allow validating fragments against future constraints
      
      * docs
      
      * relay-parent number and min code size checks
      
      * check code upgrade restriction
      
      * check max hrmp per candidate
      
      * fmt
      
      * remove GoAhead logic because it wasn't helpful
      
      * docs on code upgrade failure
      
      * test stacking
      
      * test modifications against constraints
      
      * fmt
      
      * test fragments
      
      * descending or duplicate test
      
      * fmt
      
      * remove unused imports in vstaging
      
      * wrong primitives
      
      * spellcheck
      
      * Runtime changes for Asynchronous Backing (#4786)
      
      * inclusion: utility for allowed relay-parents
      
      * inclusion: use prev number instead of prev hash
      
      * track most recent context of paras
      
      * inclusion: accept previous relay-parents
      
      * update dmp  advancement rule for async backing
      
      * fmt
      
      * add a comment about validation outputs
      
      * clean up a couple of TODOs
      
      * weights
      
      * fix weights
      
      * fmt
      
      * Resolve dmp todo
      
      * Restore inclusion tests
      
      * Restore paras_inherent tests
      
      * MostRecentContext test
      
      * Benchmark for new paras dispatchable
      
      * Prepare check_validation_outputs for upgrade
      
      * cargo run --quiet --profile=production  --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_paras.rs
      
      * cargo run --quiet --profile=production  --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_paras.rs
      
      * cargo run --quiet --profile=production  --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_paras.rs
      
      * cargo run --quiet --profile=production  --features=runtime-benchmarks -- benchmark --chain=rococo-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/rococo/src/weights/runtime_parachains_paras.rs
      
      * Implementers guide changes
      
      * More tests for allowed relay parents
      
      * Add a github issue link
      
      * Compute group index based on relay parent
      
      * Storage migration
      
      * Move allowed parents tracker to shared
      
      * Compile error
      
      * Get group assigned to core at the next block
      
      * Test group assignment
      
      * fmt
      
      * Error instead of panic
      
      * Update guide
      
      * Extend doc-comment
      
      * Update runtime/parachains/src/shared.rs
      
      Co-authored-by: default avatarRobert Habermeier <rphmeier@gmail.com>
      
      Co-authored-by: default avatarChris Sosnin <chris125_@live.com>
      Co-authored-by: default avatarParity Bot <admin@parity.io>
      Co-authored-by: default avatarChris Sosnin <48099298+slumber@users.noreply.github.com>
      
      * Prospective Parachains Subsystem (#4913)
      
      * docs and skeleton
      
      * subsystem skeleton
      
      * main loop
      
      * fragment tree basics & fmt
      
      * begin fragment trees & view
      
      * flesh out more of view update logic
      
      * further flesh out update logic
      
      * some refcount functions for fragment trees
      
      * add fatal/non-fatal errors
      
      * use non-fatal results
      
      * clear up some TODOs
      
      * ideal format for scheduling info
      
      * add a bunch of TODOs
      
      * some more fluff
      
      * extract fragment graph to submodule
      
      * begin fragment graph API
      
      * trees, not graphs
      
      * improve docs
      
      * scope and constructor for trees
      
      * add some test TODOs
      
      * limit max ancestors and store constraints
      
      * constructor
      
      * constraints: fix bug in HRMP watermarks
      
      * fragment tree population logic
      
      * set::retain
      
      * extract population logic
      
      * implement add_and_populate
      
      * fmt
      
      * add some TODOs in tests
      
      * implement child-selection
      
      * strip out old stuff based on wrong assumptions
      
      * use fatality
      
      * implement pruning
      
      * remove unused ancestor constraints
      
      * fragment tree instantiation
      
      * remove outdated comment
      
      * add message/request types and skeleton for handling
      
      * fmt
      
      * implement handle_candidate_seconded
      
      * candidate storage: handle backed
      
      * implement handle_candidate_backed
      
      * implement answer_get_backable_candidate
      
      * remove async where not needed
      
      * implement fetch_ancestry
      
      * add logic for run_iteration
      
      * add some docs
      
      * remove global allow(unused), fix warnings
      
      * make spellcheck happy (despite English)
      
      * fmt
      
      * bump Cargo.lock
      
      * replace tracing with gum
      
      * introduce PopulateFrom trait
      
      * implement GetHypotheticalDepths
      
      * revise docs slightly
      
      * first fragment tree scope test
      
      * more scope tests
      
      * test add_candidate
      
      * fmt
      
      * test retain
      
      * refactor test code
      
      * test populate is recursive
      
      * test contiguity of depth 0 is maintained
      
      * add_and_populate tests
      
      * cycle tests
      
      * remove PopulateFrom trait
      
      * fmt
      
      * test hypothetical depths (non-recursive)
      
      * have CandidateSeconded return membership
      
      * tree membership requests
      
      * Add a ProspectiveParachainsSubsystem struct
      
      * add a staging API for base constraints
      
      * add a `From` impl
      
      * add runtime API for staging_validity_constraints
      
      * implement fetch_base_constraints
      
      * implement `fetch_upcoming_paras`
      
      * remove reconstruction of candidate receipt; no obvious usecase
      
      * fmt
      
      * export message to broader module
      
      * remove last TODO
      
      * correctly export
      
      * fix compilation and add GetMinimumRelayParent request
      
      * make provisioner into a real subsystem with proper mesage bounds
      
      * fmt
      
      * fix ChannelsOut in overseer test
      
      * fix overseer tests
      
      * fix again
      
      * fmt
      
      * Integrate prospective parachains subsystem into backing: Part 1 (#5557)
      
      * BEGIN ASYNC candidate-backing CHANGES
      
      * rename & document modes
      
      * answer prospective validation data requests
      
      * GetMinimumRelayParents request is now plural
      
      * implement an implicit view utility for backing subsystems
      
      * implicit-view: get allowed relay parents
      
      * refactorings and improvements to implicit view
      
      * add some TODOs for tests
      
      * split implicit view updates into 2 functions
      
      * backing: define State to prepare for functional refactor
      
      * add some docs
      
      * backing: implement bones of new leaf activation logic
      
      * backing: create per-relay-parent-states
      
      * use new handle_active_leaves_update
      
      * begin extracting logic from CandidateBackingJob
      
      * mostly extract statement import from job logic
      
      * handle statement imports outside of job logic
      
      * do some TODO planning for prospective parachains integration
      
      * finish rewriting backing subsystem in functional style
      
      * add prospective parachains mode to relay parent entries
      
      * fmt
      
      * add a RejectedByProspectiveParachains error
      
      * notify prospective parachains of seconded and backed candidates
      
      * always validate candidates exhaustively in backing.
      
      * return persisted_validation_data from validation
      
      * handle rejections by prospective parachains
      
      * implement seconding sanity check
      
      * invoke validate_and_second
      
      * Alter statement table to allow multiple seconded messages per validator
      
      * refactor backing to have statements carry PVD
      
      * clean up all warnings
      
      * Add tests for implicit view
      
      * Improve doc comments
      
      * Prospective parachains mode based on Runtime API version
      
      * Add a TODO
      
      * Rework seconding_sanity_check
      
      * Iterate over responses
      
      * Update backing tests
      
      * collator-protocol: load PVD from runtime
      
      * Fix validator side tests
      
      * Update statement-distribution to fetch PVD
      
      * Fix statement-distribution tests
      
      * Backing tests with prospective paras #1
      
      * fix per_relay_parent pruning in backing
      
      * Test multiple leaves
      
      * Test seconding sanity check
      
      * Import statement order
      
      Before creating an entry in `PerCandidateState` map
      wait for the approval from the prospective parachains
      
      * Add a test for correct state updates
      
      * Second multiple candidates per relay parent test
      
      * Add backing tests with prospective paras
      
      * Second more than one test without prospective paras
      
      * Add a test for prospective para blocks
      
      * Update malus
      
      * typos
      
      Co-authored-by: default avatarChris Sosnin <chris125_@live.com>
      
      * Track occupied depth in backing per parachain (#5778)
      
      * provisioner: async backing changes (#5711)
      
      * Provisioner changes for async backing
      
      * Select candidates based on prospective paras mode
      
      * Revert naming
      
      * Update tests
      
      * Update TODO comment
      
      * review
      
      * provisioner: async backing changes (#5711)
      
      * Provisioner changes for async backing
      
      * Select candidates based on prospective paras mode
      
      * Revert naming
      
      * Update tests
      
      * Update TODO comment
      
      * review
      
      * fmt
      
      * Network bridge changes for asynchronous backing + update subsystems to handle versioned packets (#5991)
      
      * BEGIN STATEMENT DISTRIBUTION WORK
      
      create a vstaging network protocol which is the same as v1
      
      * mostly make network bridge amenable to vstaging
      
      * network-bridge: fully adapt to vstaging
      
      * add some TODOs for tests
      
      * fix fallout in bitfield-distribution
      
      * bitfield distribution tests + TODOs
      
      * fix fallout in gossip-support
      
      * collator-protocol: fix message fallout
      
      * collator-protocol: load PVD from runtime
      
      * add TODO for vstaging tests
      
      * make things compile
      
      * set used network protocol version using a feature
      
      * fmt
      
      * get approval-distribution building
      
      * fix approval-distribution tests
      
      * spellcheck
      
      * nits
      
      * approval distribution net protocol test
      
      * bitfield distribution net protocol test
      
      * Revert "collator-protocol: fix message fallout"
      
      This reverts commit 07cc887303e16c6b3843ecb25cdc7cc2080e2ed1.
      
      * Network bridge tests
      
      Co-authored-by: default avatarChris Sosnin <chris125_@live.com>
      
      * remove max_pov_size requirement from prospective pvd request (#6014)
      
      * remove max_pov_size requirement from prospective pvd request
      
      * fmt
      
      * Extract legacy statement distribution to its own module (#6026)
      
      * add compatibility type to v2 statement distribution message
      
      * warning cleanup
      
      * handle compatibility layer for v2
      
      * clean up an unimplemented!() block
      
      * circulate statements based on version
      
      * extract legacy v1 code into separate module
      
      * remove unimplemented
      
      * clean up naming of from_requester/responder
      
      * remove TODOs
      
      * have backing share seconded statements with PVD
      
      * fmt
      
      * fix warning
      
      * Quick fix unused warning for not yet implemented/used staging messages.
      
      * Fix network bridge test
      
      * Fix wrong merge.
      
      We now have 23 subsystems (network bridge split + prospective
      parachains)
      
      Co-authored-by: default avatarRobert Klotzner <robert.klotzner@gmx.at>
      
      * Version 3 is already live.
      
      * Fix tests (#6055)
      
      * Fix backing tests
      
      * Fix warnings.
      
      * fmt
      
      * collator-protocol: asynchronous backing changes (#5740)
      
      * Draft collator side changes
      
      * Start working on collations management
      
      * Handle peer's view change
      
      * Versioning on advertising
      
      * Versioned collation fetching request
      
      * Handle versioned messages
      
      * Improve docs for collation requests
      
      * Add spans
      
      * Add request receiver to overseer
      
      * Fix collator side tests
      
      * Extract relay parent mode to lib
      
      * Validator side draft
      
      * Add more checks for advertisement
      
      * Request pvd based on async backing mode
      
      * review
      
      * Validator side improvements
      
      * Make old tests green
      
      * More fixes
      
      * Collator side tests draft
      
      * Send collation test
      
      * fmt
      
      * Collator side network protocol versioning
      
      * cleanup
      
      * merge artifacts
      
      * Validator side net protocol versioning
      
      * Remove fragment tree membership request
      
      * Resolve todo
      
      * Collator side core state test
      
      * Improve net protocol compatibility
      
      * Validator side tests
      
      * more improvements
      
      * style fixes
      
      * downgrade log
      
      * Track implicit assignments
      
      * Limit the number of seconded candidates per para
      
      * Add a sanity check
      
      * Handle fetched candidate
      
      * fix tests
      
      * Retry fetch
      
      * Guard against dequeueing while already fetching
      
      * Reintegrate connection management
      
      * Timeout on advertisements
      
      * fmt
      
      * spellcheck
      
      * update tests after merge
      
      * validator assignment fixes for backing and collator protocol (#6158)
      
      * Rename depth->ancestry len in tests
      
      * Refactor group assignments
      
      * Remove implicit assignments
      
      * backing: consider occupied core assignments
      
      * Track a single para on validator side
      
      * Refactor prospective parachains mode request (#6179)
      
      * Extract prospective parachains mode into util
      
      * Skip activations depending on the mode
      
      * backing: don't send backed candidate to provisioner (#6185)
      
      * backing: introduce `CanSecond` request for advertisements filtering (#6225)
      
      * Drop BoundToRelayParent
      
      * draft changes
      
      * fix backing tests
      
      * Fix genesis ancestry
      
      * Fix validator side tests
      
      * more tests
      
      * cargo generate-lockfile
      
      * Implement `StagingValidityConstraints` Runtime API method (#6258)
      
      * Implement StagingValidityConstraints
      
      * spellcheck
      
      * fix ump params
      
      * Update hrmp comment
      
      * Introduce ump per candidate limit
      
      * hypothetical earliest block
      
      * refactor primitives usage
      
      * hypothetical earliest block number test
      
      * fix build
      
      * Prepare the Runtime for asynchronous backing upgrade (#6287)
      
      * Introduce async backing params to runtime config
      
      * fix cumulus config
      
      * use config
      
      * finish runtimes
      
      * Introduce new staging API
      
      * Update collator protocol
      
      * Update provisioner
      
      * Update prospective parachains
      
      * Update backing
      
      * Move async backing params lower in the config
      
      * make naming consistent
      
      * misc
      
      * Use real prospective parachains subsystem (#6407)
      
      * Backport `HypotheticalFrontier` into the feature branch (#6605)
      
      * implement more general HypotheticalFrontier
      
      * fmt
      
      * drop unneeded request
      
      Co-authored-by: default avatarRobert Habermeier <rphmeier@gmail.com>
      
      * Resolve todo about legacy leaf activation (#6447)
      
      * fix bug/warning in handling membership answers
      
      * Remove `HypotheticalDepthRequest` in favor of `HypotheticalFrontierRequest` (#6521)
      
      * Remove `HypotheticalDepthRequest` for `HypotheticalFrontierRequest`
      
      * Update tests
      
      * Fix (removed wrong docstring)
      
      * Fix can_second request
      
      * Patch some dead_code errors
      
      ---------
      
      Co-authored-by: default avatarChris Sosnin <chris125_@live.com>
      
      * Async Backing: Send Statement Distribution "Backed" messages (#6634)
      
      * Backing: Send Statement Distribution "Backed" messages
      
      Closes #6590.
      
      **TODO:**
      
      - [ ] Adjust tests
      
      * Fix compile errors
      
      * (Mostly) fix tests
      
      * Fix comment
      
      * Fix test and compile error
      
      * Test that `StatementDistributionMessage::Backed` is sent
      
      * Fix compile error
      
      * Fix some clippy errors
      
      * Add prospective parachains subsystem tests (#6454)
      
      * Add prospective parachains subsystem test
      
      * Add `should_do_no_work_if_async_backing_disabled_for_leaf` test
      
      * Implement `activate_leaf` helper, up to getting ancestry
      
      * Finish implementing `activate_leaf`
      
      * Small refactor in `activate_leaf`
      
      * Get `CandidateSeconded` working
      
      * Finish `send_candidate_and_check_if_found` test
      
      * Refactor; send more leaves & candidates
      
      * Refactor test
      
      * Implement `check_candidate_parent_leaving_view` test
      
      * Start work on `check_candidate_on_multiple_forks` test
      
      * Don’t associate specific parachains with leaf
      
      * Finish `correctly_updates_leaves` test
      
      * Fix cycle due to reused head data
      
      * Fix `check_backable_query` test
      
      * Fix `check_candidate_on_multiple_forks` test
      
      * Add `check_depth_and_pvd_queries` test
      
      * Address review comments
      
      * Remove TODO
      
      * add a new index for output head data to candidate storage
      
      * Resolve test TODOs
      
      * Fix compile errors
      
      * test candidate storage pruning, make sure new index is cleaned up
      
      ---------
      
      Co-authored-by: default avatarRobert Habermeier <rphmeier@gmail.com>
      
      * Node-side metrics for asynchronous backing (#6549)
      
      * Add metrics for `prune_view_candidate_storage`
      
      * Add metrics for `request_unblocked_collations`
      
      * Fix docstring
      
      * Couple fixes from review comments
      
      * Fix `check_depth_query` test
      
      * inclusion-emulator: mirror advancement rule check (#6361)
      
      * inclusion-emulator: mirror advancement rule check
      
      * fix build
      
      * prospective-parachains: introduce `backed_in_path_only` flag for advertisements (#6649)
      
      * Introduce `backed_in_path_only` flag for depth request
      
      * fmt
      
      * update doc comment
      
      * fmt
      
      * Add async-backing zombienet tests (#6314)
      
      * Async backing: impl guide for statement distribution (#6738)
      
      Co-authored-by: default avatarBradley Olson <34992650+BradleyOlson64@users.noreply.github.com>
      Co-authored-by: default avataralexgparity <115470171+alexgparity@users.noreply.github.com>
      
      * Asynchronous backing statement distribution: Take III (#5999)
      
      * add notification types for v2 statement-distribution
      
      * improve protocol docs
      
      * add empty vstaging module
      
      * fmt
      
      * add backed candidate packet request types
      
      * start putting down structure of new logic
      
      * handle activated leaf
      
      * some sanity-checking on outbound statements
      
      * fmt
      
      * update vstaging share to use statements with PVD
      
      * tiny refactor, candidate_hash location
      
      * import local statements
      
      * refactor statement import
      
      * first stab at broadcast logic
      
      * fmt
      
      * fill out some TODOs
      
      * start on handling incoming
      
      * split off session info into separate map
      
      * start in on a knowledge tracker
      
      * address some grumbles
      
      * format
      
      * missed comment
      
      * some docs for direct
      
      * add note on slashing
      
      * amend
      
      * simplify 'direct' code
      
      * finish up the 'direct' logic
      
      * add a bunch of tests for the direct-in-group logic
      
      * rename 'direct' to 'cluster', begin a candidate_entry module
      
      * distill candidate_entry
      
      * start in on a statement-store module
      
      * some utilities for the statement store
      
      * rewrite 'send_statement_direct' using new tools
      
      * filter sending logic on peers which have the relay-parent in their view.
      
      * some more logic for handling incoming statements
      
      * req/res: BackedCandidatePacket -> AttestedCandidate + tweaks
      
      * add a `validated_in_group` bitfield to BackedCandidateInventory
      
      * BackedCandidateInventory -> Manifest
      
      * start in on requester module
      
      * add outgoing request for attested candidate
      
      * add a priority mechanism for requester
      
      * some request dispatch logic
      
      * add seconded mask to tagged-request
      
      * amend manifest to hold group index
      
      * handle errors and set up scaffold for response validation
      
      * validate attested candidate responses
      
      * requester -> requests
      
      * add some utilities for manipulating requests
      
      * begin integrating requester
      
      * start grid module
      
      * tiny
      
      * refactor grid topology to expose more info to subsystems
      
      * fix grid_topology test
      
      * fix overseer test
      
      * implement topology group-based view construction logic
      
      * fmt
      
      * flesh out grid slightly more
      
      * add indexed groups utility
      
      * integrate Groups into per-session info
      
      * refactor statement store to borrow Groups
      
      * implement manifest knowledge utility
      
      * add a test for topology setup
      
      * don't send to group members
      
      * test for conflicting manifests
      
      * manifest knowledge tests
      
      * fmt
      
      * rename field
      
      * garbage collection for grid tracker
      
      * routines for finding correct/incorrect advertisers
      
      * add manifest import logic
      
      * tweak naming
      
      * more tests for manifest import
      
      * add comment
      
      * rework candidates into a view-wide tracker
      
      * fmt
      
      * start writing boilerplate for grid sending
      
      * fmt
      
      * some more group boilerplate
      
      * refactor handling of topology and authority IDs
      
      * fmt
      
      * send statements directly to grid peers where possible
      
      * send to cluster only if statement belongs to cluster
      
      * improve handling of cluster statements
      
      * handle incoming statements along the grid
      
      * API for introduction of candidates into the tree
      
      * backing: use new prospective parachains API
      
      * fmt prospective parachains changes
      
      * fmt statement-dist
      
      * fix condition
      
      * get ready for tracking importable candidates
      
      * prospective parachains: add Cow logic
      
      * incomplete and complete hypothetical candidates
      
      * remove keep_if_unneeded
      
      * fmt
      
      * implement more general HypotheticalFrontier
      
      * fmt, cleanup
      
      * add a by_parent_hash index to candidate tracker
      
      * more framework for future code
      
      * utilities for getting all hypothetical candidates for frontier
      
      * track origin in statement store
      
      * fmt
      
      * requests should return peer
      
      * apply post-confirmation reckoning
      
      * flesh out import/announce/circulate logic on new statements
      
      * adjust
      
      * adjust TODO comment
      
      * fix  backing tests
      
      * update statement-distribution to use new indexedvec
      
      * fmt
      
      * query hypothetical candidates
      
      * implement `note_importable_under`
      
      * extract common utility of fragment tree updates
      
      * add a helper function for getting statements unknown by backing
      
      * import fresh statements to backing
      
      * send announcements and acknowledgements over grid
      
      * provide freshly importable statements
      
      also avoid tracking backed candidates in statement distribution
      
      * do not issue requests on newly importable candidates
      
      * add TODO for later when confirming candidate
      
      * write a routine for handling backed candidate notifications
      
      * simplify grid substantially
      
      * add some test TODOs
      
      * handle confirmed candidates & grid announcements
      
      * finish implementing manifest handling, including follow up statements
      
      * send follow-up statements when acknowledging freshly backed
      
      * fmt
      
      * handle incoming acknowledgements
      
      * a little DRYing
      
      * wire up network messages to handlers
      
      * fmt
      
      * some skeleton code for peer view update handling
      
      * more peer view skeleton stuff
      
      * Fix async backing statement distribution tests (#6621)
      
      * Fix compile errors in tests
      
      * Cargo fmt
      
      * Resolve some todos in async backing statement-distribution branch (#6482)
      
      * Implement `remove_by_relay_parent`
      
      * Extract `minimum_votes` to shared primitives.
      
      * Add `can_send_statements_received_with_prejudice` test
      
      * Fix test
      
      * Update docstrings
      
      * Cargo fmt
      
      * Fix compile error
      
      * Fix compile errors in tests
      
      * Cargo fmt
      
      * Add module docs; write `test_priority_ordering` (first draft)
      
      * Fix `test_priority_ordering`
      
      * Move `insert_or_update_priority`: `Drop` -> `set_cluster_priority`
      
      * Address review comments
      
      * Remove `Entry::get_mut`
      
      * fix test compilation
      
      * add a TODO for a test
      
      * clean up a couple of TODOs
      
      * implement sending pending cluster statements
      
      * refactor utility function for sending acknowledgement and statements
      
      * mostly implement catching peers up via grid
      
      * Fix clippy error
      
      * alter grid to track all pending statements
      
      * fix more TODOs and format
      
      * tweak a TODO in requests
      
      * some logic for dispatching requests
      
      * fmt
      
      * skeleton for response receiving
      
      * Async backing statement distribution: cluster tests (#6678)
      
      * Add `pending_statements_set_when_receiving_fresh_statements`
      
      * Add `pending_statements_updated_when_sending_statements` test
      
      * fix up
      
      * fmt
      
      * update TODO
      
      * rework seconded mask in requests
      
      * change doc
      
      * change unhandledresponse not to borrow request manager
      
      * only accept responses sufficient to back
      
      * finish implementing response handling
      
      * extract statement filter to protocol crate
      
      * rework requests: use statement filter in network protocol
      
      * dispatch cluster requests correctly
      
      * rework cluster statement sending
      
      * implement request answering
      
      * fmt
      
      * only send confirmed candidate statement messages on unified relay-parent
      
      * Fix Tests In Statement Distribution Branch
      
      * Async Backing: Integrate `vstaging` of statement distribution into `lib.rs` (#6715)
      
      * Integrate `handle_active_leaves_update`
      
      * Integrate `share_local_statement`/`handle_backed_candidate_message`
      
      * Start hooking up request/response flow
      
      * Finish hooking up request/response flow
      
      * Limit number of parallel requests in responder
      
      * Fix test compilation errors
      
      * Fix missing check for prospective parachains mode
      
      * Fix some more compile errors
      
      * clean up some review comments
      
      * clean up warnings
      
      * Async backing statement distribution: grid tests (#6673)
      
      * Add `manifest_import_returns_ok_true` test
      
      * cargo fmt
      
      * Add pending_communication_receiving_manifest_on_confirmed_candidate
      
      * Add `senders_can_provide_manifests_in_acknowledgement` test
      
      * Add a couple of tests for pending statements
      
      * Add `pending_statements_cleared_when_sending` test
      
      * Add `pending_statements_respect_remote_knowledge` test
      
      * Refactor group creation in tests
      
      * Clarify docs
      
      * Address some review comments
      
      * Make some clarifications
      
      * Fix post-merge errors
      
      * Clarify test `senders_can_provide_manifests_in_acknowledgement`
      
      * Try writing `pending_statements_are_updated_after_manifest_exchange`
      
      * Document "seconding limit" and `reject_overflowing_manifests` test
      
      * Test that seconding counts are not updated for validators on error
      
      * Fix tests
      
      * Fix manifest exchange test
      
      * Add more tests in `requests.rs` (#6707)
      
      This resolves remaining TODOs in this file.
      
      * remove outdated inventory terminology
      
      * Async backing statement distribution: `Candidates` tests (#6658)
      
      * Async Backing: Fix clippy errors in statement distribution branch (#6720)
      
      * Integrate `handle_active_leaves_update`
      
      * Integrate `share_local_statement`/`handle_backed_candidate_message`
      
      * Start hooking up request/response flow
      
      * Finish hooking up request/response flow
      
      * Limit number of parallel requests in responder
      
      * Fix test compilation errors
      
      * Fix missing check for prospective parachains mode
      
      * Fix some more compile errors
      
      * Async Backing: Fix clippy errors in statement distribution branch
      
      * Fix some more clippy lints
      
      * add tests module
      
      * fix warnings in existing tests
      
      * create basic test harness
      
      * create a test state struct
      
      * fmt
      
      * create empty cluster & grid modules for tests
      
      * some TODOs for cluster test suite
      
      * describe test-suite for grid logic
      
      * describe request test suite
      
      * fix seconding-limit bug
      
      * Remove extraneous `pub`
      
      This somehow made it into my clippy PR.
      
      * Fix some test compile warnings
      
      * Remove some unneeded `allow`s
      
      * adapt some new test helpers from Marcin
      
      * add helper for activating a gossip topology
      
      * add utility for signing statements
      
      * helpers for connecting/disconnecting peers
      
      * round out network utilities
      
      * fmt
      
      * fix bug in initializing validator-meta
      
      * fix compilation
      
      * implement first cluster test
      
      * TODOs for incoming request tests
      
      * Remove unneeded `make_committed_candidate` helper
      
      * fmt
      
      * some more tests for cluster
      
      * add a TODO about grid senders
      
      * integrate inbound req/res into test harness
      
      * polish off initial cluster test suite
      
      * keep introduce candidate request
      
      * fix tests after introduce candidate request
      
      * fmt
      
      * Add grid protocol to module docs
      
      * Fix comments
      
      * Test `backed_in_path_only: true`
      
      * Update node/network/protocol/src/lib.rs
      
      Co-authored-by: default avatarChris Sosnin <48099298+slumber@users.noreply.github.com>
      
      * Update node/network/protocol/src/request_response/mod.rs
      
      Co-authored-by: default avatarChris Sosnin <48099298+slumber@users.noreply.github.com>
      
      * Mark receiver with `vstaging`
      
      * validate grid senders based on manifest kind
      
      * fix mask_seconded/valid
      
      * fix unwanted-mask check
      
      * fix build
      
      * resolve todo on leaf mode
      
      * Unify protocol naming to vstaging
      
      * fmt, fix grid test after topology change
      
      * typo
      
      Co-authored-by: default avatarChris Sosnin <48099298+slumber@users.noreply.github.com>
      
      * address review
      
      * adjust comment, make easier to understand
      
      * Fix typo
      
      ---------
      
      Co-authored-by: default avatarMarcin S <marcin@bytedude.com>
      Co-authored-by: default avatarMarcin S <marcin@realemail.net>
      Co-authored-by: default avatarChris Sosnin <48099298+slumber@users.noreply.github.com>
      Co-authored-by: default avatarChris Sosnin <chris125_@live.com>
      
      * miscellaneous fixes to make asynchronous backing work (#6791)
      
      * propagate network-protocol-staging feature
      
      * add feature to adder-collator as well
      
      * allow collation-generation of occupied cores
      
      * prospective parachains: special treatment for pending availability candidates
      
      * runtime: fetch candidates pending availability
      
      * lazily construct PVD for pending candidates
      
      * fix fallout in prospective parachains hypothetical/select_child
      
      * runtime: enact candidates when creating paras-inherent
      
      * make tests compile
      
      * test pending availability in the scope
      
      * add prospective parachains test
      
      * fix validity constraints leftovers
      
      * drop prints
      
      * Fix typos
      
      ---------
      
      Co-authored-by: default avatarChris Sosnin <chris125_@live.com>
      Co-authored-by: default avatarMarcin S <marcin@realemail.net>
      
      * Remove restart from test (#6840)
      
      * Async Backing: Statement Distribution Tests (#6755)
      
      * start on handling incoming
      
      * split off session info into separate map
      
      * start in on a knowledge tracker
      
      * address some grumbles
      
      * format
      
      * missed comment
      
      * some docs for direct
      
      * add note on slashing
      
      * amend
      
      * simplify 'direct' code
      
      * finish up the 'direct' logic
      
      * add a bunch of tests for the direct-in-group logic
      
      * rename 'direct' to 'cluster', begin a candidate_entry module
      
      * distill candidate_entry
      
      * start in on a statement-store module
      
      * some utilities for the statement store
      
      * rewrite 'send_statement_direct' using new tools
      
      * filter sending logic on peers which have the relay-parent in their view.
      
      * some more logic for handling incoming statements
      
      * req/res: BackedCandidatePacket -> AttestedCandidate + tweaks
      
      * add a `validated_in_group` bitfield to BackedCandidateInventory
      
      * BackedCandidateInventory -> Manifest
      
      * start in on requester module
      
      * add outgoing request for attested candidate
      
      * add a priority mechanism for requester
      
      * some request dispatch logic
      
      * add seconded mask to tagged-request
      
      * amend manifest to hold group index
      
      * handle errors and set up scaffold for response validation
      
      * validate attested candidate responses
      
      * requester -> requests
      
      * add some utilities for manipulating requests
      
      * begin integrating requester
      
      * start grid module
      
      * tiny
      
      * refactor grid topology to expose more info to subsystems
      
      * fix grid_topology test
      
      * fix overseer test
      
      * implement topology group-based view construction logic
      
      * fmt
      
      * flesh out grid slightly more
      
      * add indexed groups utility
      
      * integrate Groups into per-session info
      
      * refactor statement store to borrow Groups
      
      * implement manifest knowledge utility
      
      * add a test for topology setup
      
      * don't send to group members
      
      * test for conflicting manifests
      
      * manifest knowledge tests
      
      * fmt
      
      * rename field
      
      * garbage collection for grid tracker
      
      * routines for finding correct/incorrect advertisers
      
      * add manifest import logic
      
      * tweak naming
      
      * more tests for manifest import
      
      * add comment
      
      * rework candidates into a view-wide tracker
      
      * fmt
      
      * start writing boilerplate for grid sending
      
      * fmt
      
      * some more group boilerplate
      
      * refactor handling of topology and authority IDs
      
      * fmt
      
      * send statements directly to grid peers where possible
      
      * send to cluster only if statement belongs to cluster
      
      * improve handling of cluster statements
      
      * handle incoming statements along the grid
      
      * API for introduction of candidates into the tree
      
      * backing: use new prospective parachains API
      
      * fmt prospective parachains changes
      
      * fmt statement-dist
      
      * fix condition
      
      * get ready for tracking importable candidates
      
      * prospective parachains: add Cow logic
      
      * incomplete and complete hypothetical candidates
      
      * remove keep_if_unneeded
      
      * fmt
      
      * implement more general HypotheticalFrontier
      
      * fmt, cleanup
      
      * add a by_parent_hash index to candidate tracker
      
      * more framework for future code
      
      * utilities for getting all hypothetical candidates for frontier
      
      * track origin in statement store
      
      * fmt
      
      * requests should return peer
      
      * apply post-confirmation reckoning
      
      * flesh out import/announce/circulate logic on new statements
      
      * adjust
      
      * adjust TODO comment
      
      * fix  backing tests
      
      * update statement-distribution to use new indexedvec
      
      * fmt
      
      * query hypothetical candidates
      
      * implement `note_importable_under`
      
      * extract common utility of fragment tree updates
      
      * add a helper function for getting statements unknown by backing
      
      * import fresh statements to backing
      
      * send announcements and acknowledgements over grid
      
      * provide freshly importable statements
      
      also avoid tracking backed candidates in statement distribution
      
      * do not issue requests on newly importable candidates
      
      * add TODO for later when confirming candidate
      
      * write a routine for handling backed candidate notifications
      
      * simplify grid substantially
      
      * add some test TODOs
      
      * handle confirmed candidates & grid announcements
      
      * finish implementing manifest handling, including follow up statements
      
      * send follow-up statements when acknowledging freshly backed
      
      * fmt
      
      * handle incoming acknowledgements
      
      * a little DRYing
      
      * wire up network messages to handlers
      
      * fmt
      
      * some skeleton code for peer view update handling
      
      * more peer view skeleton stuff
      
      * Fix async backing statement distribution tests (#6621)
      
      * Fix compile errors in tests
      
      * Cargo fmt
      
      * Resolve some todos in async backing statement-distribution branch (#6482)
      
      * Implement `remove_by_relay_parent`
      
      * Extract `minimum_votes` to shared primitives.
      
      * Add `can_send_statements_received_with_prejudice` test
      
      * Fix test
      
      * Update docstrings
      
      * Cargo fmt
      
      * Fix compile error
      
      * Fix compile errors in tests
      
      * Cargo fmt
      
      * Add module docs; write `test_priority_ordering` (first draft)
      
      * Fix `test_priority_ordering`
      
      * Move `insert_or_update_priority`: `Drop` -> `set_cluster_priority`
      
      * Address review comments
      
      * Remove `Entry::get_mut`
      
      * fix test compilation
      
      * add a TODO for a test
      
      * clean up a couple of TODOs
      
      * implement sending pending cluster statements
      
      * refactor utility function for sending acknowledgement and statements
      
      * mostly implement catching peers up via grid
      
      * Fix clippy error
      
      * alter grid to track all pending statements
      
      * fix more TODOs and format
      
      * tweak a TODO in requests
      
      * some logic for dispatching requests
      
      * fmt
      
      * skeleton for response receiving
      
      * Async backing statement distribution: cluster tests (#6678)
      
      * Add `pending_statements_set_when_receiving_fresh_statements`
      
      * Add `pending_statements_updated_when_sending_statements` test
      
      * fix up
      
      * fmt
      
      * update TODO
      
      * rework seconded mask in requests
      
      * change doc
      
      * change unhandledresponse not to borrow request manager
      
      * only accept responses sufficient to back
      
      * finish implementing response handling
      
      * extract statement filter to protocol crate
      
      * rework requests: use statement filter in network protocol
      
      * dispatch cluster requests correctly
      
      * rework cluster statement sending
      
      * implement request answering
      
      * fmt
      
      * only send confirmed candidate statement messages on unified relay-parent
      
      * Fix Tests In Statement Distribution Branch
      
      * Async Backing: Integrate `vstaging` of statement distribution into `lib.rs` (#6715)
      
      * Integrate `handle_active_leaves_update`
      
      * Integrate `share_local_statement`/`handle_backed_candidate_message`
      
      * Start hooking up request/response flow
      
      * Finish hooking up request/response flow
      
      * Limit number of parallel requests in responder
      
      * Fix test compilation errors
      
      * Fix missing check for prospective parachains mode
      
      * Fix some more compile errors
      
      * clean up some review comments
      
      * clean up warnings
      
      * Async backing statement distribution: grid tests (#6673)
      
      * Add `manifest_import_returns_ok_true` test
      
      * cargo fmt
      
      * Add pending_communication_receiving_manifest_on_confirmed_candidate
      
      * Add `senders_can_provide_manifests_in_acknowledgement` test
      
      * Add a couple of tests for pending statements
      
      * Add `pending_statements_cleared_when_sending` test
      
      * Add `pending_statements_respect_remote_knowledge` test
      
      * Refactor group creation in tests
      
      * Clarify docs
      
      * Address some review comments
      
      * Make some clarifications
      
      * Fix post-merge errors
      
      * Clarify test `senders_can_provide_manifests_in_acknowledgement`
      
      * Try writing `pending_statements_are_updated_after_manifest_exchange`
      
      * Document "seconding limit" and `reject_overflowing_manifests` test
      
      * Test that seconding counts are not updated for validators on error
      
      * Fix tests
      
      * Fix manifest exchange test
      
      * Add more tests in `requests.rs` (#6707)
      
      This resolves remaining TODOs in this file.
      
      * remove outdated inventory terminology
      
      * Async backing statement distribution: `Candidates` tests (#6658)
      
      * Async Backing: Fix clippy errors in statement distribution branch (#6720)
      
      * Integrate `handle_active_leaves_update`
      
      * Integrate `share_local_statement`/`handle_backed_candidate_message`
      
      * Start hooking up request/response flow
      
      * Finish hooking up request/response flow
      
      * Limit number of parallel requests in responder
      
      * Fix test compilation errors
      
      * Fix missing check for prospective parachains mode
      
      * Fix some more compile errors
      
      * Async Backing: Fix clippy errors in statement distribution branch
      
      * Fix some more clippy lints
      
      * add tests module
      
      * fix warnings in existing tests
      
      * create basic test harness
      
      * create a test state struct
      
      * fmt
      
      * create empty cluster & grid modules for tests
      
      * some TODOs for cluster test suite
      
      * describe test-suite for grid logic
      
      * describe request test suite
      
      * fix seconding-limit bug
      
      * Remove extraneous `pub`
      
      This somehow made it into my clippy PR.
      
      * Fix some test compile warnings
      
      * Remove some unneeded `allow`s
      
      * adapt some new test helpers from Marcin
      
      * add helper for activating a gossip topology
      
      * add utility for signing statements
      
      * helpers for connecting/disconnecting peers
      
      * round out network utilities
      
      * fmt
      
      * fix bug in initializing validator-meta
      
      * fix compilation
      
      * implement first cluster test
      
      * TODOs for incoming request tests
      
      * Remove unneeded `make_committed_candidate` helper
      
      * fmt
      
      * Hook up request sender
      
      * Add `valid_statement_without_prior_seconded_is_ignored` test
      
      * Fix `valid_statement_without_prior_seconded_is_ignored` test
      
      * some more tests for cluster
      
      * add a TODO about grid senders
      
      * integrate inbound req/res into test harness
      
      * polish off initial cluster test suite
      
      * keep introduce candidate request
      
      * fix tests after introduce candidate request
      
      * fmt
      
      * Add grid protocol to module docs
      
      * Remove obsolete test
      
      * Fix comments
      
      * Test `backed_in_path_only: true`
      
      * Update node/network/protocol/src/lib.rs
      
      Co-authored-by: default avatarChris Sosnin <48099298+slumber@users.noreply.github.com>
      
      * Update node/network/protocol/src/request_response/mod.rs
      
      Co-authored-by: default avatarChris Sosnin <48099298+slumber@users.noreply.github.com>
      
      * Mark receiver with `vstaging`
      
      * First draft of `ensure_seconding_limit_is_respected` test
      
      * validate grid senders based on manifest kind
      
      * fix mask_seconded/valid
      
      * fix unwanted-mask check
      
      * fix build
      
      * resolve todo on leaf mode
      
      * Unify protocol naming to vstaging
      
      * Fix `ensure_seconding_limit_is_respected` test
      
      * Start `backed_candidate_leads_to_advertisement` test
      
      * fmt, fix grid test after topology change
      
      * Send Backed notification
      
      * Finish `backed_candidate_leads_to_advertisement` test
      
      * Finish `peer_reported_for_duplicate_statements` test
      
      * Finish `received_advertisement_before_confirmation_leads_to_request`
      
      * Add `advertisements_rejected_from_incorrect_peers` test
      
      * Add `manifest_rejected_*` tests
      
      * Add `manifest_rejected_when_group_does_not_match_para` test
      
      * Add `local_node_sanity_checks_incoming_requests` test
      
      * Add `local_node_respects_statement_mask` test
      
      * Add tests where peer is reported for providing invalid signatures
      
      * Add `cluster_peer_allowed_to_send_incomplete_statements` test
      
      * Add `received_advertisement_after_backing_leads_to_acknowledgement`
      
      * Add `received_advertisement_after_confirmation_before_backing` test
      
      * peer_reported_for_advertisement_conflicting_with_confirmed_candidate
      
      * Add `peer_reported_for_not_enough_statements` test
      
      * Add `peer_reported_for_providing_statements_meant_to_be_masked_out`
      
      * Add `additional_statements_are_shared_after_manifest_exchange`
      
      * Add `grid_statements_imported_to_backing` test
      
      * Add `relay_parent_entering_peer_view_leads_to_advertisement` test
      
      * Add `advertisement_not_re_sent_when_peer_re_enters_view` test
      
      * Update node/network/statement-distribution/src/vstaging/tests/grid.rs
      
      Co-authored-by: default avatarasynchronous rob <rphmeier@gmail.com>
      
      * Resolve TODOs, update test
      
      * Address unused code
      
      * Add check after every test for unhandled requests
      
      * Refactor (`make_dummy_leaf` and `handle_sent_request`)
      
      * Refactor (`make_dummy_topology`)
      
      * Minor refactor
      
      ---------
      
      Co-authored-by: default avatarRobert Habermeier <rphmeier@gmail.com>
      Co-authored-by: default avatarChris Sosnin <48099298+slumber@users.noreply.github.com>
      Co-authored-by: default avatarChris Sosnin <chris125_@live.com>
      
      * Fix some clippy lints in tests
      
      * Async backing: minor fixes (#6920)
      
      * bitfield-distribution test
      
      * implicit view tests
      
      * Refactor parameters -> params
      
      * scheduler: update storage migration (#6963)
      
      * update scheduler migration
      
      * Adjust weight to account for storage read
      
      * Statement Distribution Guide Edits (#7025)
      
      * Statement distribution guide edits
      
      * Addressed Marcin's comments
      
      * Add attested candidate request retry timeouts (#6833)
      
      Co-authored-by: default avatarChris Sosnin <48099298+slumber@users.noreply.github.com>
      Co-authored-by: default avatarasynchronous rob <rphmeier@gmail.com>
      Co-authored-by: default avatarRobert Habermeier <rphmeier@gmail.com>
      Co-authored-by: default avatarChris Sosnin <chris125_@live.com>
      Fix async backing statement distribution tests (#6621)
      Resolve some todos in async backing statement-distribution branch (#6482)
      Fix clippy errors in statement distribution branch (#6720)
      
      * Async backing: add Prospective Parachains impl guide (#6933)
      
      Co-authored-by: default avatarBradley Olson <34992650+BradleyOlson64@users.noreply.github.com>
      
      * Updates to Provisioner Guide for Async Backing (#7106)
      
      * Initial corrections and clarifications
      
      * Partial first draft
      
      * Finished first draft
      
      * Adding back wrongly removed test bit
      
      * fmt
      
      * Update roadmap/implementers-guide/src/node/utility/provisioner.md
      
      Co-authored-by: default avatarMarcin S. <marcin@realemail.net>
      
      * Addressing comments
      
      * Reorganization
      
      * fmt
      
      ---------
      
      Co-authored-by: default avatarMarcin S. <marcin@realemail.net>
      
      * fmt
      
      * Renaming Parathread Mentions (#7287)
      
      * Renaming parathreads
      
      * Renaming module to pallet
      
      * More updates
      
      * PVF: Refactor workers into separate crates, remove host dependency (#7253)
      
      * PVF: Refactor workers into separate crates, remove host dependency
      
      * Fix compile error
      
      * Remove some leftover code
      
      * Fix compile errors
      
      * Update Cargo.lock
      
      * Remove worker main.rs files
      
      I accidentally copied these from the other PR. This PR isn't intended to
      introduce standalone workers yet.
      
      * Address review comments
      
      * cargo fmt
      
      * Update a couple of comments
      
      * Update log targets
      
      * Update quote to 1.0.27 (#7280)
      
      Signed-off-by: default avatarOliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
      Co-authored-by: parity-processbot <>
      
      * pallets: implement `Default` for `GenesisConfig` in `no_std` (#7271)
      
      * pallets: implement Default for GenesisConfig in no_std
      
      This change is follow-up of: https://github.com/paritytech/substrate/pull/14108
      
      It is a step towards: https://github.com/paritytech/substrate/issues/13334
      
      * Cargo.lock updated
      
      * update lockfile for {"substrate"}
      
      ---------
      
      Co-authored-by: parity-processbot <>
      
      * cli: enable BEEFY by default on test networks (#7293)
      
      We consider BEEFY mature enough to run by default on all nodes
      for test networks (Rococo/Wococo/Versi).
      
      Right now, most nodes are not running it since it's opt-in using
      --beefy flag. Switch to an opt-out model for test networks.
      
      Replace --beefy flag from CLI with --no-beefy and have BEEFY
      client start by default on test networks.
      
      Signed-off-by: default avataracatangiu <adrian@parity.io>
      
      * runtime: past session slashing runtime API (#6667)
      
      * runtime/vstaging: unapplied_slashes runtime API
      
      * runtime/vstaging: key_ownership_proof runtime API
      
      * runtime/ParachainHost: submit_report_dispute_lost
      
      * fix key_ownership_proof API
      
      * runtime: submit_report_dispute_lost runtime API
      
      * nits
      
      * Update node/subsystem-types/src/messages.rs
      
      Co-authored-by: default avatarMarcin S. <marcin@bytedude.com>
      
      * revert unrelated fmt changes
      
      * post merge fixes
      
      * fix compilation
      
      ---------
      
      Co-authored-by: default avatarMarcin S. <marcin@bytedude.com>
      
      * Correcting git mishap
      
      * Document usage of `gum` crate (#7294)
      
      * Document usage of gum crate
      
      * Small fix
      
      * Add some more basic info
      
      * Update node/gum/src/lib.rs
      
      Co-authored-by: default avatarAndrei Sandu <54316454+sandreim@users.noreply.github.com>
      
      * Update target docs
      
      ---------
      
      Co-authored-by: default avatarAndrei Sandu <54316454+sandreim@users.noreply.github.com>
      
      * XCM: Fix issue with RequestUnlock (#7278)
      
      * XCM: Fix issue with RequestUnlock
      
      * Leave API changes for v4
      
      * Fix clippy errors
      
      * Fix tests
      
      ---------
      
      Co-authored-by: parity-processbot <>
      
      * Companion for Substrate#14228 (#7295)
      
      * Companion for Substrate#14228
      
      https://github.com/paritytech/substrate/pull/14228
      
      * update lockfile for {"substrate"}
      
      ---------
      
      Co-authored-by: parity-processbot <>
      
      * Companion for #14237: Use latest sp-crates (#7300)
      
      * To revert: Update substrate branch to "lexnv/bump_sp_crates"
      
      Signed-off-by: default avatarAlexandru Vasile <alexandru.vasile@parity.io>
      
      * Revert "To revert: Update substrate branch to "lexnv/bump_sp_crates""
      
      This reverts commit 5f1db84eac4a226c37b7f6ce6ee19b49dc7e2008.
      
      * Update cargo lock
      
      Signed-off-by: default avatarAlexandru Vasile <alexandru.vasile@parity.io>
      
      * Update cargo.lock
      
      Signed-off-by: default avatarAlexandru Vasile <alexandru.vasile@parity.io>
      
      * Update cargo.lock
      
      Signed-off-by: default avatarAlexandru Vasile <alexandru.vasile@parity.io>
      
      ---------
      
      Signed-off-by: default avatarAlexandru Vasile <alexandru.vasile@parity.io>
      
      * bounded-collections bump to 0.1.7 (#7305)
      
      * bounded-collections bump to 0.1.7
      
      Companion for: paritytech/substrate#14225
      
      * update lockfile for {"substrate"}
      
      ---------
      
      Co-authored-by: parity-processbot <>
      
      * bump to quote 1.0.28 (#7306)
      
      * `RollingSessionWindow` cleanup (#7204)
      
      * Replace `RollingSessionWindow` with `RuntimeInfo` - initial commit
      
      * Fix tests in import
      
      * Fix the rest of the tests
      
      * Remove dead code
      
      * Fix todos
      
      * Simplify session caching
      
      * Comments for `SessionInfoProvider`
      
      * Separate `SessionInfoProvider` from `State`
      
      * `cache_session_info_for_head` becomes freestanding function
      
      * Remove unneeded `mut` usage
      
      * fn session_info -> fn get_session_info() to avoid name clashes. The function also tries to initialize `SessionInfoProvider`
      
      * Fix SessionInfo retrieval
      
      * Code cleanup
      
      * Don't wrap `SessionInfoProvider` in an `Option`
      
      * Remove `earliest_session()`
      
      * Remove pre-caching -> wip
      
      * Fix some tests and code cleanup
      
      * Fix all tests
      
      * Fixes in tests
      
      * Fix comments, variable names and small style changes
      
      * Fix a warning
      
      * impl From<SessionWindowSize> for NonZeroUsize
      
      * Fix logging for `get_session_info` - remove redundant logs and decrease log level to DEBUG
      
      * Code review feedback
      
      * Storage migration removing `COL_SESSION_WINDOW_DATA` from parachains db
      
      * Remove `col_session_data` usages
      
      * Storage migration clearing columns w/o removing them
      
      * Remove session data column usages from `approval-voting` and `dispute-coordinator` tests
      
      * Add some test cases from `RollingSessionWindow` to `dispute-coordinator` tests
      
      * Fix formatting in initialized.rs
      
      * Fix a corner case in `SessionInfo` caching for `dispute-coordinator`
      
      * Remove `RollingSessionWindow` ;(
      
      * Revert "Fix formatting in initialized.rs"
      
      This reverts commit 0f94664ec9f3a7e3737a30291195990e1e7065fc.
      
      * v2 to v3 migration drops `COL_DISPUTE_COORDINATOR_DATA` instead of clearing it
      
      * Fix `NUM_COLUMNS` in `approval-voting`
      
      * Use `columns::v3::NUM_COLUMNS` when opening db
      
      * Update node/service/src/parachains_db/upgrade.rs
      
      Co-authored-by: default avatarAndrei Sandu <54316454+sandreim@users.noreply.github.com>
      
      * Don't write in `COL_DISPUTE_COORDINATOR_DATA` for `test_rocksdb_migrate_2_to_3`
      
      * Fix `NUM+COLUMNS` in approval_voting
      
      * Fix formatting
      
      * Fix columns usage
      
      * Clarification comments about the different db versions
      
      ---------
      
      Co-authored-by: default avatarAndrei Sandu <54316454+sandreim@users.noreply.github.com>
      
      * pallet-para-config: Remove remnant WeightInfo functions (#7308)
      
      * pallet-para-config: Remove remnant WeightInfo functions
      
      Signed-off-by: default avatarOliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
      
      * set_config_with_weight begone
      
      Signed-off-by: default avatarOliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
      
      * ".git/.scripts/commands/bench/bench.sh" runtime kusama-dev runtime_parachains::configuration
      
      ---------
      
      Signed-off-by: default avatarOliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
      Co-authored-by: command-bot <>
      
      * XCM: PayOverXcm config (#6900)
      
      * Move XCM query functionality to trait
      
      * Fix tests
      
      * Add PayOverXcm implementation
      
      * fix the PayOverXcm trait to compile
      
      * moved doc comment out of trait implmeentation and to the trait
      
      * PayOverXCM documentation
      
      * Change documentation a bit
      
      * Added empty benchmark methods implementation and changed docs
      
      * update PayOverXCM to convert AccountIds to MultiLocations
      
      * Implement benchmarking method
      
      * Change v3 to latest
      
      * Descend origin to an asset sender (#6970)
      
      * descend origin to an asset sender
      
      * sender as tuple of dest and sender
      
      * Add more variants to the QueryResponseStatus enum
      
      * Change Beneficiary to Into<[u8; 32]>
      
      * update PayOverXcm to return concrete errors and use AccountId as sender
      
      * use polkadot-primitives for AccountId
      
      * fix dependency to use polkadot-core-primitives
      
      * force Unpaid instruction to the top of the instructions list
      
      * modify report_outcome to accept interior argument
      
      * use new_query directly for building final xcm query, instead of report_outcome
      
      * fix usage of new_query to use the XcmQueryHandler
      
      * fix usage of new_query to use the XcmQueryHandler
      
      * tiny method calling fix
      
      * xcm query handler (#7198)
      
      * drop redundant query status
      
      * rename ReportQueryStatus to OuterQueryStatus
      
      * revert rename of QueryResponseStatus
      
      * update mapping
      
      * Update xcm/xcm-builder/src/pay.rs
      
      Co-authored-by: default avatarGavin Wood <gavin@parity.io>
      
      * Updates
      
      * Docs
      
      * Fix benchmarking stuff
      
      * Destination can be determined based on asset_kind
      
      * Tweaking API to minimise clones
      
      * Some repotting and docs
      
      ---------
      
      Co-authored-by: default avatarAnthony Alaribe <anthonyalaribe@gmail.com>
      Co-authored-by: default avatarMuharem Ismailov <ismailov.m.h@gmail.com>
      Co-authored-by: default avatarAnthony Alaribe <anthony.alaribe@parity.io>
      Co-authored-by: default avatarGavin Wood <gavin@parity.io>
      
      * Companion for #14265 (#7307)
      
      * Update Cargo.lock
      
      Signed-off-by: default avatarAlexandru Vasile <alexandru.vasile@parity.io>
      
      * Update Cargo.lock
      
      Signed-off-by: default avatarAlexandru Vasile <alexandru.vasile@parity.io>
      
      ---------
      
      Signed-off-by: default avatarAlexandru Vasile <alexandru.vasile@parity.io>
      Co-authored-by: parity-processbot <>
      
      * bump serde to 1.0.163 (#7315)
      
      * bump serde to 1.0.163
      
      * bump ci
      
      * update lockfile for {"substrate"}
      
      ---------
      
      Co-authored-by: parity-processbot <>
      
      * fmt
      
      * Updated fmt
      
      * Removing changes accidentally pulled from master
      
      * fix another master pull issue
      
      * Another master pull fix
      
      * fmt
      
      * Fixing implementers guide build
      
      * Revert "Merge branch 'rh-async-backing-feature-while-frozen' of https://github.com/paritytech/polkadot into brad-rename-parathread"
      
      This reverts commit bebc24af52ab61155e3fe02cb3ce66a592bce49c, reversing
      changes made to 1b2de662dfb11173679d6da5bd0da9d149c85547.
      
      ---------
      
      Signed-off-by: default avatarOliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
      Signed-off-by: default avataracatangiu <adrian@parity.io>
      Signed-off-by: default avatarAlexandru Vasile <alexandru.vasile@parity.io>
      Co-authored-by: default avatarMarcin S <marcin@realemail.net>
      Co-authored-by: default avatarOliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
      Co-authored-by: default avatarMichal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
      Co-authored-by: default avatarAdrian Catangiu <adrian@parity.io>
      Co-authored-by: default avatarordian <write@reusable.software>
      Co-authored-by: default avatarMarcin S. <marcin@bytedude.com>
      Co-authored-by: default avatarAndrei Sandu <54316454+sandreim@users.noreply.github.com>
      Co-authored-by: default avatarFrancisco Aguirre <franciscoaguirreperez@gmail.com>
      Co-authored-by: default avatarBastian Köcher <git@kchr.de>
      Co-authored-by: default avatarAlexandru Vasile <60601340+lexnv@users.noreply.github.com>
      Co-authored-by: default avatarSam Johnson <sam@durosoft.com>
      Co-authored-by: default avatarTsvetomir Dimitrov <tsvetomir@parity.io>
      Co-authored-by: default avatarAnthony Alaribe <anthonyalaribe@gmail.com>
      Co-authored-by: default avatarMuharem Ismailov <ismailov.m.h@gmail.com>
      Co-authored-by: default avatarAnthony Alaribe <anthony.alaribe@parity.io>
      Co-authored-by: default avatarGavin Wood <gavin@parity.io>
      
      * fix bitfield distribution test
      
      * approval distribution tests
      
      * fix bridge tests
      
      * update Cargo.lock
      
      * [async-backing-branch] Optimize collator-protocol validator-side request fetching (#7457)
      
      * Optimize collator-protocol validator-side request fetching
      
      * address feedback: replace tuples with structs
      
      * feedback: add doc comments
      
      * move collation types to subfolder
      
      ---------
      
      Signed-off-by: default avataralindima <alin@parity.io>
      
      * Update collation generation for asynchronous backing (#7405)
      
      * break candidate receipt construction and distribution into own function
      
      * update implementers' guide to include SubmitCollation
      
      * implement SubmitCollation for collation-generation
      
      * fmt
      
      * fix test compilation & remove unnecessary submodule
      
      * add some TODOs for a test suite.
      
      * Update roadmap/implementers-guide/src/types/overseer-protocol.md
      
      Co-authored-by: default avatarAndrei Sandu <54316454+sandreim@users.noreply.github.com>
      
      * add new test harness and first test
      
      * refactor to avoid requiring background sender
      
      * ensure collation gets packaged and distributed
      
      * tests for the fallback case with no hint
      
      * add parent rp-number hint tests
      
      * fmt
      
      * update uses of CollationGenerationConfig
      
      * fix remaining test
      
      * address review comments
      
      * use subsystemsender for background tasks
      
      * fmt
      
      * remove ValidationCodeHashHint and related tests
      
      ---------
      
      Co-authored-by: default avatarAndrei Sandu <54316454+sandreim@users.noreply.github.com>
      
      * fix some more fallout from merge
      
      * fmt
      
      * remove staging APIs from Rococo & Westend (#7513)
      
      * send network messages on main protocol name (#7515)
      
      * misc async backing improvements for allowed ancestry blocks (#7532)
      
      * shared: fix acquire_info
      
      * backwards-compat test for prospective parachains
      
      * same relay parent is allowed
      
      * provisioner: request candidate receipt by relay parent (#7527)
      
      * return candidates hash from prospective parachains
      
      * update provisioner
      
      * update tests
      
      * guide changes
      
      * send a single message to backing
      
      * fix test
      
      * revert to old `handle_new_activations` logic in some cases (#7514)
      
      * revert to old `handle_new_activations` logic
      
      * gate sending messages on scheduled cores to max_depth >= 2
      
      * fmt
      
      * 2->1
      
      * Omnibus asynchronous backing bugfix PR (#7529)
      
      * fix a bug in backing
      
      * add some more logs
      
      * prospective parachains: take ancestry only up to session bounds
      
      * add test
      
      * fix zombienet tests (#7614)
      
      Signed-off-by: default avatarAndrei Sandu <andrei-mihail@parity.io>
      
      * fix runtime compilation
      
      * make bitfield distribution tests compile
      
      * attempt to fix zombienet disputes (#7618)
      
      * update metric name
      
      * update some metric names
      
      * avoid cycles when creating fake candidates
      
      * make undying collator more friendly to malformed parents
      
      * fix a bug in malus
      
      * fmt
      
      * clippy
      
      * add RUN_IN_CONTAINER to new ZombieNet tests (#7631)
      
      * remove duplicated migration
      
      happened because of master-merge
      
      ---------
      
      Signed-off-by: default avatarOliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
      Signed-off-by: default avataracatangiu <adrian@parity.io>
      Signed-off-by: default avatarAlexandru Vasile <alexandru.vasile@parity.io>
      Signed-off-by: default avataralindima <alin@parity.io>
      Signed-off-by: default avatarAndrei Sandu <andrei-mihail@parity.io>
      Co-authored-by: default avatarChris Sosnin <chris125_@live.com>
      Co-authored-by: default avatarParity Bot <admin@parity.io>
      Co-authored-by: default avatarChris Sosnin <48099298+slumber@users.noreply.github.com>
      Co-authored-by: default avatarRobert Klotzner <robert.klotzner@gmx.at>
      Co-authored-by: default avatarRobert Klotzner <eskimor@users.noreply.github.com>
      Co-authored-by: default avatarMarcin S <marcin@bytedude.com>
      Co-authored-by: default avatarMarcin S <marcin@realemail.net>
      Co-authored-by: default avatarMattia L.V. Bradascio <28816406+bredamatt@users.noreply.github.com>
      Co-authored-by: default avatarBradley Olson <34992650+BradleyOlson64@users.noreply.github.com>
      Co-authored-by: default avataralexgparity <115470171+alexgparity@users.noreply.github.com>
      Co-authored-by: default avatarBradleyOlson64 <lotrftw9@gmail.com>
      Co-authored-by: default avatarOliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
      Co-authored-by: default avatarMichal Kucharczyk <1728078+michalkucharczyk@users.noreply.github.com>
      Co-authored-by: default avatarAdrian Catangiu <adrian@parity.io>
      Co-authored-by: default avatarordian <write@reusable.software>
      Co-authored-by: default avatarAndrei Sandu <54316454+sandreim@users.noreply.github.com>
      Co-authored-by: default avatarFrancisco Aguirre <franciscoaguirreperez@gmail.com>
      Co-authored-by: default avatarBastian Köcher <git@kchr.de>
      Co-authored-by: default avatarAlexandru Vasile <60601340+lexnv@users.noreply.github.com>
      Co-authored-by: default avatarSam Johnson <sam@durosoft.com>
      Co-authored-by: default avatarTsvetomir Dimitrov <tsvetomir@parity.io>
      Co-authored-by: default avatarAnthony Alaribe <anthonyalaribe@gmail.com>
      Co-authored-by: default avatarMuharem Ismailov <ismailov.m.h@gmail.com>
      Co-authored-by: default avatarAnthony Alaribe <anthony.alaribe@parity.io>
      Co-authored-by: default avatarGavin Wood <gavin@parity.io>
      Co-authored-by: default avatarAlin Dima <alin@parity.io>
      5174b9d2
  7. Aug 14, 2023
  8. May 12, 2023
    • Tsvetomir Dimitrov's avatar
      Replace `RollingSessionWindow` in approval-voting with `RuntimeInfo` (#7123) · 12dd9277
      Tsvetomir Dimitrov authored
      * Replace `RollingSessionWindow` with `RuntimeInfo` - initial commit
      
      * Fix tests in import
      
      * Fix the rest of the tests
      
      * Remove dead code
      
      * Fix todos
      
      * Simplify session caching
      
      * Comments for `SessionInfoProvider`
      
      * Separate `SessionInfoProvider` from `State`
      
      * `cache_session_info_for_head` becomes freestanding function
      
      * Remove unneeded `mut` usage
      
      * fn session_info -> fn get_session_info() to avoid name clashes. The function also tries to initialize `SessionInfoProvider`
      
      * Fix SessionInfo retrieval
      
      * Code cleanup
      
      * Don't wrap `SessionInfoProvider` in an `Option`
      
      * Remove `earliest_session()`
      
      * Remove pre-caching -> wip
      
      * Fix some tests and code cleanup
      
      * Fix all tests
      
      * Fixes in tests
      
      * Fix comments, variable names and small style changes
      
      * Fix a warning
      
      * impl From<SessionWindowSize> for NonZeroUsize
      
      * Fix logging for `get_session_info` - remove redundant logs and decrease log level to DEBUG
      
      * Code review feedback
      12dd9277
  9. May 04, 2023
  10. Apr 19, 2023
    • Davide Galassi's avatar
      Companion for Substrate #13889 (#7063) · 83fb51a2
      Davide Galassi authored
      * Companion for substrate #13889
      
      * Remove leftover
      
      * Remove removed dependency
      
      * Remove sp-consensus-vrf from lock
      
      * Revert "Remove sp-consensus-vrf from lock"
      
      This reverts commit 2269ca1e32df89272e8fd4544292204db387f436.
      
      * Fix after substrate modifications
      
      * Fix tests to use new VRF Signature type
      
      * Don't rely of Deref trait
      
      * Fix test
      
      * Further code simplification
      
      * Reuse garbage_vrf_signature
      
      * update lockfile for {"substrate"}
      
      ---------
      
      Co-authored-by: parity-processbot <>
      83fb51a2
  11. Apr 08, 2023
  12. Apr 07, 2023
    • Tsvetomir Dimitrov's avatar
      Onchain scraper in `dispute-coordinator` will scrape... · 63a805ba
      Tsvetomir Dimitrov authored
      Onchain scraper in `dispute-coordinator` will scrape `SCRAPED_FINALIZED_BLOCKS_COUNT` blocks before finality (#7013)
      
      * Onchain scraper in `dispute-coordinator` will scrape `SCRAPED_FINALIZED_BLOCKS_COUNT` blocks before finality
      
      The purpose is to make the availability of a `CandidateReceipt` for finalized candidates more likely.
      
      For details see:  https://github.com/paritytech/polkadot/issues/7009
      
      * Fix off by one error
      
      * Replace `SCRAPED_FINALIZED_BLOCKS_COUNT` with `DISPUTE_CANDIDATE_LIFETIME_AFTER_FINALIZATION`
      63a805ba
  13. Mar 24, 2023
    • Davide Galassi's avatar
      Companion for #13683 (#6944) · 260d0736
      Davide Galassi authored
      * Companion for #13683
      
      * Wraps trait is not required
      
      * update lockfile for {"substrate"}
      
      ---------
      
      Co-authored-by: parity-processbot <>
      260d0736
  14. Mar 17, 2023
    • Davide Galassi's avatar
      [Companion #13615] Keystore overhaul (#6892) · 46c36e5a
      Davide Galassi authored
      * Remove not required async calls
      
      * Fixed missing renaming
      
      * make_keystore can be sync
      
      * More fixes
      
      * Trivial nitpicks
      
      * Cherry pick test fix from master
      
      * Fixes after master merge
      
      * update lockfile for {"substrate"}
      
      ---------
      
      Co-authored-by: parity-processbot <>
      46c36e5a
  15. Mar 08, 2023
  16. Feb 16, 2023
    • Marcin S.'s avatar
      Use a `BoundedVec` in `ValidationResult` (#6603) · d5a79914
      Marcin S. authored
      * Use a `BoundedVec` in `ValidationResult`
      
      > Use a `BoundedVec` for `upward_messages` and `horizontal_messages` in order to
      > limit the number of individual messages/memory allocations right at decoding
      > time. The reason for this is that the `ValidationResult` may contain a code
      > upgrade (including a full PVF binary), so the total size limit can't be set
      > too low and this limit will still allow several millions of upward messages,
      > which will (due to the memory allocator overhead) already have a
      > non-negligible memory footprint in decoded form.
      
      * List all fields when hashing so we don't miss one
      
      * Define types for  `BoundedVec`s of messages
      
      * Fix test compile errors
      
      * Depend on `bounded-collections` 0.1.4 (fixes allocation issue)
      
      * Fix compilation issue
      
      * Derive `Hash` instead of manual `impl`
      
      * Avoid use of unwrap
      d5a79914
  17. Jan 22, 2023
    • eskimor's avatar
      Fix some unjustified disputes (#6103) · bec4168b
      eskimor authored
      * Fix indentation + add warning
      
      on participation errors.
      
      * Don't vote invalid on internal errors.
      
      * Don't dispute on code compression error.
      
      * Remove CodeDecompressionError
      
      * Candidate not invalid if PVF preparation fails.
      
      Instead: Report error.
      
      * Fix malus
      
      * Add clarifying comment.
      
      * cargo fmt
      
      * Fix indentation.
      bec4168b
  18. Jan 11, 2023
  19. Jan 10, 2023
    • eskimor's avatar
      Fix cycle dispute-coordinator <-> dispute-distribution (#6489) · cc650fe5
      eskimor authored
      
      * First iteration of message sender.
      
      * dyn Fn variant (no cloning)
      
      * Full implementation + Clone, without allocs on `Send`
      
      * Further clarifications/cleanup.
      
      * MessageSender -> NestingSender
      
      * Doc update/clarification.
      
      * dispute-coordinator: Send disputes on startup.
      
      + Some fixes, cleanup.
      
      * Fix whitespace.
      
      * Dispute distribution fixes, cleanup.
      
      * Cargo.lock
      
      * Fix spaces.
      
      * More format fixes.
      
      What is cargo fmt doing actually?
      
      * More fmt fixes.
      
      * Fix nesting sender.
      
      * Fixes.
      
      * Whitespace
      
      * Enable logging.
      
      * Guide update.
      
      * Fmt fixes, typos.
      
      * Remove unused function.
      
      * Simplifications, doc fixes.
      
      * Update roadmap/implementers-guide/src/node/disputes/dispute-coordinator.md
      
      Co-authored-by: default avatarMarcin S. <marcin@bytedude.com>
      
      * Fmt + doc example fix.
      
      Co-authored-by: default avatareskimor <eskimor@no-such-url.com>
      Co-authored-by: default avatarMarcin S. <marcin@bytedude.com>
      cc650fe5
    • Marcin S.'s avatar
      Replace async-std with tokio in PVF subsystem (#6419) · 44fd9566
      Marcin S. authored
      
      * Replace async-std with tokio in PVF subsystem
      
      * Rework workers to use `select!` instead of a mutex
      
      The improvement in code readability is more important than the thread overhead.
      
      * Remove unnecessary `fuse`
      
      * Add explanation for `expect()`
      
      * Update node/core/pvf/src/worker_common.rs
      
      Co-authored-by: default avatarBastian Köcher <info@kchr.de>
      
      * Update node/core/pvf/src/worker_common.rs
      
      Co-authored-by: default avatarBastian Köcher <info@kchr.de>
      
      * Address some review comments
      
      * Shutdown tokio runtime
      
      * Run cargo fmt
      
      * Add a small note about retries
      
      * Fix up merge
      
      * Rework `cpu_time_monitor_loop` to return when other thread finishes
      
      * Add error string to PrepareError::IoErr variant
      
      * Log when artifacts fail to prepare
      
      * Fix `cpu_time_monitor_loop`; fix test
      
      * Fix text
      
      * Fix a couple of potential minor data races.
      
      First data race was due to logging in the CPU monitor thread even if the
      job (other thread) finished. It can technically finish before or after the log.
      
      Maybe best would be to move this log to the `select!`s, where we are guaranteed
      to have chosen the timed-out branch, although there would be a bit of
      duplication.
      
      Also, it was possible for this thread to complete before we executed
      `finished_tx.send` in the other thread, which would trigger an error as the
      receiver has already been dropped. And right now, such a spurious error from
      `send` would be returned even if the job otherwise succeeded.
      
      * Update Cargo.lock
      
      Co-authored-by: default avatarBastian Köcher <info@kchr.de>
      44fd9566
  20. Dec 07, 2022
  21. Nov 30, 2022
    • ordian's avatar
      guide: remove refences to outdated secondary checkers (#6309) · 063aab23
      ordian authored
      * guide: remove refences to outdated secondary checkers
      
      * Update roadmap/implementers-guide/src/glossary.md
      
      * guide: remove refences to Fisherman
      
      * revert changes to roadmap/parachains.md
      063aab23
    • alexgparity's avatar
      Clippyfy (#6341) · 9ea14e66
      alexgparity authored
      
      * Add clippy config and remove .cargo from gitignore
      
      * first fixes
      
      * Clippyfied
      
      * Add clippy CI job
      
      * comment out rusty-cachier
      
      * minor
      
      * fix ci
      
      * remove DAG from check-dependent-project
      
      * add DAG to clippy
      
      Co-authored-by: default avataralvicsam <alvicsam@gmail.com>
      9ea14e66
  22. Nov 14, 2022
  23. Nov 01, 2022
    • Marcin S.'s avatar
      PVF timeouts follow-up (#6151) · 1f821976
      Marcin S. authored
      * Rename timeout consts and timeout parameter; bump leniency
      
      * Update implementor's guide with info about PVFs
      
      * Make glossary a bit easier to read
      
      * Add a note to LENIENT_PREPARATION_TIMEOUT
      
      * Remove PVF-specific section from glossary
      
      * Fix some typos
      1f821976
  24. Oct 22, 2022
  25. Sep 27, 2022
  26. Sep 19, 2022
    • Tsvetomir Dimitrov's avatar
      Improved dispute votes import in provisioner (#5567) · 6ae9720c
      Tsvetomir Dimitrov authored
      * Add `DisputeState` to `DisputeCoordinatorMessage::RecentDisputes`
      
      The new signature of the message is:
      ```
      RecentDisputes(oneshot::Sender<Vec<(SessionIndex, CandidateHash, DisputeStatus)>>),
      ```
      
      As part of the change also add `DispiteStatus` to
      `polkadot_node_primitives`.
      
      * Move dummy_signature() in primitives/test-helpers
      
      * Enable staging runtime api on Rococo
      
      * Implementation
      
      * Move disputes to separate module
      * Vote prioritisation
      * Duplicates handling
      * Double vote handling
      * Unit tests
      * Logs and metrics
      * Code review feedback
      * Fix ACTIVE/INACTIVE separation and update partition names
      * Add `fn dispute_is_inactive` to node primitives and refactor `fn get_active_with_status()` logic
      * Keep the 'old' logic if the staging api is not enabled
      * Fix some comments in tests
      * Add warning message if there are any inactive_unknown_onchain disputes
      * Add file headers and remove `use super::*;` usage outside tests
      * Adding doc comments
      * Fix test methods names
      
      * Fix staging api usage
      
      * Fix `get_disputes` runtime function implementation
      
      * Fix compilation error
      
      * Fix arithmetic operations in tests
      
      * Use smaller test data
      
      * Rename `RuntimeApiRequest::StagingDisputes` to `RuntimeApiRequest::Disputes`
      
      * Remove `staging-client` feature flag
      
      * fmt
      
      * Remove `vstaging` feature flag
      
      * Some comments regarding the staging api
      
      * Rename dispute selection modules in provisioner
      with_staging_api -> prioritized_selection
      without_staging_api -> random_selection
      
      * Comments for staging api
      
      * Comments
      
      * Additional logging
      
      * Code review feedback
      
      process_selected_disputes -> into_multi_dispute_statement_set
      typo
      In trait VoteType: vote_value -> is_valid
      
      * Code review feedback
      
      * Fix metrics
      
      * get_disputes -> disputes
      
      * Get time only once during partitioning
      
      * Fix partitioning
      
      * Comments
      
      * Reduce the number of hardcoded api versions
      
      * Code review feedback
      
      * Unused import
      
      * Comments
      
      * More precise log messages
      
      * Code review feedback
      
      * Code review feedback
      
      * Code review feedback - remove `trait VoteType`
      
      * Code review feedback
      
      * Trace log for DisputeCoordinatorMessage::QueryCandidateVotes counter in vote_selection
      6ae9720c
  27. Sep 05, 2022
  28. Aug 16, 2022
    • Robert Klotzner's avatar
      Reduce dispute coordinator load (#5785) · ca6297c8
      Robert Klotzner authored
      
      * Don't import backing statements directly
      
      into the dispute coordinator. This also gets rid of a redundant
      signature check. Both should have some impact on backing performance.
      In general this PR should make us scale better in the number of parachains.
      
      Reasoning (aka why this is fine):
      
      For the signature check: As mentioned, it is a redundant check. The
      signature has already been checked at this point. This is even made
      obvious by the used types. The smart constructor is not perfect as
      discussed [here](https://github.com/paritytech/polkadot/issues/3455),
      but is still a reasonable security.
      
      For not importing to the dispute-coordinator: This should be good as the
      dispute coordinator does scrape backing votes from chain. This suffices
      in practice as a super majority of validators must have seen a backing
      fork in order for a candidate to get included and only included
      candidates pose a threat to our system. The import from chain is
      preferable over direct import of backing votes for two reasons:
      
      1. The import is batched, greatly improving import performance. All
         backing votes for a candidate are imported with a single import.
         And indeed we were able to see in metrics that importing votes
         from chain is fast.
      2. We do less work in general as not every candidate for which
         statements are gossiped might actually make it on a chain. The
         dispute coordinator as with the current implementation would still
         import and keep those votes around for six sessions.
      
      While redundancy is good for reliability in the event of bugs, this also
      comes at a non negligible cost. The dispute-coordinator right now is the
      subsystem with the highest load, despite the fact that it should not be
      doing much during mormal operation and it is only getting worse
      with more parachains as the load is a direct function of the number of statements.
      
      We'll see on Versi how much of a performance improvement this PR
      
      * Get rid of dead code.
      
      * Dont send approval vote
      
      * Make it pass CI
      
      * Bring back tests for fixing them later.
      
      * Explicit signature check.
      
      * Resurrect approval-voting tests (not fixed yet)
      
      * Send out approval votes in dispute-distribution.
      
      Use BTreeMap for ordered dispute votes.
      
      * Bring back an important warning.
      
      * Fix approval voting tests.
      
      * Don't send out dispute message on import + test
      
      + Some cleanup.
      
      * Guide changes.
      
      Note that the introduced complexity is actually redundant.
      
      * WIP: guide changes.
      
      * Finish guide changes about dispute-coordinator
      
      conceputally. Requires more proof read still.
      
      Also removed obsolete implementation details, where the code is better
      suited as the source of truth.
      
      * Finish guide changes for now.
      
      * Remove own approval vote import logic.
      
      * Implement logic for retrieving approval-votes
      
      into approval-voting and approval-distribution subsystems.
      
      * Update roadmap/implementers-guide/src/node/disputes/dispute-coordinator.md
      
      Co-authored-by: default avatarasynchronous rob <rphmeier@gmail.com>
      
      * Review feedback.
      
      In particular: Add note about disputes of non included candidates.
      
      * Incorporate Review Remarks
      
      * Get rid of superfluous space.
      
      * Tidy up import logic a bit.
      
      Logical vote import is now separated, making the code more readable and
      maintainable.
      
      Also: Accept import if there is at least one invalid signer that has not
      exceeded its spam slots, instead of requiring all of them to not exceed
      their limits. This is more correct and a preparation for vote batching.
      
      * We don't need/have empty imports.
      
      * Fix tests and bugs.
      
      * Remove error prone redundancy.
      
      * Import approval votes on dispute initiated/concluded.
      
      * Add test for approval vote import.
      
      * Make guide checker happy (hopefully)
      
      * Another sanity check + better logs.
      
      * Reasoning about boundedness.
      
      * Use `CandidateIndex` as opposed to `CoreIndex`.
      
      * Remove redundant import.
      
      * Review remarks.
      
      * Add metric for calls to request signatures
      
      * More review remarks.
      
      * Add metric on imported approval votes.
      
      * Include candidate hash in logs.
      
      * More trace log
      
      * Break cycle.
      
      * Add some tracing.
      
      * Cleanup allowed messages.
      
      * fmt
      
      * Tracing + timeout for get inherent data.
      
      * Better error.
      
      * Break cycle in all places.
      
      * Clarified comment some more.
      
      * Typo.
      
      * Break cycle approval-distribution - approval-voting.
      
      Co-authored-by: default avatarasynchronous rob <rphmeier@gmail.com>
      ca6297c8
  29. May 19, 2022
  30. May 12, 2022
  31. Apr 19, 2022
    • asynchronous rob's avatar
      Reduce network bandwidth, improve parablock times: optimize approval-distribution (#5164) · 79ecc538
      asynchronous rob authored
      
      * gossip-support: be explicit about dimensions
      
      * some guide updates
      
      * update network-bridge to distinguish x and y dimensions
      
      * get everything to compile
      
      * beginnings
      
      * some TODOs
      
      * polkadot runtime: use relevant_authorities
      
      * make gossip topologies per-session
      
      * better formatting
      
      * gossip support: use current session validators
      
      * expand in comment
      
      * adjust tests and fix index bug
      
      * add past/present/future connection test and clean up code
      
      * fmt
      
      * network bridge: updated types
      
      * update protocols to new gossip topology message
      
      * guide updates
      
      * add session to BlockApprovalMeta
      
      * add session to block info
      
      * refactor knowledge and remove most unify logic
      
      * start replacing gossip_peers with new SessionTopologies
      
      * add routing information to message state
      
      * add some utilities to SessionTopology
      
      * implement new gossip topology logic
      
      * re-implement unify_with_peer
      
      * distribute assignments according to topology
      
      * finish grid topology implementation
      
      * refactor network bridge slightly
      
      * issue connection requests on all past/present/future
      
      * fmt
      
      * address grumbles
      
      * tighten invariants in unify_with_peer
      
      * implement random propagation
      
      * refactor: extract required routing adjustment logic
      
      * some block-age logic
      
      * aggressively propagate messages when finality is slow
      
      * overhaul aggression system to have 3 levels
      
      * add aggression metrics
      
      * remove aggression L3
      
      * reduce random circulation
      
      * remove PeerData
      
      * get approval tests compiling
      
      * use btree_map in known_by to make deterministic
      
      * Revert "use btree_map in known_by to make deterministic"
      
      This reverts commit 330d65343a7bb6fe4dd0f24bd8dbc15c0cbdbd9d.
      
      * test XY grid propagation
      
      * remove stray println
      
      * test unshared dimension propagation
      
      * add random gossip check
      
      * test unify_with_peer better
      
      * test sending after getting gossip topology
      
      * test L1 aggression on originator
      
      * test L1 aggression for non-originators
      
      * test non-originator aggression L2
      
      * fnt
      
      * ~spellcheck
      
      * fix statement-distribution tests
      
      * fix flaky test
      
      * fix metrics typo
      
      * re-send periodically
      
      * test resending
      
      * typo
      
      Co-authored-by: default avatarBernhard Schuster <bernhard@ahoi.io>
      
      * add more metrics about apd messages
      
      * add back unify_with_peer logs
      
      * make Resend an enum
      
      * be more explicit when resending
      
      * fmt
      
      * fix error
      
      * add a TODO for refactoring
      
      * remove debug metrics
      
      * add some guide stuff
      
      * fmt
      
      * update runtime API in test-runtim
      
      Co-authored-by: default avatarBernhard Schuster <bernhard@ahoi.io>
      79ecc538
  32. Apr 13, 2022
  33. Apr 04, 2022
  34. Mar 23, 2022
  35. Mar 09, 2022
    • asynchronous rob's avatar
      Finish migration to v2 primitives (#5037) · 49f7e5cc
      asynchronous rob authored
      * remove v0 primitives from polkadot-primitives
      
      * first pass: remove v0
      
      * fix fallout in erasure-coding
      
      * remove v1 primitives, consolidate to v2
      
      * the great import update
      
      * update runtime_api_impl_v1 to v2 as well
      
      * guide: add `Version` request for runtime API
      
      * add version query to runtime API
      
      * reintroduce OldV1SessionInfo in a limited way
      49f7e5cc
  36. Feb 07, 2022
  37. Dec 27, 2021
    • ordian's avatar
      session-info: add new fields + migration (#4545) · b342ae11
      ordian authored
      
      * session_info: v2 + migration
      
      * use primitives::v2
      
      * use polkadot_primitives::v2
      
      * impl primitives::v2
      
      * fix approval-voting tests
      
      * fix other tests
      
      * hook storage migration up
      
      * backwards compat (1)
      
      * backwards compat (2)
      
      * fmt
      
      * fix tests
      
      * FMT
      
      * do not reexport v1 in v2
      
      * fmt
      
      * set storage version to 1
      
      Co-authored-by: default avatarJavier Viola <javier@parity.io>
      b342ae11