Implementers' Guide: Chain Selection (#3262)

* high-level discussion of fork-choice and chain selection

* notes on chain-selection subsystem

* send `Approve` messages from approval-voting

* remove references to candidate-selection

* adjust grandpa voting rule docs

* Update roadmap/implementers-guide/src/node/grandpa-voting-rule.md

Co-authored-by: Lldenaurois <ljdenaurois@gmail.com>

* Update roadmap/implementers-guide/src/protocol-chain-selection.md

Co-authored-by: Lldenaurois <ljdenaurois@gmail.com>

* Update roadmap/implementers-guide/src/protocol-chain-selection.md

Co-authored-by: Lldenaurois <ljdenaurois@gmail.com>

Co-authored-by: Lldenaurois <ljdenaurois@gmail.com>

roadmap/implementers-guide/src/SUMMARY.md

@@ -7,6 +7,7 @@
   - [Approval Process](protocol-approval.md)
   - [Disputes Process](protocol-disputes.md)
     - [Dispute Flow](disputes-flow.md)
+  - [Chain Selection and Finalization](protocol-chain-selection.md)
 - [Architecture Overview](architecture.md)
   - [Messaging Overview](messaging.md)
 - [Runtime Architecture](runtime/README.md)
@@ -41,10 +42,8 @@
     - [Collation Generation](node/collators/collation-generation.md)
     - [Collator Protocol](node/collators/collator-protocol.md)
   - [Backing Subsystems](node/backing/README.md)
-    - [Candidate Selection](node/backing/candidate-selection.md)
     - [Candidate Backing](node/backing/candidate-backing.md)
     - [Statement Distribution](node/backing/statement-distribution.md)
-    - [PoV Distribution](node/backing/pov-distribution.md)
   - [Availability Subsystems](node/availability/README.md)
     - [Availability Distribution](node/availability/availability-distribution.md)
     - [Availability Recovery](node/availability/availability-recovery.md)
@@ -67,6 +66,7 @@
     - [Peer Set Manager](node/utility/peer-set-manager.md)
     - [Runtime API Requests](node/utility/runtime-api.md)
     - [Chain API Requests](node/utility/chain-api.md)
+    - [Chain Selection Request](node/utility/chain-selection.md)
 - [Data Structures and Types](types/README.md)
     - [Candidate](types/candidate.md)
     - [Backing](types/backing.md)

roadmap/implementers-guide/src/node/approval/approval-voting.md

@@ -180,6 +180,7 @@ On receiving an `OverseerSignal::ActiveLeavesUpdate(update)`:
     * If any of the runtime API calls fail, we just warn and skip the block.
   * We use the RuntimeApiSubsystem to determine the set of candidates included in these blocks and use BABE logic to determine the slot number and VRF of the blocks.
   * We also note how late we appear to have received the block. We create a `BlockEntry` for each block and a `CandidateEntry` for each candidate obtained from `CandidateIncluded` events after making a `RuntimeApiRequest::CandidateEvents` request.
+  * For each candidate, if the amount of needed approvals is more than the validators remaining after the backing group of the candidate is subtracted, then the candidate is insta-approved as approval would be impossible otherwise. If all candidates in the block are insta-approved, or there are no candidates in the block, then the block is insta-approved. If the block is insta-approved, a [`ChainSelectionMessage::Approvedl][CSM] should be sent for the block.
   * Ensure that the `CandidateEntry` contains a `block_assignments` entry for the block, with the correct backing group set.
   * If a validator in this session, compute and assign `our_assignment` for the `block_assignments`
     * Only if not a member of the backing group.
@@ -240,6 +241,7 @@ On receiving an `ApprovedAncestor(Hash, BlockNumber, response_channel)`:
   * Checks the `ApprovalEntry` for the block.
     * [determine the tranches to inspect](#determine-required-tranches) of the candidate,
     * [the candidate is approved under the block](#check-approval), set the corresponding bit in the `block_entry.approved_bitfield`.
+    * If the block is now fully approved and was not before, send a [`ChainSelectionMessage::Approved`][CSM].
     * Otherwise, [schedule a wakeup of the candidate](#schedule-wakeup)
   * If the approval vote originates locally, set the `our_approval_sig` in the candidate entry.
 
@@ -370,3 +372,5 @@ Likewise, when considering how many tranches to take, the no-show depth should b
 #### Current Tranche
   * Given the slot number of a block, and the current time, this informs about the current tranche.
   * Convert `time.saturating_sub(slot_number.to_time())` to a delay tranches value
+
+[CSM]: ../../types/overseer-protocol.md#chainselectionmessage

roadmap/implementers-guide/src/node/backing/candidate-backing.md

@@ -16,16 +16,16 @@ Output:
 
 - [`CandidateValidationMessage`][CVM]
 - [`RuntimeApiMessage`][RAM]
-- [`CandidateSelectionMessage`][CSM]
+- [`CollatorProtocolMessage`][CPM]
 - [`ProvisionerMessage`][PM]
 - [`AvailabilityDistributionMessage`][ADM]
 - [`StatementDistributionMessage`][SDM]
 
 ## Functionality
 
-The [Candidate Selection][CS] subsystem is the primary source of non-overseer messages into this subsystem. That subsystem generates appropriate [`CandidateBackingMessage`s][CBM] and passes them to this subsystem.
+The [Collator Protocol][CP] subsystem is the primary source of non-overseer messages into this subsystem. That subsystem generates appropriate [`CandidateBackingMessage`s][CBM] and passes them to this subsystem.
 
-This subsystem requests validation from the [Candidate Validation][CV] and generates an appropriate [`Statement`][Statement]. All `Statement`s are then passed on to the [Statement Distribution][SD] subsystem to be gossiped to peers. When [Candidate Validation][CV] decides that a candidate is invalid, and it was recommended to us to second by our own [Candidate Selection][CS] subsystem, a message is sent to the [Candidate Selection][CS] subsystem with the candidate's hash so that the collator which recommended it can be penalized.
+This subsystem requests validation from the [Candidate Validation][CV] and generates an appropriate [`Statement`][Statement]. All `Statement`s are then passed on to the [Statement Distribution][SD] subsystem to be gossiped to peers. When [Candidate Validation][CV] decides that a candidate is invalid, and it was recommended to us to second by our own [Collator Protocol][CP] subsystem, a message is sent to the [Collator Protocol][CP] subsystem with the candidate's hash so that the collator which recommended it can be penalized.
 
 The subsystem should maintain a set of handles to Candidate Backing Jobs that are currently live, as well as the relay-parent to which they correspond.
 
@@ -117,7 +117,7 @@ fn spawn_validation_work(candidate, parachain head, validation function) {
 ### Fetch Pov Block
 
 Create a `(sender, receiver)` pair.
-Dispatch a [`AvailabilityDistributionMessage`][PDM]`::FetchPoV{ validator_index, pov_hash, candidate_hash, tx, } and listen on the passed receiver for a response. Availability distribution will send the request to the validator specified by `validator_index`, which might not be serving it for whatever reasons, therefore we need to retry with other backing validators in that case.
+Dispatch a [`AvailabilityDistributionMessage`][ADM]`::FetchPoV{ validator_index, pov_hash, candidate_hash, tx, } and listen on the passed receiver for a response. Availability distribution will send the request to the validator specified by `validator_index`, which might not be serving it for whatever reasons, therefore we need to retry with other backing validators in that case.
 
 
 ### Validate PoV Block
@@ -127,12 +127,12 @@ Dispatch a `CandidateValidationMessage::Validate(validation function, candidate,
 
 ### Distribute Signed Statement
 
-Dispatch a [`StatementDistributionMessage`][PDM]`::Share(relay_parent, SignedFullStatement)`.
+Dispatch a [`StatementDistributionMessage`][SDM]`::Share(relay_parent, SignedFullStatement)`.
 
 [OverseerSignal]: ../../types/overseer-protocol.md#overseer-signal
 [Statement]: ../../types/backing.md#statement-type
 [STMT]: ../../types/backing.md#statement-type
-[CSM]: ../../types/overseer-protocol.md#candidate-selection-message
+[CPM]: ../../types/overseer-protocol.md#collator-protocol-message
 [RAM]: ../../types/overseer-protocol.md#runtime-api-message
 [CVM]: ../../types/overseer-protocol.md#validation-request-type
 [PM]: ../../types/overseer-protocol.md#provisioner-message
@@ -141,7 +141,7 @@ Dispatch a [`StatementDistributionMessage`][PDM]`::Share(relay_parent, SignedFul
 [SDM]: ../../types/overseer-protocol.md#statement-distribution-message
 [DCM]: ../../types/overseer-protocol.md#dispute-coordinator-message
 
-[CS]: candidate-selection.md
+[CP]: ../collators/collator-protocol.md
 [CV]: ../utility/candidate-validation.md
 [SD]: statement-distribution.md
 [RA]: ../utility/runtime-api.md

roadmap/implementers-guide/src/node/grandpa-voting-rule.md

 # GRANDPA Voting Rule
 
-[GRANDPA](https://w3f-research.readthedocs.io/en/latest/polkadot/finality.html) is the finality engine of Polkadot.
-
-One broad goal of finality, which applies across many different blockchains, is that there should exist only one finalized block at each height in the finalized chain. Before a block at a given height is finalized, it may compete with other forks.
+Specifics on the motivation and types of constraints we apply to the GRANDPA voting logic as well as the definitions of **viable** and **finalizable** blocks can be found in the [Chain Selection Protocol](../protocol-chain-selection.md) section.
+The subsystem which provides us with viable leaves is the [Chain Selection Subsystem](utility/chain-selection.md). 
 
 GRANDPA's regular voting rule is for each validator to select the longest chain they are aware of. GRANDPA proceeds in rounds, collecting information from all online validators and determines the blocks that a supermajority of validators all have in common with each other.
 
-For parachains, we extend the security guarantee of finality to be such that no invalid parachain candidate may be included in a finalized block. Candidates may be included in some fork of the relay chain with only a few backing votes behind them. After that point, we run the [Approvals Protocol](../protocol-approval.md), which is implemented as the [Approval Voting](approval/approval-voting.md) subsystem. This system involves validators self-selecting to re-check candidates included in all observed forks of the relay chain as well as an algorithm for observing validators' statements about assignment and approval in order to determine which candidates, and thus blocks, are with high probability valid. The highest approved ancestor of a given block can be determined by querying the Approval Voting subsystem via the [`ApprovalVotingMessage::ApprovedAncestor`](../types/overseer-protocol.md#approval-voting) message. If the response of `ApprovedAncestor` is `Some`, we further constrain the voting rule to avoid unfinalized blocks. The list of block hashes and candidates should be reversed, and passed to the [`DisputeCoordinatorMessage::DetermineUndisputedChain`](../types/overseer-protocol.md#dispute-coordinator-message) for a final result.
+The low-level GRANDPA logic will provide us with a **required block**. We can find the best leaf containing that block in its chain with the [`ChainSelectionMessage::BestLeafContaining`](../types/overseer-protocol.md#chain-selection-message). If the result is `None`, then we will simply cast a vote on the required block.
 
-Lastly, we refuse to finalize any block including a candidate for which we are aware of an ongoing dispute or of a dispute resolving against the candidate. The exact means of doing this has not been determined yet.
+The **viable** leaves provided from the chain selection subsystem are not necessarily **finalizable**, so we need to perform further work to discover the finalizable ancestor of the block. The first constraint is to avoid voting on any unapproved block. The highest approved ancestor of a given block can be determined by querying the Approval Voting subsystem via the [`ApprovalVotingMessage::ApprovedAncestor`](../types/overseer-protocol.md#approval-voting) message. If the response is `Some`, we continue and apply the second constraint. The second constraint is to avoid voting on any block containing a candidate undergoing an active dispute. The list of block hashes and candidates returned from `ApprovedAncestor` should be reversed, and passed to the [`DisputeCoordinatorMessage::DetermineUndisputedChain`](../types/overseer-protocol.md#dispute-coordinator-message) to determine the **finalizable** block which will be our eventual vote.

roadmap/implementers-guide/src/node/utility/chain-selection.md

+# Chain Selection Subsystem
+
+This subsystem implements the necessary metadata for the implementation of the [chain selection](../../protocol-chain-selection.md) portion of the protocol.
+
+The subsystem wraps a database component which maintains a view of the unfinalized chain and records the properties of each block: whether the block is **viable**, whether it is **stagnant**, and whether it is **reverted**. It should also maintain an updated set of active leaves in accordance with this view, which should be cheap to query.
+
+This subsystem needs to update its information on the unfinalized chain:
+  * On every leaf-activated signal
+  * On every block-finalized signal
+  * On every `ChainSelectionMessage::Approve`
+  * Periodically, to detect stagnation.
+
+Simple implementations of these updates do O(n_unfinalized_blocks) disk operations. If the amount of unfinalized blocks is relatively small, the updates should not take very much time. However, in cases where there are hundreds or thousands of unfinalized blocks the naive implementations of these update algorithms would have to be replaced with more sophisticated versions.
+
+### `OverseerSignal::ActiveLeavesUpdate`
+
+Determine all new blocks implicitly referenced by any new active leaves and add them to the view. Update the set of viable leaves accordingly
+
+### `OverseerSignal::BlockFinalized`
+
+Delete data for all orphaned chains and update all metadata descending from the new finalized block accordingly, along with the set of viable leaves. Note that finalizing a **reverted** or **stagnant** block means that the descendants of those blocks may lose that status because the definitions of those properties don't include the finalized chain. Update the set of viable leaves accordingly.
+
+### `ChainSelectionMessage::Approved`
+
+Update the approval status of the referenced block. If the block was stagnant and thus non-viable and is now viable, then the metadata of all of its descendants needs to be updated as well, as they may no longer be stagnant either. Update the set of viable leaves accordingly.
+
+### `ChainSelectionMessage::BestLeafContaining`
+
+If the required block is unknown or not viable, then return `None`.
+Iterate over all leaves, returning the first leaf containing the required block in its chain, and `None` otherwise.
+
+### Periodically
+
+Detect stagnant blocks and apply the stagnant definition to all descendants. Update the set of viable leaves accordingly.

roadmap/implementers-guide/src/protocol-chain-selection.md

+# Chain Selection
+
+Chain selection processes in blockchains are used for the purpose of selecting blocks to build on and finalize. It is important for these processes to be consistent among nodes and resilient to a maximum proportion of malicious nodes which do not obey the chain selection process.
+
+The parachain host uses both a block authoring system and a finality gadget. The chain selection strategy of the parachain host involves two key components: a _leaf-selection_ rule and a set of _finality constraints_. When it's a validator's turn to author on a block, they are expected to select the best block via the leaf-selection rule to build on top of. When a validator is participating in finality, there is a minimum block which can be voted on, which is usually the finalized block. The validator should select the best chain according to the leaf-selection rule and subsequently apply the finality constraints to arrive at the actual vote cast by that validator.
+
+Before diving into the particularities of the leaf-selection rule and the finality constraints, it's important to discuss the goals that these components are meant to achieve. For this it is useful to create the definitions of _viable_ and _finalizable_ blocks.
+
+### Property Definitions
+
+A block is considered **viable** when all of the following hold:
+  1. It is or descends from the finalized block
+  2. It is not **stagnant**
+  3. It is not **reverted**.
+
+A block is considered a **viable leaf** when all of the following hold:
+  1. It is **viable**
+  2. It has no **viable** descendant.
+
+A block is considered **stagnant** when either:
+  1. It is unfinalized, is not approved, and has not been approved within 2 minutes
+  2. Its parent is **stagnant**.
+
+A block is considered **reverted** when either:
+  1. It is unfinalized and includes a candidate which has lost a dispute
+  2. Its parent is **reverted**
+
+A block is considered **finalizable** when all of the following hold:
+  1. It is **viable**
+  2. Its parent, if unfinalized, is **finalizable**.
+  3. It is either finalized or approved.
+  4. It is either finalized or includes no candidates which have unresolved disputes or have lost a dispute.
+
+
+### The leaf-selection rule
+
+We assume that every block has an implicit weight or score which can be used to compare blocks. In BABE, this is determined by the number of primary slots included in the chain. In PoW, this is the chain with either the most work or GHOST weight.
+
+The leaf-selection rule based on our definitions above is simple: we take the maximum-scoring viable leaf we are aware of. In the case of a tie we select the one with a lower lexicographical block hash.
+
+### The best-chain-containing rule
+
+Finality gadgets, as mentioned above, will often impose an additional requirement to vote on a chain containing a specific block, known as the **required** block. Although this is typically the most recently finalized block, it is possible that it may be a block that is unfinalized. When receiving such a request:
+1. If the required block is the best finalized block, then select the best viable leaf.
+2. If the required block is unfinalized and non-viable, then select the required block and go no further. This is likely an indication that something bad will be finalized in the network, which will never happen when approvals & disputes are functioning correctly. Nevertheless we account for the case here.
+3. If the required block is unfinalized and non-viable, then iterate over the viable leaves in descending order by score and select the first one which contains the required block in its chain. Backwards iteration is a simple way to check this, but if unfinalized chains grow long then Merkle Mountain-Ranges will most likely be more efficient.
+
+Once selecting a leaf, the chain should be constrained to the maximum of the required block or the highest **finalizable** ancestor.

roadmap/implementers-guide/src/protocol-disputes.md

@@ -62,7 +62,3 @@ Validators are rewarded for providing statements to the chain as well as for par
 Disputes, roughly, are over when one side reaches a ⅔ supermajority. They may also conclude after a timeout, without either side witnessing supermajority, which will only happen if the majority of validators are unable to vote for some reason. Furthermore, disputes on-chain will stay open for some fixed amount of time even after concluding, to accept new votes.
 
 Late votes, after the dispute already reached a ⅔ supermajority, must be rewarded (albeit a smaller amount) as well.
-
-## Chain Selection / Grandpa
-
-The [Approval Checking](protocol-approval.md) protocol prevents finalization of chains that contain parablocks that are not yet approved. With disputes, we take it one step further and do not vote to finalize any chains which contain parablocks that are being disputed or have lost a dispute anywhere.

roadmap/implementers-guide/src/types/overseer-protocol.md

@@ -287,21 +287,6 @@ enum CandidateBackingMessage {
 }
 ```
 
-## Candidate Selection Message
-
-These messages are sent to the [Candidate Selection subsystem](../node/backing/candidate-selection.md) as a means of providing feedback on its outputs.
-
-```rust
-enum CandidateSelectionMessage {
-    /// A candidate collation can be fetched from a collator and should be considered for seconding.
-    Collation(RelayParent, ParaId, CollatorId),
-    /// We recommended a particular candidate to be seconded, but it was invalid; penalize the collator.
-    Invalid(RelayParent, CandidateReceipt),
-    /// The candidate we recommended to be seconded was validated successfully.
-    Seconded(RelayParent, SignedFullStatement),
-}
-```
-
 ## Chain API Message
 
 The Chain API subsystem is responsible for providing an interface to chain data.
@@ -336,6 +321,23 @@ enum ChainApiMessage {
 }
 ```
 
+## Chain Selection Message
+
+Messages received by the [Chain Selection subsystem](../node/utility/chain-selection.md)
+
+```rust
+enum ChainSelectionMessage {
+    /// Signal to the chain selection subsystem that a specific block has been approved.
+    Approved(Hash),
+    /// Request the leaves in descending order by score.
+    Leaves(ResponseChannel<Vec<Hash>>),
+    /// Request the best leaf containing the given block in its ancestry. Return `None` if
+    /// there is no such leaf.
+    BestLeafContaining(Hash, ResponseChannel<Option<Hash>>),
+    
+}
+```
+
 ## Collator Protocol Message
 
 Messages received by the [Collator Protocol subsystem](../node/collators/collator-protocol.md)