high altitude description of disputes (#2285)

polkadot/.config/lingua.dic

@@ -85,4 +85,12 @@ WND/S
 XCM/S
 XCMP/M
 include/BG
-isolate/BG
\ No newline at end of file
+isolate/BG
+dispatchable/SM
+VMP/SM
+HMP/SM
+DMP/SM
+MQC/SM
+VRF/SM
+assignee/SM
+adversary/SM
\ No newline at end of file

polkadot/.config/spellcheck.toml

@@ -7,6 +7,7 @@ extra_dictionaries = ["lingua.dic"]
 # `Type`'s
 # 5x
 # He tagged it as 'TheGreatestOfAllTimes'
-transform_regex = ["^'([^\\s])'$", "^[0-9]+(?:\\.[0-9]*)?x$", "^'s$", "^\\+$"]
+# Transforms'
+transform_regex = ["^'([^\\s])'$", "^[0-9]+(?:\\.[0-9]*)?x$", "^(.*)'$", "^\\+$"]
 allow_concatenation = true
 allow_dashes = true

polkadot/roadmap/implementers-guide/src/SUMMARY.md

@@ -5,6 +5,8 @@
 - [Whence Parachains](whence-parachains.md)
 - [Protocol Overview](protocol-overview.md)
   - [Approval Process](protocol-approval.md)
+  - [Disputes Process](protocol-disputes.md)
+    - [Dispute Flow](disputes-flow.md)
 - [Architecture Overview](architecture.md)
   - [Messaging Overview](messaging.md)
 - [Runtime Architecture](runtime/README.md)

polkadot/roadmap/implementers-guide/src/disputes-flow.md

+# Disputes Flows
+
+A component-free description in what-if form with addition state graphs of the dispute.
+
+```mermaid
+stateDiagram-v2
+    [*] --> WaitForBackingVote: negative Vote received
+    [*] --> WaitForDisputeVote: backing Vote received
+    WaitForBackingVote --> Open: negative Vote received
+    WaitForDisputeVote --> Open: backing Vote received
+    Open --> Concluded: Timeout without supermajority
+    Open --> Concluded: Incoming Vote via Gossip
+    Open --> Open: No ⅔ supermajority
+    Open --> [*]
+    Concluded --> [*]
+```
+
+---
+
+```mermaid
+stateDiagram-v2
+    [*] --> Open: First Vote(s) received
+    Open --> HasPoV : Fetch Availability Store for PoV
+
+    HasPoV --> HasCode : Fetch historical Code
+    HasCode --> VerifyWithRuntime: All Data locally avail
+
+    Open --> DisputeAvailabilityDataReceived
+    DisputeAvailabilityDataReceived --> VerifyWithRuntime: Received Gossip
+
+    HasPoV --> RequestDisputeAvailabilityData: nope
+    HasCode --> RequestDisputeAvailabilityData: nope
+    RequestDisputeAvailabilityData --> VerifyWithRuntime: Received
+    RequestDisputeAvailabilityData --> RequestDisputeAvailabilityData: Timed out - pick another peer
+
+    VerifyWithRuntime --> CastVoteValid: Block Valid
+    VerifyWithRuntime --> CastVoteInvalid: Block Invalid
+    CastVoteInvalid --> GossipVote
+    CastVoteValid --> GossipVote
+    GossipVote --> [*]
+
+```
+
+---
+
+Dispute Availability Data
+
+```mermaid
+stateDiagram-v2
+    [*] --> Open: First Vote(s) received
+    Open --> DisputeDataAvail: somehow the data became available
+    Open --> RespondUnavailable: Data not available
+    IncomingRequestDisputeAvailabilityData --> RespondUnavailable
+    IncomingRequestDisputeAvailabilityData --> DisputeDataAvail
+    DisputeDataAvail --> RespondWithDisputeAvailabilityData: Send
+    VoteGossipReceived --> Track: implies source peer has<br />dispute availablity data
+```
+
+---
+
+Peer handling
+
+```mermaid
+stateDiagram-v2
+    [*] --> Open: First Vote(s) received
+    Open --> GossipVotes: for all current peers
+    Open --> PeerConnected: another
+    PeerConnected --> GossipVotes: Peer connects
+    GossipVotes --> [*]
+```
+
+## Conditional formulation
+
+The set of validators eligible to vote consists of
+the validators that had duty at the time of backing, plus backing votes by the backing validators.
+
+If a validator receives an initial dispute message (a set of votes where there are at least two opposing votes contained), and the PoV or Code are hence not reconstructable from local storage, that validator must request the required data from its peers.
+
+The dispute availability message must contain code, persisted validation data, and the proof of validity.
+
+Only peers that already voted shall be queried for the dispute availability data.
+
+The peer to be queried for disputes data, must be picked at random.
+
+A validator must retain code, persisted validation data and PoV until a block, that contains the dispute resolution, is finalized - plus an additional 24h.
+
+Dispute availability gossip must continue beyond the dispute resolution, until the post resolution timeout expired (equiv to the timeout until which additional late votes are accepted).
+
+Remote disputes are disputes that are in relation to a chain that is not part of the local validators active heads.
+
+All incoming votes must be persisted.
+
+Persisted votes stay persisted for `N` sessions, and are cleaned up on a per session basis.
+
+Votes must be queryable by a particular validator, identified by its signing key.
+
+Votes must be queryable by a particular validator, identified by a session index and the validator index valid in that session.
+
+If there exists a negative and a positive fork for a particular block, a dispute is detected.
+
+If a dispute is detected, all currently available votes for that block must be gossiped.
+
+If an incoming dispute vote is detected, a validator must cast their own vote. The vote is determined by validating the PoV with the Code at the time of backing the block in question.
+
+If the validator was also a backer of the block, validation and casting an additional vote should be skipped.
+
+If the count of votes pro or cons regarding the disputed block, reaches the required ⅔ supermajority (including the backing votes), the conclusion must be recorded on chain and the voters on the loosing and no-shows being slashed appropriately.
+
+If a block is found invalid by a dispute resolution, it must be blacklisted to avoid resync or further build on that chain if other chains are available (to be detailed in the grandpa fork choice rule).
+
+A dispute accepts Votes after the dispute is resolved, for 1d.
+
+If a vote is received, after the dispute is resolved, the vote shall still be recorded in the state root, albeit yielding less reward.
+
+Recording in the state root might happen batched, at timeout expiry.
+
+If a new active head/chain appears, and the dispute resolution was not recorded on that chain yet, the dispute resolution or open dispute must be recorded / transplanted to that chain as well, since the disputes must be present on all chains to make sure the offender is punished.
+
+If a validator votes in two opposing ways, this composes of a double vote like in other cases (backing, approval voting).
+
+If a dispute is not resolved within due time, all validators are to be slashed for a small amount.
+
+If a dispute is not resolved within due time, governance mode shall be entered for manual resolution.
+
+If a validator unexpectedly restarts, the dispute shall be continued with the state based on votes being cast and being present in persistent storage.

polkadot/roadmap/implementers-guide/src/protocol-disputes.md

+# Disputes
+
+Fast forward to [more detailed disputes requirments](./disputes-flow.md).
+
+## Motivation
+
+All blocks that end up on chain should be valid.
+
+To ensure attempts, successful or not, of including
+a block that is invalid in respect to the validation code, must therefore be handled in some way with the offenders being punished and the whistle blowers being rewarded. Disputes and their resolution are the formal process to resolve these situations.
+
+At some point a validator claims that the `PoV` (proof of validity) - which was distributed with candidate block - for a certain block is invalid.
+
+Now the dispute can be happening quite some time later than the inclusion, but also right during backing. As such the block is stored or more so its Reed-Solomon encoded erasure chunks, from which the
+PoV can be reconstructed.
+
+A reconstructed PoV can be verified with the defined verification code, that is valid during the session the block was included or backed.
+If the block is invalid and there exists at least one backing vote and one validity challenging vote, a dispute exists.
+The existence of a dispute is detected by a backing checker
+or, if the block made it through backing stage, by an approval checker.
+In either case, the validator casts and distributes its vote via means of gossip.
+
+At this point the set of backing validators can not be trusted (since they voted for the block despite something being
+fishy at the very least). On the other hand, one must also consider, the validator that blew the whistle has ulterior motives
+to do so (i.e. it is controlled by a third party and wants to incur damage to itself).
+In either way, there are malicious validators around.
+As a consequence, all validators at the time of block backing, are being notified via broadcast of
+the first pair of backing and challenging vote.
+Validators that backed the candidate implicitly count as votes. Those validators are allowed to cast
+a regular vote (a non-backing vote) as well, but it is generally not in their interest to vote both sides, since that would
+advance the progress towards supermajority either way and have their bonds slashed.
+If both votes lean in the same direction, i.e. both positive they are only counted as one.
+Two opposing votes by the same validator would be equal to an attempted double vote and would be slashed accordingly.
+
+All validators at block inclusion time are eligible to (and should) cast their Vote. The backing votes of backing checkers
+are counted as votes as well.
+
+## Initiation
+
+A dispute is initiated by one approval checker creating and gossiping a vote, that challenges another vote.
+
+After a backing or approval checker challenged a block, all validators that received the gossiped vote, reconstruct the block
+from availability erasure code chunks and check the block's PoV themselves via the validation code.
+The result of that check is converted into a vote, and distributed via the same mechanics as the first one.
+
+Once a receiver receives ⅔ supermajority in one or the other direction, the
+vote is concluded.
+Conclusion implies that the result for this block can not be altered anymore, valid or invalid is fixed now.
+
+In order to ensure, the dispute result is not forgotten or intentionally side stepped, it has to be recorded on chain.
+This on chain recording mechanic must be vigilant, in a sense, that new emerging forks
+must also receive the dispute resolution recorded (transplantation) irrespectively of the disputed block being included in that chain or not.
+
+If the disputed block was already finalized, the chain must be put in governance mode to be resolved by human interaction
+(i.e. sudo or motion or other mechanics that are available ).
+
+As such the validator has to keep track of all votes irrespective if the disputed block is already known or not.
+All backing votes should be either kept in storage as well, or be queried on demand, since they are a kind of vote
+as well.
+
+## Late votes
+
+Late votes, after the dispute already reached a ⅔ supermajority, must be rewarded (albeit a smaller amount) as well.
+These ones must be attached to the votes after a defined period of time after the result has reached
+the required ⅔ supermajority.
+
+## Chain Selection / Grandpa
+
+Chain selection should be influenced by the chance of picking a chain that does not even include the disputed block.
+Hence removing the need to include the dispute resolution itself.
+This is only possible though, if the set of active heads contains such a fork.
+In Grandpa the Voting rule should be used to avoid finalizing chains that contain an open or negative shut (shut with supermajority that marks the block as invalid) dispute.
+In case all possible chains contains such a dispute, a metric must be used to decide which fork to use or avoid finalization until one dispute resolves positive (the
+block is valid).