Unverified Commit 8a6af441 authored by Denis_P's avatar Denis_P 🏑 Committed by GitHub
Browse files

WIP: CI: add spellcheck (#3421)



* CI: add spellcheck

* revert me

* CI: explicit command for spellchecker

* spellcheck: edit misspells

* CI: run spellcheck on diff

* spellcheck: edits

* spellcheck: edit misspells

* spellcheck: add rules

* spellcheck: mv configs

* spellcheck: more edits

* spellcheck: chore

* spellcheck: one more thing

* spellcheck: and another one

* spellcheck: seems like it doesn't get to an end

* spellcheck: new words after rebase

* spellcheck: new words appearing out of nowhere

* chore

* review edits

* more review edits

* more edits

* wonky behavior

* wonky behavior 2

* wonky behavior 3

* change git behavior

* spellcheck: another bunch of new edits

* spellcheck: new words are koming out of nowhere

* CI: finding the master

* CI: fetching master implicitly

* CI: undebug

* new errors

* a bunch of new edits

* and some more

* Update node/core/approval-voting/src/approval_db/v1/mod.rs

Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>

* Update xcm/xcm-executor/src/assets.rs

Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>

* Apply suggestions from code review

Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>

* Suggestions from the code review

* CI: scan only changed files

Co-authored-by: Andronik Ordian's avatarAndronik Ordian <write@reusable.software>
parent 43920cd7
Pipeline #147422 canceled with stages
in 7 minutes and 46 seconds
......@@ -255,11 +255,11 @@ impl<Payload, RealPayload> From<Signed<Payload, RealPayload>> for UncheckedSigne
}
}
/// This helper trait ensures that we can encode Statement as CompactStatement,
/// This helper trait ensures that we can encode `Statement` as `CompactStatement`,
/// and anything as itself.
///
/// This resembles `parity_scale_codec::EncodeLike`, but it's distinct:
/// EncodeLike is a marker trait which asserts at the typesystem level that
/// `EncodeLike` is a marker trait which asserts at the typesystem level that
/// one type's encoding is a valid encoding for another type. It doesn't
/// perform any type conversion when encoding.
///
......
......@@ -26,6 +26,6 @@ The Node-side code comes with a set of assumptions that we build upon. These ass
We assume the following constraints regarding provided basic functionality:
* The underlying **consensus** algorithm, whether it is BABE or SASSAFRAS is implemented.
* There is a **chain synchronization** protocol which will search for and download the longest available chains at all times.
* The **state** of all blocks at the head of the chain is available. There may be **state pruning** such that state of the last `k` blocks behind the last finalized block are available, as well as the state of all their descendents. This assumption implies that the state of all active leaves and their last `k` ancestors are all available. The underlying implementation is expected to support `k` of a few hundred blocks, but we reduce this to a very conservative `k=5` for our purposes.
* The **state** of all blocks at the head of the chain is available. There may be **state pruning** such that state of the last `k` blocks behind the last finalized block are available, as well as the state of all their descendants. This assumption implies that the state of all active leaves and their last `k` ancestors are all available. The underlying implementation is expected to support `k` of a few hundred blocks, but we reduce this to a very conservative `k=5` for our purposes.
* There is an underlying **networking** framework which provides **peer discovery** services which will provide us with peers and will not create "loopback" connections to our own node. The number of peers we will have is assumed to be bounded at 1000.
* There is a **transaction pool** and a **transaction propagation** mechanism which maintains a set of current transactions and distributes to connected peers. Current transactions are those which are not outdated relative to some "best" fork of the chain, which is part of the active heads, and have not been included in the best fork.
......@@ -4,9 +4,9 @@ Reading the [section on the approval protocol](../../protocol-approval.md) will
Approval votes are split into two parts: Assignments and Approvals. Validators first broadcast their assignment to indicate intent to check a candidate. Upon successfully checking, they broadcast an approval vote. If a validator doesn't broadcast their approval vote shortly after issuing an assignment, this is an indication that they are being prevented from recovering or validating the block data and that more validators should self-select to check the candidate. This is known as a "no-show".
The core of this subsystem is a Tick-based timer loop, where Ticks are 500ms. We also reason about time in terms of DelayTranches, which measure the number of ticks elapsed since a block was produced. We track metadata for all un-finalized but included candidates. We compute our local assignments to check each candidate, as well as which DelayTranche those assignments may be minimally triggered at. As the same candidate may appear in more than one block, we must produce our potential assignments for each (Block, Candidate) pair. The timing loop is based on waiting for assignments to become no-shows or waiting to broadcast and begin our own assignment to check.
The core of this subsystem is a Tick-based timer loop, where Ticks are 500ms. We also reason about time in terms of `DelayTranche`s, which measure the number of ticks elapsed since a block was produced. We track metadata for all un-finalized but included candidates. We compute our local assignments to check each candidate, as well as which `DelayTranche` those assignments may be minimally triggered at. As the same candidate may appear in more than one block, we must produce our potential assignments for each (Block, Candidate) pair. The timing loop is based on waiting for assignments to become no-shows or waiting to broadcast and begin our own assignment to check.
Another main component of this subsystem is the logic for determining when a (Block, Candidate) pair has been approved and when to broadcast and trigger our own assignment. Once a (Block, Candidate) pair has been approved, we mark a corresponding bit in the BlockEntry that indicates the candidate has been approved under the block. When we trigger our own assignment, we broadcast it via Approval Distribution, begin fetching the data from Availability Recovery, and then pass it through to the Candidate Validation. Once these steps are successful, we issue our approval vote. If any of these steps fail, we don't issue any vote and will "no-show" from the perspective of other validators. In the future we will initiate disputes as well.
Another main component of this subsystem is the logic for determining when a (Block, Candidate) pair has been approved and when to broadcast and trigger our own assignment. Once a (Block, Candidate) pair has been approved, we mark a corresponding bit in the `BlockEntry` that indicates the candidate has been approved under the block. When we trigger our own assignment, we broadcast it via Approval Distribution, begin fetching the data from Availability Recovery, and then pass it through to the Candidate Validation. Once these steps are successful, we issue our approval vote. If any of these steps fail, we don't issue any vote and will "no-show" from the perspective of other validators. In the future we will initiate disputes as well.
Where this all fits into Polkadot is via block finality. Our goal is to not finalize any block containing a candidate that is not approved. We provide a hook for a custom GRANDPA voting rule - GRANDPA makes requests of the form (target, minimum) consisting of a target block (i.e. longest chain) that it would like to finalize, and a minimum block which, due to the rules of GRANDPA, must be voted on. The minimum is typically the last finalized block, but may be beyond it, in the case of having a last-round-estimate beyond the last finalized. Thus, our goal is to inform GRANDPA of some block between target and minimum which we believe can be finalized safely. We do this by iterating backwards from the target to the minimum and finding the longest continuous chain from minimum where all candidates included by those blocks have been approved.
......@@ -164,23 +164,23 @@ Main loop:
#### `OverseerSignal::BlockFinalized`
On receiving an `OverseerSignal::BlockFinalized(h)`, we fetch the block number `b` of that block from the ChainApi subsystem. We update our `StoredBlockRange` to begin at `b+1`. Additionally, we remove all block entries and candidates referenced by them up to and including `b`. Lastly, we prune out all descendents of `h` transitively: when we remove a `BlockEntry` with number `b` that is not equal to `h`, we recursively delete all the `BlockEntry`s referenced as children. We remove the `block_assignments` entry for the block hash and if `block_assignments` is now empty, remove the `CandidateEntry`. We also update each of the `BlockNumber -> Vec<Hash>` keys in the database to reflect the blocks at that height, clearing if empty.
On receiving an `OverseerSignal::BlockFinalized(h)`, we fetch the block number `b` of that block from the `ChainApi` subsystem. We update our `StoredBlockRange` to begin at `b+1`. Additionally, we remove all block entries and candidates referenced by them up to and including `b`. Lastly, we prune out all descendants of `h` transitively: when we remove a `BlockEntry` with number `b` that is not equal to `h`, we recursively delete all the `BlockEntry`s referenced as children. We remove the `block_assignments` entry for the block hash and if `block_assignments` is now empty, remove the `CandidateEntry`. We also update each of the `BlockNumber -> Vec<Hash>` keys in the database to reflect the blocks at that height, clearing if empty.
#### `OverseerSignal::ActiveLeavesUpdate`
On receiving an `OverseerSignal::ActiveLeavesUpdate(update)`:
* We determine the set of new blocks that were not in our previous view. This is done by querying the ancestry of all new items in the view and contrasting against the stored `BlockNumber`s. Typically, there will be only one new block. We fetch the headers and information on these blocks from the ChainApi subsystem. Stale leaves in the update can be ignored.
* We determine the set of new blocks that were not in our previous view. This is done by querying the ancestry of all new items in the view and contrasting against the stored `BlockNumber`s. Typically, there will be only one new block. We fetch the headers and information on these blocks from the `ChainApi` subsystem. Stale leaves in the update can be ignored.
* We update the `StoredBlockRange` and the `BlockNumber` maps.
* We use the RuntimeApiSubsystem to determine information about these blocks. It is generally safe to assume that runtime state is available for recent, unfinalized blocks. In the case that it isn't, it means that we are catching up to the head of the chain and needn't worry about assignments to those blocks anyway, as the security assumption of the protocol tolerates nodes being temporarily offline or out-of-date.
* We use the `RuntimeApiSubsystem` to determine information about these blocks. It is generally safe to assume that runtime state is available for recent, unfinalized blocks. In the case that it isn't, it means that we are catching up to the head of the chain and needn't worry about assignments to those blocks anyway, as the security assumption of the protocol tolerates nodes being temporarily offline or out-of-date.
* We fetch the set of candidates included by each block by dispatching a `RuntimeApiRequest::CandidateEvents` and checking the `CandidateIncluded` events.
* We fetch the session of the block by dispatching a `session_index_for_child` request with the parent-hash of the block.
* If the `session index - APPROVAL_SESSIONS > state.earliest_session`, then bump `state.earliest_sessions` to that amount and prune earlier sessions.
* If the session isn't in our `state.session_info`, load the session info for it and for all sessions since the earliest-session, including the earliest-session, if that is missing. And it can be, just after pruning, if we've done a big jump forward, as is the case when we've just finished chain synchronization.
* If any of the runtime API calls fail, we just warn and skip the block.
* We use the RuntimeApiSubsystem to determine the set of candidates included in these blocks and use BABE logic to determine the slot number and VRF of the blocks.
* We use the `RuntimeApiSubsystem` to determine the set of candidates included in these blocks and use BABE logic to determine the slot number and VRF of the blocks.
* We also note how late we appear to have received the block. We create a `BlockEntry` for each block and a `CandidateEntry` for each candidate obtained from `CandidateIncluded` events after making a `RuntimeApiRequest::CandidateEvents` request.
* For each candidate, if the amount of needed approvals is more than the validators remaining after the backing group of the candidate is subtracted, then the candidate is insta-approved as approval would be impossible otherwise. If all candidates in the block are insta-approved, or there are no candidates in the block, then the block is insta-approved. If the block is insta-approved, a [`ChainSelectionMessage::Approvedl][CSM] should be sent for the block.
* For each candidate, if the amount of needed approvals is more than the validators remaining after the backing group of the candidate is subtracted, then the candidate is insta-approved as approval would be impossible otherwise. If all candidates in the block are insta-approved, or there are no candidates in the block, then the block is insta-approved. If the block is insta-approved, a [`ChainSelectionMessage::Approved`][CSM] should be sent for the block.
* Ensure that the `CandidateEntry` contains a `block_assignments` entry for the block, with the correct backing group set.
* If a validator in this session, compute and assign `our_assignment` for the `block_assignments`
* Only if not a member of the backing group.
......@@ -262,25 +262,27 @@ On receiving an `ApprovedAncestor(Hash, BlockNumber, response_channel)`:
* [Schedule a new wakeup](#schedule-wakeup) of the candidate.
#### Schedule Wakeup
* Requires `(approval_entry, candidate_entry)` which effectively denotes a `(Block Hash, Candidate Hash)` pair - the candidate, along with the block it appears in.
* Also requires `RequiredTranches`
* If the `approval_entry` is approved, this doesn't need to be woken up again.
* If `RequiredTranches::All` - no wakeup. We assume other incoming votes will trigger wakeup and potentially re-schedule.
* If `RequiredTranches::Pending { considered, next_no_show, uncovered, maximum_broadcast, clock_drift }` - schedule at the lesser of the next no-show tick, or the tick, offset positively by `clock_drift` of the next non-empty tranche we are aware of after `considered`, including any tranche containing our own unbroadcast assignment. This can lead to no wakeup in the case that we have already broadcast our assignment and there are no pending no-shows; that is, we have approval votes for every assignment we've received that is not already a no-show. In this case, we will be re-triggered by other validators broadcasting their assignments.
* If `RequiredTranches::Exact { next_no_show, .. } - set a wakeup for the next no-show tick.
* If `RequiredTranches::Exact { next_no_show, .. }` - set a wakeup for the next no-show tick.
#### Launch Approval Work
* Requires `(SessionIndex, SessionInfo, CandidateReceipt, ValidatorIndex, backing_group, block_hash, candidate_index)`
* Extract the public key of the `ValidatorIndex` from the `SessionInfo` for the session.
* Issue an `AvailabilityRecoveryMessage::RecoverAvailableData(candidate, session_index, Some(backing_group), response_sender)`
* Load the historical validation code of the parachain by dispatching a `RuntimeApiRequest::ValidationCodeByHash(`descriptor.validation_code_hash`)` against the state of `block_hash`.
* Spawn a background task with a clone of `background_tx`
* Wait for the available data
* Issue a `CandidateValidationMessage::ValidateFromExhaustive` message
* Wait for the result of validation
* Check that the result of validation, if valid, matches the commitments in the receipt.
* If valid, issue a message on `background_tx` detailing the request.
* If any of the data, the candidate, or the commitments are invalid, issue on `background_tx` a [`DisputeCoordinatorMessage::IssueLocalStatement`](../../types/overseer-protocol.md#dispute-coordinator-message) with `valid = false` to initiate a dispute.
* Requires `(SessionIndex, SessionInfo, CandidateReceipt, ValidatorIndex, backing_group, block_hash, candidate_index)`
* Extract the public key of the `ValidatorIndex` from the `SessionInfo` for the session.
* Issue an `AvailabilityRecoveryMessage::RecoverAvailableData(candidate, session_index, Some(backing_group), response_sender)`
* Load the historical validation code of the parachain by dispatching a `RuntimeApiRequest::ValidationCodeByHash(descriptor.validation_code_hash)` against the state of `block_hash`.
* Spawn a background task with a clone of `background_tx`
* Wait for the available data
* Issue a `CandidateValidationMessage::ValidateFromExhaustive` message
* Wait for the result of validation
* Check that the result of validation, if valid, matches the commitments in the receipt.
* If valid, issue a message on `background_tx` detailing the request.
* If any of the data, the candidate, or the commitments are invalid, issue on `background_tx` a [`DisputeCoordinatorMessage::IssueLocalStatement`](../../types/overseer-protocol.md#dispute-coordinator-message) with `valid = false` to initiate a dispute.
#### Issue Approval Vote
* Fetch the block entry and candidate entry. Ignore if `None` - we've probably just lost a race with finality.
......
......@@ -81,13 +81,14 @@ dispute participation subsystem.
### On `OverseerSignal::ActiveLeavesUpdate`
For each leaf in the leaves update:
* Fetch the session index for the child of the block with a [`RuntimeApiMessage::SessionIndexForChild`][RuntimeApiMessage].
* If the session index is higher than `state.highest_session`:
* update `state.highest_session`
* remove everything with session index less than `state.highest_session - DISPUTE_WINDOW` from the `"recent-disputes"` in the DB.
* Use `iter_with_prefix` to remove everything from `"earliest-session"` up to `state.highest_session - DISPUTE_WINDOW` from the DB under `"candidate-votes"`.
* Update `"earliest-session"` to be equal to `state.highest_session - DISPUTE_WINDOW`.
* For each new block, explicitly or implicitly, under the new leaf, scan for a dispute digest which indicates a rollback. If a rollback is detected, use the ChainApi subsystem to blacklist the chain.
* Fetch the session index for the child of the block with a [`RuntimeApiMessage::SessionIndexForChild`][RuntimeApiMessage].
* If the session index is higher than `state.highest_session`:
* update `state.highest_session`
* remove everything with session index less than `state.highest_session - DISPUTE_WINDOW` from the `"recent-disputes"` in the DB.
* Use `iter_with_prefix` to remove everything from `"earliest-session"` up to `state.highest_session - DISPUTE_WINDOW` from the DB under `"candidate-votes"`.
* Update `"earliest-session"` to be equal to `state.highest_session - DISPUTE_WINDOW`.
* For each new block, explicitly or implicitly, under the new leaf, scan for a dispute digest which indicates a rollback. If a rollback is detected, use the `ChainApi` subsystem to blacklist the chain.
### On `OverseerSignal::Conclude`
......@@ -144,7 +145,7 @@ Do nothing.
### On `DisputeCoordinatorMessage::QueryCandidateVotes`
* Load `"candidate-votes"` for every `(SessionIndex, CandidateHash)` in the query and return data within each `CandidateVote`.
If a particular `candidate-vote` is missing, that particular request is ommitted from the response.
If a particular `candidate-vote` is missing, that particular request is omitted from the response.
### On `DisputeCoordinatorMessage::IssueLocalStatement`
......
......@@ -21,33 +21,34 @@ There may be multiple competing blocks all ending the availability phase for a p
```dot process
digraph {
label = "Block data FSM\n\n\n";
labelloc = "t";
rankdir="LR";
st [label = "Stored"; shape = circle]
inc [label = "Included"; shape = circle]
fin [label = "Finalized"; shape = circle]
prn [label = "Pruned"; shape = circle]
st -> inc [label = "Block\nincluded"]
st -> prn [label = "Stored block\ntimed out"]
inc -> fin [label = "Block\nfinalized"]
inc -> st [label = "Competing blocks\nfinalized"]
fin -> prn [label = "Block keep time\n(1 day + 1 hour) elapsed"]
label = "Block data FSM\n\n\n";
labelloc = "t";
rankdir="LR";
st [label = "Stored"; shape = circle]
inc [label = "Included"; shape = circle]
fin [label = "Finalized"; shape = circle]
prn [label = "Pruned"; shape = circle]
st -> inc [label = "Block\nincluded"]
st -> prn [label = "Stored block\ntimed out"]
inc -> fin [label = "Block\nfinalized"]
inc -> st [label = "Competing blocks\nfinalized"]
fin -> prn [label = "Block keep time\n(1 day + 1 hour) elapsed"]
}
```
## Database Schema
We use an underlying Key-Value database where we assume we have the following operations available:
* `write(key, value)`
* `read(key) -> Option<value>`
* `iter_with_prefix(prefix) -> Iterator<(key, value)>` - gives all keys and values in lexicographical order where the key starts with `prefix`.
- `write(key, value)`
- `read(key) -> Option<value>`
- `iter_with_prefix(prefix) -> Iterator<(key, value)>` - gives all keys and values in lexicographical order where the key starts with `prefix`.
We use this database to encode the following schema:
```
```rust
("available", CandidateHash) -> Option<AvailableData>
("chunk", CandidateHash, u32) -> Option<ErasureChunk>
("meta", CandidateHash) -> Option<CandidateMeta>
......@@ -56,7 +57,7 @@ We use this database to encode the following schema:
("prune_by_time", Timestamp, CandidateHash) -> Option<()>
```
Timestamps are the wall-clock seconds since unix epoch. Timestamps and block numbers are both encoded as big-endian so lexicographic order is ascending.
Timestamps are the wall-clock seconds since Unix epoch. Timestamps and block numbers are both encoded as big-endian so lexicographic order is ascending.
The meta information that we track per-candidate is defined as the `CandidateMeta` struct
......@@ -88,84 +89,87 @@ Additionally, there is exactly one `prune_by_time` entry which holds the candida
Input: [`AvailabilityStoreMessage`][ASM]
Output:
- [`RuntimeApiMessage`][RAM]
- [`RuntimeApiMessage`][RAM]
## Functionality
For each head in the `activated` list:
- Load all ancestors of the head back to the finalized block so we don't miss anything if import notifications are missed. If a `StoreChunk` message is received for a candidate which has no entry, then we will prematurely lose the data.
- Note any new candidates backed in the head. Update the `CandidateMeta` for each. If the `CandidateMeta` does not exist, create it as `Unavailable` with the current timestamp. Register a `"prune_by_time"` entry based on the current timestamp + 1 hour.
- Note any new candidate included in the head. Update the `CandidateMeta` for each, performing a transition from `Unavailable` to `Unfinalized` if necessary. That includes removing the `"prune_by_time"` entry. Add the head hash and number to the state, if unfinalized. Add an `"unfinalized"` entry for the block and candidate.
- The `CandidateEvent` runtime API can be used for this purpose.
- Load all ancestors of the head back to the finalized block so we don't miss anything if import notifications are missed. If a `StoreChunk` message is received for a candidate which has no entry, then we will prematurely lose the data.
- Note any new candidates backed in the head. Update the `CandidateMeta` for each. If the `CandidateMeta` does not exist, create it as `Unavailable` with the current timestamp. Register a `"prune_by_time"` entry based on the current timestamp + 1 hour.
- Note any new candidate included in the head. Update the `CandidateMeta` for each, performing a transition from `Unavailable` to `Unfinalized` if necessary. That includes removing the `"prune_by_time"` entry. Add the head hash and number to the state, if unfinalized. Add an `"unfinalized"` entry for the block and candidate.
- The `CandidateEvent` runtime API can be used for this purpose.
On `OverseerSignal::BlockFinalized(finalized)` events:
- for each key in `iter_with_prefix("unfinalized")`
- Stop if the key is beyond `("unfinalized, finalized)`
- For each block number f that we encounter, load the finalized hash for that block.
- The state of each `CandidateMeta` we encounter here must be `Unfinalized`, since we loaded the candidate from an `"unfinalized"` key.
- For each candidate that we encounter under `f` and the finalized block hash,
- Update the `CandidateMeta` to have `State::Finalized`. Remove all `"unfinalized"` entries from the old `Unfinalized` state.
- Register a `"prune_by_time"` entry for the candidate based on the current time + 1 day + 1 hour.
- For each candidate that we encounter under `f` which is not under the finalized block hash,
- Remove all entries under `f` in the `Unfinalized` state.
- If the `CandidateMeta` has state `Unfinalized` with an empty list of blocks, downgrade to `Unavailable` and re-schedule pruning under the timestamp + 1 hour. We do not prune here as the candidate still may be included in a descendent of the finalized chain.
- Remove all `"unfinalized"` keys under `f`.
- Update last_finalized = finalized.
- for each key in `iter_with_prefix("unfinalized")`
- Stop if the key is beyond `("unfinalized, finalized)`
- For each block number f that we encounter, load the finalized hash for that block.
- The state of each `CandidateMeta` we encounter here must be `Unfinalized`, since we loaded the candidate from an `"unfinalized"` key.
- For each candidate that we encounter under `f` and the finalized block hash,
- Update the `CandidateMeta` to have `State::Finalized`. Remove all `"unfinalized"` entries from the old `Unfinalized` state.
- Register a `"prune_by_time"` entry for the candidate based on the current time + 1 day + 1 hour.
- For each candidate that we encounter under `f` which is not under the finalized block hash,
- Remove all entries under `f` in the `Unfinalized` state.
- If the `CandidateMeta` has state `Unfinalized` with an empty list of blocks, downgrade to `Unavailable` and re-schedule pruning under the timestamp + 1 hour. We do not prune here as the candidate still may be included in a descendant of the finalized chain.
- Remove all `"unfinalized"` keys under `f`.
- Update `last_finalized` = finalized.
This is roughly `O(n * m)` where n is the number of blocks finalized since the last update, and `m` is the number of parachains.
On `QueryAvailableData` message:
- Query `("available", candidate_hash)`
- Query `("available", candidate_hash)`
This is `O(n)` in the size of the data, which may be large.
On `QueryDataAvailability` message:
- Query whether `("meta", candidate_hash)` exists and `data_available == true`.
- Query whether `("meta", candidate_hash)` exists and `data_available == true`.
This is `O(n)` in the size of the metadata which is small.
On `QueryChunk` message:
- Query `("chunk", candidate_hash, index)`
- Query `("chunk", candidate_hash, index)`
This is `O(n)` in the size of the data, which may be large.
On `QueryAllChunks` message:
- Query `("meta", candidate_hash)`. If `None`, send an empty response and return.
- For all `1` bits in the `chunks_stored`, query `("chunk", candidate_hash, index)`. Ignore but warn on errors, and return a vector of all loaded chunks.
On `QueryChunkAvailability message:
- Query `("meta", candidate_hash)`. If `None`, send an empty response and return.
- For all `1` bits in the `chunks_stored`, query `("chunk", candidate_hash, index)`. Ignore but warn on errors, and return a vector of all loaded chunks.
On `QueryChunkAvailability` message:
- Query whether `("meta", candidate_hash)` exists and the bit at `index` is set.
- Query whether `("meta", candidate_hash)` exists and the bit at `index` is set.
This is `O(n)` in the size of the metadata which is small.
On `StoreChunk` message:
- If there is a `CandidateMeta` under the candidate hash, set the bit of the erasure-chunk in the `chunks_stored` bitfield to `1`. If it was not `1` already, write the chunk under `("chunk", candidate_hash, chunk_index)`.
- If there is a `CandidateMeta` under the candidate hash, set the bit of the erasure-chunk in the `chunks_stored` bitfield to `1`. If it was not `1` already, write the chunk under `("chunk", candidate_hash, chunk_index)`.
This is `O(n)` in the size of the chunk.
On `StoreAvailableData` message:
- If there is no `CandidateMeta` under the candidate hash, create it with `State::Unavailable(now)`. Load the `CandidateMeta` otherwise.
- Store `data` under `("available", candidate_hash)` and set `data_available` to true.
- Store each chunk under `("chunk", candidate_hash, index)` and set every bit in `chunks_stored` to `1`.
- If there is no `CandidateMeta` under the candidate hash, create it with `State::Unavailable(now)`. Load the `CandidateMeta` otherwise.
- Store `data` under `("available", candidate_hash)` and set `data_available` to true.
- Store each chunk under `("chunk", candidate_hash, index)` and set every bit in `chunks_stored` to `1`.
This is `O(n)` in the size of the data as the aggregate size of the chunks is proportional to the data.
Every 5 minutes, run a pruning routine:
- for each key in `iter_with_prefix("prune_by_time")`:
- If the key is beyond ("prune_by_time", now), return.
- Remove the key.
- Extract `candidate_hash` from the key.
- Load and remove the `("meta", candidate_hash)`
- For each erasure chunk bit set, remove `("chunk", candidate_hash, bit_index)`.
- If `data_available`, remove `("available", candidate_hash)
- for each key in `iter_with_prefix("prune_by_time")`:
- If the key is beyond `("prune_by_time", now)`, return.
- Remove the key.
- Extract `candidate_hash` from the key.
- Load and remove the `("meta", candidate_hash)`
- For each erasure chunk bit set, remove `("chunk", candidate_hash, bit_index)`.
- If `data_available`, remove `("available", candidate_hash)`
This is O(n * m) in the amount of candidates and average size of the data stored. This is probably the most expensive operation but does not need
to be run very often.
......@@ -193,7 +197,7 @@ Basically we need to test the correctness of data flow through state FSMs descri
- Wait until the data should have been pruned.
- The data is no longer available.
- Forkfulness of the relay chain is taken into account
- Fork-awareness of the relay chain is taken into account
- Block `B1` is added to the store.
- Block `B2` is added to the store.
- Notify the subsystem that both `B1` and `B2` were included in different leafs of relay chain.
......
......@@ -34,7 +34,7 @@ Note that the candidate can fail to be included in any of the following ways:
This process can be divided further down. Steps 2 & 3 relate to the work of the collator in collating and distributing the candidate to validators via the Collation Distribution Subsystem. Steps 3 & 4 relate to the work of the validators in the Candidate Backing Subsystem and the block author (itself a validator) to include the block into the relay chain. Steps 6, 7, and 8 correspond to the logic of the relay-chain state-machine (otherwise known as the Runtime) used to fully incorporate the block into the chain. Step 7 requires further work on the validators' parts to participate in the Availability Distribution Subsystem and include that information into the relay chain for step 8 to be fully realized.
This brings us to the second part of the process. Once a parablock is considered available and part of the parachain, it is still "pending approval". At this stage in the pipeline, the parablock has been backed by a majority of validators in the group assigned to that parachain, and its data has been guaranteed available by the set of validators as a whole. Once it's considered available, the host will even begin to accept children of that block. At this point, we can consider the parablock as having been tentatively included in the parachain, although more confirmations are desired. However, the validators in the parachain-group (known as the "Parachain Validators" for that parachain) are sampled from a validator set which contains some proportion of byzantine, or arbitrarily malicious members. This implies that the Parachain Validators for some parachain may be majority-dishonest, which means that (secondary) approval checks must be done on the block before it can be considered approved. This is necessary only because the Parachain Validators for a given parachain are sampled from an overall validator set which is assumed to be up to <1/3 dishonest - meaning that there is a chance to randomly sample Parachain Validators for a parachain that are majority or fully dishonest and can back a candidate wrongly. The Approval Process allows us to detect such misbehavior after-the-fact without allocating more Parachain Validators and reducing the throughput of the system. A parablock's failure to pass the approval process will invalidate the block as well as all of its descendents. However, only the validators who backed the block in question will be slashed, not the validators who backed the descendents.
This brings us to the second part of the process. Once a parablock is considered available and part of the parachain, it is still "pending approval". At this stage in the pipeline, the parablock has been backed by a majority of validators in the group assigned to that parachain, and its data has been guaranteed available by the set of validators as a whole. Once it's considered available, the host will even begin to accept children of that block. At this point, we can consider the parablock as having been tentatively included in the parachain, although more confirmations are desired. However, the validators in the parachain-group (known as the "Parachain Validators" for that parachain) are sampled from a validator set which contains some proportion of byzantine, or arbitrarily malicious members. This implies that the Parachain Validators for some parachain may be majority-dishonest, which means that (secondary) approval checks must be done on the block before it can be considered approved. This is necessary only because the Parachain Validators for a given parachain are sampled from an overall validator set which is assumed to be up to <1/3 dishonest - meaning that there is a chance to randomly sample Parachain Validators for a parachain that are majority or fully dishonest and can back a candidate wrongly. The Approval Process allows us to detect such misbehavior after-the-fact without allocating more Parachain Validators and reducing the throughput of the system. A parablock's failure to pass the approval process will invalidate the block as well as all of its descendants. However, only the validators who backed the block in question will be slashed, not the validators who backed the descendants.
The Approval Process, at a glance, looks like this:
......@@ -170,7 +170,7 @@ digraph {
}
```
In this example, group 1 has received block C while the others have not due to network asynchrony. Now, a validator from group 2 may be able to build another block on top of B, called C'. Assume that afterwards, some validators become aware of both C and C', while others remain only aware of one.
In this example, group 1 has received block C while the others have not due to network asynchrony. Now, a validator from group 2 may be able to build another block on top of B, called `C'`. Assume that afterwards, some validators become aware of both C and `C'`, while others remain only aware of one.
```dot process
digraph {
......
......@@ -4,7 +4,7 @@ Runtime APIs are the means by which the node-side code extracts information from
Every block in the relay-chain contains a *state root* which is the root hash of a state trie encapsulating all storage of runtime modules after execution of the block. This is a cryptographic commitment to a unique state. We use the terminology of accessing the *state at* a block to refer accessing the state referred to by the state root of that block.
Although Runtime APIs are often used for simple storage access, they are actually empowered to do arbitrary computation. The implementation of the Runtime APIs lives within the Runtime as Wasm code and exposes extern functions that can be invoked with arguments and have a return value. Runtime APIs have access to a variety of host functions, which are contextual functions provided by the Wasm execution context, that allow it to carry out many different types of behaviors.
Although Runtime APIs are often used for simple storage access, they are actually empowered to do arbitrary computation. The implementation of the Runtime APIs lives within the Runtime as Wasm code and exposes `extern` functions that can be invoked with arguments and have a return value. Runtime APIs have access to a variety of host functions, which are contextual functions provided by the Wasm execution context, that allow it to carry out many different types of behaviors.
Abilities provided by host functions includes:
......
......@@ -21,9 +21,9 @@ We will split the logic of the runtime up into these modules:
* Scheduler: manages parachain and parathread scheduling as well as validator assignments.
* Inclusion: handles the inclusion and availability of scheduled parachains and parathreads.
* Validity: handles secondary checks and dispute resolution for included, available parablocks.
* Hrmp: handles horizontal messages between paras.
* Ump: Handles upward messages from a para to the relay chain.
* Dmp: Handles downward messages from the relay chain to the para.
* HRMP: handles horizontal messages between paras.
* UMP: Handles upward messages from a para to the relay chain.
* DMP: Handles downward messages from the relay chain to the para.
The [Initializer module](initializer.md) is special - it's responsible for handling the initialization logic of the other modules to ensure that the correct initialization order and related invariants are maintained. The other modules won't specify a on-initialize logic, but will instead expose a special semi-private routine that the initialization module will call. The other modules are relatively straightforward and perform the roles described above.
......@@ -31,7 +31,7 @@ The Parachain Host operates under a changing set of validators. Time is split up
The relay chain is intended to use BABE or SASSAFRAS, which both have the property that a session changing at a block is determined not by the number of the block but instead by the time the block is authored. In some sense, sessions change in-between blocks, not at blocks. This has the side effect that the session of a child block cannot be determined solely by the parent block's identifier. Being able to unilaterally determine the validator-set at a specific block based on its parent hash would make a lot of Node-side logic much simpler.
In order to regain the property that the validator set of a block is predictable by its parent block, we delay session changes' application to Parachains by 1 block. This means that if there is a session change at block X, that session change will be stored and applied during initialization of direct descendents of X. This principal side effect of this change is that the Parachains runtime can disagree with session or consensus modules about which session it currently is. Misbehavior reporting routines in particular will be affected by this, although not severely. The parachains runtime might believe it is the last block of the session while the system is really in the first block of the next session. In such cases, a historical validator-set membership proof will need to accompany any misbehavior report, although they typically do not need to during current-session misbehavior reports.
In order to regain the property that the validator set of a block is predictable by its parent block, we delay session changes' application to Parachains by 1 block. This means that if there is a session change at block X, that session change will be stored and applied during initialization of direct descendants of X. This principal side effect of this change is that the Parachains runtime can disagree with session or consensus modules about which session it currently is. Misbehavior reporting routines in particular will be affected by this, although not severely. The parachains runtime might believe it is the last block of the session while the system is really in the first block of the next session. In such cases, a historical validator-set membership proof will need to accompany any misbehavior report, although they typically do not need to during current-session misbehavior reports.
So the other role of the initializer module is to forward session change notifications to modules in the initialization order. Session change is also the point at which the [Configuration Module](configuration.md) updates the configuration. Most of the other modules will handle changes in the configuration during their session change operation, so the initializer should provide both the old and new configuration to all the other
modules alongside the session change notification. This means that a session change notification should consist of the following data:
......
......@@ -78,12 +78,12 @@ struct CandidateDescriptor {
/// derived from relay-chain state that influence the validity of the block which
/// must also be kept available for secondary checkers.
persisted_validation_data_hash: Hash,
/// The blake2-256 hash of the pov-block.
/// The blake2-256 hash of the `pov-block`.
pov_hash: Hash,
/// The root of a block's erasure encoding Merkle tree.
erasure_root: Hash,
/// Signature on blake2-256 of components of this receipt:
/// The parachain index, the relay parent, the validation data hash, and the pov_hash.
/// The parachain index, the relay parent, the validation data hash, and the `pov_hash`.
signature: CollatorSignature,
/// Hash of the para header that is being generated by this candidate.
para_head: Hash,
......@@ -92,7 +92,7 @@ struct CandidateDescriptor {
}
```
## PersistedValidationData
## `PersistedValidationData`
The validation data provides information about how to create the inputs for validation of a candidate. This information is derived from the chain state and will vary from para to para, although some of the fields may be the same for every para.
......@@ -102,7 +102,7 @@ Furthermore, the validation data acts as a way to authorize the additional data
Since the commitments of the validation function are checked by the relay-chain, secondary checkers can rely on the invariant that the relay-chain only includes para-blocks for which these checks have already been done. As such, there is no need for the validation data used to inform validators and collators about the checks the relay-chain will perform to be persisted by the availability system.
The `PersistedValidationData` should be relatively lightweight primarly because it is constructed during inclusion for each candidate and therefore lies on the critical path of inclusion.
The `PersistedValidationData` should be relatively lightweight primarily because it is constructed during inclusion for each candidate and therefore lies on the critical path of inclusion.
```rust
struct PersistedValidationData {
......@@ -124,7 +124,7 @@ struct PersistedValidationData {
}
```
## HeadData
## `HeadData`
Head data is a type-safe abstraction around bytes (`Vec<u8>`) for the purposes of representing heads of parachains or parathreads.
......
......@@ -81,7 +81,7 @@ enum PoVDistributionV1Message {
/// specific relay-parent hash.
Awaiting(Hash, Vec<Hash>),
/// Notification of an awaited PoV, in a given relay-parent context.
/// (relay_parent, pov_hash, pov)
/// (`relay_parent`, `pov_hash`, `pov`)
SendPoV(Hash, Hash, PoV),
}
```
......
......@@ -14,7 +14,7 @@
// You should have received a copy of the GNU General Public License
// along with Polkadot. If not, see <http://www.gnu.org/licenses/>.
//! A helper macro for generating SlotRange enum.
//! A helper macro for generating `SlotRange` enum.
#![cfg_attr(not(feature = "std"), no_std)]
......
......@@ -89,7 +89,7 @@ pub mod pallet {
/// The length of each sample to take during the ending period.
///
/// EndingPeriod / SampleLength = Total # of Samples
/// `EndingPeriod` / `SampleLength` = Total # of Samples
#[pallet::constant]
type SampleLength: Get<Self::BlockNumber>;
......@@ -114,24 +114,24 @@ pub mod pallet {
pub enum Event<T: Config> {
/// An auction started. Provides its index and the block number where it will begin to
/// close and the first lease period of the quadruplet that is auctioned.
/// [auction_index, lease_period, ending]
/// `[auction_index, lease_period, ending]`
AuctionStarted(AuctionIndex, LeasePeriodOf<T>, T::BlockNumber),
/// An auction ended. All funds become unreserved. [auction_index]
/// An auction ended. All funds become unreserved. `[auction_index]`
AuctionClosed(AuctionIndex),
/// Funds were reserved for a winning bid. First balance is the extra amount reserved.
/// Second is the total. [bidder, extra_reserved, total_amount]
/// Second is the total. `[bidder, extra_reserved, total_amount]`
Reserved(T::AccountId, BalanceOf<T>, BalanceOf<T>),
/// Funds were unreserved since bidder is no longer active. [bidder, amount]
/// Funds were unreserved since bidder is no longer active. `[bidder, amount]`
Unreserved(T::AccountId, BalanceOf<T>),
/// Someone attempted to lease the same slot twice for a parachain. The amount is held in reserve
/// but no parachain slot has been leased.
/// \[parachain_id, leaser, amount\]
/// `[parachain_id, leaser, amount]`
ReserveConfiscated(ParaId, T::AccountId, BalanceOf<T>),
/// A new bid has been accepted as the current winner.
/// \[who, para_id, amount, first_slot, last_slot\]
/// `[who, para_id, amount, first_slot, last_slot]`
BidAccepted(T::AccountId, ParaId, BalanceOf<T>, LeasePeriodOf<T>, LeasePeriodOf<T>),
/// The winning offset was chosen for an auction. This will map into the `Winning` storage map.
/// \[auction_index, block_number\]
/// `[auction_index, block_number]`
WinningOffset(AuctionIndex, T::BlockNumber),
}
......@@ -565,7 +565,7 @@ impl<T: Config> Pallet<T> {
/// Calculate the final winners from the winning slots.
///
/// This is a simple dynamic programming algorithm designed by Al, the original code is at:
/// https://github.com/w3f/consensus/blob/master/NPoS/auctiondynamicthing.py
/// `https://github.com/w3f/consensus/blob/master/NPoS/auctiondynamicthing.py`
fn calculate_winners(
mut winning: WinningData<T>
) -> WinnersData<T> {
......
......@@ -157,7 +157,7 @@ pub mod pallet {
#[pallet::generate_deposit(pub(super) fn deposit_event)]
#[pallet::metadata(T::AccountId = "AccountId", BalanceOf<T> = "Balance")]
pub enum Event<T: Config> {
/// Someone claimed some DOTs. [who, ethereum_address, amount]
/// Someone claimed some DOTs. `[who, ethereum_address, amount]`
Claimed(T::AccountId, EthereumAddress, BalanceOf<T>),
}
......@@ -167,7 +167,7 @@ pub mod pallet {
InvalidEthereumSignature,
/// Ethereum address has no claim.
SignerHasNoClaim,
/// Account ID sending tx has no claim.
/// Account ID sending transaction has no claim.
SenderHasNoClaim,
/// There's not enough in the pot to pay out some unvested amount. Generally implies a logic
/// error.
......
......@@ -14,7 +14,7 @@
// You should have received a copy of the GNU General Public License
// along with Polkadot. If not, see <http://www.gnu.org/licenses/>.
//! # Parachain Crowdloaning pallet
//! # Parachain `Crowdloaning` pallet
//!
//! The point of this pallet is to allow parachain projects to offer the ability to help fund a
//! deposit for the parachain. When the crowdloan has ended, the funds are returned.
......@@ -136,11 +136,11 @@ pub struct FundInfo<AccountId, Balance, BlockNumber, LeasePeriod> {
/// If this is `Ending(n)`, this fund received a contribution during the current ending period,
/// where `n` is how far into the ending period the contribution was made.
last_contribution: LastContribution<BlockNumber>,
/// First lease period in range to bid on; it's actually a LeasePeriod, but that's the same type
/// as BlockNumber.
/// First lease period in range to bid on; it's actually a `LeasePeriod`, but that's the same type
/// as `BlockNumber`.
first_period: LeasePeriod,
/// Last lease period in range to bid on; it's actually a LeasePeriod, but that's the same type
/// as BlockNumber.
/// Last lease period in range to bid on; it's actually a `LeasePeriod`, but that's the same type
/// as `BlockNumber`.
last_period: LeasePeriod,
/// Index used for the child trie of this fund
trie_index: TrieIndex,
......@@ -160,7 +160,7 @@ pub mod pallet {
pub trait Config: frame_system::Config {
type Event: From<Event<Self>> + IsType<<Self as frame_system::Config>::Event>;
/// PalletId for the crowdloan pallet. An appropriate value could be ```PalletId(*b"py/cfund")```
/// `PalletId` for the crowdloan pallet. An appropriate value could be `PalletId(*b"py/cfund")`
#[pallet::constant]
type PalletId: Get<PalletId>;
......@@ -168,7 +168,7 @@ pub mod pallet {
type SubmissionDeposit: Get<BalanceOf<Self>>;
/// The minimum amount that may be contributed into a crowdloan. Should almost certainly be at
/// least ExistentialDeposit.
/// least `ExistentialDeposit`.
#[pallet::constant]
type MinContribution: Get<BalanceOf<Self>>;
......@@ -176,7 +176,7 @@ pub mod pallet {
#[pallet::constant]
type RemoveKeysLimit: Get<u32>;
/// The parachain registrar type. We jus use this to ensure that only the manager of a para is able to
/// The parachain registrar type. We just use this to ensure that only the manager of a para is able to
/// start a crowdloan for its slot.
type Registrar: Registrar<AccountId=Self::AccountId>;
......@@ -223,26 +223,26 @@ pub mod pallet {
#[pallet::generate_deposit(pub(super) fn deposit_event)]
#[pallet::metadata(T::AccountId = "AccountId", BalanceOf<T> = "Balance")]
pub enum Event<T: Config> {
/// Create a new crowdloaning campaign. [fund_index]
/// Create a new crowdloaning campaign. `[fund_index]`
Created(ParaId),
/// Contributed to a crowd sale. [who, fund_index, amount]
/// Contributed to a crowd sale. `[who, fund_index, amount]`
Contributed(T::AccountId, ParaId, BalanceOf<T>),
/// Withdrew full balance of a contributor. [who, fund_index, amount]
/// Withdrew full balance of a contributor. `[who, fund_index, amount]`
Withdrew(T::AccountId, ParaId, BalanceOf<T>),
/// The loans in a fund have been partially dissolved, i.e. there are some left
/// over child keys that still need to be killed. [fund_index]
/// over child keys that still need to be killed. `[fund_index]`
PartiallyRefunded(ParaId),
/// All loans in a fund have been refunded. [fund_index]
/// All loans in a fund have been refunded. `[fund_index]`
AllRefunded(ParaId),
/// Fund is dissolved. [fund_index]
/// Fund is dissolved. `[fund_index]`
Dissolved(ParaId),
/// The result of trying to submit a new bid to the Slots pallet.
HandleBidResult(ParaId, DispatchResult),
/// The configuration to a crowdloan has been edited. [fund_index]
/// The configuration to a crowdloan has been edited. `[fund_index]`
Edited(ParaId),
/// A memo has been updated. [who, fund_index, memo]