Candidate Validation Subsystem (#1432)

* skeleton for candidate-validation

* add to workspace

* implement candidate validation logic

* guide: note occupied-core assumption for candidate validation

* adjust message doc

* wire together `run` asynchronously

* add a Subsystem implementation

* clean up a couple warnings

* fix compilation errors due to merge

* improve candidate-validation.md

* remove old reference to subsystem-test helpers crate

* update Cargo.lock

* add a couple new Runtime API methods

* add a candidate validation message

* fetch validation data from the chain state

* some tests for assumption checking

* make spawn_validate_exhaustive mockable

* more tests on the error handling side

* fix all other grumbles except for wasm validation API change

* wrap a SpawnNamed in candidate-validation

* warn

* amend guide

* squanch warning

* remove duplicate after merge

polkadot/Cargo.lock

@@ -4635,6 +4635,24 @@ dependencies = [
  "wasm-timer",
 ]
 
+[[package]]
+name = "polkadot-node-core-candidate-validation"
+version = "0.1.0"
+dependencies = [
+ "assert_matches",
+ "derive_more 0.99.9",
+ "futures 0.3.5",
+ "log 0.4.8",
+ "parity-scale-codec",
+ "polkadot-node-primitives",
+ "polkadot-node-subsystem",
+ "polkadot-parachain",
+ "polkadot-primitives",
+ "sp-blockchain",
+ "sp-core",
+ "sp-keyring",
+]
+
 [[package]]
 name = "polkadot-node-core-proposer"
 version = "0.1.0"

polkadot/Cargo.toml

@@ -44,7 +44,9 @@ members = [
 	"validation",
 
 	"node/core/av-store",
+	"node/core/backing",
 	"node/core/bitfield-signing",
+	"node/core/candidate-validation",
 	"node/core/proposer",
 	"node/core/runtime-api",
 	"node/network/bridge",
@@ -54,7 +56,6 @@ members = [
 	"node/overseer",
 	"node/primitives",
 	"node/service",
-	"node/core/backing",
 	"node/subsystem",
 	"node/test-service",
 

polkadot/node/core/candidate-validation/Cargo.toml

+[package]
+name = "polkadot-node-core-candidate-validation"
+version = "0.1.0"
+authors = ["Parity Technologies <admin@parity.io>"]
+edition = "2018"
+
+[dependencies]
+futures = "0.3.5"
+sp-blockchain = { git = "https://github.com/paritytech/substrate", branch = "master" }
+sp-core = { package = "sp-core", git = "https://github.com/paritytech/substrate", branch = "master" }
+parity-scale-codec = { version = "1.3.0", default-features = false, features = ["bit-vec", "derive"] }
+
+polkadot-primitives = { path = "../../../primitives" }
+polkadot-parachain = { path = "../../../parachain" }
+polkadot-node-primitives = { path = "../../primitives" }
+polkadot-subsystem = { package = "polkadot-node-subsystem", path = "../../subsystem" }
+derive_more = "0.99.9"
+log = "0.4.8"
+
+[dev-dependencies]
+sp-keyring = { git = "https://github.com/paritytech/substrate", branch = "master" }
+futures = { version = "0.3.5", features = ["thread-pool"] }
+assert_matches = "1.3.0"
+polkadot-subsystem = { package = "polkadot-node-subsystem", path = "../../subsystem", features = ["test-helpers"] }

polkadot/node/subsystem/src/messages.rs

@@ -105,7 +105,9 @@ pub enum CandidateValidationMessage {
 	/// This will implicitly attempt to gather the `OmittedValidationData` and `ValidationCode`
 	/// from the runtime API of the chain, based on the `relay_parent`
 	/// of the `CandidateDescriptor`.
-	/// If there is no state available which can provide this data, an error is returned.
+	///
+	/// If there is no state available which can provide this data or the core for
+	/// the para is not free at the relay-parent, an error is returned.
 	ValidateFromChainState(
 		CandidateDescriptor,
 		Arc<PoV>,

polkadot/roadmap/implementers-guide/src/node/utility/candidate-validation.md

@@ -12,12 +12,38 @@ Output: Validation result via the provided response side-channel.
 
 ## Functionality
 
-Given the hashes of a relay parent and a parachain candidate block, and either its PoV or the information with which to retrieve the PoV from the network, spawn a short-lived async job to determine whether the candidate is valid.
+This subsystem answers two types of requests: one which draws out validation data from the state, and another which accepts all validation data exhaustively. The goal of both request types is to validate a candidate. There are three possible outputs of validation: either the candidate is valid, the candidate is invalid, or an internal error occurred. Whatever the end result is, it will be returned on the response channel to the requestor.
 
-Each job follows this process:
+Parachain candidates are validated against their validation function: A piece of Wasm code that is describes the state-transition of the parachain. Validation function execution is not metered. This means that an execution which is an infinite loop or simply takes too long must be forcibly exited by some other means. For this reason, we recommend dispatching candidate validation to be done on subprocesses which can be killed if they time-out.
 
-- Get the full candidate from the current relay chain state
-- Check the candidate's proof
-   > TODO: that's extremely hand-wavey. What does that actually entail?
-- Generate either `Statement::Valid` or `Statement::Invalid`. Note that this never generates `Statement::Seconded`; Candidate Backing is the only subsystem which upgrades valid to seconded.
-- Return the statement on the provided channel.
+Upon receiving a validation request, the first thing the candidate validation subsystem should do is make sure it has all the necessary parameters to the validation function. These are:
+  * The Validation Function itself.
+  * The [`CandidateDescriptor`](../../types/candidate.md#candidatedescriptor).
+  * The [`LocalValidationData`](../../types/candidate.md#localvalidationdata).
+  * The [`GlobalValidationSchedule](../../types/candidate.md#globalvalidationschedule).
+  * The [`PoV`](../../types/availability.md#proofofvalidity).
+
+### Determining Parameters
+
+For a [`CandidateValidationMessage`][CVM]`::ValidateFromExhaustive`, these parameters are exhaustively provided. The [`OmittedValidationData`](../../types/availability.md#omittedvalidationdata) can be deconstructed into the validation data.
+
+For a [`CandidateValidationMessage`][CVM]`::ValidateFromChainState`, some more work needs to be done. Due to the uncertainty of Availability Cores (implemented in the [`Scheduler`](../../runtime/scheduler.md) module of the runtime), a candidate at a particular relay-parent and for a particular para may have two different valid validation-data to be executed under depending on what is assumed to happen if the para is occupying a core at the onset of the new block. This is encoded as an `OccupiedCoreAssumption` in the runtime API.
+
+The way that we can determine which assumption the candidate is meant to be executed under is simply to do an exhaustive check of both possibilities based on the state of the relay-parent. First we fetch the validation data under the assumption that the block occupying becomes available. If the `validation_data_hash` of the `CandidateDescriptor` matches this validation data, we use that. Otherwise, if the `validation_data_hash` matches the validation data fetched under the `TimedOut` assumption, we use that. Otherwise, we return a `ValidationResult::Invalid` response and conclude.
+
+Then, we can fetch the validation code from the runtime based on which type of candidate this is. This gives us all the parameters. The descriptor and PoV come from the request itself, and the other parameters have been derived from the state.
+
+> TODO: This would be a great place for caching to avoid making lots of runtime requests. That would need a job, though.
+
+### Execution of the Parachain Wasm
+
+Once we have all parameters, we can spin up a background task to perform the validation in a way that doesn't hold up the entire event loop. Before invoking the validation function itself, this should first do some basic checks:
+  * The collator signature is valid
+  * The PoV provided matches the `pov_hash` field of the descriptor
+
+After that, we can invoke the validation function. Lastly, we do some final checks on the output:
+  * The produced head-data is no larger than the maximum allowed.
+  * The produced code upgrade, if any, is no larger than the maximum allowed, and a code upgrade was allowed to be signaled.
+  * The amount and size of produced upward messages is not too large.
+
+[CVM]: ../../types/overseer-protocol.md#validationrequesttype

polkadot/roadmap/implementers-guide/src/types/overseer-protocol.md

@@ -324,7 +324,9 @@ enum CandidateValidationMessage {
 	/// Validate a candidate with provided parameters. This will implicitly attempt to gather the
 	/// `OmittedValidationData` and `ValidationCode` from the runtime API of the chain,
 	/// based on the `relay_parent` of the `CandidateDescriptor`.
-	/// If there is no state available which can provide this data, an error is returned.
+	///
+	/// If there is no state available which can provide this data or the core for
+	/// the para is not free at the relay-parent, an error is returned.
 	ValidateFromChainState(CandidateDescriptor, PoV, ResponseChannel<Result<ValidationResult>>),
 
 	/// Validate a candidate with provided parameters. Explicitly provide the `OmittedValidationData`