Basically combines all the recommended calls into one
`build_using_defaults()` call or `init_with_defaults()` when there are
some custom changes required.

**Update:** Pushed additional changes based on the review comments.

**This pull request fixes various spelling mistakes in this
repository.**

Most of the changes are contained in the first **3** commits:

- `Fix spelling mistakes in comments and docs`

- `Fix spelling mistakes in test names`

- `Fix spelling mistakes in error messages, panic messages, logs and
tracing`

Other source code spelling mistakes are separated into individual
commits for easier reviewing:

- `Fix the spelling of 'authority'`

- `Fix the spelling of 'REASONABLE_HEADERS_IN_JUSTIFICATION_ANCESTRY'`

- `Fix the spelling of 'prev_enqueud_messages'`

- `Fix the spelling of 'endpoint'`

- `Fix the spelling of 'children'`

- `Fix the spelling of 'PenpalSiblingSovereignAccount'`

- `Fix the spelling of 'PenpalSudoAccount'`

- `Fix the spelling of 'insufficient'`

- `Fix the spelling of 'PalletXcmExtrinsicsBenchmark'`

- `Fix the spelling of 'subtracted'`

- `Fix the spelling of 'CandidatePendingAvailability'`

- `Fix the spelling of 'exclusive'`

- `Fix the spelling of 'until'`

- `Fix the spelling of 'discriminator'`

- `Fix the spelling of 'nonexistent'`

- `Fix the spelling of 'subsystem'`

- `Fix the spelling of 'indices'`

- `Fix the spelling of 'committed'`

- `Fix the spelling of 'topology'`

- `Fix the spelling of 'response'`

- `Fix the spelling of 'beneficiary'`

- `Fix the spelling of 'formatted'`

- `Fix the spelling of 'UNKNOWN_PROOF_REQUEST'`

- `Fix the spelling of 'succeeded'`

- `Fix the spelling of 'reopened'`

- `Fix the spelling of 'proposer'`

- `Fix the spelling of 'InstantiationNonce'`

- `Fix the spelling of 'depositor'`

- `Fix the spelling of 'expiration'`

- `Fix the spelling of 'phantom'`

- `Fix the spelling of 'AggregatedKeyValue'`

- `Fix the spelling of 'randomness'`

- `Fix the spelling of 'defendant'`

- `Fix the spelling of 'AquaticMammal'`

- `Fix the spelling of 'transactions'`

- `Fix the spelling of 'PassingTracingSubscriber'`

- `Fix the spelling of 'TxSignaturePayload'`

- `Fix the spelling of 'versioning'`

- `Fix the spelling of 'descendant'`

- `Fix the spelling of 'overridden'`

- `Fix the spelling of 'network'`

Let me know if this structure is adequate.

**Note:** The usage of the words `Merkle`, `Merkelize`, `Merklization`,
`Merkelization`, `Merkleization`, is somewhat inconsistent but I left it
as it is.

~~**Note:** In some places the term `Receival` is used to refer to
message reception, IMO `Reception` is the correct word here, but I left
it as it is.~~

~~**Note:** In some places the term `Overlayed` is used instead of the
more acceptable version `Overlaid` but I also left it as it is.~~

~~**Note:** In some places the term `Applyable` is used instead of the
correct version `Applicable` but I also left it as it is.~~

**Note:** Some usage of British vs American english e.g. `judgement` vs
`judgment`, `initialise` vs `initialize`, `optimise` vs `optimize` etc.
are both present in different places, but I suppose that's
understandable given the number of contributors.

~~**Note:** There is a spelling mistake in `.github/CODEOWNERS` but it
triggers errors in CI when I make changes to it, so I left it as it
is.~~

This PR adds a new PolkaVM-based executor to Substrate.

- The executor can now be used to actually run a PolkaVM-based runtime,
and successfully produces blocks.
- The executor is always compiled-in, but is disabled by default.
- The `SUBSTRATE_ENABLE_POLKAVM` environment variable must be set to `1`
to enable the executor, in which case the node will accept both WASM and
PolkaVM program blobs (otherwise it'll default to WASM-only). This is
deliberately undocumented and not explicitly exposed anywhere (e.g. in
the command line arguments, or in the API) to disincentivize anyone from
enabling it in production. If/when we'll move this into production usage
I'll remove the environment variable and do it "properly".
- I did not use our legacy runtime allocator for the PolkaVM executor,
so currently every allocation inside of the runtime will leak guest
memory until that particular instance is destroyed. The idea here is
that I will work on the https://github.com/polkadot-fellows/RFCs/pull/4
which will remove the need for the legacy allocator under WASM, and that
will also allow us to use a proper non-leaking allocator under PolkaVM.
- I also did some minor cleanups of the WASM executor and deleted some
dead code.

No prdocs included since this is not intended to be an end-user feature,
but an unofficial experiment, and shouldn't affect any current
production user. Once this is production-ready a full Polkadot
Fellowship RFC will be necessary anyway.

resolve https://github.com/paritytech/polkadot-sdk/issues/3116

a follow-up on
https://github.com/paritytech/polkadot-sdk/pull/3061#pullrequestreview-1847530265

:

- [x] reuse collator overseer builder for polkadot-node and collator
- [x] run zombienet test (0001-parachains-smoke-test.toml)
- [x] make wasm build errors more user-friendly for an easier problem
detection when using different toolchains in Rust

---------

Co-authored-by: ordian <[email protected]>
Co-authored-by: s0me0ne-unkn0wn <[email protected]>

This PR fixes a subtle bug in `wasm-builder` first introduced in
https://github.com/paritytech/polkadot-sdk/pull/1851 (sorry, my bad! I
should have caught this during review) where the status code of the
`cargo` subprocess is not properly checked, which results in builds
silently succeeding when they shouldn't (that is: if we successfully
build a runtime blob, and then modify the code so that it won't compile,
and recompile it again, then the build will succeed and silently use the
*old* blob).

cc @athei This is the bug you were seeing.

[edit]Also fixes a similar PolkaVM-specific bug where I accidentally
used the wrong comparison operator.[/edit]

This PR improves compatibility with RISC-V and PolkaVM, allowing more
runtimes to successfully compile.

In particular, it makes the following changes:

- The `sp-mmr-primitives` and `sp-consensus-beefy` crates
unconditionally required an `std`-only dependency; now they only require
those dependencies when the `std` feature is actually enabled. (Our
RISC-V target is, unlike WASM, a true `no_std` target where you can't
accidentally use stuff from `std` anymore.)
- One of our dependencies (the `bitvec` trace) uses a crate called
`radium` which doesn't compile under RISC-V due to incomplete
autodetection logic in their `build.rs` file. The good news is that this
is already fixed in the newest upstream version of `radium`, and the
newest version of `bitvec` uses it. The bad news is that the newest
version of `bitvec` is not currently released on crates.io, so we can't
use it. I've [created an
issue](https://github.com/ferrilab/ferrilab/issues/5) asking for a new
release, but in the meantime I forked the currently used `radium` 0.7,
[fixed the faulty
logic](https://github.com/paritytech/radium-0.7-fork/commit/ed66c8a294b138c67f93499644051d97d4c7fbda)
and used cargo's patching capabilities to use it for the RISC-V runtime
builds. This might be a little hacky, but it is the least intrusive way
to fix the problem, doesn't affect WASM builds at all, and we can
trivially remove it once a new `bitvec` is released.
- The new runtimes are added to the CI to make sure their compilation
doesn't break.

This PR adds initial support for building RISC-V runtimes targeting
PolkaVM.

- Setting the `SUBSTRATE_RUNTIME_TARGET=riscv` environment variable will
now build a RISC-V runtime instead of a WASM runtime.
- This only adds support for *building* runtimes; running them will need
a PolkaVM-based executor, which I will add in a future PR.
- Only building the minimal runtime is supported (building the Polkadot
runtime doesn't work *yet* due to one of the dependencies).
- The builder now sets a `substrate_runtime` cfg flag when building the
runtimes, with the idea being that instead of doing `#[cfg(not(feature =
"std"))]` or `#[cfg(target_arch = "wasm32")]` to detect that we're
building a runtime you'll do `#[cfg(substrate_runtime)]`. (Switching the
whole codebase to use this will be done in a future PR; I deliberately
didn't do this here to keep this PR minimal and reviewable.)
- Further renaming of things (e.g. types, environment variables and proc
macro attributes having "wasm" in their name) to be target-agnostic will
also be done in a future refactoring PR (while keeping backwards
compatibility where it makes sense; I don't intend to break anyone's
workflow or create unnecessary churn).
- This PR also fixes two bugs in the `wasm-builder` crate:
* The `RUSTC` environment variable is now removed when invoking the
compiler. This prevents the toolchain version from being overridden when
called from a `build.rs` script.
* When parsing the `rustup toolchain list` output the `(default)` is now
properly stripped and not treated as part of the version.
- I've also added a minimal CI job that makes sure this doesn't break in
the future. (cc @paritytech/ci)

cc @athei



------

Also, just a fun little tidbit: quickly comparing the size of the built
runtimes it seems that the PolkaVM runtime is slightly smaller than the
WASM one. (`production` build, with the `names` section substracted from
the WASM's size to keep things fair, since for the PolkaVM runtime we're
currently stripping out everything)

- `.wasm`: 625505 bytes
- `.wasm` (after wasm-opt -O3): 563205 bytes
- `.wasm` (after wasm-opt -Os): 562987 bytes
- `.wasm` (after wasm-opt -Oz): 536852 bytes
- `.polkavm`: ~~580338 bytes~~ 550476 bytes (after enabling extra target
features; I'll add those in another PR once we have an executor working)

---------

Co-authored-by: Bastian Köcher <[email protected]>

Hi folks!

Thank for the well organized codebase and an outstanding engineering!

I am trying to compile a substrate node template from source
(https://github.com/paritytech/polkadot-sdk) and encountered a
dependency conflict
![Screenshot 2024-01-11 at 12 22
16](https://github.com/paritytech/polkadot-sdk/assets/16288656/b630773f-9d58-4abc-a15c-45f0e6b96b48)
and a deprecation warning from advisory db for `ansi_term` (I see you
replace it with some alternatives in other crates).

While for `ansi_term` there is an adopted fork
(https://github.com/rustadopt/ansiterm-rs) and it was my first commit in
the PR, I've decided to use https://github.com/console-rs/console

 as you
already use it to reduce dependencies (as I believe other substrate
crates will remove ansi_term eventually)

---------

Co-authored-by: Bastian Köcher <[email protected]>

The [chainSpec RPC API from the v2
spec](https://paritytech.github.io/json-rpc-interface-spec/api/chainSpec.html)
was only added to substrate-node and should be added to polkadot as well

/cc @lexnv

Followup of https://github.com/paritytech/polkadot-sdk/pull/2217

This PR deletes the README of the `wasm-builder` crate and moves its
docs back into the rustdoc, [as requested
here](https://github.com/paritytech/polkadot-sdk/pull/2217#discussion_r1406401175).
(:

Our executor currently only supports the WASM MVP feature set, however
nowadays when compiling WASM the Rust compiler has more features enabled
by default.

We do set the `-C target-cpu=mvp` flag to make sure that *our* code gets
compiled in a way that is compatible with our executor, however this
doesn't affect Rust's standard library crates (`std`, `core` and
`alloc`) which are by default precompiled and still can make use of
these extra features.

So in this PR we force the compiler to also compile the standard library
crates for us to make sure that they also only use the MVP features.

I've added the `WASM_BUILD_STD` environment variable which can be used
to disable this behavior if set to `0`.

Unfortunately this *will* slow down the compile times when building
runtimes, but there isn't much that we can do about that.

Fixes https://github.com/paritytech/polkadot-sdk/issues/1755



---------

Co-authored-by: Bastian Köcher <[email protected]>

Optimizes the `rerun-if-changed` logic by ignoring `dev-dependencies`
and also not outputting paths. Because outputting paths could lead to
include unwanted crates in the rerun checks.

`wasm-builder` was adjusted to default to building wasm blobs in
`release` mode even when cargo is in `debug` because `debug` wasm is too
slow.

A side effect of this was `.compact` and `.compact.compressed` getting
built when the dev is running build in `debug`, adding ~5s to the build
time of every wasm runtime.

I think it's reasonable to assume if the dev is running `debug` build
they want to optimise speed and do not care about the size of the wasm
binary. Compacting a blob has negligible impact on its actual
performance.

In this PR, I adjusted the behavior of the wasm builder so it does not
produce `.compact` or `.compact.compressed` wasm when the user is
running in `debug`. The builder will continue to produce the bloaty wasm
in release mode unless it is overriden with an env var.

As suggested by @koute in review, also refactored the
`maybe_compact_wasm_and_copy_blobs` into multiple funuctions, and
renamed things to better support RISC-V in the future.

---

There is no `T-runtime` label so @KiChjang

 told me to put `T1-FRAME` :)

---------

Co-authored-by: Koute <[email protected]>

✄
-----------------------------------------------------------------------------

Thank you for your Pull Request! 🙏 Please make sure it follows the
contribution guidelines outlined in
[this
document](https://github.com/paritytech/polkadot-sdk/blob/master/docs/CONTRIBUTING.md)
and fill
out the sections below. Once you're ready to submit your PR for review,
please
delete this section and leave only the text under the "Description"
heading.

# Description

*Please include a summary of the changes and the related issue. Please
also include relevant motivation and context,
including:*

- What does this PR do?

make 'substrate-wasm-builder' manually set 'CARGO_TARGET_DIR' to
'$project_dir/target' while building instead of unset
'CARGO_TARGET_DIR';

- Why are these changes needed?

If you using this in the `build.rs` with following content in your
`~/.cargo/config.toml':

    [build]
    target-dir = "target"

the build process will stuck because of dead lock -- two `cargo build`
on same target directory in the same time.
There is already an attempt to avoid such dead lock by unset the
`CARGO_TARGET_DIR`, but for users with config above in his build
enviroment (like me), this workaround won't work.

- How were these changes implemented and what do they affect?

Instead of unset 'CARGO_TARGET_DIR', we set 'CARGO_TARGET_DIR' to
'$project/target/', which is already assumed to be true by rest of the
code.

*Use [Github semantic

linking](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword)
to address any open issues this PR relates to or closes.*

Fixes # (issue number, *if applicable*)

Closes # (issue number, *if applicable*)

# Checklist

- [x] My PR includes a detailed description as outlined in the
"Description" section above
- [ ] My PR follows the [labeling requirements](CONTRIBUTING.md#Process)
of this project (at minimum one label for `T`
  required)
- [ ] I have made corresponding changes to the documentation (if
applicable)
- [ ] I have added tests that prove my fix is effective or that my
feature works (if applicable)

You can remove the "Checklist" section once all have been checked. Thank
you for your contribution!

✄
-----------------------------------------------------------------------------

I have built my project with this fix, there's still some warnings with
`build.target-dir` set but the building process won't hang.
I haven't found related issue in this repo. But I did find one issue
[here](https://github.com/substrate-developer-hub/substrate-node-template/issues/116).

This pull request changes the `wasm-builder` to skip building the wasm
files when the build process is running on docs.rs.

* Added ECIES encryption

* tweaks

* fmt

* Make clippy happy

* Use local keystore

* qed

* wasm-builder: Make `hash` and `date` optional

Apparently there are installations where the `hash` and `date` is optional.

Closes: https://github.com/paritytech/substrate/issues/14335

* ".git/.scripts/commands/fmt/fmt.sh"

---------

Co-authored-by: command-bot <>

* frame-benchmarking-cli: Remove native dispatch requirement

No need for this, we can just use the `WasmExecutor` directly.

* Fixes

* Pass benchmarking host functions

* Ensure we can pass custom host functions

* wasm-builder: Enforce `runtime_version` wasm section

This pr changes the `wasm-builder` to enforce the `runtime_version` wasm section. This wasm section
is being created by the `sp_version::runtime_version` attribute macro. This attribute macro now
exists since quite some time and `runtime_version` also is the only way for parachains to support
reading the `RuntimeVersion` from the runtime.

\# Disabling the check

By default the `WasmBuilder` will now check for this wasm section and if not found, exit with an
error. However, there are situations where you may want to disable this check (like for tests). In
this case there exists the `disable_runtime_version_section_check` function.

```
WasmBuilder::new()
   ...
   ...
   ...
   .disable_runtime_version_section_check()
   .build()
```

By using this method you get back the old behavior.

* Review comment

* Fix

* Fix issue with `enum-as-inner`

Co-authored-by: parity-processbot <>

* Support stable rust for compiling the runtime

This pull request brings support for compiling the runtime with stable Rust. This requires at least
rust 1.68.0 to work on stable. The code is written in a way that it is backwards compatible and
should automatically work when someone compiles with 1.68.0+ stable.

* We always support nightlies!

* 🤦



* Sort by version

* Review feedback

* Review feedback

* Fix version parsing

* Apply suggestions from code review

Co-authored-by: Koute <[email protected]>

---------

Co-authored-by: Koute <[email protected]>

* Change copyright year to 2023 from 2022

* Fix incorrect update of copyright year

* Remove years from copy right header

* Fix remaining files

* Fix typo in a header and remove update-copyright.sh

* Fixup some wrong dependencies

Dev dependencies should not appear in the feature list. If features are required, they should be
directly enabled for the `dev-dependency`.

* More fixups

* Fix fix

* Remove deprecated feature

* Make all work properly and nice!!

* FMT

* Fix formatting

* implement crate publishing from CI

* fix indentation

* use resource_group for job exclusivity

ensure that at most one instance of the publish-crates job is running at any given time to prevent race conditions

* correct publish = false

* Remove YAML anchors as GitLab's `extends:` doesn't need it

* Temporarily force cache upload for the new jobs

* Revert `RUSTY_CACHIER_FORCE_UPLOAD`

* pin libp2p-tcp=0.37.0 for sc-telemetry

* Revert "pin libp2p-tcp=0.37.0 for sc-telemetry"

This reverts commit 29146bfad6c31e8cf0e2f17ad92a71bb81a373af.

* always collect generated crates

* increase timeout for publish-crates-template

* Force upload the new job cache again

* Revert "Force upload the new job cache again"

This reverts commit 5a5feee1b2c51fdef768b25a76be4c3949ec1c99.

* reformat

* improve timeout explanation

* s/usual/average

Co-authored-by: Vladimir Istyufeev <[email protected]>

* Use wasm-opt on runtime

* Optimize for size

* Simplify fn compact_wasm_file

* Run a lighter pass for non production builds

* Disable optimizations and keep name section

* Update wasm-opt

* Remove dward sections

* Update wasm-opt

* Update wasm-opt

* Rename node=runtime to kithensink-runtime

* Undo md formatting

* Support offline env variable in wasm builder

* Clean up

* Improve checks

Co-authored-by: Bastian Köcher <[email protected]>

* Update crate docs

* Add docs to `lib.rs` and introduce helper method `offline_build`

Co-authored-by: Bastian Köcher <[email protected]>

Recently we added the wasm binaries to the `rerun-if-changed` list. The problem with that is that
they have a later mtime than the `invoked.timestamp` file and this file's mtime is used to determine
if the `build.rs` script needs to be re-run. The solution to this is that we copy the mtime of this
`invoked.timestamp` file and add it to the wasm binaries. Then cargo/rustc doesn't constantly wants
to rerun the `build.rs` script.

* Apply some clippy hints

* Revert clippy ci changes

* Update client/cli/src/commands/generate.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/cli/src/commands/inspect_key.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/db/src/bench.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/db/src/bench.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/service/src/client/block_rules.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/service/src/client/block_rules.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/network/src/transactions.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/network/src/protocol.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Revert due to missing `or_default` function.

* Fix compilation and simplify code

* Undo change that corrupts benchmark.

* fix clippy

* Update client/service/test/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/state-db/src/noncanonical.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update client/state-db/src/noncanonical.rs

remove leftovers!

* Update client/tracing/src/logging/directives.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update utils/fork-tree/src/lib.rs

Co-authored-by: Bastian Köcher <[email protected]>

* added needed ref

* Update frame/referenda/src/benchmarking.rs

* Simplify byte-vec creation

* let's just not overlap the ranges

* Correction

* cargo fmt

* Update utils/frame/benchmarking-cli/src/shared/stats.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update utils/frame/benchmarking-cli/src/pallet/command.rs

Co-authored-by: Bastian Köcher <[email protected]>

* Update utils/frame/benchmarking-cli/src/pallet/command.rs

Co-authored-by: Bastian Köcher <[email protected]>

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Giles Cope <[email protected]>

* wasm-builder: Support latest nightly

With latest nightly, aka rust version 1.60+ namespaced features are added. This changes the handling
of optional dependencies. We currently have features that enable optional dependencies when `std` is
enabled. This was before no problem, but now the wasm-builder detects them as enabled. To support
the transition period until 1.60 is released as stable, this pr adds an heuristic to not enable these
optional crates in the wasm build when they are enabled in the `std` feature. This heuristic fails
when someones enables these optional dependencies from the outside as well as via the `std` feature,
however we hope that no one is doing this at the moment. When namespaced features are enabled, these
dependencies needs to be enabled using `dep:dependency-name` to solve this properly.

https://doc.rust-lang.org/cargo/reference/unstable.html#namespaced-features

* Remove accidentally added features

* Add production profile to wasm builder

* Fix profile detection

* Apply suggestions from code review

Co-authored-by: Bastian Köcher <[email protected]>

* Replace panic! by println! + exit

* Default to `release` for wasm on debug builds

* Replaced unwrap by expect

* Update all weights

Rerun on the bm2 server.

Signed-off-by: Oliver Tale-Yazdi <[email protected]>

* cargo run --quiet --profile=production  --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs

Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Oliver Tale-Yazdi <[email protected]>
Co-authored-by: Parity Bot <[email protected]>

This is required for projects like Cumulus that have dependencies on `westend-runtime`, but this
runtime is only added when the `runtime-benchmarks` feature is enabled. By having all features
"enabled" in `cargo metadata` we ensure that all crates can be found.

When building a wasm binary from a different repo inside a local workspace, we did not used the
correct `Cargo.toml` to find the correct patches and features. The solution to this is to just walk
up from the target directory until we find the workspace we are currently compiling. If this
heuristic isn't working, we print a warning and let the user set an env variable
`WASM_BUILD_WORKSPACE_HINT` to tell the `wasm-builder` where the actual workspace is.

* Set current dir at check_wasm_toolchain_installed

* Add comments

* add TypeInfo for DispatchTime

* upgrade wasm-builder to Rust 2021

* remove resolver

* revert resolver in virtual workspace

* Do not propagate host RUSTFLAGS when checking for WASM toolchain

* Update utils/wasm-builder/src/prerequisites.rs

* Update utils/wasm-builder/src/prerequisites.rs

Co-authored-by: Bastian Köcher <[email protected]>