- Feb 03, 2024
-
-
Koute authored
This PR adds initial support for building RISC-V runtimes targeting PolkaVM. - Setting the `SUBSTRATE_RUNTIME_TARGET=riscv` environment variable will now build a RISC-V runtime instead of a WASM runtime. - This only adds support for *building* runtimes; running them will need a PolkaVM-based executor, which I will add in a future PR. - Only building the minimal runtime is supported (building the Polkadot runtime doesn't work *yet* due to one of the dependencies). - The builder now sets a `substrate_runtime` cfg flag when building the runtimes, with the idea being that instead of doing `#[cfg(not(feature = "std"))]` or `#[cfg(target_arch = "wasm32")]` to detect that we're building a runtime you'll do `#[cfg(substrate_runtime)]`. (Switching the whole codebase to use this will be done in a future PR; I deliberately didn't do this here to keep this PR minimal and reviewable.) - Further renaming of things (e.g. types, environment variables and proc macro attributes having "wasm" in their name) to be target-agnostic will also be done in a future refactoring PR (while keeping backwards compatibility where it makes sense; I don't intend to break anyone's workflow or create unnecessary churn). - This PR also fixes two bugs in the `wasm-builder` crate: * The `RUSTC` environment variable is now removed when invoking the compiler. This prevents the toolchain version from being overridden when called from a `build.rs` script. * When parsing the `rustup toolchain list` output the `(default)` is now properly stripped and not treated as part of the version. - I've also added a minimal CI job that makes sure this doesn't break in the future. (cc @paritytech/ci) cc @athei ------ Also, just a fun little tidbit: quickly comparing the size of the built runtimes it seems that the PolkaVM runtime is slightly smaller than the WASM one. (`production` build, with the `names` section substracted from the WASM's size to keep things fair, since for the PolkaVM runtime we're currently stripping out everything) - `.wasm`: 625505 bytes - `.wasm` (after wasm-opt -O3): 563205 bytes - `.wasm` (after wasm-opt -Os): 562987 bytes - `.wasm` (after wasm-opt -Oz): 536852 bytes - `.polkavm`: ~~580338 bytes~~ 550476 bytes (after enabling extra target features; I'll add those in another PR once we have an executor working) --------- Co-authored-by: Bastian Köcher <[email protected]>
-
- Jan 13, 2024
-
-
maksimryndin authored
Hi folks! Thank for the well organized codebase and an outstanding engineering! I am trying to compile a substrate node template from source (https://github.com/paritytech/polkadot-sdk) and encountered a dependency conflict ![Screenshot 2024-01-11 at 12 22 16](https://github.com/paritytech/polkadot-sdk/assets/16288656/b630773f-9d58-4abc-a15c-45f0e6b96b48) and a deprecation warning from advisory db for `ansi_term` (I see you replace it with some alternatives in other crates). While for `ansi_term` there is an adopted fork (https://github.com/rustadopt/ansiterm-rs) and it was my first commit in the PR, I've decided to use https://github.com/console-rs/console as you already use it to reduce dependencies (as I believe other substrate crates will remove ansi_term eventually) --------- Co-authored-by: Bastian Köcher <[email protected]>
-
- Jan 08, 2024
-
-
Niklas Adolfsson authored
The [chainSpec RPC API from the v2 spec](https://paritytech.github.io/json-rpc-interface-spec/api/chainSpec.html) was only added to substrate-node and should be added to polkadot as well /cc @lexnv
-
- Nov 28, 2023
-
-
Koute authored
Followup of https://github.com/paritytech/polkadot-sdk/pull/2217 This PR deletes the README of the `wasm-builder` crate and moves its docs back into the rustdoc, [as requested here](https://github.com/paritytech/polkadot-sdk/pull/2217#discussion_r1406401175). (:
-
- Nov 27, 2023
-
-
Koute authored
Our executor currently only supports the WASM MVP feature set, however nowadays when compiling WASM the Rust compiler has more features enabled by default. We do set the `-C target-cpu=mvp` flag to make sure that *our* code gets compiled in a way that is compatible with our executor, however this doesn't affect Rust's standard library crates (`std`, `core` and `alloc`) which are by default precompiled and still can make use of these extra features. So in this PR we force the compiler to also compile the standard library crates for us to make sure that they also only use the MVP features. I've added the `WASM_BUILD_STD` environment variable which can be used to disable this behavior if set to `0`. Unfortunately this *will* slow down the compile times when building runtimes, but there isn't much that we can do about that. Fixes https://github.com/paritytech/polkadot-sdk/issues/1755 --------- Co-authored-by: Bastian Köcher <[email protected]>
-
- Nov 13, 2023
-
-
Bastian Köcher authored
Optimizes the `rerun-if-changed` logic by ignoring `dev-dependencies` and also not outputting paths. Because outputting paths could lead to include unwanted crates in the rerun checks.
-
- Oct 25, 2023
-
-
Liam Aharon authored
`wasm-builder` was adjusted to default to building wasm blobs in `release` mode even when cargo is in `debug` because `debug` wasm is too slow. A side effect of this was `.compact` and `.compact.compressed` getting built when the dev is running build in `debug`, adding ~5s to the build time of every wasm runtime. I think it's reasonable to assume if the dev is running `debug` build they want to optimise speed and do not care about the size of the wasm binary. Compacting a blob has negligible impact on its actual performance. In this PR, I adjusted the behavior of the wasm builder so it does not produce `.compact` or `.compact.compressed` wasm when the user is running in `debug`. The builder will continue to produce the bloaty wasm in release mode unless it is overriden with an env var. As suggested by @koute in review, also refactored the `maybe_compact_wasm_and_copy_blobs` into multiple funuctions, and renamed things to better support RISC-V in the future. --- There is no `T-runtime` label so @KiChjang told me to put `T1-FRAME` :) --------- Co-authored-by: Koute <[email protected]>
-
- Oct 23, 2023
-
-
aj3n authored
✄ ----------------------------------------------------------------------------- Thank you for your Pull Request!
🙏 Please make sure it follows the contribution guidelines outlined in [this document](https://github.com/paritytech/polkadot-sdk/blob/master/docs/CONTRIBUTING.md) and fill out the sections below. Once you're ready to submit your PR for review, please delete this section and leave only the text under the "Description" heading. # Description *Please include a summary of the changes and the related issue. Please also include relevant motivation and context, including:* - What does this PR do? make 'substrate-wasm-builder' manually set 'CARGO_TARGET_DIR' to '$project_dir/target' while building instead of unset 'CARGO_TARGET_DIR'; - Why are these changes needed? If you using this in the `build.rs` with following content in your `~/.cargo/config.toml': [build] target-dir = "target" the build process will stuck because of dead lock -- two `cargo build` on same target directory in the same time. There is already an attempt to avoid such dead lock by unset the `CARGO_TARGET_DIR`, but for users with config above in his build enviroment (like me), this workaround won't work. - How were these changes implemented and what do they affect? Instead of unset 'CARGO_TARGET_DIR', we set 'CARGO_TARGET_DIR' to '$project/target/', which is already assumed to be true by rest of the code. *Use [Github semantic linking](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword) to address any open issues this PR relates to or closes.* Fixes # (issue number, *if applicable*) Closes # (issue number, *if applicable*) # Checklist - [x] My PR includes a detailed description as outlined in the "Description" section above - [ ] My PR follows the [labeling requirements](CONTRIBUTING.md#Process) of this project (at minimum one label for `T` required) - [ ] I have made corresponding changes to the documentation (if applicable) - [ ] I have added tests that prove my fix is effective or that my feature works (if applicable) You can remove the "Checklist" section once all have been checked. Thank you for your contribution! ✄ ----------------------------------------------------------------------------- I have built my project with this fix, there's still some warnings with `build.target-dir` set but the building process won't hang. I haven't found related issue in this repo. But I did find one issue [here](https://github.com/substrate-developer-hub/substrate-node-template/issues/116).
-
- Sep 13, 2023
-
-
Bastian Köcher authored
This pull request changes the `wasm-builder` to skip building the wasm files when the build process is running on docs.rs.
-
- Jul 17, 2023
-
-
Arkadiy Paronyan authored
* Added ECIES encryption * tweaks * fmt * Make clippy happy * Use local keystore * qed
-
- Jul 03, 2023
-
-
Bastian Köcher authored
* wasm-builder: Make `hash` and `date` optional Apparently there are installations where the `hash` and `date` is optional. Closes: https://github.com/paritytech/substrate/issues/14335 * ".git/.scripts/commands/fmt/fmt.sh" --------- Co-authored-by: command-bot <>
-
- Jun 29, 2023
-
-
Bastian Köcher authored
* frame-benchmarking-cli: Remove native dispatch requirement No need for this, we can just use the `WasmExecutor` directly. * Fixes * Pass benchmarking host functions * Ensure we can pass custom host functions
-
- May 28, 2023
-
-
Bastian Köcher authored
* wasm-builder: Enforce `runtime_version` wasm section This pr changes the `wasm-builder` to enforce the `runtime_version` wasm section. This wasm section is being created by the `sp_version::runtime_version` attribute macro. This attribute macro now exists since quite some time and `runtime_version` also is the only way for parachains to support reading the `RuntimeVersion` from the runtime. \# Disabling the check By default the `WasmBuilder` will now check for this wasm section and if not found, exit with an error. However, there are situations where you may want to disable this check (like for tests). In this case there exists the `disable_runtime_version_section_check` function. ``` WasmBuilder::new() ... ... ... .disable_runtime_version_section_check() .build() ``` By using this method you get back the old behavior. * Review comment * Fix * Fix issue with `enum-as-inner`
-
- Apr 03, 2023
-
-
Koute authored
Co-authored-by: parity-processbot <>
-
- Mar 30, 2023
-
-
Alexander Theißen authored
-
- Mar 29, 2023
-
-
Bastian Köcher authored
* Support stable rust for compiling the runtime This pull request brings support for compiling the runtime with stable Rust. This requires at least rust 1.68.0 to work on stable. The code is written in a way that it is backwards compatible and should automatically work when someone compiles with 1.68.0+ stable. * We always support nightlies! *
🤦 * Sort by version * Review feedback * Review feedback * Fix version parsing * Apply suggestions from code review Co-authored-by: Koute <[email protected]> --------- Co-authored-by: Koute <[email protected]>
-
- Feb 21, 2023
-
-
Vivek Pandya authored
* Change copyright year to 2023 from 2022 * Fix incorrect update of copyright year * Remove years from copy right header * Fix remaining files * Fix typo in a header and remove update-copyright.sh
-
- Dec 13, 2022
-
-
Bastian Köcher authored
* Fixup some wrong dependencies Dev dependencies should not appear in the feature list. If features are required, they should be directly enabled for the `dev-dependency`. * More fixups * Fix fix * Remove deprecated feature * Make all work properly and nice!! * FMT * Fix formatting
-
- Dec 07, 2022
-
-
João Paulo Silva de Souza authored
* implement crate publishing from CI * fix indentation * use resource_group for job exclusivity ensure that at most one instance of the publish-crates job is running at any given time to prevent race conditions * correct publish = false * Remove YAML anchors as GitLab's `extends:` doesn't need it * Temporarily force cache upload for the new jobs * Revert `RUSTY_CACHIER_FORCE_UPLOAD` * pin libp2p-tcp=0.37.0 for sc-telemetry * Revert "pin libp2p-tcp=0.37.0 for sc-telemetry" This reverts commit 29146bfad6c31e8cf0e2f17ad92a71bb81a373af. * always collect generated crates * increase timeout for publish-crates-template * Force upload the new job cache again * Revert "Force upload the new job cache again" This reverts commit 5a5feee1b2c51fdef768b25a76be4c3949ec1c99. * reformat * improve timeout explanation * s/usual/average Co-authored-by: Vladimir Istyufeev <[email protected]>
-
- Oct 30, 2022
-
-
Alexander Theißen authored
* Use wasm-opt on runtime * Optimize for size * Simplify fn compact_wasm_file * Run a lighter pass for non production builds * Disable optimizations and keep name section * Update wasm-opt * Remove dward sections * Update wasm-opt * Update wasm-opt
-
- Aug 02, 2022
-
-
Nikos Kontakis authored
* Rename node=runtime to kithensink-runtime * Undo md formatting
-
- Jun 22, 2022
-
-
Sebastian Kunert authored
* Support offline env variable in wasm builder * Clean up * Improve checks Co-authored-by: Bastian Köcher <[email protected]> * Update crate docs * Add docs to `lib.rs` and introduce helper method `offline_build` Co-authored-by: Bastian Köcher <[email protected]>
-
- Jun 14, 2022
-
-
Bastian Köcher authored
Recently we added the wasm binaries to the `rerun-if-changed` list. The problem with that is that they have a later mtime than the `invoked.timestamp` file and this file's mtime is used to determine if the `build.rs` script needs to be re-run. The solution to this is that we copy the mtime of this `invoked.timestamp` file and add it to the wasm binaries. Then cargo/rustc doesn't constantly wants to rerun the `build.rs` script.
-
- Jun 02, 2022
-
-
Bastian Köcher authored
-
- Apr 30, 2022
-
-
Falco Hirschenberger authored
* Apply some clippy hints * Revert clippy ci changes * Update client/cli/src/commands/generate.rs Co-authored-by: Bastian Köcher <[email protected]> * Update client/cli/src/commands/inspect_key.rs Co-authored-by: Bastian Köcher <[email protected]> * Update client/db/src/bench.rs Co-authored-by: Bastian Köcher <[email protected]> * Update client/db/src/bench.rs Co-authored-by: Bastian Köcher <[email protected]> * Update client/service/src/client/block_rules.rs Co-authored-by: Bastian Köcher <[email protected]> * Update client/service/src/client/block_rules.rs Co-authored-by: Bastian Köcher <[email protected]> * Update client/network/src/transactions.rs Co-authored-by: Bastian Köcher <[email protected]> * Update client/network/src/protocol.rs Co-authored-by: Bastian Köcher <[email protected]> * Revert due to missing `or_default` function. * Fix compilation and simplify code * Undo change that corrupts benchmark. * fix clippy * Update client/service/test/src/lib.rs Co-authored-by: Bastian Köcher <[email protected]> * Update client/state-db/src/noncanonical.rs Co-authored-by: Bastian Köcher <[email protected]> * Update client/state-db/src/noncanonical.rs remove leftovers! * Update client/tracing/src/logging/directives.rs Co-authored-by: Bastian Köcher <[email protected]> * Update utils/fork-tree/src/lib.rs Co-authored-by: Bastian Köcher <[email protected]> * added needed ref * Update frame/referenda/src/benchmarking.rs * Simplify byte-vec creation * let's just not overlap the ranges * Correction * cargo fmt * Update utils/frame/benchmarking-cli/src/shared/stats.rs Co-authored-by: Bastian Köcher <[email protected]> * Update utils/frame/benchmarking-cli/src/pallet/command.rs Co-authored-by: Bastian Köcher <[email protected]> * Update utils/frame/benchmarking-cli/src/pallet/command.rs Co-authored-by: Bastian Köcher <[email protected]> Co-authored-by: Bastian Köcher <[email protected]> Co-authored-by: Giles Cope <[email protected]>
-
- Feb 14, 2022
-
-
Bastian Köcher authored
* wasm-builder: Support latest nightly With latest nightly, aka rust version 1.60+ namespaced features are added. This changes the handling of optional dependencies. We currently have features that enable optional dependencies when `std` is enabled. This was before no problem, but now the wasm-builder detects them as enabled. To support the transition period until 1.60 is released as stable, this pr adds an heuristic to not enable these optional crates in the wasm build when they are enabled in the `std` feature. This heuristic fails when someones enables these optional dependencies from the outside as well as via the `std` feature, however we hope that no one is doing this at the moment. When namespaced features are enabled, these dependencies needs to be enabled using `dep:dependency-name` to solve this properly. https://doc.rust-lang.org/cargo/reference/unstable.html#namespaced-features * Remove accidentally added features
-
- Feb 02, 2022
-
-
Alexander Theißen authored
-
- Jan 31, 2022
-
-
Alexander Theißen authored
* Add production profile to wasm builder * Fix profile detection * Apply suggestions from code review Co-authored-by: Bastian Köcher <[email protected]> * Replace panic! by println! + exit * Default to `release` for wasm on debug builds * Replaced unwrap by expect * Update all weights Rerun on the bm2 server. Signed-off-by: Oliver Tale-Yazdi <[email protected]> * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs Co-authored-by: Bastian Köcher <[email protected]> Co-authored-by: Oliver Tale-Yazdi <[email protected]> Co-authored-by: Parity Bot <[email protected]>
-
- Jan 22, 2022
-
-
Bastian Köcher authored
This is required for projects like Cumulus that have dependencies on `westend-runtime`, but this runtime is only added when the `runtime-benchmarks` feature is enabled. By having all features "enabled" in `cargo metadata` we ensure that all crates can be found.
-
- Jan 21, 2022
-
-
Bastian Köcher authored
When building a wasm binary from a different repo inside a local workspace, we did not used the correct `Cargo.toml` to find the correct patches and features. The solution to this is to just walk up from the target directory until we find the workspace we are currently compiling. If this heuristic isn't working, we print a warning and let the user set an env variable `WASM_BUILD_WORKSPACE_HINT` to tell the `wasm-builder` where the actual workspace is.
-
- Jan 03, 2022
-
-
Bastian Köcher authored
-
- Nov 18, 2021
-
-
MOZGIII authored
* Set current dir at check_wasm_toolchain_installed * Add comments
-
- Nov 10, 2021
-
-
zjb0807 authored
* add TypeInfo for DispatchTime * upgrade wasm-builder to Rust 2021 * remove resolver * revert resolver in virtual workspace
-
- Oct 04, 2021
-
-
Koute authored
* Do not propagate host RUSTFLAGS when checking for WASM toolchain * Update utils/wasm-builder/src/prerequisites.rs * Update utils/wasm-builder/src/prerequisites.rs Co-authored-by: Bastian Köcher <[email protected]>
-
- Sep 24, 2021
-
-
Bastian Köcher authored
* Prepare UI tests for rust 1.54 * Delete wrong_page.stderr * CI: run with a staging CI image * Revert "CI: run with a staging CI image" This reverts commit 66f5b00d14b50fd9d8fbf773f7e884f380697591. * CI: debug, again * LOG_TARGET is only used in std * Remove unnecessary unsafe * Fixes * Use correct rustc locally * FMT * Compile with benchmarking * Review feedback * Some ui tests * I know... * Fix wasm tests Co-authored-by: Denis P <[email protected]> Co-authored-by: thiolliere <[email protected]>
-
- Aug 11, 2021
-
-
Kian Paimani authored
* reformat everything again * manual formatting * last manual fix * Fix build
-
- Jul 22, 2021
-
-
Squirrel authored
* Remove extra commas made redundent after rustfmt
-
- Jul 21, 2021
-
-
Bastian Köcher authored
* Run cargo fmt on the whole code base * Second run * Add CI check * Fix compilation * More unnecessary braces * Handle weights * Use --all * Use correct attributes... * Fix UI tests * AHHHHHHHHH *
🤦 * Docs * Fix compilation *🤷 * Please stop *🤦 x 2 * More * make rustfmt.toml consistent with polkadot Co-authored-by: André Silva <[email protected]>
-
- Jun 15, 2021
-
-
Ethan Brierley authored
-
- Jun 07, 2021
-
-
cheme authored
* validation extension in sp_io * need paths * arc impl * missing host function in executor * io to pkdot * decode function. * encode primitive. * trailing tab * multiple patch * fix child trie logic * restore master versionning * bench compact proof size * trie-db 22.3 is needed * line width * split line * fixes for bench (additional root may not be needed as original issue was with empty proof). * revert compact from block size calculation. * New error type for compression. * Adding test (incomplete (failing)). Also lacking real proof checking (no good primitives in sp-trie crate). * There is currently no proof recording utility in sp_trie, removing test. * small test of child root in proof without a child proof. * remove empty test. * remove non compact proof size * Missing revert. * proof method to encode decode.
-