Skip to content
Unverified Commit fee481f9 authored by tmpolaczyk's avatar tmpolaczyk Committed by GitHub
Browse files

Use jobserver in wasm-builder to limit concurrency of spawned cargo processes (#4946)



When building multiple runtimes in parallel, each of them will try to
use the concurrency set by the parent cargo process. For example, in a
system with 8 cpu cores, building 3 runtimes in parallel creates 8 * 3
tasks. This results in the system hanging because of the high cpu and
memory usage.

This PR allows the substrate_wasm_builder to use the same [jobserver][1]
as the parent cargo process, making all invocations of cargo share the
same concurrency pool. So in a system with 8 cores, there will never be
more than 8 tasks running at the same time.

Implementation roughly based on [cargo][2] but with less unsafe.

This can be tested by telling cargo to use half the cpu cores, like
`cargo build -j4` in an 8 core machine. Before this PR it will use 100%
cpu when building 2 runtimes in parallel, after this PR it will always
use 50%.

[1]:
https://doc.rust-lang.org/cargo/reference/build-scripts.html#jobserver
[2]:
https://github.com/rust-lang/cargo/blob/d1b5f0759eedf5f1126c781c64232856956069ad/src/cargo/util/context/mod.rs#L271

---------

Co-authored-by: default avatarBastian Köcher <[email protected]>
parent 2982bf3e
Pipeline #486144 waiting for manual action with stages
in 1 hour, 8 minutes, and 32 seconds
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment