Skip to content
Snippets Groups Projects
  • tmpolaczyk's avatar
    Use jobserver in wasm-builder to limit concurrency of spawned cargo processes (#4946) · fee481f9
    tmpolaczyk authored
    
    When building multiple runtimes in parallel, each of them will try to
    use the concurrency set by the parent cargo process. For example, in a
    system with 8 cpu cores, building 3 runtimes in parallel creates 8 * 3
    tasks. This results in the system hanging because of the high cpu and
    memory usage.
    
    This PR allows the substrate_wasm_builder to use the same [jobserver][1]
    as the parent cargo process, making all invocations of cargo share the
    same concurrency pool. So in a system with 8 cores, there will never be
    more than 8 tasks running at the same time.
    
    Implementation roughly based on [cargo][2] but with less unsafe.
    
    This can be tested by telling cargo to use half the cpu cores, like
    `cargo build -j4` in an 8 core machine. Before this PR it will use 100%
    cpu when building 2 runtimes in parallel, after this PR it will always
    use 50%.
    
    [1]:
    https://doc.rust-lang.org/cargo/reference/build-scripts.html#jobserver
    [2]:
    https://github.com/rust-lang/cargo/blob/d1b5f0759eedf5f1126c781c64232856956069ad/src/cargo/util/context/mod.rs#L271
    
    ---------
    
    Co-authored-by: default avatarBastian Köcher <git@kchr.de>
    Unverified
    fee481f9
Code owners
Assign users and groups as approvers for specific file changes. Learn more.