Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OOM/surprising memory consumption with Rust 1.72 #115216

Open
amiyatulu opened this issue Aug 25, 2023 · 12 comments
Open

OOM/surprising memory consumption with Rust 1.72 #115216

amiyatulu opened this issue Aug 25, 2023 · 12 comments
Labels
A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. C-bug Category: This is a bug. I-compilemem Issue: Problems and improvements with respect to memory usage during compilation. O-wasm Target: WASM (WebAssembly), http://webassembly.org/ P-high High priority regression-from-stable-to-stable Performance or correctness regression from one stable version to another. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@amiyatulu
Copy link

amiyatulu commented Aug 25, 2023 •
edited
Loading

I was using trunk serve in a yew app,

Caused by:
  process didn't exit successfully: `/home/amiya/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/rustc --crate-name shivarthu_client --edition=2021 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=80 --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 -C metadata=2fb06b1d42d3f1a3 -C extra-filename=-2fb06b1d42d3f1a3 --out-dir /home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps --target wasm32-unknown-unknown -C incremental=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/incremental -L dependency=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps -L dependency=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/debug/deps --extern futures=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libfutures-9238ff4e317c3aac.rmeta --extern gloo=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libgloo-60896b1ae6a42d49.rmeta --extern gloo_storage=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libgloo_storage-a6fdc8399e55cb57.rmeta --extern gloo_timers=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libgloo_timers-64084cd68bafd650.rmeta --extern hex=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libhex-5e7bde3579f1f1a6.rmeta --extern js_sys=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libjs_sys-e590384aba116f81.rmeta --extern json=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libjson-a742c24145fa9768.rmeta --extern jsonrpsee_core=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libjsonrpsee_core-7b87c5a28395f6a9.rmeta --extern jsonrpsee_wasm_client=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libjsonrpsee_wasm_client-fb5fb48c5225dd1b.rmeta --extern magic_crypt=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libmagic_crypt-4ac60596f6303e71.rmeta --extern codec=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libparity_scale_codec-caa20e3ef61bd74b.rmeta --extern pulldown_cmark=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libpulldown_cmark-2f215fdb4fc370a2.rmeta --extern rand=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/librand-6a1cd1454f5f2ed3.rmeta --extern regex=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libregex-0ea9999c5fd0a43a.rmeta --extern serde=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libserde-77e7121d7f29378b.rmeta --extern serde_json=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libserde_json-0cde3d295cb3341b.rmeta --extern sp_core_hashing=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libsp_core_hashing-4abbca5b36da070b.rmeta --extern stylist=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libstylist-8bf6dde19155c506.rlib --extern subxt=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libsubxt-85e91451cd672d8b.rmeta --extern subxt_signer=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libsubxt_signer-30a7c737b8121a1e.rmeta --extern bip39=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libbip39-a9da80cbe86f172d.rmeta --extern tracing_subscriber=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libtracing_subscriber-cc770f7b154a03a5.rmeta --extern wasm_bindgen=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libwasm_bindgen-238c011dda2fc36f.rmeta --extern wasm_bindgen_futures=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libwasm_bindgen_futures-f9e3a36327e867b0.rmeta --extern web_sys=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libweb_sys-bd30a5ab258c4098.rmeta --extern yew=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libyew-1f77145103937fbf.rmeta --extern yew_hooks=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libyew_hooks-5af603117099c85b.rmeta --extern yew_router=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libyew_router-8ff113f54e98b6bc.rmeta --extern yew_icons=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libyew_icons-f15f701dda4c112c.rmeta --extern yewdux=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/deps/libyewdux-2543950b3ee67c71.rmeta -L native=/home/amiya/Documents/workspace/shivarthu/working_directory/shivarthu-client/target/wasm32-unknown-unknown/debug/build/secp256k1-sys-ceedfda36f3d6248/out` (signal: 9, SIGKILL: kill)
2023-08-25T15:27:32.978449Z ERROR  error
error from HTML pipeline

Caused by:
    0: error from asset pipeline
    1: error during cargo build execution
    2: cargo call returned a bad status
Error: error from HTML pipeline

Caused by:
    0: error from asset pipeline
    1: error during cargo build execution

I have 32 GB of ram, running trunk serve consumes all and terminates without producing build.

I am using rustc 1.72.0 (5680fa1 2023-08-23)
System: Ubuntu 20.04.2 LTS
Processor: 12th Gen Intel® Core™ i7-12650H × 16
Memory: 32 GiB

It also shows error in rustc 1.71.0
cargo build --release successfully works, but error comes with trunk build or trunk serve or just cargo build

Reproducible code link
Please change the src/constants/auth.rs.bak to auth.rs
@amiyatulu amiyatulu added the C-bug Category: This is a bug. label Aug 25, 2023
@rustbot rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Aug 25, 2023
@saethlin saethlin added T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. I-compilemem Issue: Problems and improvements with respect to memory usage during compilation. and removed needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. labels Aug 25, 2023
@junbl
Copy link

junbl commented Aug 25, 2023

Believe I have the same issue. Not using yew, the project is a web server using actix_web. Both local and CI builds appear to be taking all available memory and never finishing.

Just built the same code with 1.72 and 1.71 on a 8CPU 8GB RAM machine: CI builds running 1.72 time out at 2h, where building with 1.71 took 5-10 minutes. Same 2h timeout with a 32CPU 32GB RAM machine.

@SanchithHegde
Copy link
Contributor

We've noticed similar build time differences with Rust 1.72 as well for hyperswitch: our CI builder (used to build Docker release images) got killed because of going out of memory.

Building the same branch on a 32 core, 64 GB RAM machine took more than 1 hour 20 minutes on Rust 1.72 (and we had to cancel it), while building with Rust 1.71 took ~20 minutes.

@saethlin saethlin added the regression-from-stable-to-stable Performance or correctness regression from one stable version to another. label Aug 25, 2023
@rustbot rustbot added the I-prioritize Issue: Indicates that prioritization has been requested for this issue. label Aug 25, 2023
@saethlin saethlin changed the title Memory error while trunk build, signal: 9, SIGKILL: kill OOM/surprising memory consumption with Rust 1.72 Aug 25, 2023
@saethlin
Copy link
Member

@SanchithHegde I have a similarly sized machine and I'm having trouble reproducing your numbers. I cloned the repo you linked and on commit 6247996ddead66086551eef0de8f0b5d678eec27 I'm running cargo build --release on 1.72. The build for me takes ~17 minutes.

I glanced through the repo and I didn't notice any other build options being set, but maybe I missed something. Can you help me reproduce the problem you're having?

@amiyatulu
Copy link
Author

amiyatulu commented Aug 26, 2023 •
edited
Loading

It also shows error in rustc 1.71.0
cargo build --release successfully works in my code, but error comes with trunk build or trunk serve or just cargo build

@amiyatulu
Copy link
Author

Successfully cargo build with

[build]
jobs = 1

in .cargo/config.toml

Not sure why so much memory even if my code base isn't very large.

@gen-xu gen-xu mentioned this issue Aug 26, 2023
Closed
@SanchithHegde
Copy link
Contributor

SanchithHegde commented Aug 27, 2023 •
edited
Loading

@SanchithHegde I have a similarly sized machine and I'm having trouble reproducing your numbers. I cloned the repo you linked and on commit 6247996ddead66086551eef0de8f0b5d678eec27 I'm running cargo build --release on 1.72. The build for me takes ~17 minutes.

I glanced through the repo and I didn't notice any other build options being set, but maybe I missed something. Can you help me reproduce the problem you're having?

Hey @saethlin, sure, I'll include relevant information to help you try and reproduce the build time increase.

Building the same branch on a 32 core, 64 GB RAM machine took more than 1 hour 20 minutes on Rust 1.72 (and we had to cancel it), while building with Rust 1.71 took ~20 minutes.

The exact repository and branch that was compiled in the above case is not public, and is a wrapper over the linked hyperswitch repository, with the hyperswitch repository being the major contributor to the code compiled (~80%).

However, I was still able to reproduce the increase in build time with the corresponding branch on the hyperswitch repository. The command that was run in each case was the following Docker command, with a suitable Docker image tag:

$ docker build --build-arg BINARY=router --tag <image_tag> --file Dockerfile .

This in turn runs the following cargo command:

$ cargo build --release --features release

The first row in the table contains the branch name and the commit hash being compiled, and the first column contains the Rust version used to compile each time.

main (6247996) rust-1.72-build-time-repro (3f7ddd4)
Rust 1.71 13m 37s 13m 42s
Rust 1.72 13m 46s 19m 56s

By the way, to build with 1.71, I changed the first line in the Dockerfile from:

FROM rust:slim-bookworm as builder

to:

FROM rust:1.71-slim-bookworm as builder

I'll also include some details about the host machine used:

  • Distribution: Ubuntu 22.04.3 LTS x86_64
  • Kernel: 5.15.0-79-generic
  • CPU: 13th Gen Intel i9-13900K
  • Memory: 64070MiB
  • docker --version: Docker version 24.0.5, build ced0996

Hope this helps, let me know if you need any more information from me.

@saethlin saethlin mentioned this issue Aug 28, 2023
Merged
@apiraino
Copy link
Contributor

WG-prioritization assigning priority (Zulip discussion).

@rustbot label -I-prioritize +P-high

@rustbot rustbot added P-high High priority and removed I-prioritize Issue: Indicates that prioritization has been requested for this issue. labels Aug 29, 2023
@wesleywiser
Copy link
Member

Thanks for sharing your code @amiyatulu! Unfortunately, I wasn't able to observe much of a difference in memory usage between 1.71, 1.72 and current nightly:

1.71 
Finished dev [unoptimized + debuginfo] target(s) in 4m 51s
        Command being timed: "trunk build"
        User time (seconds): 501.92
        System time (seconds): 45.32
        Percent of CPU this job got: 181%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 5:01.29
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 31958488
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 461015
        Minor (reclaiming a frame) page faults: 12335945
        Voluntary context switches: 303592
        Involuntary context switches: 4159
        Swaps: 0
        File system inputs: 7377432
        File system outputs: 14170592
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
1.72 
Finished dev [unoptimized + debuginfo] target(s) in 5m 04s
        Command being timed: "trunk build"
        User time (seconds): 461.05
        System time (seconds): 41.95
        Percent of CPU this job got: 160%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 5:12.77
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 31898432
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 571925
        Minor (reclaiming a frame) page faults: 13081495
        Voluntary context switches: 429478
        Involuntary context switches: 11022
        Swaps: 0
        File system inputs: 9868848
        File system outputs: 14188016
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
1.74.0-nightly (59a8294 2023-08-30) 
Finished dev [unoptimized + debuginfo] target(s) in 4m 33s
        Command being timed: "trunk build"
        User time (seconds): 462.40
        System time (seconds): 40.69
        Percent of CPU this job got: 175%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 4:46.39
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 31975988
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 467899
        Minor (reclaiming a frame) page faults: 11890715
        Voluntary context switches: 309180
        Involuntary context switches: 7680
        Swaps: 0
        File system inputs: 7035752
        File system outputs: 14475576
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Even in 1.71, the build is right on the cusp of using ~32gb of RAM so perhaps it's just taking slightly more in 1.72 and getting OOM killed.

@amiyatulu
Copy link
Author

amiyatulu commented Sep 1, 2023 •
edited
Loading

@wesleywiser Thanks for checking out. Have you tried the commit. It works fine with jobs = 1, but gives build error on both 1.71 and 1.72 without it.

@Skgland
Copy link
Contributor

Skgland commented May 28, 2024 •
edited
Loading

I am having a similar problem when compiling scryer-prolog for wasm32-unknown-unknown at mthom/scryer-prolog#2413 (the linked PR is not directly related, all CI currently failes due to the wasm32 build)
I also made a asciinema recording of the compilation with htop side-by-side, where the peak usage is around 11:41 with 13.6 GB RES memory usage.

$> cargo --version
cargo 1.78.0 (54d8815d0 2024-03-26)
$> rustc --version
rustc 1.78.0 (9b00956e5 2024-04-29)
$> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy

On Windows 11 using WSL 2

@saethlin
Copy link
Member

saethlin commented May 28, 2024 •
edited
Loading

Ah! That reproduces easily.

perf top shows this while memory is growing:

 60.99%  libLLVM.so.18.1-rust-1.80.0-nightly  [.] llvm::SmallPtrSetImplBase::insert_imp_big(void const*)
 38.69%  libLLVM.so.18.1-rust-1.80.0-nightly  [.] (anonymous namespace)::WebAssemblyFixIrreducibleControlFlow::processRegion(llvm::MachineBasicBlock*, llvm::SmallPtrSet<llvm::MachineBasicBlock*, 4u>&, llvm::MachineFunction&)

@rustbot label +A-LLVM +O-wasm

@rustbot rustbot added A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. O-wasm Target: WASM (WebAssembly), http://webassembly.org/ labels May 28, 2024
@Skgland
Copy link
Contributor

Skgland commented May 31, 2024 •
edited
Loading

Based on that it looks like this/my problem might be related to llvm/llvm-project#47793

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. C-bug Category: This is a bug. I-compilemem Issue: Problems and improvements with respect to memory usage during compilation. O-wasm Target: WASM (WebAssembly), http://webassembly.org/ P-high High priority regression-from-stable-to-stable Performance or correctness regression from one stable version to another. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@wesleywiser @amiyatulu @Skgland @apiraino @saethlin @SanchithHegde @junbl @rustbot