ICE: MemDecoder exhausted

[erikh]

Code

I do not have a minimum verifiable example. It's hard to decipher this backtrace. However, I have erikh/megalock@daed9c0 which should demonstrate this error on 1.75.0.

EDIT: sorry, I had the url syntax wrong. That link should point at the correct commit.

Meta

rustc --version --verbose:

rustc 1.75.0 (82e1608df 2023-12-21)
binary: rustc
commit-hash: 82e1608dfa6e0b5569232559e3d385fea5a93112
commit-date: 2023-12-21
host: x86_64-unknown-linux-gnu
release: 1.75.0
LLVM version: 17.0.6

Error output

thread 'rustc' panicked at compiler/rustc_serialize/src/opaque.rs:257:9:
MemDecoder exhausted
stack backtrace:
   0:     0x7fc9fbea462c - std::backtrace_rs::backtrace::libunwind::trace::ha637c64ce894333a
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/../../backtrace/src/backtrace/libunwi
nd.rs:104:5
   1:     0x7fc9fbea462c - std::backtrace_rs::backtrace::trace_unsynchronized::h47f62dea28e0c88d
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/../../backtrace/src/backtrace/mod.rs:
66:5
   2:     0x7fc9fbea462c - std::sys_common::backtrace::_print_fmt::h9eef0abe20ede486
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x7fc9fbea462c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hed7f999df88cc644
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x7fc9fbef7630 - core::fmt::rt::Argument::fmt::h1539a9308b8d058d
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/fmt/rt.rs:142:9
   5:     0x7fc9fbef7630 - core::fmt::write::h3a39390d8560d9c9
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/fmt/mod.rs:1120:17
   6:     0x7fc9fbe9854f - std::io::Write::write_fmt::h5fc9997dfe05f882
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/io/mod.rs:1762:15
   7:     0x7fc9fbea4414 - std::sys_common::backtrace::_print::h894006fb5c6f3d45
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x7fc9fbea4414 - std::sys_common::backtrace::print::h23a2d212c6fff936
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x7fc9fbea70a7 - std::panicking::default_hook::{{closure}}::h8a1d2ee00185001a
  10:     0x7fc9fbea6e0f - std::panicking::default_hook::h6038f2eba384e475
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:292:9
  11:     0x7fc9f8d37190 - std[409886f6357001f0]::panicking::update_hook::<alloc[c1b021ad36e35877]::boxed::Box<rustc_driver_impl[7d23c5
715ff089db]::install_ice_hook::{closure#0}>>::{closure#0}
  12:     0x7fc9fbea77e8 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h1f8f335eaa9cfaee
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/alloc/src/boxed.rs:2021:9
  13:     0x7fc9fbea77e8 - std::panicking::rust_panic_with_hook::h2b5517d590cab22e
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:783:13
  14:     0x7fc9fbea7509 - std::panicking::begin_panic_handler::{{closure}}::h233112c06e0ef43e
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:649:13
  15:     0x7fc9fbea4af6 - std::sys_common::backtrace::__rust_end_short_backtrace::h6e893f24d7ebbff8
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:170:18
  16:     0x7fc9fbea72a2 - rust_begin_unwind
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:645:5
  17:     0x7fc9fbef3d15 - core::panicking::panic_fmt::hbf0e066aabfa482c
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:72:14
  18:     0x7fc9f95eaf9a - <rustc_serialize[5d0f213b508bf701]::opaque::MemDecoder>::decoder_exhausted
  19:     0x7fc9fb03c696 - <rustc_middle[aca4860da4e5a967]::query::on_disk_cache::OnDiskCache>::new
  20:     0x7fc9fae3caae - rustc_incremental[8fad7206137c3013]::persist::load::load_query_result_cache
  21:     0x7fc9fae3ccc8 - rustc_interface[fbb0cb4be6c0ba34]::passes::create_global_ctxt
  22:     0x7fc9faf201df - <rustc_interface[fbb0cb4be6c0ba34]::queries::Queries>::global_ctxt
  23:     0x7fc9fae06dc0 - rustc_interface[fbb0cb4be6c0ba34]::interface::run_compiler::<core[21cdcf8e8af4c2d9]::result::Result<(), rust
c_span[3d5dc97049ad8d50]::ErrorGuaranteed>, rustc_driver_impl[7d23c5715ff089db]::run_compiler::{closure#1}>::{closure#0}
  24:     0x7fc9fae0215b - std[409886f6357001f0]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[fbb0cb4be6c0ba3
4]::util::run_in_thread_with_globals<rustc_interface[fbb0cb4be6c0ba34]::interface::run_compiler<core[21cdcf8e8af4c2d9]::result::Result<
(), rustc_span[3d5dc97049ad8d50]::ErrorGuaranteed>, rustc_driver_impl[7d23c5715ff089db]::run_compiler::{closure#1}>::{closure#0}, core[
21cdcf8e8af4c2d9]::result::Result<(), rustc_span[3d5dc97049ad8d50]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[21cdcf8e8af4c2d9]
::result::Result<(), rustc_span[3d5dc97049ad8d50]::ErrorGuaranteed>>
  25:     0x7fc9fae01fb3 - <<std[409886f6357001f0]::thread::Builder>::spawn_unchecked_<rustc_interface[fbb0cb4be6c0ba34]::util::run_in_
thread_with_globals<rustc_interface[fbb0cb4be6c0ba34]::interface::run_compiler<core[21cdcf8e8af4c2d9]::result::Result<(), rustc_span[3d
5dc97049ad8d50]::ErrorGuaranteed>, rustc_driver_impl[7d23c5715ff089db]::run_compiler::{closure#1}>::{closure#0}, core[21cdcf8e8af4c2d9]
::result::Result<(), rustc_span[3d5dc97049ad8d50]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[21cdcf8e8af4c2d9]::result::Result<
(), rustc_span[3d5dc97049ad8d50]::ErrorGuaranteed>>::{closure#1} as core[21cdcf8e8af4c2d9]::ops::function::FnOnce<()>>::call_once::{shi
m:vtable#0}
  26:     0x7fc9fbeb16a5 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc7eafaff61e32df9
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/alloc/src/boxed.rs:2007:9
  27:     0x7fc9fbeb16a5 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h6ba4a5de48dd2304
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/alloc/src/boxed.rs:2007:9
  28:     0x7fc9fbeb16a5 - std::sys::unix::thread::Thread::new::thread_start::he469335aef763e45
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys/unix/thread.rs:108:17
  29:     0x7fc9f60aa9eb - <unknown>
  30:     0x7fc9f612e7cc - <unknown>
  31:                0x0 - <unknown>

error: the compiler unexpectedly panicked. this is a bug.

Backtrace

The backtrace comes from rustc itself.

Just doing my part, looks like my repo did something to hose the compiler. Next steps I'm happy to help with, but the repo is probably the easiest way to investigate this independently. I'm sorry I'm not more helpful.

[erikh]

Since I don't know if it's relevant, here's a ldd megalock on a recent optimized bin because it does link against numerous C shared libraries:

	linux-vdso.so.1 (0x00007fff2f555000)
	libxkbcommon.so.0 => /usr/lib/libxkbcommon.so.0 (0x00007f97388a4000)
	libxkbcommon-x11.so.0 => /usr/lib/libxkbcommon-x11.so.0 (0x00007f973889b000)
	libcairo.so.2 => /usr/lib/libcairo.so.2 (0x00007f9738762000)
	libxcb.so.1 => /usr/lib/libxcb.so.1 (0x00007f9738737000)
	libxcb-randr.so.0 => /usr/lib/libxcb-randr.so.0 (0x00007f9738725000)
	libxcb-image.so.0 => /usr/lib/libxcb-image.so.0 (0x00007f973871f000)
	libxcb-xrm.so.0 => /usr/lib/libxcb-xrm.so.0 (0x00007f9738715000)
	libxcb-xkb.so.1 => /usr/lib/libxcb-xkb.so.1 (0x00007f97386f6000)
	libxcb-util.so.1 => /usr/lib/libxcb-util.so.1 (0x00007f97386ef000)
	libpam.so.0 => /usr/lib/libpam.so.0 (0x00007f97386de000)
	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f97386b9000)
	libm.so.6 => /usr/lib/libm.so.6 (0x00007f97385cc000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f97383e8000)
	libz.so.1 => /usr/lib/libz.so.1 (0x00007f97383ce000)
	libpng16.so.16 => /usr/lib/libpng16.so.16 (0x00007f9738394000)
	libfontconfig.so.1 => /usr/lib/libfontconfig.so.1 (0x00007f9738344000)
	libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x00007f9738276000)
	libX11.so.6 => /usr/lib/libX11.so.6 (0x00007f9738133000)
	libXext.so.6 => /usr/lib/libXext.so.6 (0x00007f973811c000)
	libXrender.so.1 => /usr/lib/libXrender.so.1 (0x00007f973810f000)
	libxcb-render.so.0 => /usr/lib/libxcb-render.so.0 (0x00007f9738100000)
	libxcb-shm.so.0 => /usr/lib/libxcb-shm.so.0 (0x00007f97380fb000)
	libpixman-1.so.0 => /usr/lib/libpixman-1.so.0 (0x00007f973804e000)
	libXau.so.6 => /usr/lib/libXau.so.6 (0x00007f9738049000)
	libXdmcp.so.6 => /usr/lib/libXdmcp.so.6 (0x00007f973803f000)
	libaudit.so.1 => /usr/lib/libaudit.so.1 (0x00007f9738015000)
	/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f97389d7000)
	libexpat.so.1 => /usr/lib/libexpat.so.1 (0x00007f9737fea000)
	libbz2.so.1.0 => /usr/lib/libbz2.so.1.0 (0x00007f9737fd7000)
	libharfbuzz.so.0 => /usr/lib/libharfbuzz.so.0 (0x00007f9737ec9000)
	libbrotlidec.so.1 => /usr/lib/libbrotlidec.so.1 (0x00007f9737eb8000)
	libcap-ng.so.0 => /usr/lib/libcap-ng.so.0 (0x00007f9737eb0000)
	libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0x00007f9737d64000)
	libgraphite2.so.3 => /usr/lib/libgraphite2.so.3 (0x00007f9737d41000)
	libbrotlicommon.so.1 => /usr/lib/libbrotlicommon.so.1 (0x00007f9737d1e000)
	libpcre2-8.so.0 => /usr/lib/libpcre2-8.so.0 (0x00007f9737c81000)

[erikh]

Upgrading to the latest version of my crates, proc_macro2 was in there as well as syn, seemed to have corrected the error. Could also be lurking somewhere in the build cache. If you want that I suppose I can provide it somewhere.

Not sure what to do with this bug now, but feel free to close it if you no longer feel it's relevant.

[Noratrieb]

@saethlin does your PR fix this?

[saethlin]

Hm, it looks like the problem here is that the OnDiskCache footer is off the end of the file. When that happens due to out-of-disk, people tend to notice that they are out of disk. @erikh Am I correct in assuming you had some disk space available when this crash happened?

If this is an out-of-disk, it was probably "solved" by #117301 then will be actually solved by #119510. But if isn't an out-of-disk situation, this is probably an incremental compilation bug. The build directory state at the time the crash happened would be useful, but if the crash doesn't recur anymore on your end then the problematic state in the build directory is probably gone. In that case, this issue is still useful to say that something is wrong here, but that's kind of all.

[erikh]

@saethlin correct, I had about 900GB free on this mountpoint at the time of compilation.

[erikh]

It does also make sense that my build cache was corrupt; refreshing crates seemed to correct the issue. Maybe a truncated object write or something? Just throwing things at the wall here.

Either way, my personal issue is resolved, but I figure this is probably worth keeping around for resolution on a more reasonable time frame.

[saethlin]

Crashes like this in incremental compilation are usually due to subtle internal things like a Hash impl used in the incremental compilation cache that accidentally has a high collision rate in some rare situation.

[erikh]

Ah, interesting. So heat death of the universe before this bug probably gets closed, eh? :)

I'll leave ya to it.