Future of jemallocator

[gnzlbg]

I think that we should move jemallocator into the nursery, and have rustc use it instead of the current liballoc_jemalloc implementation.

That would make it much easier for us to contribute to liballoc_jemalloc, add benchmarks, etc.

For example, I have my own fork of jemallocator here: https://github.com/gnzlbg/jemallocator where I have added benchmarks (in the gbg branches) that I would like to contribute to liballoc_jemalloc upstream, but then running does benchmarks becomes really painful (typically requires at least a stage1 compiler, etc.).

Another thing that the current jemallocator setup easily allows is to improve jemalloc itself. For example, in the gbg3 branch I test jemallocator with my own jemalloc fork and by slightly tweaking jemallocs API I am able to to show some nice speed-ups.

I wouldn't have been able to do any of this if I would have had to hack in liballoc_jemalloc within the compiler itself.

[alexcrichton]

Eventually the plan is to remove the jemalloc crate from the standard library in favor of "some crate on crates.io". I was planning on this crate becoming that "some crate" barring any other developments. In the meantime I don't think there's much to gain from active development in libstd itself, but improvements/benchmarks would be most welcome here!

[gnzlbg]

Eventually the plan is to remove the jemalloc crate from the standard library in favor of "some crate on crates.io". I was planning on this crate becoming that "some crate" barring any other developments.

Which steps would be required to make this change, and what could I do to speed it up?

[alexcrichton]

We'd need to stabilize rust-lang/rust#27389 for global allocators (already in FCP) and rust-lang/rust#32838 for defining an allocator (now in FCP). To help speed it up I will say the objection to continue to figure out the excess-related APIs on rust-lang/rust#32838 probably isn't helping the speed of stabilizing...

[gnzlbg]

Is compiling on stable Rust a requirement for using jemallocator instead of liballoc_jemalloc in rustc ?

[sfackler]

It's a requirement for switching the default allocator for binaries to alloc_system, but not for using jemallocator in rustc AFAIK.

[gnzlbg]

It's a requirement for switching the default allocator for binaries to alloc_system,

How come?

[sfackler]

It's not really reasonable to switch off of jemalloc providing people with the means to switch back.

[gnzlbg]

Makes sense. I thought there might just be a compiler flag for selecting the allocator, but the allocator API is a much better option.

[SimonSapin]

Update: (for future readers, since people who commented so far are already aware :))

#[global_allocator] is now stable. rust-lang/rust#36963 tracks the removal of liballoc_jemalloc, which might be blocked on switching rustc to jemallocator: rust-lang/rust#51038

[gnzlbg]

Now making progress in: rust-lang/rust#55238

EDIT: progress was made!