SIGSEGV when using StringBuilder with jemalloc

[alamb]

Note: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-8202

I have a Rust app which is just appending strings into many StringBuilders.  I tried using jemalloc and the app crashes with SIGSEGV (Address boundary error)

 

rust-lldb backtrace:

{{* frame #0: 0x00000001004073f1 memoird`_rjem_mallocx at sz.h:158:18 [opt]}}
{{ frame #1: 0x00000001004073e3 memoird`_rjem_mallocx [inlined] sz_s2u_lookup(size=<unavailable>) at sz.h:238 [opt]}}
{{ frame #2: 0x00000001004073e3 memoird`_rjem_mallocx [inlined] sz_s2u(size=<unavailable>) at sz.h:252 [opt]}}
{{ frame #3: 0x00000001004073d6 memoird`_rjem_mallocx [inlined] sz_sa2u(size=<unavailable>, alignment=64) at sz.h:283 [opt]}}
{{ frame #4: 0x00000001004073ac memoird`_rjem_mallocx [inlined] imalloc_body at jemalloc.c:1841 [opt]}}
{{ frame #5: 0x0000000100407394 memoird`_rjem_mallocx [inlined] imalloc(sopts=<unavailable>, dopts=<unavailable>) at jemalloc.c:2005 [opt]}}
{{ frame #6: 0x0000000100407345 memoird`_rjem_mallocx(size=<unavailable>, flags=<unavailable>) at jemalloc.c:2588 [opt]}}
{{ frame #7: 0x0000000100370187 memoird`arrow::array::builder::ListBuilder$LT$T$GT$::new::h16819112466ced47 [inlined] alloc::alloc::alloc_zeroed::hc53d8d0d6ed944ef(layout=<unavailable>) at alloc.rs:165:4 [opt]}}
{{ frame #8: 0x000000010037017a memoird`arrow::array::builder::ListBuilder$LT$T$GT$::new::h16819112466ced47 at memory.rs:29 [opt]}}
{{ frame #9: 0x000000010037017a memoird`arrow::array::builder::ListBuilder$LT$T$GT$::new::h16819112466ced47 at buffer.rs:419 [opt]}}
{{ frame #10: 0x000000010037017a memoird`arrow::array::builder::ListBuilder$LT$T$GT$::new::h16819112466ced47 at builder.rs:138 [opt]}}
{{ frame #11: 0x0000000100370169 memoird`arrow::array::builder::ListBuilder$LT$T$GT$::new::h16819112466ced47(values_builder=PrimitiveBuilder<arrow::datatypes::UInt8Type> {}}
{{values_builder: BufferBuilder<arrow::datatypes::UInt8Type> {}}
{{buffer: MutableBuffer {}}
{{data: &0x100b96000,}}
{{len: 0,}}
{{capacity: 8192}}
{{},}}
{{len: 0,}}
{{_marker: PhantomData<arrow::datatypes::UInt8Type> {}}

{{}}}
{{},}}
{{bitmap_builder: BufferBuilder<arrow::datatypes::BooleanType> {}}
{{buffer: MutableBuffer {}}
{{data: &0x100be3000,}}
{{len: 0,}}
{{capacity: 1024}}
{{},}}
{{len: 0,}}
{{_marker: PhantomData<arrow::datatypes::BooleanType> {}}

{{}}}
{{}}}
{{}) at builder.rs:368 [opt]}}
{{ frame #12: 0x0000000100370d4c memoird`arrow::array::builder::BinaryBuilder::new::h8f11851f0863e756(capacity=<unavailable>) at builder.rs:670:21 [opt]}}

[velvia]

@alamb are we moving off of the Apache JIRA for arrow-rs issues?

[alamb]

@velvia yes that is the plan -- we have moved to use github issues instead of Apache JIRA.

Here is the proposal with more details: https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit#heading=h.kv1hwbhi3cmi

[velvia]

Awesome plan. What is the pyarrow testing stuff for? I was trying out Jorge’s datafusion-python and made me think of that. There are some rough edges with pyarrow -> Rust integration (but I’ll carry that convo somewhere else)

…

On Apr 29, 2021, at 3:00 AM, Andrew Lamb ***@***.***> wrote: @velvia <https://github.com/velvia> yes that is the plan -- we have moved to use github issues instead of Apache JIRA. Here is the proposal with more details: https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit#heading=h.kv1hwbhi3cmi <https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit#heading=h.kv1hwbhi3cmi> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#62 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIDPW5BWV62ZW2FJX5WJETTLEU5JANCNFSM43SZEHMA>.

[alamb]

I *think* the py-arrow stuff is related to flight integration testing, perhaps?

…

On Sat, May 1, 2021 at 3:07 AM Evan Chan ***@***.***> wrote: Awesome plan. What is the pyarrow testing stuff for? I was trying out Jorge’s datafusion-python and made me think of that. There are some rough edges with pyarrow -> Rust integration (but I’ll carry that convo somewhere else) > On Apr 29, 2021, at 3:00 AM, Andrew Lamb ***@***.***> wrote: > > > @velvia <https://github.com/velvia> yes that is the plan -- we have moved to use github issues instead of Apache JIRA. > > Here is the proposal with more details: https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit#heading=h.kv1hwbhi3cmi < https://docs.google.com/document/d/1TyrUP8_UWXqk97a8Hvb1d0UYWigch0HAephIjW7soSI/edit#heading=h.kv1hwbhi3cmi > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub < #62 (comment)>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAIDPW5BWV62ZW2FJX5WJETTLEU5JANCNFSM43SZEHMA >. > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#62 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADXZMII4IMNULPQYSENHFDTLOSDDANCNFSM43SZEHMA> .

[vertexclique]

This is because jemalloc behaves undefined when allocation length is requested as 0 in non-standard allocation API that it has. Since that API doesn't check aligned access, it internally rejects these. Builder requests for this allocation should check allocation before doing an allocation request. :)

[tustvold]

I'm not entirely sure when this was fixed, but buffer allocation now checks for 0 length allocations