Memory allocator problem with OpenVZ kernel

[ariya]

[email protected] commented:

Which version of PhantomJS are you using?
1.5.0

What steps will reproduce the problem?

1. Download binary or compile phantomjs per instructions.
2. Extract from tar
3. Run example scripts from example/

What is the expected output? What do you see instead?
None of example scripts run
--version and --help run as expected.

Which operating system are you using?

Ubuntu 10.04 (2.6.26-2-openvz-amd64 #1 SMP x86_64 GNU/Linux)

Did you use binary PhantomJS or did you compile it from source?

Have tried both.

Misc information:

root@host # ldd phantomjs
linux-vdso.so.1 => (0x00007fff0dbff000)
libdl.so.2 => /lib/libdl.so.2 (0x00007f1dafcfc000)
libQtWebKit.so.4 => /opt/phantomjs/bin/./../lib/libQtWebKit.so.4 (0x00007f1dae3a1000)
libQtGui.so.4 => /opt/phantomjs/bin/./../lib/libQtGui.so.4 (0x00007f1dad831000)
libQtNetwork.so.4 => /opt/phantomjs/bin/./../lib/libQtNetwork.so.4 (0x00007f1dad4d5000)
libQtCore.so.4 => /opt/phantomjs/bin/./../lib/libQtCore.so.4 (0x00007f1dacf63000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007f1dacd45000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f1daca31000)
libm.so.6 => /lib/libm.so.6 (0x00007f1dac7ae000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f1dac596000)
libc.so.6 => /lib/libc.so.6 (0x00007f1dac213000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1daff08000)
libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x00007f1dabf8d000)
libfontconfig.so.1 => /usr/lib/libfontconfig.so.1 (0x00007f1dabd57000)
librt.so.1 => /lib/librt.so.1 (0x00007f1dabb4f000)
libz.so.1 => /lib/libz.so.1 (0x00007f1dab937000)
libexpat.so.1 => /lib/libexpat.so.1 (0x00007f1dab70e000)

GDB:

gdb phantomjs
GNU gdb (GDB) 7.1-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/phantomjs/bin/phantomjs...(no debugging symbols found)...done.
(gdb) run -X
Starting program: /opt/phantomjs/bin/phantomjs -X
[Thread debugging using libthread_db enabled]
[New Thread 0x7ffff35ef700 (LWP 30228)]
[New Thread 0x7ffff2cee700 (LWP 30229)]

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff72b9ec9 in WTF::OSAllocator::reserveUncommitted(unsigned long, WTF::OSAllocator::Usage, bool, bool) () from /opt/phantomjs/bin/../lib/libQtWebKit.so.4

(gdb) info registers
rax 0x0 0
rbx 0xffffffffffffffff -1
rcx 0xffffffffffffff80 -128
rdx 0x0 0
rsi 0x40000000 1073741824
rdi 0x0 0
rbp 0x7ffff2d15820 0x7ffff2d15820
rsp 0x7fffffffd9d0 0x7fffffffd9d0
r8 0xffffffff 4294967295
r9 0x0 0
r10 0x4022 16418
r11 0x246 582
r12 0x7ffff7bd8c38 140737349782584
r13 0x7ffff75b9168 140737343361384
r14 0x0 0
r15 0x0 0
rip 0x7ffff72b9ec9 0x7ffff72b9ec9 <WTF::OSAllocator::reserveUncommitted(unsigned long, WTF::OSAllocator::Usage, bool, bool)+89>
eflags 0x10246 [ PF ZF IF RF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
(gdb) backtrace

0 0x00007ffff72b9ec9 in WTF::OSAllocator::reserveUncommitted(unsigned long, WTF::OSAllocator::Usage, bool, bool) () from /opt/phantomjs/bin/../lib/libQtWebKit.so.4

1 0x00007ffff7399a13 in JSC::ExecutableAllocator::isValid() const () from /opt/phantomjs/bin/../lib/libQtWebKit.so.4

2 0x00007ffff7308b17 in JSC::ExecutableAllocator::ExecutableAllocator() () from /opt/phantomjs/bin/../lib/libQtWebKit.so.4

3 0x00007ffff73036ac in JSC::JSGlobalData::JSGlobalData(JSC::JSGlobalData::GlobalDataType, JSC::ThreadStackType) () from /opt/phantomjs/bin/../lib/libQtWebKit.so.4

4 0x00007ffff7304e33 in JSC::JSGlobalData::createLeaked(JSC::ThreadStackType) () from /opt/phantomjs/bin/../lib/libQtWebKit.so.4

5 0x00007ffff6a91372 in WebCore::JSDOMWindowBase::commonJSGlobalData() () from /opt/phantomjs/bin/../lib/libQtWebKit.so.4

6 0x00007ffff6a5d896 in WebCore::FrameLoaderClientQt::createDocumentLoader(WebCore::ResourceRequest const&, WebCore::SubstituteData const&) ()

from /opt/phantomjs/bin/../lib/libQtWebKit.so.4

7 0x00007ffff6d7dfa6 in WebCore::FrameLoader::load(WebCore::ResourceRequest const&, WebCore::SubstituteData const&, bool) ()

from /opt/phantomjs/bin/../lib/libQtWebKit.so.4

8 0x00007ffff6a245c6 in QWebFrame::setHtml(QString const&, QUrl const&) () from /opt/phantomjs/bin/../lib/libQtWebKit.so.4

9 0x0000000000418241 in WebPage::WebPage(QObject_, Config const_, QUrl const&) ()

10 0x00000000004121d7 in Phantom::Phantom(QObject*) ()

11 0x0000000000421f54 in main ()

Disclaimer:
This issue was migrated on 2013-03-15 from the project's former issue tracker on Google Code, Issue #481.
🌟   3 people had starred this issue at the time of migration.

[ariya]

[email protected] commented:

(gdb) info threads
3 Thread 0x7ffff2cee700 (LWP 350) 0x00007ffff4ffad20 in QObject::connect(QObject const_, char const_, QObject const_, char const_, Qt::ConnectionType) ()
from /opt/phantomjs/bin/../lib/libQtCore.so.4
2 Thread 0x7ffff35ef700 (LWP 349) 0x00007ffff4c2f85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0

• 1 Thread 0x7ffff7fea720 (LWP 348) 0x00007ffff72b9ec9 in WTF::OSAllocator::reserveUncommitted(unsigned long, WTF::OSAllocator::Usage, bool, bool) ()
  from /opt/phantomjs/bin/../lib/libQtWebKit.so.4

Will provide any additional data if requested.

[ariya]

[email protected] commented:

I wonder if this is related to the openvz kernel.

[ariya]

[email protected] commented:

I'm also seeing the exact same crash on an ubuntu 10.04 64bit openvz vm.

[diwu1989]

[email protected] commented:

I think the problem is that webkit's allocated but uncommitted memory is being counted by OpenVZ, even though the memory isn't actually used.

[diwu1989]

[email protected] commented:

Is javascript core built with a fixed executable allocator?

I checked the code in src\qt\src\3rdparty\webkit\Source\JavaScriptCore\jit\ExecutableAllocatorFixedVMPool.cpp and it looks like if we're building it with a fixed vm pool, it always tried to allocate 1gb of space on x64, so it would explain why we're getting seg faults.

See around line 395 where it sets the page table size

I hacked that file to have x64 use a 32mb page table and I'm recompiling to see if it works.

[diwu1989]

[email protected] commented:

in JavaScriptCore/wtf/Platform.h

This code picks the allocator:
/* Pick which allocator to use; we only need an executable allocator if the assembler is compiled in.
On x86-64 we use a single fixed mmap, on other platforms we mmap on demand. */
#if ENABLE(ASSEMBLER)
#if CPU(X86_64)
#define ENABLE_EXECUTABLE_ALLOCATOR_FIXED 1
#else
#define ENABLE_EXECUTABLE_ALLOCATOR_DEMAND 1
#endif
#endif

It looks like my guess is right, the mmap call in ExecutableAllocatorFixedVMPool is killing the openvz.

So, I guess an alternative fix instead of changing x64 to use a 32mb page table is to just switch to ENABLE_EXECUTABLE_ALLOCATOR_DEMAND 1 for x64

[diwu1989]

[email protected] commented:

Just tested my new build, and it works!

[diwu1989]

[email protected] commented:

I'm going to learn how to use GitHub and send patch fix to the repo, it'll be my first contribution to PhantomJS!

[ariya]

[email protected] commented:

I've got the segfault on x86 system. gdb output for HelloWorld example is following:
gdb phantomjs
GNU gdb (GDB) 7.4
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /usr/bin/phantomjs...(no debugging symbols found)...done.
(gdb) run test.js
Starting program: /usr/bin/phantomjs test.js
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
[New Thread 0xb54b4b40 (LWP 10024)]
[New Thread 0xb4b8bb40 (LWP 10025)]
Hello, world!
[Thread 0xb4b8bb40 (LWP 10025) exited]

Program received signal SIGSEGV, Segmentation fault.
0xb6140d70 in QEventDispatcherQPAPrivate::~QEventDispatcherQPAPrivate() () from /home/kraplax/builds/phantomjs/bin/../lib/libQtGui.so.4
(gdb) backtrace
#0 0xb6140d70 in QEventDispatcherQPAPrivate::~QEventDispatcherQPAPrivate() () from /home/kraplax/builds/phantomjs/bin/../lib/libQtGui.so.4
#1 0xb5ca120b in QObject::~QObject() () from /home/kraplax/builds/phantomjs/bin/../lib/libQtCore.so.4
#2 0xb5c7a246 in QAbstractEventDispatcher::~QAbstractEventDispatcher() () from /home/kraplax/builds/phantomjs/bin/../lib/libQtCore.so.4
#3 0xb5cb8fc3 in QEventDispatcherUNIX::~QEventDispatcherUNIX() () from /home/kraplax/builds/phantomjs/bin/../lib/libQtCore.so.4
#4 0xb614011d in QEventDispatcherQPA::~QEventDispatcherQPA() () from /home/kraplax/builds/phantomjs/bin/../lib/libQtGui.so.4
#5 0xb5c9ce51 in QObjectPrivate::deleteChildren() () from /home/kraplax/builds/phantomjs/bin/../lib/libQtCore.so.4
#6 0xb5ca11d1 in QObject::~QObject() () from /home/kraplax/builds/phantomjs/bin/../lib/libQtCore.so.4
#7 0xb5c8a254 in QCoreApplication::~QCoreApplication() () from /home/kraplax/builds/phantomjs/bin/../lib/libQtCore.so.4
#8 0xb60d57a9 in QApplication::~QApplication() () from /home/kraplax/builds/phantomjs/bin/../lib/libQtGui.so.4
#9 0x080516f8 in main ()

[ariya]

[email protected] commented:

 

 
Metadata Updates

• Status updated: Accepted
• Title updated: Memory allocator problem with OpenVZ kernel

[JamesMGreene]

Is this the same issue that PR #295 was trying to address? If so, @ariya recommended that the fix needed to be upstreamed in WebKit.

I'd be curious to see if it is resolved in Qt 5.0.

[JamesMGreene]

Conversation with @chrislea from Media Temple, who also offer Virtuozzo-based VMs:

me: What is your virtualization platform?
chris.lea: Parallels / Virtuozzo. It's basically the commercial OpenVZ.
me: PhantomJS 💔 OpenVZ. It gets angry about memory allocation....
chris.lea: Yes, we're well aware of that issue. It's the same reason you can't safely run MongoDB on it.
me: Bummer.
chris.lea: It basically takes the concept of virtual memory and throws it out the window.
me: Limitation, or is some sort of fix in the works?
chris.lea: But it gives us amazing density!
Newer versions apparently do some things to get around the issue, but we haven't gotten to upgrading yet as we have tens of thousands of them to do, so it's a major undertaking. It's essentially a limitation, but there's apparently some new syscalls in newer kernels that lets them work around it all to some extent.
Essentially: we know it's a problem, but it affects an extremely tiny number of people that might want to host with us, and the density increase we get from it is worth it economically.
The only people it affects tend to be really technical, like yourself. And most of our userbase simply isn't that technical. Sorry, I hate to be the bearer of bad tidings… like I said, I really love Phantom a ton. It's sooooo cool what you guys are doing!
me: Makes sense.
Thanks for the praise, we like it too! 😛
Always short on time and contributors, though. 😕
chris.lea: Of course.

[stale]

Due to our very limited maintenance capacity (see #14541 for more details), we need to prioritize our development focus on other tasks. Therefore, this issue will be automatically closed. In the future, if we see the need to attend to this issue again, then it will be reopened. Thank you for your contribution!