Large virtual memory (13-20 GB) when running Peertube without --jitless

[pbeyssac]

Version

v18.7.0 (also tested on node14)

Platform

FreeBSD tube 12.3-RELEASE FreeBSD 12.3-RELEASE GENERIC amd64 (also tested on Linux)

Subsystem

No idea (but no idea how to check for this)

What steps will reproduce the bug?

• install and start Peertube (https://github.com/Chocobozzz/PeerTube)
• nodejs will grow up to 13 GB - 20 GB memory (depending on node version/OS)

How often does it reproduce? Is there a required condition?

Same behaviour at every run. Seems easy to reproduce.

What is the expected behavior?

13-20 GB seems to be a lot more than necessary.

Starting with --jitless seems to shrink the memory size significantly to ~1 GB, which looks more reasonable:

Perhaps there's a specific Javascript module with specific code which triggers excessive memory use in the JIT compiler, but I have no idea how to check for this.

What do you see instead?

nodejs grows up to 13 GB - 20 GB memory (depending on node version/OS -- same observations on node14/FreeBSD, Linux, and on other Peertube instances)

Additional information

Posted here on the Peertube forum, where I was advised to come here and ask for help.

https://framacolibri.org/t/high-memory-use-for-node-process-v4-3-0-v4-3-1/16155/7

[tniessen]

@pbeyssac Could you provide the output of cat /proc/74144/status please, replacing 74144 with the actual PID of the node process when its memory usage appears to be high?

[pbeyssac]

@pbeyssac Could you provide the output of cat /proc/74144/status please, replacing 74144 with the actual PID of the node process when its memory usage appears to be high?

Thanks for your reply.

Here goes:

% cat /proc/94670/status
node 94670 67081 94670 67081 pts/5 ctty 1669029587,527265 4,154345 0,557096 kqread 1001 1001 1001,1001 ioc-tube

Note that it is FreeBSD's /proc, which is different from the Linux output, so here's the relevant bit in the man page:

     status  The process status.  This file is read-only and returns a single
             line containing multiple space-separated fields as follows:

             •   command name
             •   process id
             •   parent process id
             •   process group id
             •   session id
             •   device name of the controlling terminal, or a minus sign
                 (“-”) if there is no controlling terminal.
             •   a list of process flags: ctty if there is a controlling
                 terminal, sldr if the process is a session leader, noflags if
                 neither of the other two flags are set.
             •   the process start time in seconds and microseconds, comma
                 separated.
             •   the user time in seconds and microseconds, comma separated.
             •   the system time in seconds and microseconds, comma separated.
             •   the wait channel message
             •   the process credentials consisting of the effective user id
                 and the list of groups (whose first member is the effective
                 group id) all comma separated.
             •   the hostname of the jail in which the process runs, or ‘-’ to
                 indicate that the process is not running within a jail.

Please find attached the memory map of the process as returned by /proc/${pid}/map.
map.txt

Another remark (I'm not sure this is relevant, but just in case): the process I started with --jitless died on an error which looks like a Peertube error, but which never happens when running with JIT. Perhaps a race condition due to the code running at a different speed.

[...:443] 2022-11-21 12:06:17.761 error: uncaughtException: Cannot read properties of undefined (reading 'timeoutType')
TypeError: Cannot read properties of undefined (reading 'timeoutType')
    at _resume (/var/www/peertube/versions/peertube-v4.3.1/node_modules/undici/lib/client.js:1167:29)
    at resume (/var/www/peertube/versions/peertube-v4.3.1/node_modules/undici/lib/client.js:1131:3)
    at connect (/var/www/peertube/versions/peertube-v4.3.1/node_modules/undici/lib/client.js:1116:3) {
  "error": {
    "stack": "TypeError: Cannot read properties of undefined (reading 'timeoutType')\n    at _resume (/var/www/peertube/versions/peertube-v4.3.1/node_modules/undici/lib/client.js:1167:29)\n    at resume (/var/www/peertube/versions/peertube-v4.3.1/node_modules/undici/lib/client.js:1131:3)\n    at connect (/var/www/peertube/versions/peertube-v4.3.1/node_modules/undici/lib/client.js:1116:3)",
    "message": "Cannot read properties of undefined (reading 'timeoutType')"
  },
(...)

[tniessen]

% cat /proc/94670/status
node 94670 67081 94670 67081 pts/5 ctty 1669029587,527265 4,154345 0,557096 kqread 1001 1001 1001,1001 ioc-tube

Note that it is FreeBSD's /proc

This looks like /stat, not /status. /stat does not contain the accumulated memory info.

[pbeyssac]

This looks like /stat, not /status. /stat does not contain the accumulated memory info.

I confirm it is really /proc/${pid}/status on FreeBSD (see the "cat", directly copied from my terminal). FreeBSD /proc differs from Linux and has no /proc/*/stat anyway.

Here /rlimit just in case (but these are static limits, not usage):

% cat /proc/94670/rlimit 
cpu -1 -1
fsize -1 -1
data 34359738368 34359738368
stack 536870912 536870912
core -1 -1
rss -1 -1
memlock 65536 65536
nproc 63680 63680
nofile 1882368 1882368
sbsize -1 -1
vmem -1 -1
npts -1 -1
swap -1 -1
kqueues -1 -1
umtx -1 -1

man page excerpt:

     rlimit  This is a read-only file containing the process current and
             maximum limits.  Each line is of the format rlimit current max,
             with -1 indicating infinity.

Excerpt from ps:

USER       PID  %CPU %MEM      VSZ     RSS TT  STAT  STARTED        TIME COMMAND
peertube 94670   0,0  0,3 13254356  203024  5  S+J   12:19       0:12,45 node: peertube (node)

[tniessen]

Sorry, I was looking at the man page for CentOS, not FreeBSD.

The RSS looks reasonable. Maybe the JIT compiler creates a bunch of large memory mappings thus blowing up the reserved address space in virtual memory, but that's usually not a problem. (For example, V8 might decide to reserve a 4GiB address range for every single WebAssembly instance for performance reasons.)

[bnoordhuis]

Maybe the JIT compiler creates a bunch of large memory mappings

That's what happens. I'm going to guess peertube loads wasm files (undici does, of that I'm sure) and V8 makes large reservations for wasm modules, basically XL guard pages. That's virtual memory though, not RSS, so it's harmless.