Node v4.2.6 - FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory #1868

busticated · 2016-02-02T19:57:33Z

hey there -

i'm hitting the following error after upgrading to node v4 LTS.

02 02 2016 11:13:08.092:INFO [karma]: Karma v0.13.19 server started at http://localhost:9876/
02 02 2016 11:13:08.122:INFO [launcher]: Starting browser PhantomJS

<--- Last few GCs --->

    22555 ms: Scavenge 1441.4 (1502.2) -> 1441.4 (1502.2) MB, 1.6 / 0 ms (+ 2.4 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
    22610 ms: Mark-sweep 1441.4 (1502.2) -> 1441.2 (1502.2) MB, 55.5 / 0 ms (+ 3.6 ms in 2 steps since start of marking, biggest step 2.4 ms) [last resort gc].
    22664 ms: Mark-sweep 1441.2 (1502.2) -> 1441.2 (1502.2) MB, 54.0 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x3acf8b8b4629 <JS Object>
    1: exec [native regexp.js:~85] [pc=0x1ab48201d158] (this=0x3477de668ed1 <JS RegExp>,k=0x1dc821cffa39 <String[106]: Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1>)
    2: parse [/path/to/my/project/node_modules/karma/node_modules/useragent/index.js:427] [pc=0x1ab4821e1cbf] (this=0x276f61054869 <JS Funct...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
Abort trap: 6`

Likewise, i see a similar error when running tests in chrome:

>>karma start
02 02 2016 11:49:26.366:WARN [karma]: No captured browser, open http://localhost:9876/
02 02 2016 11:49:26.447:INFO [karma]: Karma v0.13.19 server started at http://localhost:9876/
02 02 2016 11:49:26.455:INFO [launcher]: Starting browser Chrome

<--- Last few GCs --->

21051 ms: Mark-sweep 1429.9 (1488.2) -> 1429.4 (1488.2) MB, 26.3 / 0 ms (+ 124.4 ms in 20 steps since start of marking, biggest step 82.2 ms) [allocation failure] [GC in old space requested].
21163 ms: Mark-sweep 1429.4 (1488.2) -> 1429.4 (1488.2) MB, 110.6 / 0 ms [last resort gc].
21298 ms: Mark-sweep 1429.4 (1488.2) -> 1429.3 (1488.2) MB, 135.5 / 0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x24bd825b4629 <JS Object>
    2: decodeURIComponent(aka decodeURIComponent) [native uri.js:248] [pc=0x2575a442da41] (this=0x24bd825041b9 <undefined>,W=0x111d0ab40cd9 <String[2]: id>)
    3: /* anonymous */(aka /* anonymous */) [querystring.js:83] [pc=0x2575a4426aee] (this=0x24bd825041b9 <undefined>,s=0x111d0ab40cd9 <String[2]: id>,decodeSpaces=0x24bd825041b9 <undefined>)
    4: arguments adaptor frame: 1->2
    5: decod...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
Abort trap: 6

My environment:

node: v4.2.6
phantomjs: v1.9.8
karma: v0.13.19
karma-phantomjs-launcher: v0.2.3
karma-chrome-launcher: v0.2.2
karma-chai: v0.1.0
karma-mocha: v0.2.1
karma-sinon: v1.0.4
karma-sourcemap-loader: v0.3.7
karma-webpack: v1.7.0
webpack: v1.12.12

I tried updating to the latest phantomjs but the error persisted - upgrades:

phantomjs phantomjs-prebuilt: v2.1.1
karma-phantomjs-launcher: v1.0.0

Running under the old configuration works fine:

node: v0.10.41
phantomjs: v1.9.8
karma: v0.13.19
karma-phantomjs-launcher: v0.2.3
karma-chrome-launcher: v0.2.2
karma-chai: v0.1.0
karma-mocha: v0.2.1
karma-sinon: v1.0.4
karma-sourcemap-loader: v0.3.7
karma-webpack: v1.7.0
webpack: v1.12.12

I'm on os-x v10.10.5

The text was updated successfully, but these errors were encountered:

busticated · 2016-02-03T17:30:49Z

more info: if i limit the number of test files - e.g.

instead of:

    files: [
        'public/js/**/*.test.js'
    ]

i use:

    files: [
        'public/js/util/*.test.js'
    ]

both chrome and phantomjs run the reduced test suite fine. not headline news i realize but still :-D

is there a way to set the V8 option --max_old_space_size to increase the max memory limit of the karma process? i'm not seeing anything obvious in the docs.

edit: fwiw, looks like i can run ~800 ~~files~~ tests across 87 files before hitting node's 1.7gb memory limit and falling over

busticated · 2016-02-04T20:09:50Z

fwiw, my work-around is to run karma as a child process using the public api - this allows me to set the --max_old_space_size flag (whereas running using karma's CLI does not).

var karma = fork('./karma-worker', {
    execArgv: ['--max_old_space_size=4096'] // increase v8's available memory to 4gb
});

karma.send({ type: 'start' /*, config: { ... } */});

and then in the worker:

process.on('message', function(msg){
    if (msg.type === 'start'){
        return new KarmaServer(config, done).start();
    }
});

function done(code){
    if (process.connected){
        process.send({ type: 'done', code: code });
    }
    process.exit(code);
}

there's some additional fiddly bits around error / disconnect handling, etc but this is the basic gist. only thing i'm missing at the moment is colored output - for whatever reason the only messages that retain their color (green) are 'karma started', 'starting browser', and 'connected' messages (the first 3 you typically see during a run).

dignifiedquire · 2016-02-04T20:14:08Z

I'm not sure but have you tried karma start -- --max_old_space_size=4096 ? If not I'd be happy to accept a PR that allows for that possibility

busticated · 2016-02-04T20:15:00Z

re: memory consumption - comparing node v4 LTS to v0.10.41 i see similar memory use. v0.10.41 maxes out at ~1.9gb, node v4 at ~2.3gb. in both cases, that seems really high to me (as a reference point, my build task - LESS & webpack - uses ~100mb of memory). the really weird thing is that v4 crashes but v0.10 does not yet my understanding is that both use the same default memory limit.

¯_(ツ)_/¯

busticated · 2016-02-04T20:21:20Z

@dignifiedquire thanks for taking a look.

have you tried karma start -- --max_old_space_size=4096 ?

yep, it doesn't work (same for setting it via config.client.args)

I'd be happy to accept a PR that allows for that possibility

i poked at that approach a bit but it seemed like a dead-end - in part b/c of this.

dignifiedquire · 2016-02-04T20:22:08Z

A lot of things have changed in the v8 versions between 0.10 and 4.0 so it might be that sth there is the result of the crashing in one point and working fine in the other.

From the file count these limits should never be reached (I've run karma on larger projects, e.g. take a look at the angular main source) so it seems to be that there is probably some memory leak/bug why this is happening :/

dignifiedquire · 2016-02-04T20:23:12Z

Could you try reducing the amount of files/tests you are loading and see if there is a limit/specific file that triggers this?

busticated · 2016-02-04T20:28:38Z

@dignifiedquire yep, can do. buried in my note above --> i can run ~800 files tests across 87 files before hitting the limit and crashing. that was just a quick try though. i'll be more deliberate and see what i see :)

dignifiedquire · 2016-02-04T20:30:50Z

What's the target file count? (test files and source files being loaded)

busticated · 2016-02-04T21:29:40Z

@dignifiedquire just looking at chrome's network tab while running in the debug view, i see 232 files / 458mb(!). this includes helpers like chai.js, mocha.js, etc but is mostly local source. (NOTE: this is while running to completion w/ --max_old_space_size=4096)

it's not uncommon to see individual test files > 4mb since these are built by webpack and include external dependencies (e.g. react, etc) as well as sourcemaps. that certainly seems like something i can tune a bit better - maybe by loading common deps in a single file or something.

edit: like this (>_<)

havenchyk · 2016-07-19T09:39:53Z

@dignifiedquire do you need any additional info? I faced the same issue

ianks · 2016-08-12T00:21:55Z

This issue is also coming up with me. I can confirm that karma is using ~1600Mb of virtual memory and ~600Mb of actual memory.

Some notes about my config:

uses karma-webpack
it is a react React application
uses typescript

I attempted to take some heap snapshots of karma using node-inspector, and noticed that when I would attempt to snapshot the heap for when I was testing my React components, i would trigger the Allocation failed - process out of memory, which left me unable to get the final results of the snapshot 😞

Does anyone have any pointers about further debugging this issue?

havenchyk · 2016-08-12T08:05:30Z

@ianks what I did is, maybe it can be helpful

update babel to the latest version that supports passing params like max_old_space_size
set max_old_space_size

teleaziz · 2016-08-12T22:01:32Z

@ianks just wondering if this is related to karma-sourcemap-loader plugin? same setup except it's angular 2 project with TS

daviddias · 2016-08-16T11:03:14Z

We are facing the same issue with https://github.com/ipfs/js-ipfs-api, both in Node.js v4 and Node.js v6.

@dignifiedquire do you recommend any solution? Running it on a child_process? Adding a option to karma to enable using more memory for the proc?

dignifiedquire · 2016-08-16T13:10:57Z

After some investigation, this seems to be related with webpack, some issues for reference that could influence this

out of memory webpack/webpack#2157
proposed fix: Fixing memory leak causing node go out of memory webpack/webpack#2497 (does not work for me)
Process out of memory - Webpack webpack/webpack#1914

dignifiedquire · 2016-08-16T13:11:58Z

It might be that we just need to give it more memory on larger projects though, like suggested in webpack/webpack#1875

dignifiedquire · 2016-08-16T13:14:24Z

When using gulp this works fine:

> node --max_old_space_size=4096 node_modules/.bin/gulp task

havenchyk · 2016-08-16T13:15:36Z

@dignifiedquire the thing is we're using babel-node and there wasn't any way to increase space size.

havenchyk · 2016-08-16T13:16:14Z

proposed fix: webpack/webpack#2497 (does not work for me)

doesn't work for me as well. still have crashes

dignifiedquire · 2016-08-16T13:17:11Z

Thanks @havenchyk. From your comment above it sounded like the latest version of babel-node does support the flag passing though?

havenchyk · 2016-08-16T13:20:57Z

yep, it works (node takes 4GB of my memory), but I don't like this solution, it's a workaround. I believe that the problem is in webpack itself, but didn't find any quick or even not a quick fix.

So for you as for karma maintainer, I'd say it's not a problem of karma itself and issue can be closed.

I have the same error (rarely) with webpack in dev mode

Anmo · 2016-12-02T14:13:17Z

I had the same problem and I used the require.context like is suggested in karma-webpack docs.

The problem with this approach is that you can't include test dynamically (just test one spec for instance).
I did a work around with webpack.definePlugin. With the karma config pointing to the test/test_index.js I changed that file to have something like this: var testsContext = require.context(".", true, __REQUIRE_CONTEXT_REGEX__) and in the webpackConfig I just add a new definePlugin({__REQUIRE_CONTEXT_REGEX__: new RegExp(...)}) (this regex will depends on process.argv).
With this I now can do something like: npm test -- --spec sagas

Note: the require.context will not fix the Allocation failed - process out of memory issue.

pbio · 2017-01-25T12:38:25Z

If others are having this issue still, I solved it buy using

node --max_old_space_size=4096 node_modules/karma/bin/karma start

instead of

karma start

Make sure karma is included in your project's node_modules.

stevemao · 2017-04-21T05:22:41Z

another workaround is:

Create a test loader file: loadtests.js
require all the tests in loadtests.js

const testsContext = require.context('./src', true, /\.spec\.js$/);
testsContext.keys().forEach(testsContext);

Include loadtests.js in karma.conf.js

files: [
      'loadtests.js',
],
preprocessors: {
      'loadtests.js': ['webpack', 'coverage'],
},

aaronisme · 2017-05-28T04:23:24Z

@stevemao thanks, facing the same issue, in webpack use this workaround to fix that

Opty1712 · 2017-06-09T13:09:06Z

Got same error. After trying a lot of methods for me helped this one (npm script in package.json):

"test": "node --max_old_space_size=4096 ./node_modules/karma/bin/karma start --single-run || exit 0",

Also waiting for new TS-loader, see details here: TypeStrong/ts-loader#502 (comment)

chennas · 2017-10-20T00:21:29Z

Hi,

After increasing the size, we are facing syntax issues on windows machines, on MAC machines working fine with out any issues, any help?

stevemao · 2017-10-20T00:30:28Z

@chennas your problem doesn't seem to related to this issue. More like different versions of node.

chennas · 2017-10-20T00:33:47Z

@stevemao Thanks for your quick reply. If I remove the "max size setting" karma is running, but with heap memory issues? Any thoughts are welcomed.

Thanks

My node version is v8.3.0

anothermarkus · 2017-11-06T19:45:30Z

I found that with a growing number of tests, this was a problem even though I allocated more memory with the flag --max_old_space_size=4096

karma start karma.conf.js --max_old_space_size=4096

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

Removing source mapping as defined here: https://webpack.github.io/docs/usage-with-karma.html#source-maps helped get over this

karma.conf.js

webpack: {
// devtool: 'inline-source-map',

Good luck out there!

epernice · 2017-11-13T20:03:06Z

@stevemao's solution is the only one that worked for me. Setting max_old_space_size to 4096 did not help (we have about 1500 tests).

jfcwagner · 2018-03-06T18:47:13Z

Agree with @epernice. @stevemao's solution worked for me, too. My team and I have been dealing with this problem on and off for months. Thanks @epernice for calling it out; thanks @stevemao for the workaround!

GalitTugi · 2018-04-22T11:17:18Z

Happened to me too, and found out that caused by webpack's
EvalSourceMapDevToolPlugin (aso happened with SourceMapDevToolPlugin).
I need this plugin for debugging, did anybody else handle this situation?

nodkrot · 2018-09-15T22:47:52Z

So is this going to be addressed?

johnjbarton · 2018-09-18T23:52:45Z

Based on all of the information posted here, the issue is not something karma-runner can fix. The runner itself is not causing memory overflow but some combination of webpack and sourcemaps. It looks like @stevemao has a workaround.

nosus · 2018-10-19T03:21:39Z

to avoid the above comments confuse people in the future
karma start karma.conf.js --max_old_space_size=4096 does NOT work
and
node --max_old_space_size=4096 node_modules/karma/bin/karma start works

At least for me.

Although the test are passing, the process crash with error Assertion failed: (0), function uv_close, file ../deps/uv/src/unix/core.c, line 187. Abort trap: 6 the problem is somewhere between karma and webpack most hopeful lead I follow is from the is issue: karma-runner/karma#1868 not very successful so far, but this change doesn't hurt.

This exception was fixed in past and that time main root cause was tied to Helmet library. Now this OOM exception more likely is caused by increasing code base. Previously, webpack configuration provided to Karma contained `devtool = sourcemap` and might caused growing memory usage along with increased codebase. To mitigate this problem as a quick fix during stabilization period, following non-production changes were made: - Changed source maps generation for karma tests to use lightweight 'eval' option - `Sourcemap` preprocessors are removed While testing locally, I couldn't get OOM exception with following configs. Related issue in Karma repo: karma-runner/karma#1868 (comment) Release justification: Non-production code change Release note: None

46047: ui: Fix OOM exception while building Karma tests r=dhartunian a=koorosh This exception was fixed in past and that time main root cause was tied to Helmet library. Now this OOM exception more likely is caused by increasing code base. Previously, webpack configuration provided to Karma contained `devtool = sourcemap` and might caused growing memory usage along with increased codebase. To mitigate this problem as a quick fix during stabilization period, following non-production changes were made: - Changed source maps generation for karma tests to use lightweight 'eval' option - `Sourcemap` preprocessors are removed While testing locally, I couldn't get OOM exception with following configs. Related issue in Karma repo: karma-runner/karma#1868 (comment) Release justification: Non-production code change Release note: None 46132: sql: trace execution stats for query diagnostics r=RaduBerinde a=RaduBerinde The distsql processors are initialized with the Context in the eval context. If this context contains a tracing span that is recording, the processors will set up statistics collection and put them in the span as tags. The statement diagnostics code sets up a span but doesn't change this context, so statistics collection doesn't happen. We want these statistics in the trace, as they will soon be used to generate EXPLAIN ANALYZE diagrams for the bundles. This change fixes this issue and moves up the initialization of the planner so we can tweak it directly, which simplifies code. Release note (bug fix): statement diagnostics traces now contain processor statistics. Release justification: Bug fixes and low-risk updates to new functionality 46179: sql: decrease job adopt interval for more schema change tests r=lucy-zhang a=ajwerner This commit is a follow up to #45992. It adds some tests that were missed and refactors the setup to a helper. Fixes #45970. ``` --- PASS: TestPrimaryKeyChangeWithPrecedingIndexCreation (1.04s) --- PASS: TestCRUDWhileColumnBackfill (0.73s) --- PASS: TestIndexBackfillAfterGC (0.74s) --- PASS: TestSchemaChangeGRPCError (0.67s) --- PASS: TestFKReferencesAddedOnlyOnceOnRetry (0.50s) ``` Release justification: non-production code changes Release note: None Co-authored-by: Andrii Vorobiov <[email protected]> Co-authored-by: Radu Berinde <[email protected]> Co-authored-by: Andrew Werner <[email protected]>

It is one more attempt to fix OOM issue on ui test runs. Previous fixes were based mainly on suggestion that the root cause of issue is source maps generated during builds by karma. Actually the problem was in webpack and the way it preprocess files by karma `preprocessors: [...]`. Before we provided a bunch of files (source and test files) and webpack processed each matched file separately what caused memory consumption overhead. Current solution: - removes source files from karma configuration entirely. Required sources are already imported in test files so they will be included as well. - added `tests-loader` file as a single entry point for all test files. It allows us to provide and preprocess one file with all required test file at once. Current fix is based on found work around: karma-runner/karma#1868 (comment) Release note: None Release justification: non-production code changes

46640: ui: (fix) OOM exception on ui-test build r=nathanstilwell a=koorosh It is one more attempt to fix OOM issue on ui test runs. Previous fixes were based mainly on suggestion that the root cause of issue is source maps generated during builds by karma. Actually the problem was in webpack and the way it preprocess files by karma `preprocessors: [...]`. Before we provided a bunch of files (source and test files) and webpack processed each matched file separately what caused memory consumption overhead. Current solution: - removes source files from karma configuration entirely. Required sources are already imported in test files so they will be included as well. - added `tests-loader` file as a single entry point for all test files. It allows us to provide and preprocess one file with all required test file at once. Current fix is based on found work around: karma-runner/karma#1868 (comment) Release note: None Release justification: non-production code changes Co-authored-by: Andrii Vorobiov <[email protected]>

dignifiedquire added the needs: investigation label Feb 4, 2016

tleunen mentioned this issue Feb 26, 2016

npm test fails with out of memory tleunen/react-mdl#230

Closed

daviddias mentioned this issue Aug 16, 2016

next generation ipfs-inactive/js-ipfs-http-client#305

Merged

65 tasks

This was referenced Oct 7, 2016

Meta issue: Plan of action TypeStrong/ts-loader#296

Closed

TypeScript 2.0 related garbage collection issue affecting ts-loader microsoft/TypeScript#11452

Closed

busticated mentioned this issue Jan 3, 2017

Ability to set max_old_space_size jakejs/jake#328

Closed

filipesilva mentioned this issue May 5, 2017

Karma fails with coverage reports angular/angular-cli#5983

Closed

netil mentioned this issue Oct 1, 2017

Out of memory running karma naver/billboard.js#156

Closed

Izhaki mentioned this issue Mar 6, 2018

Karma fails with coverage reports #2661

Open

This was referenced Jul 28, 2018

turn off source-maps for test FormidableLabs/victory#1065

Merged

Infrastructure: karma fails out of memory FormidableLabs/victory#1066

Closed

johnjbarton closed this as completed Sep 18, 2018

koorosh mentioned this issue Mar 12, 2020

ui: Fix OOM exception while building Karma tests cockroachdb/cockroach#46047

Merged

koorosh mentioned this issue Mar 26, 2020

ui: (fix) OOM exception on ui-test build cockroachdb/cockroach#46640

Merged

Node v4.2.6 - FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory #1868

Node v4.2.6 - FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory #1868

Comments

busticated commented Feb 2, 2016

busticated commented Feb 3, 2016

busticated commented Feb 4, 2016

dignifiedquire commented Feb 4, 2016

busticated commented Feb 4, 2016

busticated commented Feb 4, 2016

dignifiedquire commented Feb 4, 2016

dignifiedquire commented Feb 4, 2016

busticated commented Feb 4, 2016

dignifiedquire commented Feb 4, 2016

busticated commented Feb 4, 2016

havenchyk commented Jul 19, 2016

ianks commented Aug 12, 2016

havenchyk commented Aug 12, 2016

teleaziz commented Aug 12, 2016 • edited Loading

daviddias commented Aug 16, 2016

dignifiedquire commented Aug 16, 2016

dignifiedquire commented Aug 16, 2016 • edited Loading

dignifiedquire commented Aug 16, 2016 • edited Loading

havenchyk commented Aug 16, 2016

havenchyk commented Aug 16, 2016

dignifiedquire commented Aug 16, 2016

havenchyk commented Aug 16, 2016

Anmo commented Dec 2, 2016 • edited Loading

pbio commented Jan 25, 2017 • edited Loading

stevemao commented Apr 21, 2017 • edited Loading

aaronisme commented May 28, 2017

Opty1712 commented Jun 9, 2017 • edited Loading

chennas commented Oct 20, 2017

stevemao commented Oct 20, 2017

chennas commented Oct 20, 2017

anothermarkus commented Nov 6, 2017

epernice commented Nov 13, 2017

jfcwagner commented Mar 6, 2018

GalitTugi commented Apr 22, 2018

nodkrot commented Sep 15, 2018

johnjbarton commented Sep 18, 2018

nosus commented Oct 19, 2018

teleaziz commented Aug 12, 2016 •

edited

Loading

dignifiedquire commented Aug 16, 2016 •

edited

Loading

dignifiedquire commented Aug 16, 2016 •

edited

Loading

Anmo commented Dec 2, 2016 •

edited

Loading

pbio commented Jan 25, 2017 •

edited

Loading

stevemao commented Apr 21, 2017 •

edited

Loading

Opty1712 commented Jun 9, 2017 •

edited

Loading