ppcbe-ubuntu1404 jobs are not finishing

[refack]

https://ci.nodejs.org/job/node-test-commit-plinux/10137/nodes=ppcbe-ubuntu1404/console
https://ci.nodejs.org/job/node-test-commit-plinux/10138/nodes=ppcbe-ubuntu1404/console
https://ci.nodejs.org/job/node-test-commit-plinux/10139/nodes=ppcbe-ubuntu1404/console

[----------] 3 tests from EnvironmentTest
[ RUN      ] EnvironmentTest.AtExitWithEnvironment
[       OK ] EnvironmentTest.AtExitWithEnvironment (362 ms)
[ RUN      ] EnvironmentTest.AtExitWithArgument
[       OK ] EnvironmentTest.AtExitWithArgument (354 ms)
[ RUN      ] EnvironmentTest.MultipleEnvironmentsPerIsolate
Build was aborted
Run condition [Always] enabling perform for step [[]]

@nodejs/platform-ppc

[Trott]

I think this is primarily maintained by the IBM folks, so... /ping @mhdawson @gibfahn

[gibfahn]

I think this is primarily maintained by the IBM folks, so... /ping @mhdawson @gibfahn

Yep, we're looking at it (that's who @nodejs/platform-ppc is!)

[gibfahn]

The jobs are hanging here:

out/Release/cctest --gtest_output=tap:cctest.tap
[==========] Running 48 tests from 6 test cases.
[----------] Global test environment set-up.
[----------] 2 tests from Base64Test
[ RUN      ] Base64Test.Encode
[       OK ] Base64Test.Encode (0 ms)
[ RUN      ] Base64Test.Decode
[       OK ] Base64Test.Decode (0 ms)
[----------] 2 tests from Base64Test (0 ms total)

[----------] 3 tests from EnvironmentTest
[ RUN      ] EnvironmentTest.AtExitWithEnvironment
[       OK ] EnvironmentTest.AtExitWithEnvironment (375 ms)
[ RUN      ] EnvironmentTest.AtExitWithArgument
[       OK ] EnvironmentTest.AtExitWithArgument (354 ms)
[ RUN      ] EnvironmentTest.MultipleEnvironmentsPerIsolate

Looks like the hanging test is EnvironmentTest.MultipleEnvironmentsPerIsolate.

[gibfahn]

ps -ef on the machine shows nothing out of the ordinary:

iojs      7205 29041  0 05:31 ?        00:00:00 make test-ci
iojs      9930  7205  0 05:32 ?        00:00:01 out/Release/cctest --gtest_output=tap:cctest.tap
iojs     29038  1005  0 05:30 ?        00:00:00 /bin/sh -xe /tmp/hudson6064430299175217652.sh
iojs     29041 29038  0 05:30 ?        00:00:00 make run-ci -j 2

[mhdawson]

Looking at the recent history I only see a single failure that looks like that hang.

[refack]

Did anyone add an auto-termination?
This one seems to have stoped on it own
https://ci.nodejs.org/job/node-test-commit-plinux/10156/nodes=ppcbe-ubuntu1404/

[----------] 3 tests from EnvironmentTest
[ RUN      ] EnvironmentTest.AtExitWithEnvironment
[       OK ] EnvironmentTest.AtExitWithEnvironment (369 ms)
[ RUN      ] EnvironmentTest.AtExitWithArgument
[       OK ] EnvironmentTest.AtExitWithArgument (359 ms)
[ RUN      ] EnvironmentTest.MultipleEnvironmentsPerIsolate
make[1]: *** [test-ci] Terminated
make[1]: Leaving directory `/home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcbe-ubuntu1404'
make: *** [run-ci] Error 2
Build step 'Execute shell' marked build as failure
Run condition [Always] enabling perform for step [[]]
TAP Reports Processing: START
Looking for TAP results report in workspace using pattern: *.tap
Did not find any matching files. Setting build result to FAILURE.
Checking ^not ok
Jenkins Text Finder: File set '*.tap' is empty
Sending e-mails to: [email protected] [email protected]
Notifying upstream projects of job completion
Finished: FAILURE

[Trott]

Yep, we're looking at it (that's who @nodejs/platform-ppc is!)

Oops, I missed that @refack had already pinged that team. Sorry!

[mhdawson]

The hangs seems to have occurred on both be-1 and be-2 from the history.

[mhdawson]

Jobs seem to be running/completing on both be-1 and be-2 now. The failures are related to nodejs/node#14177 which is not specific to PPC

[gibfahn]

I killed the out/Release/cctest --gtest_output=tap:cctest.tap process on both machines, this seems to have fixed it, I have no idea why. It was definitely hanging before.

[mhdawson]

Ok I guess we can just keep an eye on it for today.

[mhdawson]

Just noticed same failure on arm job as well: https://ci.nodejs.org/job/node-test-commit-arm/10826/nodes=armv7-wheezy/console

[refack]

https://ci.nodejs.org/job/node-test-commit-plinux/10165/nodes=ppcbe-ubuntu1404/console

[mhdawson]

No instance of cctest running this time on the machine.

[Trott]

Happening here too: https://ci.nodejs.org/job/node-test-commit-plinux/10166/nodes=ppcbe-ubuntu1404/console

[mhdawson]

This is the test

TEST_F(EnvironmentTest, MultipleEnvironmentsPerIsolate) {
  const v8::HandleScope handle_scope(isolate_);
  const Argv argv;
  Env env1 {handle_scope, isolate_, argv};
  Env env2 {handle_scope, isolate_, argv};

  AtExit(*env1, at_exit_callback1);
  AtExit(*env2, at_exit_callback2);
  RunAtExit(*env1);
  EXPECT_TRUE(called_cb_1);
  EXPECT_FALSE(called_cb_2);

  RunAtExit(*env2);
  EXPECT_TRUE(called_cb_2);
}

From
test/cctest/test_environment.cc

[jBarz]

I will investigate this and provide an update

[mhdawson]

@jBarz thanks :)

[refack]

https://ci.nodejs.org/job/node-test-commit-linux/11117/nodes=centos5-64

[refack]

https://ci.nodejs.org/job/node-test-commit-plinux/10172/nodes=ppcbe-ubuntu1404/console

[mhdawson]

Given that we now have seen this on centos and ARM in addition to PPC, this should likely be moved to the normal repo as a problem with the test across platforms. @refack any objections ?

[mhdawson]

My guess is that disabling snapshots has shifted the timing to expose and existing problem either in the test or v8 itself.

[refack]

No objection, just a suggestion to implement some sort of timeout on jobs spawned by node-test-commit, they should not run (actual time on runners) more the 2h...

[refack]

Closing in favor of nodejs/node#14206