V8 CI is not working #886

MylesBorins · 2017-09-16T22:59:34Z

Looks like an error with glient fetching v8

Fetching V8 from chromium.googlesource.com
Error: Command 'download_from_google_storage --no_resume --platform=linux* --no_auth --bucket chromium-clang-format -s v8/buildtools/linux64/clang-format.sha1' returned non-zero exit status 1 in /home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/deps
Running: gclient root
Running: gclient config --spec 'solutions = [
  {
    "url": "https://chromium.googlesource.com/v8/v8.git",
    "managed": False,
    "name": "v8",
    "deps_file": "DEPS",
    "custom_deps": {},
  },
]
'
Running: gclient sync --with_branch_heads
Traceback (most recent call last):
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/depot_tools/fetch.py", line 299, in <module>
    sys.exit(main())
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/depot_tools/fetch.py", line 294, in main
    return run(options, spec, root)
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/depot_tools/fetch.py", line 288, in run
    return checkout.init()
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/depot_tools/fetch.py", line 131, in init
    self.run_gclient(*sync_cmd)
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/depot_tools/fetch.py", line 78, in run_gclient
    return self.run(cmd_prefix + cmd, **kwargs)
  File "/home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/depot_tools/fetch.py", line 68, in run
    return subprocess.check_output(cmd, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '('gclient', 'sync', '--with_branch_heads')' returned non-zero exit status 2
V8 fetch failed

/cc @mhdawson @nodejs/v8

The text was updated successfully, but these errors were encountered:

MylesBorins · 2017-09-16T22:59:56Z

Worth mentioning this is happening for any + all attempts to run CI right now

gibfahn · 2017-09-16T23:09:27Z

To be clear, this is happening for all runs of node-test-commit-v8-linux, not node-test-commit right?

@joransiu @jBarz do I recall you guys talking about something like this on Slack?

jbajwa · 2017-09-17T12:43:04Z

This started failing after https://chromium-review.googlesource.com/c/chromium/tools/depot_tools/+/656565.
@john-yan is working with chromium's infra team on this https://bugs.chromium.org/p/chromium/issues/detail?id=764087 . In the meantime could add the following to the jenkins job

cd <path>/depot_tools
git revert -n 509776e
cd -
export DEPOT_TOOLS_UPDATE=0

mhdawson · 2017-09-18T13:28:43Z

Have put the work around into the job and kicked off a run to see if it resolves the issue.

gibfahn · 2017-09-18T15:46:14Z

Assuming it's https://ci.nodejs.org/job/node-test-commit-v8-linux/912/ then it looks like it worked.

We should leave this open to track the proper fix though, sounds like John is making progress.

mhdawson · 2017-09-18T16:05:20Z

Looks like work around did the trick, lets leave this open until the proper fix is in place.

mhdawson · 2017-10-03T19:52:37Z

@jBarz @joransiu this looks different than this issue https://bugs.chromium.org/p/chromium/issues/detail?id=764087&can=2&start=0&num=100&q=&colspec=ID%20Pri%20M%20Stars%20ReleaseBlock%20Component%20Status%20Owner%20Summary%20OS%20Modified&groupby=&sort=

But it did remind me to ask where we are on this one. At some point I should remove the work around.

john-yan · 2017-10-03T20:11:39Z

It should be fixed now. All we need to do is adding an extra env variable.
export VPYTHON_BYPASS="manually managed python not supported by chrome operations"

MylesBorins · 2017-10-09T20:14:22Z

Looks like the temporary patch is now failing 😭

I'm hacking on the CI job right now to make a permanent fix

MylesBorins · 2017-10-09T21:01:16Z

Was not able to fix it. I attempted to revert all of the hacks we had currently put in and added export VPYTHON_BYPASS="manually managed python not supported by chrome operations"

Same error reported in OP

refack · 2017-10-09T21:23:50Z

~~The error I'm seeing is:~~ env issue

Using https://ci.nodejs.org/job/node-test-commit-v8-linux-fedora24/nodes=fedora24,v8test=v8test/398/console as reference.
Manual on test-rackspace-fedora24-x64-1

refack · 2017-10-09T22:29:51Z

Ok found some stuff:

The fedora machines (https://ci.nodejs.org/job/node-test-commit-v8-linux-fedora24/) are failing because they don't have bzip2 (and their yum repos have been archived to http://archives.fedoraproject.org/pub/archive/fedora/linux), but I see this is only run by timer...

ppc getting:

iojs@test-osuosl-ubuntu14-ppc64-be-3:~/build/workspace/node-test-commit-v8-linux/nodes/ppcbe-
ubuntu1404/v8test/v8test/depot_tools$ ./update_depot_tools
UNKNOWN Machine architecture: ppc64

which lead me to https://bugs.chromium.org/p/chromium/issues/detail?id=764087.
I think it's a depot_tools regression of that issue.

The best I got is to revive the test on the fedora24 - https://ci.nodejs.org/job/node-test-commit-v8-linux-fedora24/399/nodes=fedora24,v8test=v8test/

MylesBorins · 2017-10-09T23:02:52Z

@refack were you testing with VPYTHON_BYPASS="manually managed python not supported by chrome operations"

refack · 2017-10-09T23:08:48Z

@refack were you testing with VPYTHON_BYPASS="manually managed python not supported by chrome operations"

In multiple variations with no success (with " without, in the same line, with export, before make after make, with just calling gclient sync --with_branch_heads).
Also it wasn't there yesterday when the last green job run https://ci.nodejs.org/view/All/job/node-test-commit-v8-linux/953/nodes=ppcbe-ubuntu1404,v8test=v8test/consoleFull

But now I see that we used to roll back depot_tools, and it broke sometime in the last 24h:

+ git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
Cloning into 'depot_tools'...
+ cd depot_tools
+ git revert -n 509776e
error: could not revert 509776e... [gsutil] run through "vpython" (2)
hint: after resolving the conflicts, mark the corrected paths

refack · 2017-10-09T23:26:13Z

@MylesBorins need someone with job edit capas.
add this
git checkout 633a9830a9fce875d277b248c6e6fb14d72e8183
just after
cd depot_tools
and before
git revert -n 509776e

That should do it.

gibfahn · 2017-10-09T23:32:44Z

@MylesBorins need someone with job edit capas.

My copy and paste skills are pretty sharp, will have a go

gibfahn · 2017-10-09T23:34:12Z

https://ci.nodejs.org/view/All/job/node-test-commit-v8-linux/962/

New log:

# temporary work around for https://github.com/nodejs/build/issues/886
cd depot_tools
git checkout 633a9830a9fce875d277b248c6e6fb14d72e8183
git revert -n 509776e

Fails with:

+ cd depot_tools
+ git checkout 633a9830a9fce875d277b248c6e6fb14d72e8183
Note: checking out '633a9830a9fce875d277b248c6e6fb14d72e8183'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 633a983... [gsutil.vpython] Update packages.
+ git revert -n 509776e
error: could not revert 509776e... [gsutil] run through "vpython" (2)
hint: after resolving the conflicts, mark the corrected paths

refack · 2017-10-09T23:48:23Z

sorry new SHA - 77b7687e4837223f820e1e98c369d36696f3f2b3

gibfahn · 2017-10-09T23:52:47Z

git checkout 633a9830a9fce875d277b248c6e6fb14d72e8183
git revert -n 77b7687e4837223f820e1e98c369d36696f3f2b3 || true

Seems healthier. Actually the previous SHA was fine, there was a conflict, but we're not committing so not an issue.

https://ci.nodejs.org/view/All/job/node-test-commit-v8-linux/967/ 🙏

refack · 2017-10-10T00:10:30Z

I think it should be:

git checkout 77b7687e4837223f820e1e98c369d36696f3f2b3
git revert -n 509776e

And don't add the || true cause if it fails, it won't build.

refack · 2017-10-10T00:10:47Z

P.S. the conflict is the crux of the issue

gibfahn · 2017-10-10T01:37:20Z

Okay, my flight is delayed so lots of time to fix this...

Seems to be working better.

gibfahn · 2017-10-10T01:47:54Z

FWIW this, and other things like the FreeBSD bugs, are really regressions in V8 and tools on platforms that Node supports but V8 doesn't (so for V8 I guess not technically regressions, but whatever). I'm not really sure how to solve this, but if we're running Node on V8 to make sure V8 commits don't break Node, we should probably think about platform support. Most regressions arent going to be on xLinux or macOS, they're going to be on less-used platforms.

refack · 2017-10-10T02:13:32Z

I think we can with not too much effort remove the dependency on depot_tools, it's a very fragile monstrosity (this current issue is because of vpython which is a frankenstein of python and virtualenv compiled together in go 🤦‍♂️ . For that, depot_tools brings it's own python, clang, and go compiler).

I have a partial recipe for syncing V8 without depot_tools (partial as in works-on-my-computer), I can try to mature it a little.

refack · 2017-10-10T02:14:14Z

Holly molly! It's green https://ci.nodejs.org/view/All/job/node-test-commit-v8-linux/968/

MylesBorins · 2017-10-10T19:01:08Z

Looks like the git changes broke stuff... failures across all environments rn

refack · 2017-10-10T19:34:47Z

@MylesBorins that's a v4.x PR right? (https://ci.nodejs.org/view/All/job/node-test-commit-v8-linux/970/)
Do we know when was the last known good revision for testing v4.x?

MylesBorins · 2017-10-11T07:19:03Z

nvm... I think I broke stuff (in my branch)... Will check in again in am and confirm

mhdawson · 2017-10-11T17:00:31Z

@john-yan I tried putting in the final fix by adding

# temporary work around for https://github.com/nodejs/build/issues/886
#export DEPOT_TOOLS_UPDATE=0 
export VPYTHON_BYPASS="manually managed python not supported by chrome operations"

But it is still failing on the PPC platforms. See this job which is a clone of the regular CI job with the change in place:

https://ci.nodejs.org/view/All/job/node-test-commit-v8-linux-test-mdawson/configure

mhdawson · 2017-10-11T19:20:26Z

Is it just me or does it look like the job is no longer running all of the tests ?

mhdawson · 2017-10-11T19:32:56Z

Ignore my last comment, its just that they run in the middle of the job since there is recompilation for the hashseed test.

john-yan · 2017-10-11T22:18:28Z

open https://bugs.chromium.org/p/chromium/issues/detail?id=773857 to fix depot_tools on ppc

john-yan · 2017-10-17T20:15:58Z

Hello, depot_tools should be fixed on ppc linux now.

>fetch v8
Running: gclient root
Running: gclient config --spec 'solutions = [
  { 
    "url": "https://chromium.googlesource.com/v8/v8.git",
    "managed": False,
    "name": "v8",
    "deps_file": "DEPS",
    "custom_deps": {},
  },
]
'
Running: gclient sync --with_branch_heads
Running: git submodule foreach 'git config -f $toplevel/.git/config submodule.$name.ignore all'
Running: git config --add remote.origin.fetch '+refs/tags/*:refs/tags/*'
Running: git config diff.ignoreSubmodules all

refack · 2017-10-17T20:42:04Z

crbug pretty much resolved. we can remove the git checkout and git revert and instead add:
export VPYTHON_BYPASS="manually managed python not supported by chrome operations"

/cc @mhdawson @gibfahn

refack · 2017-10-17T20:43:09Z

Thank @john-yan 🎉

mhdawson · 2017-10-18T14:13:55Z

Tested here: https://ci.nodejs.org/view/All/job/node-test-commit-v8-linux-test-mdawson/8/

mhdawson · 2017-10-18T14:18:58Z

Main job updated, run to validate works as expected: https://ci.nodejs.org/job/node-test-commit-v8-linux/999/

mhdawson · 2017-10-18T14:41:26Z

Looks good closing.

gibfahn mentioned this issue Sep 18, 2017

[v8.x] deps: backport V8 update to 6.1 nodejs/node#15393

Closed

2 tasks

refack closed this as completed Oct 9, 2017

refack reopened this Oct 9, 2017

gibfahn closed this as completed Oct 19, 2017

refack mentioned this issue Nov 13, 2018

CI-Change: tweaks to make-v8 test script #1536

Closed

V8 CI is not working #886

V8 CI is not working #886

Comments

MylesBorins commented Sep 16, 2017

MylesBorins commented Sep 16, 2017

gibfahn commented Sep 16, 2017

jbajwa commented Sep 17, 2017

mhdawson commented Sep 18, 2017

gibfahn commented Sep 18, 2017

mhdawson commented Sep 18, 2017

mhdawson commented Oct 3, 2017

john-yan commented Oct 3, 2017

MylesBorins commented Oct 9, 2017

MylesBorins commented Oct 9, 2017

refack commented Oct 9, 2017 • edited by gibfahn Loading

refack commented Oct 9, 2017

MylesBorins commented Oct 9, 2017

refack commented Oct 9, 2017 • edited Loading

refack commented Oct 9, 2017

gibfahn commented Oct 9, 2017 • edited Loading

gibfahn commented Oct 9, 2017 • edited Loading

refack commented Oct 9, 2017

gibfahn commented Oct 9, 2017

refack commented Oct 10, 2017

refack commented Oct 10, 2017

gibfahn commented Oct 10, 2017

gibfahn commented Oct 10, 2017

refack commented Oct 10, 2017

refack commented Oct 10, 2017

MylesBorins commented Oct 10, 2017

refack commented Oct 10, 2017

MylesBorins commented Oct 11, 2017 • edited Loading

mhdawson commented Oct 11, 2017

mhdawson commented Oct 11, 2017

mhdawson commented Oct 11, 2017

john-yan commented Oct 11, 2017

john-yan commented Oct 17, 2017

refack commented Oct 17, 2017

refack commented Oct 17, 2017

mhdawson commented Oct 18, 2017

mhdawson commented Oct 18, 2017

mhdawson commented Oct 18, 2017

refack commented Oct 9, 2017 •

edited by gibfahn

Loading

refack commented Oct 9, 2017 •

edited

Loading

gibfahn commented Oct 9, 2017 •

edited

Loading

gibfahn commented Oct 9, 2017 •

edited

Loading

MylesBorins commented Oct 11, 2017 •

edited

Loading