Test-sanity.system-JDK11-win jlink tests failing

[pshipton]

47 - - JlinkTest_RequiredMod_0
48 - - JlinkTest_AddModLimitMod_0
50 - - JlinkPluginOptionsTest_GeneralOptionsTest_0

https://ci.eclipse.org/openj9/job/Test-sanity.system-JDK11-win_x86-64_cmprssptrs/129
win2012r2-x86-2

STF 01:04:12.978 - Running command: C:/Users/jenkins/workspace/Test-sanity.system-JDK11-win_x86-64_cmprssptrs/openjdkbinary/j2sdk-image/bin/../bin/jmod create --class-path C:\Users\jenkins\workspace\Test-sanity.system-JDK11-win_x86-64_cmprssptrs\openjdk-tests\TestConfig\test_output_15471011417768\JlinkTest_RequiredMod_0\20190110-010408-JlinkTest\tmp\..\com.test.jLinkDirCopy --config C:\Users\jenkins\workspace\Test-sanity.system-JDK11-win_x86-64_cmprssptrs\jvmtest\systemtest\openjdk-systemtest\openjdk.test.modularity\bin\tests\com.test.jlink\conf --libs C:\Users\jenkins\workspace\Test-sanity.system-JDK11-win_x86-64_cmprssptrs\jvmtest\systemtest\openjdk-systemtest\openjdk.test.modularity\bin\tests\com.test.jlink\native\bin\win_x86-64 C:/Users/jenkins/workspace/Test-sanity.system-JDK11-win_x86-64_cmprssptrs/openjdk-tests/TestConfig/test_output_15471011417768/JlinkTest_RequiredMod_0/20190110-010408-JlinkTest/tmp/com.test.jlink.jmod
STF 01:04:12.979 - Redirecting stderr to C:/Users/jenkins/workspace/Test-sanity.system-JDK11-win_x86-64_cmprssptrs/openjdk-tests/TestConfig/test_output_15471011417768/JlinkTest_RequiredMod_0/20190110-010408-JlinkTest/results/8.JMOD.stderr
STF 01:04:12.979 - Redirecting stdout to C:/Users/jenkins/workspace/Test-sanity.system-JDK11-win_x86-64_cmprssptrs/openjdk-tests/TestConfig/test_output_15471011417768/JlinkTest_RequiredMod_0/20190110-010408-JlinkTest/results/8.JMOD.stdout
STF 01:04:12.984 - Monitoring processes: JMOD
JMOD Error: Packages that are exported or open in com.test.jlink are not present: [adoptopenjdk.test.modularity.jlink]
STF 01:04:13.380 - **FAILED** Process JMOD ended with exit code (2) and not the expected exit code/s (0)
STF 01:04:13.381 - Monitoring Report Summary:
STF 01:04:13.381 -   o Process JMOD ended with exit code (2) and not the expected exit code/s (0)
STF 01:04:13.381 - Killing processes: JMOD
STF 01:04:13.381 -   o Process JMOD is not running
**FAILED** at step 8 (Run jmod). Expected return value=0 Actual=1 at C:/Users/jenkins/workspace/Test-sanity.system-JDK11-win_x86-64_cmprssptrs/openjdk-tests/\TestConfig\test_output_15471011417768\JlinkTest_RequiredMod_0/20190110-010408-JlinkTest/setUp.pl line 230.
STF 01:04:13.415 - **FAILED** setup script failed. Expected return value=0 Actual=1

[pshipton]

@Mesbah-Alam

[Mesbah-Alam]

JlinkPluginOptionsTest_GeneralOptionsTest_0 fails for path being too long - something I'd fix by shortening the unnecessarily long name of this test.

JlinkTest_RequiredMod_0 and JlinkTest_AddModLimitMod_0 are failing for common reason:

JMOD Error: Packages that are exported or open in com.test.jlink are not present: [adoptopenjdk.test.modularity.jlink]

Checking the result zip, there is no file in com.test.jLinkDirCopy, which is used in the jmod command to create com.test.jlink.jmod - which fails. This may explain why we get the above error.

The only reason why it could fail is if there was a failure in copying over of the files into com.test.jLinkDirCopy -- but the test does not indicate any failure regarding this cp step-- nothing in output or log files.

Also, I can't seem to reproduce the issue:

Passes openj9 5x Grinder: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/775/tapResults/
Passes hotspot 1x Grinder: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/773/console

So, this could well be an intermittent issue we have bumped into.

The test should be made more robust -- perhaps we should remove the copy step altogether. It only does this copy to a temp folder because it wants to remove some unnecessary files (native) from there, before the jmod is created out of it. I do not see any reason why this should be an absolute must. It seems to be introducing flakiness in the test instead.

[Mesbah-Alam]

Turns out we do need the intermediate copy stage that creates the com.test.jLinkDirCopy folder. It not only removes the unnecessary native files from the copy folder, but also removes the "conf" folder - without this step, the creation of runtime image fails: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/779/console

So, simply not using the cp stage is not possible.

[pshipton]

https://ci.eclipse.org/openj9/job/Test-sanity.system-JDK11-win_x86-64_cmprssptrs/130
win2012r2-x86-2

[Mesbah-Alam]

@pshipton : could you please run a 5x grinder in openj9 Test-Grinder with same settings as the Grinder I ran in Adopt: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/775/parameters/? (I do not have permission to).

This should flush out any differences we may have in the Adopt Windows Machine and those in OpenJ9. The test had passed a 5x Grinder I ran at Adopt, even though it seems to continue to fail at OpenJ9.

[pshipton]

https://ci.eclipse.org/openj9/view/Test/job/Test-Grinder/166/

[pshipton]

I just ran 1x since the tests failed consistently in the last 2 builds.

[pshipton]

Previous grinder never got started, here is a new one.
https://ci.eclipse.org/openj9/view/Test/job/Test-Grinder/167
win2012r2-x86-4

[pshipton]

Grinder passed, started 5x https://ci.eclipse.org/openj9/view/Test/job/Test-Grinder/169/
win2012r2-x86-2

[pshipton]

The 5x passed as well.

[Mesbah-Alam]

Could it be a machine issue then?

[pshipton]

Maybe there is a side affect from running other tests? They failed again in the nightly build
https://ci.eclipse.org/openj9/job/Test-sanity.system-JDK11-win_x86-64_cmprssptrs/131
win2012r2-x86-1

[pshipton]

I updated the descriptions with machine information. The original failures were on win2012r2-x86-2, however the 5x grinder ran on this machine and passed

[Mesbah-Alam]

The only solution from test perspective that I can think of is have the "copy" folder created and checked into the repository itself so that the test can simply use that instead of attempting to create a copy folder at runtime. This will introduce duplicate code in the repo, but make the test robust -- even though I don't know the real reason why the test is failing inconsistently (seemingly at the copy stage-- since the failure indicates no file being copied).

[pshipton]

These failures didn't occur before the 10th. What happened to cause them to start failing?

[pshipton]

Seems these tests were not running previously.

[Mesbah-Alam]

They were running on other platforms -- Windows was enabled recently - and the failure is on Windows only.

[pshipton]

This is 1 of 2 remaining failures (non osx) in the nightly builds for the 0.12 release.

@Mesbah-Alam what is the outlook for fixing? Can we exclude these tests on Windows if they won't be fixed in the next day or two?

[Mesbah-Alam]

I have not found any solution to this intermittent issue. It will be excluded from Windows temporarily while investigation goes on : adoptium/aqa-tests#807