AIX LTS builds 8 and 11 failing when built on AIX 7.2 machines.

[sxa]

adoptium/ci-jenkins-pipelines#622 is in to adjust the configurations to use AIX 7.2 for building AIX LTS releases, on the basis that AIX 7.1 is going out of support soon.

The build for JDK17 is running ok but as per one of the comments in there (repeasted below) it's currently failing for 8 and 11 and will need some remedial work:

• JDK8: jdk/src/share/classes/jdk/internal/platform/Metrics.java:57: error: cannot find symbol on SystemMetrics
• JDK11: Not quite sure where the underlying failure is

[sxa]

@backwaterred has confirmed to me that he has been able to successfully build JDK11 on one of his AIX 7.2 machines using the same XLC 16.0.10000.0011 compiler that's on ours.

[sxa]

@backwaterred confirmed that JDK8u doesn't build on AIX 7.2 for him either with some issues in SystemMetrics.java

[sxa]

Re-running JDK8 on AIX 7.2 with last GA tag at https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-aix-ppc64-temurin/235/
PASSED on build-osuosl-aix72-ppc64-1:

10:23:43  =JAVA VERSION OUTPUT=
10:23:45  openjdk version "1.8.0_362-beta"
10:23:45  OpenJDK Runtime Environment (Temurin)(build 1.8.0_362-beta-202302240952-b09)
10:23:45  OpenJDK 64-Bit Server VM (Temurin)(build 25.362-b09, mixed mode)
10:23:45  =/JAVA VERSION OUTPUT=

So just 11 that needs to be fixed

[backwaterred]

FYI, jdk8u should be building again after openjdk/jdk8u#33 fixed the SystemMetrics issue.

[sxa]

New attempts:

• JDK8: https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-aix-ppc64-temurin/268
• JDK11: https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-aix-ppc64-temurin/214

[sxa]

8 failed quite spectacularly during the CACerts part of the build.

15:11:03  Done (140 CA certs processed, 23 skipped).
15:11:03  mk-ca-bundle.pl generates 140 certificates
15:11:03  Processing certificate with alias:  ,C=BE,O=GlobalSign nv-sa,OU=Root CA,CN=GlobalSign Root CA
15:11:03  keytool error: java.lang.Exception: Input not an X.509 certificate
15:11:04  
15:11:04  real	0m1.291s
15:11:04  user	0m0.458s
15:11:04  sys	0m0.052s
[Second attempt](https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-aix-ppc64-temurin/269/console) didn't even get that far:

16:09:46 Using default /tmp for temporary files as /ramdisk0/build/tmp does not exist
16:09:46 LDR_CNTRL=MAXDATA=0x80000000
16:09:46 BOOT JDK: /home/jenkins/workspace/build-scripts/jobs/jdk8u/jdk8u-aix-ppc64-temurin/build-farm/platform-specific-configurations/aix.sh: line 109: /home/jenkins/workspace/build-scripts/jobs/jdk8u/jdk8u-aix-ppc64-temurin/jdk-7/bin/java: No such file or directory

 11 has got past that point without issues but failed on `ar`:

15:20:08 ar: 0707-126 /home/jenkins/workspace/build-scripts/jobs/jdk11u/jdk11u-aix-ppc64-temurin/workspace/build/src/build/aix-ppc64-normal-server-release/support/native/java.instrument/libinstrument/static/Utilities.o is not valid with the current object file mode.
15:20:08 Use the -X option to specify the desired object mode.
15:20:08 gmake[3]: *** [Lib-java.instrument.gmk:40: /home/jenkins/workspace/build-scripts/jobs/jdk11u/jdk11u-aix-ppc64-temurin/workspace/build/src/build/aix-ppc64-normal-server-release/support/native/java.instrument/libinstrument/static/libinstrument.a] Error 12

[sxa]

Rebuilding 8 on the build- machine instead of the test- machine which has been labelled with build for redundancy. It's possible that the issues seen above are due to the default and/or boot JDKs available (or not) on the machines.
https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-aix-ppc64-temurin/276/

11 is almost certainly the default being 32-bit and can be resolved as per #2734 (comment) so I've added OBJECT_MODE=64 on the definitions for build-osuosl-aix72-ppc64-1 and test-osuosl-aix72-ppc64-1

I suspect it would make more sense to have this in the aix.sh script though, so I would suggest that regardless of how well this works we do not close this issue until that is done.

Rebuilding 11 on https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-aix-ppc64-temurin/220/ (Note that 8 and 11 are building on the opposite machiens to the ones they were in the earlier test.

[sxa]

8 passed: https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-aix-ppc64-temurin/220/
11 failed but that's because there's a third machine that needed the OBJECT_MODE option. Re-running at https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-aix-ppc64-temurin/221/

[adamfarley]

The only failures left here seem to be the 715 test issue which has been fixed.

@sxa - Please close if you agree this is resolved.

[sxa]

Is like to ensure that we've successfully run a test on all 7.2 machines labelled as build before officially closing this. I thought we had at least one still offline but I could be wrong.

[sxa]

See adoptium/infrastructure#3038 as an example

[aixtools]

See adoptium/infrastructure#3052 and adoptium/infrastructure#3053 re: machine definitions (ansible inventory).

Note: there is (currently) only one AIX 7.2 system labeled as build.

[sxa]

Note: there is (currently) only one AIX 7.2 system labelled as build.

Yeah - that's down to adoptium/infrastructure#3038, but I'm happy to close this since the original problem is likely with the specific machines in that issue now and not a general issue with all AIX 7.2 systems based on the fact we completed the April releases on AIX 7.2

[adamfarley]

FYI: As mentioned here, a JDK8 build of aix on build-osuosl-aix72-ppc64-2 was recently found to have failed with this error:

13:05:57  BOOT JDK: /home/jenkins/workspace/build-scripts/jobs/jdk8u/jdk8u-aix-ppc64-temurin/build-farm/platform-specific-configurations/aix.sh: line 109: /home/jenkins/workspace/build-scripts/jobs/jdk8u/jdk8u-aix-ppc64-temurin/jdk-7/bin/java: A file or directory in the path name does not exist.

This appears to have been caused by an incorrect boot jdk location for building JDK8 in the machine's jenkins environment variables. A JDK8 was found to be present in a different folder (as per the variable setting for another aix machine), so the variable was modified on Jenkins and the build was re-run here.

This appears to have solved the problem. Noting here for posterity.

[sxa]

FYI: As mentioned #3389 (comment), a JDK8 build of aix on build-osuosl-aix72-ppc64-2 was recently found to have failed

Noting that this occurred on the new machine added in adoptium/infrastructure#3069

[adamfarley]

Odd that a new machine would have this set wrong. I wonder if this was a manual typo, or some sort of automated setup flaw in ansible or somesuch.

Following up on Slack here.

[sxa]

It's not quite a new machine - it was cloned from another one. Do you have the info on what the path was and what you updated it it? I suspect we cloned the new AIX box from one machine but the jenkins definition got copied from a different one that was slightly different.

[adamfarley]

I think the path began "/home/jenkins" and ended with "jdk-7". At one point, I thought this could be to blame, though I'm unclear as to how it went wrong.

[adamfarley]

Update:

On test-osuosl-aix72-ppc64-* machines 1, 3, and 4, JDK7_BOOT_DIR was set to "/usr/java7_64", which is an IBM build that lacks the x509 certificate, failing the build with this issue. This (jenkins) variable has since been replaced with /usr/java8_64, which is a Temurin build.

test-osuosl-aix72-ppc64-2's issue was a combination of "could not find java" (older jdk, likely expired and deleted), and the above issue when it tried to use the IBM JDK instead. This variable was also replaced with /usr/java8_64.

[adamfarley]

Ok, now we're having the same problem on test-osuosl-aix72-ppc64-6.

/home/jenkins/workspace/build-scripts/jobs/jdk8u/jdk8u-aix-ppc64-temurin/build-farm/platform-specific-configurations/aix.sh: line 109: /home/jenkins/workspace/build-scripts/jobs/jdk8u/jdk8u-aix-ppc64-temurin/jdk-7/bin/java: A file or directory in the path name does not exist.

Can someone with Jenkins admin access please review the JDK7 boot JDK setting and see if there's a java 8 build in /usr/java8_64 that can be used to replace the current value?

Also, please check test-osuosl-aix72-ppc64-5 as well. I'd like to get out ahead of this.

Slack chat here

[steelhead31]

@adamfarley I've updated this on test-osuosl-aix72-ppc64-5 & test-osuosl-aix72-ppc64-6

[adamfarley]

Thanks!

[sxa]

Noting that using /usr/j2sdk-image as a JDK7 boot JDK works correctly.
I've removed the build and build72 labels from all of the test machines as we now have two dedicated build systems so we should not be building on the test systems.