OpenJDK java/lang/String/CompactString/MaxSizeUTF16String -XX:-CompactStrings failing some platforms

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_aarch64_mac_Nightly_testList_0/13/
jdk_lang_1 (-XX:-UseCompressedOops)
-Xcomp (-Xjit:count=0) -Xmx8g
java/lang/String/CompactString/MaxSizeUTF16String.java

21:30:07  STARTED    MaxSizeUTF16String::testMaxCharArray 'testMaxCharArray()'
21:30:07  Checking max UTF16 string len: 1073741824
21:30:07  Checking max UTF16 string len: 1073741823
21:30:07  org.opentest4j.AssertionFailedError: Expected OutOfMemoryError with message prefix: UTF16 String size is
21:30:07  	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
21:30:07  	at org.junit.jupiter.api.Assertions.fail(Assertions.java:135)
21:30:07  	at MaxSizeUTF16String.testMaxCharArray(MaxSizeUTF16String.java:112)
21:30:07  	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
21:30:07  	at java.base/java.lang.reflect.Method.invoke(Method.java:586)
21:30:07  	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
21:30:07  	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
21:30:07  	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
21:30:07  	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
21:30:07  	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
21:30:07  	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
21:30:07  	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
21:30:07  	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
21:30:07  	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
21:30:07  	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
21:30:07  	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
21:30:07  	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
21:30:07  	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
21:30:07  	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
21:30:07  	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
21:30:07  	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
21:30:07  	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
21:30:07  	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
21:30:07  	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
21:30:07  	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
21:30:07  	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
21:30:07  	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
21:30:07  	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
21:30:07  	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
21:30:07  	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
21:30:07  	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
21:30:07  	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
21:30:07  	at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
21:30:07  	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
21:30:07  	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
21:30:07  	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
21:30:07  	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
21:30:07  	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
21:30:07  	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
21:30:07  	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
21:30:07  	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
21:30:07  	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
21:30:07  	at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
21:30:07  	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
21:30:07  	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
21:30:07  	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
21:30:07  	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
21:30:07  	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
21:30:07  	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
21:30:07  	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
21:30:07  	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
21:30:07  	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
21:30:07  	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
21:30:07  	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
21:30:07  	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
21:30:07  	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
21:30:07  	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
21:30:07  	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
21:30:07  	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
21:30:07  	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
21:30:07  	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
21:30:07  	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
21:30:07  	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
21:30:07  	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
21:30:07  	at com.sun.javatest.regtest.agent.JUnitRunner.runWithJUnitPlatform(JUnitRunner.java:142)
21:30:07  	at com.sun.javatest.regtest.agent.JUnitRunner.main(JUnitRunner.java:95)
21:30:07  	at com.sun.javatest.regtest.agent.JUnitRunner.main(JUnitRunner.java:61)
21:30:07  	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
21:30:07  	at java.base/java.lang.reflect.Method.invoke(Method.java:586)
21:30:07  	at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
21:30:07  	at java.base/java.lang.Thread.run(Thread.java:1587)
21:30:07  FAILED     MaxSizeUTF16String::testMaxCharArray 'testMaxCharArray()'

https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_x86-64_linux_Nightly_testList_1/38/
jdk_lang_1

https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Nightly_testList_0/11
jdk_lang_0, jdk_lang_j9_0
https://openj9-jenkins.osuosl.org/job/Test_openjdk23_j9_sanity.openjdk_x86-64_mac_Nightly_testList_1/11/
jdk_lang_1

[pshipton]

Test case

public class MaxChar {

private final static int MAX_UTF16_STRING_LENGTH = Integer.MAX_VALUE / 2;

    private static char[] generateCharData(int size) {
        char[] nonAscii = "\u0100".toCharArray();
        char[] arr = new char[size];
        System.arraycopy(nonAscii, 0, arr, 0, nonAscii.length); // non-latin1 at start
        return arr;
    }

public static void main(String[] args) throws Throwable {
        final char[] large_char_array = generateCharData(MAX_UTF16_STRING_LENGTH + 1);
        int[] sizes = new int[] {
                MAX_UTF16_STRING_LENGTH + 1,
                MAX_UTF16_STRING_LENGTH,
                MAX_UTF16_STRING_LENGTH - 1};
        for (int size : sizes) {
                try {
                        new String(large_char_array, 0, size);
                        if (size >= MAX_UTF16_STRING_LENGTH) {
                                System.out.println("Expected OOM for " + size);
                        }
                } catch (OutOfMemoryError e) {
                        System.out.println(e);
                }
        }
}
}

When running with -Xcomp (-Xjit:count=0) on at least aarch64 or x86, the OOM which is expected for size 1073741823 doesn't occur. It didn't fail on zlinux, some platforms (AIX, Windows) didn't run.

@hzongaro pls take a look

[pshipton]

This does seem related to #19309 after all.

[hzongaro]

In JDK 21 and earlier, the implementation of StringUTF16.newBytesFor tested whether the length was greater than 0x3fffffff. If so, it would throw an OutOfMemoryError.

In JDK 23, the implementation of StringUTF16.newBytes, by way of String.UTF16.newBytesLength, will throw an OutOfMemoryError for length values greater than or equal to 0x3fffffff.

The IL the JIT compiler is generating is consistent with the expectation that an OutOfMemoryError will not be thrown for a length of 0x3fffffff. Before I make any further changes to the JIT compiler, is the inconsistency between the different JDK versions expected, or is one of them a bug?

[hzongaro]

It looks like the handling of a length of 0x3fffffff was deemed to be a bug, and the change in behaviour was introduced in JDK Level 23 in commit ibmruntimes/openj9-openjdk-jdk23@38f0d53.

I will update the JIT compiler's handling of this by falling back to the out-of-line call to StringUTF16.toBytes for lengths outside the range [0,0x3ffffffe]. That way, if the bug is ever fixed in other JDK levels, the JIT-compiled code will not show any difference in behaviour to interpreted bytecode.

[hzongaro]

Reopening, as I will need to port the fix to the v0.47.0-release and v0.48.0-release branches

[hzongaro]

Opened pull requests #20077 and #20078 to port the fix to the v0.47.0-release and v0.48.0-release branches