System.Runtime.Numerics.Tests tests fail on Android

[mdh1418]

Several tests from System.Runtime.Numerics.Tests fail on Android x86

System.Runtime.Numerics.Tests

• System.Numerics.Tests.ComplexTests.Reciprocal(real: 1.7976931348623157E+308, imaginary: 1.7976931348623157E+308)
• System.Numerics.Tests.ComplexTests.Reciprocal(real: -1.7976931348623157E+308, imaginary: -1.7976931348623157E+308)
• System.Numerics.Tests.GetBitLengthTests.RunGetBitLengthTestsLarge - disabled in https://github.com/dotnet/runtime/pull/73773/files
• System.Numerics.Tests.BigIntegerConstructorTest.RunCtorByteArrayTests

System.Runtime.Numerics.Tests.dll   Failed: 3

Test collection for System.Numerics.Tests.GetBitLengthTests
System.Numerics.Tests.GetBitLengthTests.RunGetBitLengthTestsLarge
    System.OutOfMemoryException : Insufficient memory to continue the execution of the program.

Test collection for System.Numerics.Tests.BigIntegerConstructorTest
System.Numerics.Tests.BigIntegerConstructorTest.RunCtorByteArrayTests
    System.OutOfMemoryException : Insufficient memory to continue the execution of the program.

Test collection for System.Numerics.Tests.ComplexTests
System.Numerics.Tests.ComplexTests.Reciprocal(real: 1.7976931348623157E+308, imaginary: 1.7976931348623157E+308)
    Failure at line 1325. Expected real: 0. Actual real: 2.781342323134E-309\nExpected: True\nActual:   False
System.Numerics.Tests.ComplexTests.Reciprocal(real: -1.7976931348623157E+308, imaginary: -1.7976931348623157E+308)
    Failure at line 1325. Expected real: -0. Actual real: -2.781342323134E-309\nExpected: True\nActual:   False

Edit: Updated based on #50095

[VincentBu]

Failed again in runtime-coreclr libraries-jitstressregs 20210704.1

Failed test:

net6.0-windows-Release-x86-CoreCLR_checked-jitstressregs3-Windows.10.Amd64.Open

- System.Numerics.Tests.GetBitLengthTests.RunGetBitLengthTestsLarge

Error message:

System.OutOfMemoryException : Exception of type 'System.OutOfMemoryException' was thrown.


Stack trace
   at System.Numerics.BigInteger..ctor(ReadOnlySpan`1 value, UInt32[] valueArray, Boolean negative) in /_/src/libraries/System.Runtime.Numerics/src/System/Numerics/BigInteger.cs:line 487
   at System.Numerics.BigInteger.Add(UInt32[] leftBits, Int32 leftSign, UInt32[] rightBits, Int32 rightSign) in /_/src/libraries/System.Runtime.Numerics/src/System/Numerics/BigInteger.cs:line 1529
   at System.Numerics.Tests.GetBitLengthTests.Log2BitLength(BigInteger integer) in /_/src/libraries/System.Runtime.Numerics/tests/BigInteger/GetBitLengthTests.cs:line 89
   at System.Numerics.Tests.GetBitLengthTests.RunGetBitLengthTestsLarge() in /_/src/libraries/System.Runtime.Numerics/tests/BigInteger/GetBitLengthTests.cs:line 48

[jtschuster]

System.Numerics.Tests.GetBitLengthTests.RunGetBitLengthTestsLarge failed in runtime-extra-platforms rolling build on tvOS.
https://runfo.azurewebsites.net/view/build/?number=1871170

[ghost]

Tagging subscribers to 'os-tvos': @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Several tests from System.Runtime.Numerics.Tests fail on Android x86

System.Runtime.Numerics.Tests

• System.Numerics.Tests.ComplexTests.Reciprocal(real: 1.7976931348623157E+308, imaginary: 1.7976931348623157E+308)
• System.Numerics.Tests.ComplexTests.Reciprocal(real: -1.7976931348623157E+308, imaginary: -1.7976931348623157E+308)
• System.Numerics.Tests.GetBitLengthTests.RunGetBitLengthTestsLarge
• System.Numerics.Tests.BigIntegerConstructorTest.RunCtorByteArrayTests

System.Runtime.Numerics.Tests.dll   Failed: 3

Test collection for System.Numerics.Tests.GetBitLengthTests
System.Numerics.Tests.GetBitLengthTests.RunGetBitLengthTestsLarge
    System.OutOfMemoryException : Insufficient memory to continue the execution of the program.

Test collection for System.Numerics.Tests.BigIntegerConstructorTest
System.Numerics.Tests.BigIntegerConstructorTest.RunCtorByteArrayTests
    System.OutOfMemoryException : Insufficient memory to continue the execution of the program.

Test collection for System.Numerics.Tests.ComplexTests
System.Numerics.Tests.ComplexTests.Reciprocal(real: 1.7976931348623157E+308, imaginary: 1.7976931348623157E+308)
    Failure at line 1325. Expected real: 0. Actual real: 2.781342323134E-309\nExpected: True\nActual:   False
System.Numerics.Tests.ComplexTests.Reciprocal(real: -1.7976931348623157E+308, imaginary: -1.7976931348623157E+308)
    Failure at line 1325. Expected real: -0. Actual real: -2.781342323134E-309\nExpected: True\nActual:   False

Edit: Updated based on #50095

Author:	mdh1418
Assignees:	mkhamoyan
Labels:	area-System.Runtime, os-android, os-tvos
Milestone:	Future

[steveisok]

Looks like the failure was OOM. It may be a fluke. Will monitor to see.

[karelz]

@steveisok it keeps happening on regular basis -- lately mostly in Mono (with only 1 exception since 5/25)

Failures:

• 3/27-7/22 ... 23 failures ... 1-2x per week

let failedTests = (testNameSubstring : string, methodName : string, includePR : bool, messageSubstr: string, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    | where Message contains messageSubstr
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests('System.Numerics', '', true, 'OutOfMemoryException', true);

[steveisok]

Thanks for the query. Excluding wasm, the mono failures have occurred on devices, which makes me think this is random and is based upon how much contention at the time. Ideally, we should retry the test, but we don't have a good way to let the infra know in cases of OOM.

@akoeplinger Thoughts? I'm not yet sure skipping the tests is the right move here.

[danmoseley]

Would it help to mark this test library so it runs tests serially only on Android? Not sure if we have a mechanism to do that only for Android.

[steveisok]

Would it help to mark this test library so it runs tests serially only on Android? Not sure if we have a mechanism to do that only for Android.

When we tried it in the crypto tests, it did not make a difference. That said, it is worth a try.