MatrixStats when all values are 1-> UnexpectedElasticsearchClientException

[msw16]

Similar issue from #5007

NEST/Elasticsearch.Net version:
7.17
Elasticsearch version:
8.10 / 8.15
.NET runtime version:
8
Operating system version:
Windows 11
Description of the problem including expected versus actual behavior:
A clear and concise description of what the bug is.

"message": "An error has occurred.",
 "exceptionMessage": "expected:'Number Token', actual:'\"NaN\"', at offset:179002",
 "exceptionType": "Elasticsearch.Net.UnexpectedElasticsearchClientException",

Steps to reproduce:

1. One field, which is mapped as double, more than 1 data but values are all 1
2. While querying aggregation with MatrixStats, skewness and kurtosis is "NaN", which is right calculation but got the exception while creating a response
3. Seems Extended Stats are patched but not MatrixStats

var searchResponse = await _Nest.SearchAsync<Document>(s => s
    .Index(indexName)
    .Size(0)
    .Query(_ => queryContainer)
    .Aggregations(a => a
        .MatrixStats("matrix_stats", m => m
            .Field(new[] { field })
        )
        .ExtendedStats("extended_stats", e => e
            .Field(field)
        )
        .Percentiles("percentiles", p => p
            .Field(field)
            .Percents(5.0, 25.0, 50.0, 75.0, 95.0)
        )
        .Missing("missing", m => m
            .Field(field)
        )
    )
    .TrackTotalHits()
);

Expected behavior
A clear and concise description of what you expected to happen.
SearchApi shouldn't be failed while returns query

Provide ConnectionSettings (if relevant):

Provide DebugInformation (if relevant):

[flobernd]

Hi @msw16, it seems that the JSON library used in the old NEST versions of the client does not correctly handle special float values like NaN.

Is there any chance you can upgrade to the new package Elastic.Clients.Elasticsearch?

[msw16]

I might need to check all migration guidance but just quickly deleted previous Nest 7.17.5 and NEST.JsonNetSerializer 7.17.5, replaced it with Elastic.Clients.Elasticsearch 8.16.3 but still works same and SearchAsync is still pointing to Nest Version 7.0.0.0. Could you check SearchAsync is still returning "NaN" as it is with latest version of client, not like ExtendedStats?

[flobernd]

@msw16

quickly deleted previous Nest 7.17.5 and NEST.JsonNetSerializer 7.17.5, replaced it with Elastic.Clients.Elasticsearch 8.16.3 but still works same and SearchAsync is still pointing to Nest Version 7.0.0.0.

Maybe you can create a completely fresh test project to try out this scenario with the new Elastic.Clients.Elasticsearch package? It seems like your dependencies got confused and you are still using NEST under the hood.

Could you check SearchAsync is still returning "NaN" as it is with latest version of client

The NaN value is returned by the server. The problem is that the NEST package fails to deserialize the JSON string "NaN" as a floating-point value. The new client uses System.Text.Json for (de-)serialization which handles "NaN" strings just fine 🙂