[client-v2] "java.lang.IllegalArgumentException: array element type mismatch" for groupUniqArray with Tuple

[andris-rauda]

Describe the bug

We are encountering an IllegalArgumentException when reading response from query, containing groupUniqArray function with Tuples.

Steps to reproduce

1. Submit RowBinaryWithNamesAndTypes query containing groupUniqArray function with Tuples, e.g.

SELECT surname, groupUniqArray((id, name)) names FROM users GROUP BY surname ORDER BY surname

2. Try to read using Client.newBinaryFormatReader (also tried using CSVWithNames and Client.queryRecords)
3. This produces an exception:

java.lang.IllegalArgumentException: Failed to set value at index: 0 value [Ljava.lang.Object;@adcfad9 of class [Ljava.lang.Object;

(See full stack trace below)

Expected behaviour

Expected for query to work.

Code example

Here's a complete integration test demonstrating the problem. Workaround is to replace Tuple with Array: groupUniqArray([id, name])

package my.service.clickhouse;

import com.clickhouse.client.api.Client;
import com.clickhouse.client.api.command.CommandResponse;
import com.clickhouse.client.api.data_formats.ClickHouseBinaryFormatReader;
import com.clickhouse.client.api.data_formats.internal.BinaryStreamReader;
import com.clickhouse.client.api.query.*;
import com.clickhouse.data.ClickHouseFormat;
import org.junit.*;
import org.testcontainers.clickhouse.ClickHouseContainer;

import java.util.*;
import java.util.concurrent.*;

import static org.junit.Assert.*;

public class ClickHouseClientIntegrationTest {
    private static final ClickHouseContainer clickHouseContainer =
            new ClickHouseContainer("clickhouse/clickhouse-server")
                    .withCreateContainerCmdModifier(cmd -> cmd.withName("test-clickhouse-server"));

    private static Client client;

    @BeforeClass
    public static void setUpBeforeClass() throws Exception {
        clickHouseContainer.start();

        client = new Client.Builder()
//                .addEndpoint("http://localhost:8123")
                .addEndpoint("http://" + clickHouseContainer.getHost() + ":" + clickHouseContainer.getFirstMappedPort())
                .setUsername("default")
                .setPassword(null)
                .build();

        CompletableFuture<CommandResponse> responseFuture = client.execute( "CREATE TABLE users(" +
                "id Int64, name String, surname String) " +
                "ENGINE = ReplacingMergeTree PRIMARY KEY (id) ORDER BY (id)");
        assertEquals(0, responseFuture.get().getWrittenRows());

        responseFuture = client.execute( "INSERT INTO users(id, name, surname) VALUES (1, 'John', 'Smith')(2, 'Bob', 'Smith')");
        assertEquals(2, responseFuture.get().getWrittenRows());
    }

    @AfterClass
    public static void tearDownAfterClass() throws Exception {
        client.close();
    }

    @Test
    public void groupUniqArrayTuple() throws Exception {
        QuerySettings settings = new QuerySettings().setFormat(ClickHouseFormat.RowBinaryWithNamesAndTypes);
        CompletableFuture<QueryResponse> responseFuture = client.query(
                "SELECT surname, groupUniqArray(id, name)) names FROM users GROUP BY surname ORDER BY surname", settings);

        try (QueryResponse response = responseFuture.get();
             ClickHouseBinaryFormatReader reader = client.newBinaryFormatReader(response)) {

            List<Map<String, Object>> result = new ArrayList<>();
            while (reader.hasNext()) { // << Fails here
                result.add(new HashMap<>(reader.next()));
            }
            assertEquals(1, result.size());
            assertEquals("Smith", result.get(0).get("surname"));
            assertEquals("[[2, Bob], [1, John]]", ((BinaryStreamReader.ArrayValue) result.get(0).get("names")).asList().toString());
        }
    }

    @Test
    public void groupUniqArrayCSV() throws Exception {
        QuerySettings settings = new QuerySettings().setFormat(ClickHouseFormat.CSVWithNames);
        CompletableFuture<Records> responseFuture = client.queryRecords(
                "SELECT surname, groupUniqArray((id, name)) names FROM users GROUP BY surname ORDER BY surname", settings);

        try (Records records = responseFuture.get();) {  // << Fails here
            Iterator<GenericRecord> iterator = records.iterator();
            assertTrue(iterator.hasNext());
            GenericRecord record = iterator.next();
            assertEquals("Smith", record.getString("surname"));
            assertEquals("[Bob, John]", ((BinaryStreamReader.ArrayValue)record.getObject("names")).asList().toString());
            assertFalse(iterator.hasNext());
        }
    }

    @Test
    public void groupUniqArrayOfArraysWorkaround() throws Exception {
        QuerySettings settings = new QuerySettings().setFormat(ClickHouseFormat.RowBinaryWithNamesAndTypes);
        CompletableFuture<QueryResponse> responseFuture = client.query(
                "SELECT surname, groupUniqArray([toString(id), name]) names FROM users GROUP BY surname ORDER BY surname", settings);

        try (QueryResponse response = responseFuture.get();
             ClickHouseBinaryFormatReader reader = client.newBinaryFormatReader(response)) {

            List<Map<String, Object>> result = new ArrayList<>();
            while (reader.hasNext()) {
                result.add(new HashMap<>(reader.next()));
            }
            assertEquals(1, result.size());
            assertEquals("Smith", result.get(0).get("surname"));
            assertEquals("[[2, Bob], [1, John]]", ((BinaryStreamReader.ArrayValue) result.get(0).get("names")).asList().toString());
        }
    }
}

Error log

Failed to set value at index: 0 value [Ljava.lang.Object;@adcfad9 of class [Ljava.lang.Object;
java.lang.IllegalArgumentException: Failed to set value at index: 0 value [Ljava.lang.Object;@adcfad9 of class [Ljava.lang.Object;
	at com.clickhouse.client.api.data_formats.internal.BinaryStreamReader$ArrayValue.set(BinaryStreamReader.java:580)
	at com.clickhouse.client.api.data_formats.internal.BinaryStreamReader.readArray(BinaryStreamReader.java:534)
	at com.clickhouse.client.api.data_formats.internal.BinaryStreamReader.readValue(BinaryStreamReader.java:209)
	at com.clickhouse.client.api.data_formats.internal.BinaryStreamReader.readValue(BinaryStreamReader.java:82)
	at com.clickhouse.client.api.data_formats.internal.AbstractBinaryFormatReader.readRecord(AbstractBinaryFormatReader.java:130)
	at com.clickhouse.client.api.data_formats.internal.AbstractBinaryFormatReader.readNextRecord(AbstractBinaryFormatReader.java:176)
	at com.clickhouse.client.api.data_formats.internal.AbstractBinaryFormatReader.hasNext(AbstractBinaryFormatReader.java:165)
	at my.service.clickhouse.ClickHouseClientIntegrationTest.groupUniqArrayTuple(ClickHouseClientIntegrationTest.java:120)
	...
Caused by: java.lang.IllegalArgumentException: array element type mismatch
	at java.base/java.lang.reflect.Array.set(Native Method)
	at com.clickhouse.client.api.data_formats.internal.BinaryStreamReader$ArrayValue.set(BinaryStreamReader.java:577)
	...

Configuration

Environment

• Client version: 0.7.0
• Language version: Java 11
• OS: MacOS Sonoma 14.6

ClickHouse server

• ClickHouse Server version: 24.9.2.42
• CREATE TABLE statements for tables involved:

CREATE TABLE users(id Int64, name String, surname String) ENGINE = ReplacingMergeTree PRIMARY KEY (id) ORDER BY (id);

• Sample data for all these tables:

INSERT INTO users(id, name, surname) VALUES (1, 'John', 'Smith')(2, 'Bob', 'Smith');

[chernser]

@andris-rauda thank you for reporting!
I will look into it shortly.