Elasticsearch search method returning 0 hits

[aliktb]

I'm trying to write a few integration tests for some application code that interacts with elasticsearch in a spring boot application. However, I seem to have an issue with the search method of elasticsearch's RestHighLevelClient running in a testcontainer. I have tried writing a sample test as shown below where I am indexing a document and then searching for that document.

I can connect to the container fine and other methods work ok. I can index the document and retrieve it using the get method so I know it is being saved in the testcontainer. However trying the search method, it always returns 0 hits, i.e. it is not failing as a request but the response from elastic is that it finds no matching documents.

I know the queries work since I have tried running them in the application code and they work fine against a standard elasticsearch container. I've tried with and without an index template in case there is something about the document that would prevent it from being searchable but that has not solved it. I've tried a few other configurable options as shown in the commented out code but that again hasn't changed anything.

For reference I am running elasticsearch version 7.10.2 and testcontainers version 1.17.6.

I'm not sure if I am missing something fairly obvious since I can't imagine that testcontainers won't support the search functionality of elasticsearch.

   @BeforeEach
    void testIsContainerRunning(){
        Assertions.assertTrue(esContainer.isRunning());
    }

    @Test
    void saveAndSearchDocument(){

        RestHighLevelClient elasticSearchClient = new RestHighLevelClient(RestClient.builder(
                HttpHost.create(esContainer.getHttpHostAddress())
        ));

        PutIndexTemplateRequest templateRequest = new PutIndexTemplateRequest("template-name");

        List<String> patterns = new ArrayList<>();
        patterns.add("my*");

        templateRequest.patterns(patterns);

        try{
            XContentBuilder templateBuilder = XContentFactory.jsonBuilder();

            templateBuilder.startObject();
            {
                templateBuilder.startObject("_source");
                {
                    templateBuilder.field("enabled", true);
                }
                templateBuilder.endObject();

                templateBuilder.startObject("properties");
                {
                    templateBuilder.startObject("mykey");
                    {
                        templateBuilder.field("type", "text");
                        templateBuilder.field("analyzer", "whitespace");
                    }
                    templateBuilder.endObject();
                }
                templateBuilder.endObject();
            }
            templateBuilder.endObject();

            templateRequest.mapping(templateBuilder);

            elasticSearchClient.indices().putTemplate(templateRequest, RequestOptions.DEFAULT);

        }catch(Exception e){
            throw new RuntimeException(e);
        }

        IndexRequest request = new IndexRequest();
        request.id("xyz");
        request.index("myindex");

        JSONObject indexSource = new JSONObject();
        indexSource.put("mykey", "myval");
        request.source(indexSource.toString(), XContentType.JSON);

        RequestOptions.Builder builder = RequestOptions.DEFAULT.toBuilder();
        builder.addHeader("Content-Type", "application/json");

        System.out.println("SAVE REQUEST " + request);

        try {
            elasticSearchClient.index(request, builder.build());

            GetRequest gr = new GetRequest();
            gr.index("myindex");
            gr.id("xyz");
            GetResponse gresp = elasticSearchClient.get(gr, RequestOptions.DEFAULT);

            System.out.println("GRESP " + gresp);

            SearchRequest searchRequest = new SearchRequest("m*");
            SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

//            searchSourceBuilder.query(QueryBuilders.matchQuery("mykey","myval"));
            searchSourceBuilder.query(QueryBuilders.matchAllQuery());
            searchRequest.source(searchSourceBuilder);
//            searchRequest.allowPartialSearchResults(true);

//            IndicesOptions opt1 = IndicesOptions.fromOptions(false,
//                    true, true, true);

//            searchRequest.indicesOptions(opt1);

            RequestOptions.Builder optBuilder = RequestOptions.DEFAULT.toBuilder();
            optBuilder.addHeader("Content-Type", "application/json");

            System.out.println(optBuilder.build());

            System.out.println("sRequest " + searchRequest);

            SearchResponse sResponse = elasticSearchClient.search(searchRequest, optBuilder.build());

            System.out.println("sResponse " + sResponse);


        } catch (Exception e) {
            throw new DataUploadFailedException(e);
        }


    }

[aliktb]

This issue has been resolved. For future reference, the search API within elastic is not real-time like the get API. Therefore it is not possible for the java application to index a document and immediately search for it. The dummy test has been changed to below. In my actual test, I have called elasticSearchClient.indices.refresh(RefreshRequest) after indexing the data as a workaround to avoid using Thread.sleep.

@Testcontainers
public class SampleTest {

    static String ELASTICSEARCH_VERSION = "7.10.2";

    @Shared
    @Container
    private static ElasticsearchContainer esContainer =
            new ElasticsearchContainer(
                    DockerImageName.parse("docker.elastic.co/elasticsearch/elasticsearch-oss")
                            .withTag(ELASTICSEARCH_VERSION)
            );

    static RestHighLevelClient elasticSearchClient;

    static String id = "testid";
    static String index = "testindex";
    static String key = "testkey";
    static String value = "testvalue";

    @BeforeAll
    static void setUp(){

        elasticSearchClient = new RestHighLevelClient(RestClient.builder(HttpHost.create(esContainer.getHttpHostAddress())));

        Assertions.assertTrue(esContainer.isRunning());

        indexInElastic(id, index, key, value);

    }

    private static void indexInElastic(String id, String index, String key, String value){

        IndexRequest request = new IndexRequest();
        request.id(id);
        request.index(index);

        JSONObject indexSource = new JSONObject();
        indexSource.put(key, value);
        request.source(indexSource.toString(), XContentType.JSON);

        RequestOptions.Builder builder = RequestOptions.DEFAULT.toBuilder();
        builder.addHeader("Content-Type", "application/json");

        System.out.println("SAVE REQUEST " + request);

        try{

            IndexResponse indexResponse = elasticSearchClient.index(request, builder.build());
            System.out.println("INDEXRESPONSE" + indexResponse);

        }catch (Exception e){
            throw new RuntimeException(e);
        }

    }

    @Test
    void searchDocument(){

        try {

            GetRequest gr = new GetRequest();
            gr.index(index);
            gr.id(id);
            GetResponse gresp = elasticSearchClient.get(gr, RequestOptions.DEFAULT);

            System.out.println("GRESP " + gresp);

            SearchRequest searchRequest = new SearchRequest(index);
            SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

            searchSourceBuilder.query(QueryBuilders.matchAllQuery());
            searchRequest.source(searchSourceBuilder);


            RequestOptions.Builder optBuilder = RequestOptions.DEFAULT.toBuilder();
            optBuilder.addHeader("Content-Type", "application/json");

            System.out.println(optBuilder.build());

            System.out.println("sRequest " + searchRequest);

            SearchResponse sResponse = elasticSearchClient.search(searchRequest, optBuilder.build());

            System.out.println("sResponse " + sResponse);


        } catch (Exception e) {
            throw new RuntimeException(e);
        }


    }
}

[kiview]

Thanks for sharing your solution. I'd also suggest to look into using https://github.com/awaitility/awaitility for testing async behavior and code.

[aliktb]

Thanks @kiview, I'll definitely look into Awaitility 😄