Support Alternate Datetime Formats (#268)

* Add OpenSearchDateType as a datatype for matching with Date/Time OpenSearch types Signed-off-by: Andrew Carbonetto <[email protected]> * Added IT Test Signed-off-by: GabeFernandez310 <[email protected]> * Added Basic Implementation Signed-off-by: GabeFernandez310 <[email protected]> * Added A Comment Signed-off-by: GabeFernandez310 <[email protected]> * Added Support For Default OpenSearch Formats Signed-off-by: GabeFernandez310 <[email protected]> * Code Cleanup Signed-off-by: GabeFernandez310 <[email protected]> * Refactored 'parsetimestamp' Function Signed-off-by: GabeFernandez310 <[email protected]> * Cleanup Signed-off-by: GabeFernandez310 <[email protected]> * Removed Unnecessary Try-Catch Block Signed-off-by: GabeFernandez310 <[email protected]> * Fixed Unit Tests Signed-off-by: GabeFernandez310 <[email protected]> * Fixed IT Test Signed-off-by: GabeFernandez310 <[email protected]> * Fixed Text Type and Disabled JSON Format Test Signed-off-by: GabeFernandez310 <[email protected]> * Fix Mapping Type Reporting As Date Instead Of Timestamp In Schema Signed-off-by: GabeFernandez310 <[email protected]> * Revert Breaking Test Changes Signed-off-by: GabeFernandez310 <[email protected]> * Fixed Returned ExprCoreTypes Signed-off-by: GabeFernandez310 <[email protected]> * Fixed OpenSearch Type Names In Mapping Signed-off-by: GabeFernandez310 <[email protected]> * Fixed LegacyTypeName Function And Refactored Code To Fix Calculations With Datetime Types Signed-off-by: GabeFernandez310 <[email protected]> * Changed Support For Passing Fields Into Text Type Signed-off-by: GabeFernandez310 <[email protected]> * Reverted Some Changes And Commented Out A Test Signed-off-by: GabeFernandez310 <[email protected]> * Commented Out A Breaking Change Signed-off-by: GabeFernandez310 <[email protected]> * Minor Refactoring Changes Signed-off-by: GabeFernandez310 <[email protected]> * Reworked parseTimestamp Function To Format Return Based On Enum Signed-off-by: GabeFernandez310 <[email protected]> * Fixed Returned ExprValue Signed-off-by: GabeFernandez310 <[email protected]> * Reverted Changes To Test Files Signed-off-by: GabeFernandez310 <[email protected]> * Reverted Some Changes To Enum And Reworked Code Signed-off-by: GabeFernandez310 <[email protected]> * Added Type Mapping To Allow Query To Be Parsed Signed-off-by: GabeFernandez310 <[email protected]> * Fixed Some Unit Tests Signed-off-by: GabeFernandez310 <[email protected]> * Fixed TypeActionMap To Still Support ExprCoreTypes Signed-off-by: GabeFernandez310 <[email protected]> * Wrapped Some Function Returns in ExprValueFactory Signed-off-by: GabeFernandez310 <[email protected]> * Fixed Integration Test Bug Signed-off-by: GabeFernandez310 <[email protected]> (cherry picked from commit 2c16a554573015fcd752721353b2713858caad1a) * Slight Refactoring And Code Cleanup Signed-off-by: GabeFernandez310 <[email protected]> (cherry picked from commit ebecec79c79d75e851a4a0914cf4abf5aa6d3efd) * Removed Unneeded Class Member Signed-off-by: GabeFernandez310 <[email protected]> (cherry picked from commit 14de39815ebeecd36c4155099d875a45a7e008f6) * Removed Unneccessary MappingType Parameters Signed-off-by: GabeFernandez310 <[email protected]> (cherry picked from commit e6a8bc3a6a3d7a11dacd03fc885871f3b055d2d2) * Refactored Code To Fix Integration Test Signed-off-by: GabeFernandez310 <[email protected]> (cherry picked from commit c91f48d9763499940ecf302212fcd592aecb6018) * Addressed JaCoCo Issues Signed-off-by: GabeFernandez310 <[email protected]> * Fixed Checkstyle Signed-off-by: GabeFernandez310 <[email protected]> * Changed gradle version and removed values iterator Signed-off-by: Guian Gumpac <[email protected]> * Addressed PR Comments Signed-off-by: GabeFernandez310 <[email protected]> * Update a test to match new indexResponse.aliases() type. Signed-off-by: MaxKsyunz <[email protected]> * Changed To Throw Exception When Encountering Parsing Failures Signed-off-by: GabeFernandez310 <[email protected]> * Fixed Checkstyle Signed-off-by: GabeFernandez310 <[email protected]> * Removed Unneeded Imports Signed-off-by: GabeFernandez310 <[email protected]> * Re-enabled Some Tests Signed-off-by: GabeFernandez310 <[email protected]> * Addressed PR Comments Signed-off-by: GabeFernandez310 <[email protected]> * Changed test data Signed-off-by: Guian Gumpac <[email protected]> * Pulled out support for user-defined datetime formats Signed-off-by: Guian Gumpac <[email protected]> * Modified IT test data Signed-off-by: Guian Gumpac <[email protected]> * Added IT tests Signed-off-by: Guian Gumpac <[email protected]> * Removed support for just time and just date formats Signed-off-by: Guian Gumpac <[email protected]> * Removed unsupported formats from IT Signed-off-by: Guian Gumpac <[email protected]> * Removed unit tests for date only or time only formats Signed-off-by: Guian Gumpac <[email protected]> * Addressed PR comments from previous PR Signed-off-by: Guian Gumpac <[email protected]> * Addressed PR comment regarding unnecessary OpenSearchDateType instance in OpenSearchDataType Signed-off-by: Guian Gumpac <[email protected]> * Updates to include build-in date and time formats for OpenSearchDateTypes Signed-off-by: Andrew Carbonetto <[email protected]> * Add tests and fix checkstyle Signed-off-by: Andrew Carbonetto <[email protected]> * Fixed bugs regarding custom formats Signed-off-by: Guian Gumpac <[email protected]> * Add unit test to cover custom formats Signed-off-by: Andrew Carbonetto <[email protected]> * Addressed PR comments Signed-off-by: Guian Gumpac <[email protected]> * Fixed checkstyle errors Signed-off-by: Guian Gumpac <[email protected]> * Modified integration test Signed-off-by: Guian Gumpac <[email protected]> * Fixed Date and time aggregation Signed-off-by: Guian Gumpac <[email protected]> * Fixed failing integ tests Signed-off-by: Guian Gumpac <[email protected]> * Modified integ test based on date format changes Signed-off-by: Guian Gumpac <[email protected]> * Added default formatters for date and time. Added unit tests for coverage Signed-off-by: Guian Gumpac <[email protected]> * Some minor review comments Signed-off-by: Andrew Carbonetto <[email protected]> * Addressed PR comments Signed-off-by: Guian Gumpac <[email protected]> * Fixed checktsyle and jacoco Signed-off-by: Guian Gumpac <[email protected]> * Changed link in rst file Signed-off-by: Guian Gumpac <[email protected]> * Changed link in rst file Signed-off-by: Guian Gumpac <[email protected]> * Addressed PR comments Signed-off-by: Guian Gumpac <[email protected]> --------- Signed-off-by: Andrew Carbonetto <[email protected]> Signed-off-by: GabeFernandez310 <[email protected]> Signed-off-by: Guian Gumpac <[email protected]> Signed-off-by: MaxKsyunz <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> Co-authored-by: MaxKsyunz <[email protected]>
opensearch-project · May 29, 2023 · 0e5ec71 · 0e5ec71
1 parent a924d5a
commit 0e5ec71
Show file tree

Hide file tree

Showing 22 changed files with 1,552 additions and 278 deletions.
diff --git a/docs/user/general/datatypes.rst b/docs/user/general/datatypes.rst
@@ -91,7 +91,7 @@ The table below list the mapping between OpenSearch Data Type, OpenSearch SQL Da
 +-----------------+---------------------+-----------+
 | text            | text                | VARCHAR   |
 +-----------------+---------------------+-----------+
-| date            | timestamp           | TIMESTAMP |
+| date*           | timestamp           | TIMESTAMP |
 +-----------------+---------------------+-----------+
 | date_nanos      | timestamp           | TIMESTAMP |
 +-----------------+---------------------+-----------+
@@ -104,7 +104,11 @@ The table below list the mapping between OpenSearch Data Type, OpenSearch SQL Da
 | nested          | array               | STRUCT    |
 +-----------------+---------------------+-----------+
 
-Notes: Not all the OpenSearch SQL Type has correspond OpenSearch Type. e.g. data and time. To use function which required such data type, user should explicitly convert the data type.
+Notes:
+* Not all the OpenSearch SQL Type has correspond OpenSearch Type. e.g. data and time. To use function which required such data type, user should explicitly convert the data type.
+* date*: Maps to `timestamp` by default. Based on the "format" property `date` can map to `date` or `time`. See list of supported named formats `here <https://opensearch.org/docs/latest/field-types/supported-field-types/date/>`_.
+For example, `basic_date` will map to a `date` type, and `basic_time` will map to a `time` type.
+
 
 Data Type Conversion
 ====================

diff --git a/integ-test/src/test/java/org/opensearch/sql/legacy/SQLIntegTestCase.java b/integ-test/src/test/java/org/opensearch/sql/legacy/SQLIntegTestCase.java
@@ -651,6 +651,10 @@ public enum Index {
         "calcs",
         getMappingFile("calcs_index_mappings.json"),
         "src/test/resources/calcs.json"),
+    DATE_FORMATS(TestsConstants.TEST_INDEX_DATE_FORMATS,
+        "date_formats",
+        getMappingFile("date_formats_index_mapping.json"),
+        "src/test/resources/date_formats.json"),
     WILDCARD(TestsConstants.TEST_INDEX_WILDCARD,
         "wildcard",
         getMappingFile("wildcard_index_mappings.json"),

diff --git a/integ-test/src/test/java/org/opensearch/sql/legacy/TestsConstants.java b/integ-test/src/test/java/org/opensearch/sql/legacy/TestsConstants.java
@@ -56,6 +56,7 @@ public class TestsConstants {
   public final static String TEST_INDEX_BEER = TEST_INDEX + "_beer";
   public final static String TEST_INDEX_NULL_MISSING = TEST_INDEX + "_null_missing";
   public final static String TEST_INDEX_CALCS = TEST_INDEX + "_calcs";
+  public final static String TEST_INDEX_DATE_FORMATS = TEST_INDEX + "_date_formats";
   public final static String TEST_INDEX_WILDCARD = TEST_INDEX + "_wildcard";
   public final static String TEST_INDEX_MULTI_NESTED_TYPE = TEST_INDEX + "_multi_nested";
   public final static String TEST_INDEX_NESTED_WITH_NULLS = TEST_INDEX + "_nested_with_nulls";

diff --git a/integ-test/src/test/java/org/opensearch/sql/sql/AggregationIT.java b/integ-test/src/test/java/org/opensearch/sql/sql/AggregationIT.java
@@ -383,17 +383,17 @@ public void testAvgDoubleInMemory() throws IOException {
 
   @Test
   public void testMaxDatePushedDown() throws IOException {
-    var response = executeQuery(String.format("SELECT max(CAST(date0 AS date))"
+    var response = executeQuery(String.format("SELECT max(date0)"
         + " from %s", TEST_INDEX_CALCS));
-    verifySchema(response, schema("max(CAST(date0 AS date))", null, "date"));
+    verifySchema(response, schema("max(date0)", null, "date"));
     verifyDataRows(response, rows("2004-06-19"));
   }
 
   @Test
   public void testAvgDatePushedDown() throws IOException {
-    var response = executeQuery(String.format("SELECT avg(CAST(date0 AS date))"
+    var response = executeQuery(String.format("SELECT avg(date0)"
         + " from %s", TEST_INDEX_CALCS));
-    verifySchema(response, schema("avg(CAST(date0 AS date))", null, "date"));
+    verifySchema(response, schema("avg(date0)", null, "date"));
     verifyDataRows(response, rows("1992-04-23"));
   }
 
@@ -423,25 +423,25 @@ public void testAvgDateTimePushedDown() throws IOException {
 
   @Test
   public void testMinTimePushedDown() throws IOException {
-    var response = executeQuery(String.format("SELECT min(CAST(time1 AS time))"
+    var response = executeQuery(String.format("SELECT min(time1)"
         + " from %s", TEST_INDEX_CALCS));
-    verifySchema(response, schema("min(CAST(time1 AS time))", null, "time"));
+    verifySchema(response, schema("min(time1)", null, "time"));
     verifyDataRows(response, rows("00:05:57"));
   }
 
   @Test
   public void testMaxTimePushedDown() throws IOException {
-    var response = executeQuery(String.format("SELECT max(CAST(time1 AS time))"
+    var response = executeQuery(String.format("SELECT max(time1)"
         + " from %s", TEST_INDEX_CALCS));
-    verifySchema(response, schema("max(CAST(time1 AS time))", null, "time"));
+    verifySchema(response, schema("max(time1)", null, "time"));
     verifyDataRows(response, rows("22:50:16"));
   }
 
   @Test
   public void testAvgTimePushedDown() throws IOException {
-    var response = executeQuery(String.format("SELECT avg(CAST(time1 AS time))"
+    var response = executeQuery(String.format("SELECT avg(time1)"
         + " from %s", TEST_INDEX_CALCS));
-    verifySchema(response, schema("avg(CAST(time1 AS time))", null, "time"));
+    verifySchema(response, schema("avg(time1)", null, "time"));
     verifyDataRows(response, rows("13:06:36.25"));
   }
 
@@ -471,28 +471,28 @@ public void testAvgTimeStampPushedDown() throws IOException {
 
   @Test
   public void testMinDateInMemory() throws IOException {
-    var response = executeQuery(String.format("SELECT min(CAST(date0 AS date))"
+    var response = executeQuery(String.format("SELECT min(date0)"
         + " OVER(PARTITION BY datetime1) from %s", TEST_INDEX_CALCS));
     verifySchema(response,
-        schema("min(CAST(date0 AS date)) OVER(PARTITION BY datetime1)", null, "date"));
+        schema("min(date0) OVER(PARTITION BY datetime1)", null, "date"));
     verifySome(response.getJSONArray("datarows"), rows("1972-07-04"));
   }
 
   @Test
   public void testMaxDateInMemory() throws IOException {
-    var response = executeQuery(String.format("SELECT max(CAST(date0 AS date))"
+    var response = executeQuery(String.format("SELECT max(date0)"
         + " OVER(PARTITION BY datetime1) from %s", TEST_INDEX_CALCS));
     verifySchema(response,
-        schema("max(CAST(date0 AS date)) OVER(PARTITION BY datetime1)", null, "date"));
+        schema("max(date0) OVER(PARTITION BY datetime1)", null, "date"));
     verifySome(response.getJSONArray("datarows"), rows("2004-06-19"));
   }
 
   @Test
   public void testAvgDateInMemory() throws IOException {
-    var response = executeQuery(String.format("SELECT avg(CAST(date0 AS date))"
+    var response = executeQuery(String.format("SELECT avg(date0)"
         + " OVER(PARTITION BY datetime1) from %s", TEST_INDEX_CALCS));
     verifySchema(response,
-        schema("avg(CAST(date0 AS date)) OVER(PARTITION BY datetime1)", null, "date"));
+        schema("avg(date0) OVER(PARTITION BY datetime1)", null, "date"));
     verifySome(response.getJSONArray("datarows"), rows("1992-04-23"));
   }
 
@@ -525,28 +525,28 @@ public void testAvgDateTimeInMemory() throws IOException {
 
   @Test
   public void testMinTimeInMemory() throws IOException {
-    var response = executeQuery(String.format("SELECT min(CAST(time1 AS time))"
+    var response = executeQuery(String.format("SELECT min(time1)"
         + " OVER(PARTITION BY datetime1) from %s", TEST_INDEX_CALCS));
     verifySchema(response,
-        schema("min(CAST(time1 AS time)) OVER(PARTITION BY datetime1)", null, "time"));
+        schema("min(time1) OVER(PARTITION BY datetime1)", null, "time"));
     verifySome(response.getJSONArray("datarows"), rows("00:05:57"));
   }
 
   @Test
   public void testMaxTimeInMemory() throws IOException {
-    var response = executeQuery(String.format("SELECT max(CAST(time1 AS time))"
+    var response = executeQuery(String.format("SELECT max(time1)"
         + " OVER(PARTITION BY datetime1) from %s", TEST_INDEX_CALCS));
     verifySchema(response,
-        schema("max(CAST(time1 AS time)) OVER(PARTITION BY datetime1)", null, "time"));
+        schema("max(time1) OVER(PARTITION BY datetime1)", null, "time"));
     verifySome(response.getJSONArray("datarows"), rows("22:50:16"));
   }
 
   @Test
   public void testAvgTimeInMemory() throws IOException {
-    var response = executeQuery(String.format("SELECT avg(CAST(time1 AS time))"
+    var response = executeQuery(String.format("SELECT avg(time1)"
         + " OVER(PARTITION BY datetime1) from %s", TEST_INDEX_CALCS));
     verifySchema(response,
-        schema("avg(CAST(time1 AS time)) OVER(PARTITION BY datetime1)", null, "time"));
+        schema("avg(time1) OVER(PARTITION BY datetime1)", null, "time"));
     verifySome(response.getJSONArray("datarows"), rows("13:06:36.25"));
   }
 

diff --git a/integ-test/src/test/java/org/opensearch/sql/sql/DateTimeFormatsIT.java b/integ-test/src/test/java/org/opensearch/sql/sql/DateTimeFormatsIT.java
@@ -0,0 +1,70 @@
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+
+package org.opensearch.sql.sql;
+
+import static org.opensearch.sql.legacy.TestsConstants.TEST_INDEX_DATE_FORMATS;
+import static org.opensearch.sql.legacy.plugin.RestSqlAction.QUERY_API_ENDPOINT;
+import static org.opensearch.sql.util.MatcherUtils.rows;
+import static org.opensearch.sql.util.MatcherUtils.schema;
+import static org.opensearch.sql.util.MatcherUtils.verifyDataRows;
+import static org.opensearch.sql.util.MatcherUtils.verifySchema;
+import static org.opensearch.sql.util.TestUtils.getResponseBody;
+
+import java.io.IOException;
+import java.util.Locale;
+import org.json.JSONObject;
+import org.junit.jupiter.api.Test;
+import org.opensearch.client.Request;
+import org.opensearch.client.RequestOptions;
+import org.opensearch.client.Response;
+import org.opensearch.sql.legacy.SQLIntegTestCase;
+
+public class DateTimeFormatsIT extends SQLIntegTestCase {
+
+  @Override
+  public void init() throws Exception {
+    super.init();
+    loadIndex(Index.DATE_FORMATS);
+  }
+
+  @Test
+  public void testReadingDateFormats() throws IOException {
+    String query = String.format("SELECT weekyear_week_day, hour_minute_second_millis," +
+        " strict_ordinal_date_time FROM %s LIMIT 1", TEST_INDEX_DATE_FORMATS);
+    JSONObject result = executeQuery(query);
+    verifySchema(result,
+        schema("weekyear_week_day", null, "date"),
+        schema("hour_minute_second_millis", null, "time"),
+        schema("strict_ordinal_date_time", null, "timestamp"));
+    verifyDataRows(result,
+        rows("1984-04-12",
+            "09:07:42",
+            "1984-04-12 09:07:42.000123456"
+        ));
+  }
+
+  @Test
+  public void testDateFormatsWithOr() throws IOException {
+    String query = String.format("SELECT yyyy-MM-dd_OR_epoch_millis FROM %s", TEST_INDEX_DATE_FORMATS);
+    JSONObject result = executeQuery(query);
+    verifyDataRows(result,
+        rows("1984-04-12 00:00:00"),
+        rows("1984-04-12 09:07:42.000123456"));
+  }
+
+  protected JSONObject executeQuery(String query) throws IOException {
+    Request request = new Request("POST", QUERY_API_ENDPOINT);
+    request.setJsonEntity(String.format(Locale.ROOT, "{\n" + "  \"query\": \"%s\"\n" + "}", query));
+
+    RequestOptions.Builder restOptionsBuilder = RequestOptions.DEFAULT.toBuilder();
+    restOptionsBuilder.addHeader("Content-Type", "application/json");
+    request.setOptions(restOptionsBuilder);
+
+    Response response = client().performRequest(request);
+    return new JSONObject(getResponseBody(response));
+  }
+}