-
Notifications
You must be signed in to change notification settings - Fork 141
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support Alternate Datetime Formats #1664
Merged
acarbonetto
merged 2 commits into
opensearch-project:main
from
Bit-Quill:integ-support-datetime-formats
May 30, 2023
Merged
Support Alternate Datetime Formats #1664
acarbonetto
merged 2 commits into
opensearch-project:main
from
Bit-Quill:integ-support-datetime-formats
May 30, 2023
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
GumpacG
requested review from
pjfitzgibbons,
ps48,
kavithacm,
derek-ho,
joshuali925,
dai-chen,
YANG-DB,
rupal-bq,
mengweieric,
vmmusings,
Swiddis,
penghuo,
seankao-az,
MaxKsyunz,
Yury-Fridlyand,
anirudha,
forestmvey and
acarbonetto
as code owners
May 29, 2023 22:19
Codecov Report
@@ Coverage Diff @@
## main #1664 +/- ##
============================================
+ Coverage 97.16% 97.19% +0.02%
- Complexity 4120 4165 +45
============================================
Files 371 372 +1
Lines 10373 10478 +105
Branches 704 720 +16
============================================
+ Hits 10079 10184 +105
Misses 287 287
Partials 7 7
Flags with carried forward coverage won't be shown. Click here to find out more.
|
acarbonetto
approved these changes
May 29, 2023
acarbonetto
approved these changes
May 30, 2023
Yury-Fridlyand
previously approved these changes
May 30, 2023
...n/java/org/opensearch/sql/opensearch/storage/script/aggregation/AggregationQueryBuilder.java
Outdated
Show resolved
Hide resolved
...search/src/main/java/org/opensearch/sql/opensearch/storage/script/core/ExpressionScript.java
Outdated
Show resolved
Hide resolved
matthewryanwells
previously approved these changes
May 30, 2023
acarbonetto
reviewed
May 30, 2023
...n/java/org/opensearch/sql/opensearch/storage/script/aggregation/AggregationQueryBuilder.java
Outdated
Show resolved
Hide resolved
acarbonetto
previously approved these changes
May 30, 2023
...search/src/main/java/org/opensearch/sql/opensearch/storage/script/core/ExpressionScript.java
Outdated
Show resolved
Hide resolved
* Add OpenSearchDateType as a datatype for matching with Date/Time OpenSearch types Signed-off-by: Andrew Carbonetto <[email protected]> * Added IT Test Signed-off-by: GabeFernandez310 <[email protected]> * Added Basic Implementation Signed-off-by: GabeFernandez310 <[email protected]> * Added A Comment Signed-off-by: GabeFernandez310 <[email protected]> * Added Support For Default OpenSearch Formats Signed-off-by: GabeFernandez310 <[email protected]> * Code Cleanup Signed-off-by: GabeFernandez310 <[email protected]> * Refactored 'parsetimestamp' Function Signed-off-by: GabeFernandez310 <[email protected]> * Cleanup Signed-off-by: GabeFernandez310 <[email protected]> * Removed Unnecessary Try-Catch Block Signed-off-by: GabeFernandez310 <[email protected]> * Fixed Unit Tests Signed-off-by: GabeFernandez310 <[email protected]> * Fixed IT Test Signed-off-by: GabeFernandez310 <[email protected]> * Fixed Text Type and Disabled JSON Format Test Signed-off-by: GabeFernandez310 <[email protected]> * Fix Mapping Type Reporting As Date Instead Of Timestamp In Schema Signed-off-by: GabeFernandez310 <[email protected]> * Revert Breaking Test Changes Signed-off-by: GabeFernandez310 <[email protected]> * Fixed Returned ExprCoreTypes Signed-off-by: GabeFernandez310 <[email protected]> * Fixed OpenSearch Type Names In Mapping Signed-off-by: GabeFernandez310 <[email protected]> * Fixed LegacyTypeName Function And Refactored Code To Fix Calculations With Datetime Types Signed-off-by: GabeFernandez310 <[email protected]> * Changed Support For Passing Fields Into Text Type Signed-off-by: GabeFernandez310 <[email protected]> * Reverted Some Changes And Commented Out A Test Signed-off-by: GabeFernandez310 <[email protected]> * Commented Out A Breaking Change Signed-off-by: GabeFernandez310 <[email protected]> * Minor Refactoring Changes Signed-off-by: GabeFernandez310 <[email protected]> * Reworked parseTimestamp Function To Format Return Based On Enum Signed-off-by: GabeFernandez310 <[email protected]> * Fixed Returned ExprValue Signed-off-by: GabeFernandez310 <[email protected]> * Reverted Changes To Test Files Signed-off-by: GabeFernandez310 <[email protected]> * Reverted Some Changes To Enum And Reworked Code Signed-off-by: GabeFernandez310 <[email protected]> * Added Type Mapping To Allow Query To Be Parsed Signed-off-by: GabeFernandez310 <[email protected]> * Fixed Some Unit Tests Signed-off-by: GabeFernandez310 <[email protected]> * Fixed TypeActionMap To Still Support ExprCoreTypes Signed-off-by: GabeFernandez310 <[email protected]> * Wrapped Some Function Returns in ExprValueFactory Signed-off-by: GabeFernandez310 <[email protected]> * Fixed Integration Test Bug Signed-off-by: GabeFernandez310 <[email protected]> (cherry picked from commit 2c16a554573015fcd752721353b2713858caad1a) * Slight Refactoring And Code Cleanup Signed-off-by: GabeFernandez310 <[email protected]> (cherry picked from commit ebecec79c79d75e851a4a0914cf4abf5aa6d3efd) * Removed Unneeded Class Member Signed-off-by: GabeFernandez310 <[email protected]> (cherry picked from commit 14de39815ebeecd36c4155099d875a45a7e008f6) * Removed Unneccessary MappingType Parameters Signed-off-by: GabeFernandez310 <[email protected]> (cherry picked from commit e6a8bc3a6a3d7a11dacd03fc885871f3b055d2d2) * Refactored Code To Fix Integration Test Signed-off-by: GabeFernandez310 <[email protected]> (cherry picked from commit c91f48d9763499940ecf302212fcd592aecb6018) * Addressed JaCoCo Issues Signed-off-by: GabeFernandez310 <[email protected]> * Fixed Checkstyle Signed-off-by: GabeFernandez310 <[email protected]> * Changed gradle version and removed values iterator Signed-off-by: Guian Gumpac <[email protected]> * Addressed PR Comments Signed-off-by: GabeFernandez310 <[email protected]> * Update a test to match new indexResponse.aliases() type. Signed-off-by: MaxKsyunz <[email protected]> * Changed To Throw Exception When Encountering Parsing Failures Signed-off-by: GabeFernandez310 <[email protected]> * Fixed Checkstyle Signed-off-by: GabeFernandez310 <[email protected]> * Removed Unneeded Imports Signed-off-by: GabeFernandez310 <[email protected]> * Re-enabled Some Tests Signed-off-by: GabeFernandez310 <[email protected]> * Addressed PR Comments Signed-off-by: GabeFernandez310 <[email protected]> * Changed test data Signed-off-by: Guian Gumpac <[email protected]> * Pulled out support for user-defined datetime formats Signed-off-by: Guian Gumpac <[email protected]> * Modified IT test data Signed-off-by: Guian Gumpac <[email protected]> * Added IT tests Signed-off-by: Guian Gumpac <[email protected]> * Removed support for just time and just date formats Signed-off-by: Guian Gumpac <[email protected]> * Removed unsupported formats from IT Signed-off-by: Guian Gumpac <[email protected]> * Removed unit tests for date only or time only formats Signed-off-by: Guian Gumpac <[email protected]> * Addressed PR comments from previous PR Signed-off-by: Guian Gumpac <[email protected]> * Addressed PR comment regarding unnecessary OpenSearchDateType instance in OpenSearchDataType Signed-off-by: Guian Gumpac <[email protected]> * Updates to include build-in date and time formats for OpenSearchDateTypes Signed-off-by: Andrew Carbonetto <[email protected]> * Add tests and fix checkstyle Signed-off-by: Andrew Carbonetto <[email protected]> * Fixed bugs regarding custom formats Signed-off-by: Guian Gumpac <[email protected]> * Add unit test to cover custom formats Signed-off-by: Andrew Carbonetto <[email protected]> * Addressed PR comments Signed-off-by: Guian Gumpac <[email protected]> * Fixed checkstyle errors Signed-off-by: Guian Gumpac <[email protected]> * Modified integration test Signed-off-by: Guian Gumpac <[email protected]> * Fixed Date and time aggregation Signed-off-by: Guian Gumpac <[email protected]> * Fixed failing integ tests Signed-off-by: Guian Gumpac <[email protected]> * Modified integ test based on date format changes Signed-off-by: Guian Gumpac <[email protected]> * Added default formatters for date and time. Added unit tests for coverage Signed-off-by: Guian Gumpac <[email protected]> * Some minor review comments Signed-off-by: Andrew Carbonetto <[email protected]> * Addressed PR comments Signed-off-by: Guian Gumpac <[email protected]> * Fixed checktsyle and jacoco Signed-off-by: Guian Gumpac <[email protected]> * Changed link in rst file Signed-off-by: Guian Gumpac <[email protected]> * Changed link in rst file Signed-off-by: Guian Gumpac <[email protected]> * Addressed PR comments Signed-off-by: Guian Gumpac <[email protected]> --------- Signed-off-by: Andrew Carbonetto <[email protected]> Signed-off-by: GabeFernandez310 <[email protected]> Signed-off-by: Guian Gumpac <[email protected]> Signed-off-by: MaxKsyunz <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> Co-authored-by: MaxKsyunz <[email protected]>
GumpacG
force-pushed
the
integ-support-datetime-formats
branch
from
May 30, 2023 17:00
0e5ec71
to
1345794
Compare
Signed-off-by: Guian Gumpac <[email protected]>
GumpacG
dismissed stale reviews from matthewryanwells, acarbonetto, and Yury-Fridlyand
via
May 30, 2023 17:12
1591a85
Yury-Fridlyand
approved these changes
May 30, 2023
MitchellGale
approved these changes
May 30, 2023
forestmvey
approved these changes
May 30, 2023
matthewryanwells
approved these changes
May 30, 2023
6 tasks
acarbonetto
approved these changes
May 30, 2023
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
May 30, 2023
* Support Alternate Datetime Formats (#268) * Add OpenSearchDateType as a datatype for matching with Date/Time OpenSearch types Signed-off-by: Andrew Carbonetto <[email protected]> --------- Signed-off-by: Andrew Carbonetto <[email protected]> Signed-off-by: GabeFernandez310 <[email protected]> Signed-off-by: Guian Gumpac <[email protected]> Signed-off-by: MaxKsyunz <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> Co-authored-by: MaxKsyunz <[email protected]> (cherry picked from commit 986db39)
This was referenced May 30, 2023
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
May 30, 2023
* Support Alternate Datetime Formats (#268) * Add OpenSearchDateType as a datatype for matching with Date/Time OpenSearch types Signed-off-by: Andrew Carbonetto <[email protected]> --------- Signed-off-by: Andrew Carbonetto <[email protected]> Signed-off-by: GabeFernandez310 <[email protected]> Signed-off-by: Guian Gumpac <[email protected]> Signed-off-by: MaxKsyunz <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> Co-authored-by: MaxKsyunz <[email protected]> (cherry picked from commit 986db39)
acarbonetto
pushed a commit
that referenced
this pull request
May 30, 2023
* Support Alternate Datetime Formats (#268) * Add OpenSearchDateType as a datatype for matching with Date/Time OpenSearch types Signed-off-by: Andrew Carbonetto <[email protected]> --------- Signed-off-by: Andrew Carbonetto <[email protected]> Signed-off-by: GabeFernandez310 <[email protected]> Signed-off-by: Guian Gumpac <[email protected]> Signed-off-by: MaxKsyunz <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> Co-authored-by: MaxKsyunz <[email protected]> (cherry picked from commit 986db39) Co-authored-by: Guian Gumpac <[email protected]>
acarbonetto
pushed a commit
that referenced
this pull request
May 30, 2023
* Support Alternate Datetime Formats (#268) * Add OpenSearchDateType as a datatype for matching with Date/Time OpenSearch types Signed-off-by: Andrew Carbonetto <[email protected]> --------- Signed-off-by: Andrew Carbonetto <[email protected]> Signed-off-by: GabeFernandez310 <[email protected]> Signed-off-by: Guian Gumpac <[email protected]> Signed-off-by: MaxKsyunz <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> Co-authored-by: MaxKsyunz <[email protected]> (cherry picked from commit 986db39) Co-authored-by: Guian Gumpac <[email protected]>
MitchellGale
pushed a commit
to Bit-Quill/opensearch-project-sql
that referenced
this pull request
Jun 12, 2023
* Support Alternate Datetime Formats (#268) * Add OpenSearchDateType as a datatype for matching with Date/Time OpenSearch types Signed-off-by: Andrew Carbonetto <[email protected]> --------- Signed-off-by: Andrew Carbonetto <[email protected]> Signed-off-by: GabeFernandez310 <[email protected]> Signed-off-by: Guian Gumpac <[email protected]> Signed-off-by: MaxKsyunz <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> Co-authored-by: MaxKsyunz <[email protected]>
MitchellGale
pushed a commit
to Bit-Quill/opensearch-project-sql
that referenced
this pull request
Jun 12, 2023
* Support Alternate Datetime Formats (#268) * Add OpenSearchDateType as a datatype for matching with Date/Time OpenSearch types Signed-off-by: Andrew Carbonetto <[email protected]> --------- Signed-off-by: Andrew Carbonetto <[email protected]> Signed-off-by: GabeFernandez310 <[email protected]> Signed-off-by: Guian Gumpac <[email protected]> Signed-off-by: MaxKsyunz <[email protected]> Co-authored-by: Andrew Carbonetto <[email protected]> Co-authored-by: GabeFernandez310 <[email protected]> Co-authored-by: MaxKsyunz <[email protected]> Signed-off-by: Mitchell Gale <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Supports using date/time/timestamp formats obtained from OpenSearch. Contains some changes from OpenSearchDataType rework (Bit-Quill@8b2f65d). Resolves some issues previously seen on the forum (please see Issues Resolved section below).
Formats that require only dates will return type of
date
. For example, the format "basic_date" will have typedate
in the result. Formats that require only times will return type oftime
. For example, the format "basic_time" will have typetime
in the result. Formats that can have eitherdate
ortime
will return type oftimestamp
. For example, the format "basic_date||basic_time" will have typetimestamp
in the result. This is also the case when using aggregation likeAVG
.Out of scope
Custom formats in the index mapping will be supported in a follow-up issue
TODO: Follow up with support for user-defined formats
Based on PoC (Here: https://github.com/Bit-Quill/opensearch-project-sql/pull/169/files).
Note: With these changes, issues may be introduced where data is read differently if a user's mapping defines a format. This is because the plugin previously used a hardcoded formatter to parse the data, while with these changes it will use the formatter defined in the mapping.
Supported formats include named formats listed in: opensearch/src/main/java/org/opensearch/sql/opensearch/data/type/OpenSearchDateType.java or https://opensearch.org/docs/latest/field-types/supported-field-types/date/
Issues Resolved
#794
https://forum.opensearch.org/t/sql-select-fails-on-date-fields-format-epoch-second/11521
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.