Add datetime functions `FROM_UNIXTIME` and `UNIX_TIMESTAMP` #114

Yury-Fridlyand · 2022-09-08T19:38:54Z

FROM_UNIXTIME

(DOUBLE) -> DATETIME
(DOUBLE, STRING) -> STRING

Implementation details

For string formatting DATE_FORMAT function is used
Function returns 0 for values outside of range 1970-01-01 00:00:00 - 3001-01-18 23:59:59.999999

NOTES

Returning an integer in numeric context is not supported.
Operations with DOUBLE affected by Floating Point Imprecision

Example

opensearchsql> select FROM_UNIXTIME(1220249547.12);
fetched rows / total rows = 1/1
+--------------------------------+
| FROM_UNIXTIME(1220249547.12)   |
|--------------------------------|
| 2008-09-01 06:12:27.119999885  |
+--------------------------------+

Test samples

SQL

select FROM_UNIXTIME(1337.42), FROM_UNIXTIME(123235435, '%T %M %D, %Y');

+--------------------------+--------------------------------------------+
| FROM_UNIXTIME(1337.42)   | FROM_UNIXTIME(123235435, '%T %M %D, %Y')   |
|--------------------------+--------------------------------------------|
| 1970-01-01 00:22:17.42   | 08:03:55 November 27th, 1973               |
+--------------------------+--------------------------------------------+

PPL

source=date | top 1 date_keyword | eval `FROM_UNIXTIME(1337.42)` = FROM_UNIXTIME(1337.42), `FROM_UNIXTIME(123235435, '%T %M %D, %Y')` = FROM_UNIXTIME(123235435, '%T %M %D, %Y') | fields `FROM_UNIXTIME(1337.42)`, `FROM_UNIXTIME(123235435, '%T %M %D, %Y')`;

+--------------------------+--------------------------------------------+
| FROM_UNIXTIME(1337.42)   | FROM_UNIXTIME(123235435, '%T %M %D, %Y')   |
|--------------------------+--------------------------------------------|
| 1970-01-01 00:22:17.42   | 08:03:55 November 27th, 1973               |
+--------------------------+--------------------------------------------+

UNIX_TIMESTAMP

() -> LONG
DATE -> DOUBLE
DATETIME -> DOUBLE
TIMESTAMP -> DOUBLE
DOUBLE -> DOUBLE

Implementation details

First function signature calls for LocalDateTime.now() without caching the value. Sequential calls in one query might return different values.
Function returns 0 for values outside of range 1970-01-01 00:00:00 - 3001-01-18 23:59:59.999999.

NOTES

Operations with DOUBLE affected by Floating Point Imprecision

Example

opensearchsql> select UNIX_TIMESTAMP(20771122143845.2345);
fetched rows / total rows = 1/1
+---------------------------------------+
| UNIX_TIMESTAMP(20771122143845.2345)   |
|---------------------------------------|
| 3404817524.234375                     |
+---------------------------------------+

Test samples

SQL

select UNIX_TIMESTAMP(), UNIX_TIMESTAMP(20771122143845), UNIX_TIMESTAMP(881226), UNIX_TIMESTAMP(881226231058.123), UNIX_TIMESTAMP(DATE('1995-05-06')), UNIX_TIMESTAMP(TIMESTAMP('2022-08-09 12:16:12')), UNIX_TIMESTAMP(ADDDATE(DATE('2020-08-26'), INTERVAL 1 HOUR));

UNIX_TIMESTAMP()                                             | 1662664900
UNIX_TIMESTAMP(20771122143845)                               | 3404817525.0
UNIX_TIMESTAMP(881226)                                       | 599097600.0
UNIX_TIMESTAMP(881226231058.123)                             | 599181057.1230469
UNIX_TIMESTAMP(DATE('1995-05-06'))                           | 799718400.0
UNIX_TIMESTAMP(TIMESTAMP('2022-08-09 12:16:12'))             | 1660047372.0
UNIX_TIMESTAMP(ADDDATE(DATE('2020-08-26'), INTERVAL 1 HOUR)) | 1598403600.0

PPL

source=date | eval `UNIX_TIMESTAMP()` = UNIX_TIMESTAMP(), `UNIX_TIMESTAMP(20771122143845)` = UNIX_TIMESTAMP(20771122143845), `UNIX_TIMESTAMP(881226)` = UNIX_TIMESTAMP(881226), `UNIX_TIMESTAMP(881226231058.123)` = UNIX_TIMESTAMP(881226231058.123), `UNIX_TIMESTAMP(DATE('1995-05-06'))` = UNIX_TIMESTAMP(DATE('1995-05-06')), `UNIX_TIMESTAMP(TIMESTAMP('2022-08-09 12:16:12'))` = UNIX_TIMESTAMP(TIMESTAMP('2022-08-09 12:16:12')), `UNIX_TIMESTAMP(ADDDATE(DATE('2020-08-26'), INTERVAL 1 HOUR))` = UNIX_TIMESTAMP(ADDDATE(DATE('2020-08-26'), INTERVAL 1 HOUR)) | fields `UNIX_TIMESTAMP()`, `UNIX_TIMESTAMP(20771122143845)`, `UNIX_TIMESTAMP(881226)`, `UNIX_TIMESTAMP(881226231058.123)`, `UNIX_TIMESTAMP(DATE('1995-05-06'))`, `UNIX_TIMESTAMP(TIMESTAMP('2022-08-09 12:16:12'))`, `UNIX_TIMESTAMP(ADDDATE(DATE('2020-08-26'), INTERVAL 1 HOUR))`;

UNIX_TIMESTAMP()                                             | 1662664900
UNIX_TIMESTAMP(20771122143845)                               | 3404817525.0
UNIX_TIMESTAMP(881226)                                       | 599097600.0
UNIX_TIMESTAMP(881226231058.123)                             | 599181057.1230469
UNIX_TIMESTAMP(DATE('1995-05-06'))                           | 799718400.0
UNIX_TIMESTAMP(TIMESTAMP('2022-08-09 12:16:12'))             | 1660047372.0
UNIX_TIMESTAMP(ADDDATE(DATE('2020-08-26'), INTERVAL 1 HOUR)) | 1598403600.0

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…, UT and IT. Signed-off-by: Yury-Fridlyand <[email protected]>

Signed-off-by: Yury-Fridlyand <[email protected]>

acarbonetto · 2022-09-08T22:33:07Z

core/src/main/java/org/opensearch/sql/expression/datetime/DateTimeFunction.java

@@ -6,6 +6,7 @@

 package org.opensearch.sql.expression.datetime;

+import static java.time.temporal.ChronoField.YEAR;


is there no YEAR field in ExprCoreType?

No.. Why it should be? Aren't you messing ChronoField and ExprCoreType? ExprCoreType is a enum of STRING, DOUBLE, etc

core/src/main/java/org/opensearch/sql/expression/datetime/DateTimeFunction.java

Signed-off-by: Yury-Fridlyand <[email protected]>

docs/user/ppl/functions/datetime.rst

core/src/main/java/org/opensearch/sql/expression/datetime/DateTimeFunction.java

core/src/test/java/org/opensearch/sql/expression/datetime/FromUnixTimeTest.java

docs/user/dql/functions.rst

docs/user/ppl/functions/datetime.rst

Co-authored-by: MaxKsyunz <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]>

Signed-off-by: Yury-Fridlyand <[email protected]>

MitchellGale · 2022-09-13T07:29:47Z

MySQL:

mysql> select FROM_UNIXTIME(1337.4200), FROM_UNIXTIME(123235435, '%T %M %D, %Y');
+--------------------------+------------------------------------------+
| FROM_UNIXTIME(1337.4200) | FROM_UNIXTIME(123235435, '%T %M %D, %Y') |
+--------------------------+------------------------------------------+
| 1970-01-01 00:22:17.4200 | 08:03:55 November 27th, 1973             |
+--------------------------+------------------------------------------+
1 row in set (0.00 sec)

OpenSearchSQL:

opensearchsql> select FROM_UNIXTIME(1337.4200), FROM_UNIXTIME(123235435, '%T %M %D, %Y');                                                                                                                                                                   
fetched rows / total rows = 1/1
+----------------------------+--------------------------------------------+
| FROM_UNIXTIME(1337.4200)   | FROM_UNIXTIME(123235435, '%T %M %D, %Y')   |
|----------------------------+--------------------------------------------|
| 1970-01-01 00:22:17.42     | 08:03:55 November 27th, 1973               |
+----------------------------+--------------------------------------------+

Missing trailing zeros in FROM_UNIXTIME

MitchellGale

select UNIX_TIMESTAMP(TIMESTAMP('1996-13-15 17:05:42'));

Should return null (similar with other invalid dates (or times!)) instead of

TransportError(500, 'SemanticCheckException', {'error': {'type': 'SemanticCheckException', 'reason': 'Invalid Query', 'details': 'timestamp:1996-13-15 17:05:42 in unsupported format, please use yyyy-MM-dd HH:mm:ss[.SSSSSSSSS]'}, 'status': 400})

MitchellGale · 2022-09-13T07:33:08Z

docs/user/dql/functions.rst

+    | 3404817525.0                     |
+    +----------------------------------+
+
+    os> select UNIX_TIMESTAMP(TIMESTAMP('1996-11-15 17:05:42'))


MySQL shows this as returning without a floating point.

mysql> select UNIX_TIMESTAMP(TIMESTAMP('1996-11-15 17:05:42')); +--------------------------------------------------+ | UNIX_TIMESTAMP(TIMESTAMP('1996-11-15 17:05:42')) | +--------------------------------------------------+ | 848077542 | +--------------------------------------------------+ 1 row in set (0.00 sec)

It is added by the double serializer, it is out of my control. We can discuss to fix this globally in scope of another fix.

MitchellGale · 2022-09-13T07:34:12Z

ppl/src/main/antlr/OpenSearchPPLParser.g4

@@ -375,7 +375,7 @@ trigonometricFunctionName
 dateAndTimeFunctionBase
    : ADDDATE | DATE | DATE_ADD | DATE_SUB | DAY | DAYNAME | DAYOFMONTH | DAYOFWEEK | DAYOFYEAR | FROM_DAYS
    | HOUR | MICROSECOND | MINUTE | MONTH | MONTHNAME | QUARTER | SECOND | SUBDATE | TIME | TIME_TO_SEC
-    | TIMESTAMP | TO_DAYS | YEAR | WEEK | DATE_FORMAT | MAKETIME | MAKEDATE
+    | TIMESTAMP | TO_DAYS | YEAR | WEEK | DATE_FORMAT | MAKETIME | MAKEDATE | FROM_UNIXTIME | UNIX_TIMESTAMP


Add in alphabetical order please.

Thank you, fixed in faf0372.

MitchellGale · 2022-09-13T07:34:26Z

sql/src/main/antlr/OpenSearchSQLParser.g4

@@ -385,7 +385,7 @@ trigonometricFunctionName
 dateTimeFunctionName
    : ADDDATE | DATE | DATE_ADD | DATE_SUB | DAY | DAYNAME | DAYOFMONTH | DAYOFWEEK | DAYOFYEAR | FROM_DAYS
    | HOUR | MICROSECOND | MINUTE | MONTH | MONTHNAME | QUARTER | SECOND | SUBDATE | TIME | TIME_TO_SEC
-    | TIMESTAMP | TO_DAYS | YEAR | WEEK | DATE_FORMAT | MAKETIME | MAKEDATE
+    | TIMESTAMP | TO_DAYS | YEAR | WEEK | DATE_FORMAT | MAKETIME | MAKEDATE | FROM_UNIXTIME | UNIX_TIMESTAMP


Add in alphabetical order.

Thank you, fixed in faf0372.

Signed-off-by: Yury-Fridlyand <[email protected]>

Yury-Fridlyand · 2022-09-13T20:01:08Z

Missing trailing zeros in FROM_UNIXTIME

Interesting. It is impossible to track this in our framework. We convert input to Double, it doesn't store or serialize non-significant digits. The same applicable to the output value.

MaxKsyunz

@Yury-Fridlyand please update FromUnixTimeTest, UnixTimeStampTeset, and UnixTwoWayConversionTest to match existing expression tests in core/src/test/java/org/opensearch/sql/expression.

The following belong in core/src/main/java/org/opensearch/sql/expression/DSL.java

FromUnixTimeTest.fromUnixTime(Expression)
FromUnixTimeTest.fromUnixTime(Expression, Expression)
UnixTimeStampTest.unixTimeStampExpr()
UnixTimeStampTest.unixTimeStampOf(Expression)

Once you do that, duplicated functions from UnixTwoWayConversionTest can be removed.

core/src/test/java/org/opensearch/sql/expression/datetime/UnixTwoWayConversionTest.java

Signed-off-by: Yury-Fridlyand <[email protected]>

core/src/test/java/org/opensearch/sql/expression/datetime/DateTimeTestBase.java

…TimeTestBase.java Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: Max Ksyunz <[email protected]>

Signed-off-by: Yury-Fridlyand <[email protected]>

* Add implementation for `FROM_UNIXTIME` and `UNIX_TIMESTAMP` functions, UT and IT. Signed-off-by: Yury-Fridlyand <[email protected]>

…ch-project#835) * Add datetime functions `FROM_UNIXTIME` and `UNIX_TIMESTAMP` (#114) * Add implementation for `FROM_UNIXTIME` and `UNIX_TIMESTAMP` functions, UT and IT. Signed-off-by: Yury-Fridlyand <[email protected]> * Collent all DateTime formatters into one place. Signed-off-by: Yury-Fridlyand <[email protected]> * Rename `DateFormatters` -> `DateTimeFormatters`. Signed-off-by: Yury-Fridlyand <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]>

Yury-Fridlyand added 5 commits September 7, 2022 20:41

Add implementation for FROM_UNIXTIME and UNIX_TIMESTAMP functions…

3dd12e0

…, UT and IT. Signed-off-by: Yury-Fridlyand <[email protected]>

Fix checkstyle.

c954d07

Signed-off-by: Yury-Fridlyand <[email protected]>

Optimize FROM_UNIXTIME signature.

ab766f2

Signed-off-by: Yury-Fridlyand <[email protected]>

Add doctests.

6872cf0

Signed-off-by: Yury-Fridlyand <[email protected]>

Minor clean-up.

73ca29e

Signed-off-by: Yury-Fridlyand <[email protected]>

Yury-Fridlyand requested review from acarbonetto, forestmvey, MaxKsyunz and MitchellGale September 8, 2022 19:45

Bit-Quill deleted a comment from codecov bot Sep 8, 2022

acarbonetto approved these changes Sep 8, 2022

View reviewed changes

Add a constant instead of magic numbers.

65776ed

Signed-off-by: Yury-Fridlyand <[email protected]>

This comment was marked as spam.

Sign in to view

MaxKsyunz requested changes Sep 13, 2022

View reviewed changes

Yury-Fridlyand and others added 2 commits September 12, 2022 18:52

Update docs according to PR feedback.

dfd2c1e

Co-authored-by: MaxKsyunz <[email protected]> Signed-off-by: Yury-Fridlyand <[email protected]>

Update tests according to PR feedback.

541c63c

Signed-off-by: Yury-Fridlyand <[email protected]>

MitchellGale reviewed Sep 13, 2022

View reviewed changes

Reorder functions in ANTLR syntax file.

faf0372

Signed-off-by: Yury-Fridlyand <[email protected]>

Yury-Fridlyand requested review from MaxKsyunz and MitchellGale September 13, 2022 20:01

MaxKsyunz requested changes Sep 14, 2022

View reviewed changes

core/src/test/java/org/opensearch/sql/expression/datetime/UnixTwoWayConversionTest.java Outdated Show resolved Hide resolved

core/src/test/java/org/opensearch/sql/expression/datetime/UnixTwoWayConversionTest.java Outdated Show resolved Hide resolved

Yury-Fridlyand added 2 commits September 14, 2022 09:08

Move test methods to common class DateTimeTestBase.

36cb226

Signed-off-by: Yury-Fridlyand <[email protected]>

Fix EOL.

ad1523c

Signed-off-by: Yury-Fridlyand <[email protected]>

Yury-Fridlyand requested a review from MaxKsyunz September 14, 2022 19:57

MaxKsyunz reviewed Sep 14, 2022

View reviewed changes

core/src/test/java/org/opensearch/sql/expression/datetime/DateTimeTestBase.java Outdated Show resolved Hide resolved

Update core/src/test/java/org/opensearch/sql/expression/datetime/Date…

eca585b

…TimeTestBase.java Signed-off-by: Yury-Fridlyand <[email protected]> Co-authored-by: Max Ksyunz <[email protected]>

Yury-Fridlyand requested a review from MaxKsyunz September 15, 2022 02:19

Fix using @Autowired.

9d0c52f

Signed-off-by: Yury-Fridlyand <[email protected]>

acarbonetto self-requested a review September 15, 2022 16:39

acarbonetto approved these changes Sep 15, 2022

View reviewed changes

MitchellGale approved these changes Sep 15, 2022

View reviewed changes

Yury-Fridlyand mentioned this pull request Sep 15, 2022

SQL ingestion and ops - standup opensearch-project/sql#791

Open

MaxKsyunz approved these changes Sep 15, 2022

View reviewed changes

Yury-Fridlyand merged commit c01f1df into integ-datetime-unix-convert Sep 15, 2022

Yury-Fridlyand deleted the dev-datetime-unix-convert branch September 15, 2022 23:27

Yury-Fridlyand added a commit that referenced this pull request Sep 15, 2022

Add datetime functions FROM_UNIXTIME and UNIX_TIMESTAMP (#114)

9fb786b

* Add implementation for `FROM_UNIXTIME` and `UNIX_TIMESTAMP` functions, UT and IT. Signed-off-by: Yury-Fridlyand <[email protected]>

Yury-Fridlyand added a commit that referenced this pull request Sep 24, 2022

Add datetime functions FROM_UNIXTIME and UNIX_TIMESTAMP (#114)

c1513f9

* Add implementation for `FROM_UNIXTIME` and `UNIX_TIMESTAMP` functions, UT and IT. Signed-off-by: Yury-Fridlyand <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add datetime functions `FROM_UNIXTIME` and `UNIX_TIMESTAMP` #114

Add datetime functions `FROM_UNIXTIME` and `UNIX_TIMESTAMP` #114

Yury-Fridlyand commented Sep 8, 2022

acarbonetto Sep 8, 2022

Yury-Fridlyand Sep 8, 2022

This comment was marked as spam.

MitchellGale commented Sep 13, 2022

MitchellGale left a comment •

edited

Loading

MitchellGale Sep 13, 2022

Yury-Fridlyand Sep 13, 2022

MitchellGale Sep 13, 2022

Yury-Fridlyand Sep 13, 2022

MitchellGale Sep 13, 2022

Yury-Fridlyand Sep 13, 2022

Yury-Fridlyand commented Sep 13, 2022

MaxKsyunz left a comment

		@@ -6,6 +6,7 @@

		package org.opensearch.sql.expression.datetime;

		import static java.time.temporal.ChronoField.YEAR;

Add datetime functions FROM_UNIXTIME and UNIX_TIMESTAMP #114

Add datetime functions FROM_UNIXTIME and UNIX_TIMESTAMP #114

Conversation

Yury-Fridlyand commented Sep 8, 2022

Implementation details

NOTES

Test samples

Implementation details

NOTES

Test samples

Check List

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as spam.

MitchellGale commented Sep 13, 2022

MitchellGale left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Yury-Fridlyand commented Sep 13, 2022

MaxKsyunz left a comment

Choose a reason for hiding this comment

Add datetime functions `FROM_UNIXTIME` and `UNIX_TIMESTAMP` #114

Add datetime functions `FROM_UNIXTIME` and `UNIX_TIMESTAMP` #114

MitchellGale left a comment •

edited

Loading