feat!: Bump datafusion, prost, hyper, tonic, tower, axum #5417

evenyag · 2025-01-21T08:55:02Z

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

This PR bumps a lot of crates:

datafusion 2464703c84c400a09cc59277018813f0e797bb4e (After 43.0.0)
sqllparser 0.52.x
prost 0.13
hyper 1.x
tonic 0.12
tower 0.5
axum 0.8
arrow 53
parquet 53

There are too many changes due to many breaking changes in these crates. Here are some of them:

Implements Debug for plans and related structs.
Uses SessionStateBuilder to build SessionState
Removes the usage of HandleErrorLayer.
Implements From<arrow interval> for interval types.
Adds a new method new_with_count() to RecordBatch to allow constructing an empty RecordBatch that only contains row count.
DFLogicalSubstraitConvertor doesn't require catalog list now.
Supports Some(Vec::new()) projection in mito.
mito uses min_bytes_opt, max_bytes_opt, min_opt, max_opt.
ColumnSchema will ignore the metadata of time index if it doesn't have timestamp type. (For SQL CAST).
Some protobuf structs support Copy now.
DfContextProviderAdapter implements all missing methods.
Uses SortExpr and the new function API of datafusion.
Uses Bytes to replace RawBody in axum handlers.
Refactors all middleware for gRPC server and HTTP server.
Updates all sqlness results.
udf functions with zero arguements need Nullary signature now

Breaking changes

id and tag are SQL keywords now

Notes

The HTTP server doesn't support setting the TCP keepalive timeout. We may need to check if it has regressions.
DatafusionQueryEngine::optimize_physical_plan() doesn't optimize the physical plan now, as the planner already optimize it.

PR Checklist

Please convert it to a draft if some of the following conditions are not met.

I have written the necessary rustdoc comments.
I have added the necessary unit tests and integration tests.
This PR requires documentation updates.
API changes are backward compatible.
Schema or data changes are backward compatible.

Signed-off-by: Ruihang Xia <[email protected]>

* remove handle_error handler * refactor timeout layer * serve axum

* boxed() * use TokioIo

* feat: exclude script crate * chore: simplify feature * feat: remove the script crate * chore: remove python feature and some comments * chore: fix warning

src/catalog/src/system_schema/information_schema/cluster_info.rs

src/common/function/src/scalars/date/date_add.rs

src/flow/src/expr/func.rs

src/flow/src/transform.rs

src/flow/src/transform/literal.rs

src/mito2/src/read/projection.rs

src/mito2/src/sst/parquet/format.rs

src/promql/src/extension_plan/scalar_calculate.rs

Signed-off-by: Ruihang Xia <[email protected]>

tests/cases/standalone/common/view/view.result

tests/cases/standalone/common/flow/flow_null.sql

src/servers/Cargo.toml

src/sql/src/statements/transform/type_alias.rs

waynexia

I've reviewed them and hereby cast my vote 🫡

src/query/src/range_select/plan.rs

evenyag · 2025-01-22T13:18:58Z

rustc may panic...
https://github.com/GreptimeTeam/greptimedb/actions/runs/12908415598/job/35994232050

I'll retry that test.

fengjiachun

Rest LGTM

waynexia and others added 30 commits December 6, 2024 16:33

change dep

2db8107

Signed-off-by: Ruihang Xia <[email protected]>

feat: adapt to arrow's interval array

db9dfba

chore: fix compile errors in datatypes crate

c757feb

chore: fix api crate compiler errors

181f28e

chore: fix compiler errors in common-grpc

75f9ea2

chore: fix common-datasource errors

46bc65e

chore: fix deprecated code in common-datasource

4514db9

fix promql and physical plan related

ce73058

Signed-off-by: Ruihang Xia <[email protected]>

wip: upgrading network deps

8dad368

Signed-off-by: Ruihang Xia <[email protected]>

block on updating sqlparser

8f1c760

upgrade sqlparser

3845086

Signed-off-by: Ruihang Xia <[email protected]>

adapt new df's trait requirements

3047af4

Signed-off-by: Ruihang Xia <[email protected]>

chore: fix compiler errors in mito2

9caf689

chore: fix common-function crate errors

5986565

chore: fix catalog errors

761ff28

change import path

93213dd

Signed-off-by: Ruihang Xia <[email protected]>

chore: fix some errors in query crate

55add17

chore: fix some errors in query crate

069fff0

aggr expr and some other tiny fixes

176645b

Signed-off-by: Ruihang Xia <[email protected]>

chore: fix expr related errors in query crate

885bc52

chore: fix query serializer and admin command

2b1caa6

chore: fix grpc services

0207c37

feat: axum serve

9c575d8

chore: fix http server

b1a3490

* remove handle_error handler * refactor timeout layer * serve axum

chore: fix flow aggr functions

91abacf

chore: fix flow

ba6442f

feat: fix errors in meta-srv

04871e2

* boxed() * use TokioIo

feat!: Remove script crate and python feature (#5321)

b68f727

* feat: exclude script crate * chore: simplify feature * feat: remove the script crate * chore: remove python feature and some comments * chore: fix warning

chore: fix servers tests compiler errors

124115e

feat: fix tests-integration errors

c6e3452

killme2008 reviewed Jan 22, 2025

View reviewed changes

discord9 and others added 2 commits January 22, 2025 16:41

refactor: typed precision

3aed891

update sqlness view case

257e8dd

Signed-off-by: Ruihang Xia <[email protected]>

killme2008 reviewed Jan 22, 2025

View reviewed changes

tests/cases/standalone/common/view/view.result Outdated Show resolved Hide resolved

chore: flow per review

500a8c7

waynexia reviewed Jan 22, 2025

View reviewed changes

tests/cases/standalone/common/flow/flow_null.sql Show resolved Hide resolved

evenyag added 2 commits January 22, 2025 17:38

chore: add example in comment

bad35dc

chore: warn if parquet stats of timestamp is not INT64

30a78a9

waynexia reviewed Jan 22, 2025

View reviewed changes

src/servers/Cargo.toml Outdated Show resolved Hide resolved

evenyag force-pushed the bump-df-202412 branch from de9e55b to 30a78a9 Compare January 22, 2025 10:50

evenyag added 3 commits January 22, 2025 18:52

style: add a newline before derive to make the comment more clear

aaacdbd

test: update sqlness result

5b9d412

chore: Merge branch 'main' into bump-df-202412

5e6b7e8

waynexia reviewed Jan 22, 2025

View reviewed changes

src/sql/src/statements/transform/type_alias.rs Show resolved Hide resolved

waynexia reviewed Jan 22, 2025

View reviewed changes

src/sql/src/statements/transform/type_alias.rs Show resolved Hide resolved

fix: flow from substrait

74b34aa

waynexia approved these changes Jan 22, 2025

View reviewed changes

src/query/src/range_select/plan.rs Outdated Show resolved Hide resolved

evenyag added 2 commits January 22, 2025 21:22

chore: change update_range_context log to debug level

fbbbfee

chore: move axum-extra axum-macros to workspace

3a28240

fengjiachun approved these changes Jan 22, 2025

View reviewed changes

evenyag enabled auto-merge January 23, 2025 06:13

evenyag added this pull request to the merge queue Jan 23, 2025

evenyag mentioned this pull request Jan 23, 2025

refactor: support to flatten json object in greptime_identity pipeline #5358

Merged

5 tasks

Merged via the queue into main with commit 35b635f Jan 23, 2025
37 checks passed

evenyag deleted the bump-df-202412 branch January 23, 2025 06:35

waynexia mentioned this pull request Jan 24, 2025

Build time regression apache/datafusion#14256

Open

yihong0618 mentioned this pull request Jan 26, 2025

fix: no need for special case since datafusion updated #5458

Merged

5 tasks

evenyag mentioned this pull request Feb 8, 2025

Support setting tcp keepalive duration #5493

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: Bump datafusion, prost, hyper, tonic, tower, axum #5417

feat!: Bump datafusion, prost, hyper, tonic, tower, axum #5417

evenyag commented Jan 21, 2025 •

edited by waynexia

Loading

waynexia left a comment

evenyag commented Jan 22, 2025

fengjiachun left a comment

feat!: Bump datafusion, prost, hyper, tonic, tower, axum #5417

feat!: Bump datafusion, prost, hyper, tonic, tower, axum #5417

Conversation

evenyag commented Jan 21, 2025 • edited by waynexia Loading

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

Breaking changes

Notes

PR Checklist

waynexia left a comment

Choose a reason for hiding this comment

evenyag commented Jan 22, 2025

fengjiachun left a comment

Choose a reason for hiding this comment

evenyag commented Jan 21, 2025 •

edited by waynexia

Loading