Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge paginated plan optimizer into the regular optimizer. #1516

Conversation

MaxKsyunz
Copy link
Collaborator

Address this comment in #1497

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

This also made the paginationPlanner in QueryService unnecessary.

Signed-off-by: MaxKsyunz <[email protected]>
MaxKsyunz and others added 6 commits April 12, 2023 15:05
PaginateOperator is not optimized away when scan builder is created.

Signed-off-by: MaxKsyunz <[email protected]>
…lPlanOptimizerTest.java

Co-authored-by: Yury-Fridlyand <[email protected]>
Signed-off-by: MaxKsyunz <[email protected]>
…zer' into feature/pagination/P1-one-optimizer
CreatePagingTableScanBuilder now does the same thing.

Signed-off-by: MaxKsyunz <[email protected]>
…ature/pagination/P1-one-optimizer

# Conflicts:
#	core/src/test/java/org/opensearch/sql/executor/execution/ContinuePaginatedPlanTest.java
#	core/src/test/java/org/opensearch/sql/executor/execution/PaginatedPlanTest.java
#	integ-test/src/test/java/org/opensearch/sql/ppl/StandaloneIT.java
#	plugin/src/main/java/org/opensearch/sql/plugin/config/OpenSearchPluginModule.java
var plan = plans.pop();
if (plan.getChild().stream().anyMatch(LogicalRelation.class::isInstance)) {
if (plan.getChild().size() > 1) {
throw new UnsupportedOperationException();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a message to explain what's wrong here

* {@link this.logicalPaginate}, false otherwise.
*/
private boolean findLogicalRelation(LogicalPaginate logicalPaginate) {
Deque<LogicalPlan> plans = new ArrayDeque<>();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: you could create ArrayDeque with logicalPagination

new ArrayDeque<>(logicalPaginate);

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sadly, I cannot -- that constructor expects a collection.

Compared to adding a conversion to collection, just calling push is simpler.

relationParent = plan;
return true;
}
plan.getChild().forEach(plans::push);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's an addAll() but no pushAll() method... hmm
Maybe use addAll() and removeFirst() instead?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll switch for the sake of removing forEach(plans:push).

verify(table).createScanBuilder();
verify(table, never()).createPagedScanBuilder(anyInt());
// Assert that createPagedTableScan was not called
// Assert that createTableScan was called
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO?

@codecov-commenter
Copy link

codecov-commenter commented Apr 13, 2023

Codecov Report

Merging #1516 (9df41c7) into feature/pagination/P1 (bc8c73f) will decrease coverage by 0.01%.
The diff coverage is 100.00%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@                     Coverage Diff                     @@
##             feature/pagination/P1    #1516      +/-   ##
===========================================================
- Coverage                    98.51%   98.51%   -0.01%     
+ Complexity                    4018     4013       -5     
===========================================================
  Files                          362      361       -1     
  Lines                         9911     9905       -6     
  Branches                       644      643       -1     
===========================================================
- Hits                          9764     9758       -6     
  Misses                         142      142              
  Partials                         5        5              
Flag Coverage Δ
sql-engine 98.51% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...pensearch/sql/planner/logical/LogicalRelation.java 100.00% <ø> (ø)
...ch/sql/planner/optimizer/LogicalPlanOptimizer.java 100.00% <ø> (ø)
...java/org/opensearch/sql/executor/QueryService.java 100.00% <100.00%> (ø)
...opensearch/sql/planner/logical/LogicalPlanDSL.java 100.00% <100.00%> (ø)
...r/optimizer/rule/CreatePagingTableScanBuilder.java 100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: MaxKsyunz <[email protected]>
@MaxKsyunz MaxKsyunz merged commit b9cb0d0 into opensearch-project:feature/pagination/P1 Apr 14, 2023
@Yury-Fridlyand Yury-Fridlyand mentioned this pull request Apr 14, 2023
6 tasks
acarbonetto pushed a commit to Bit-Quill/opensearch-project-sql that referenced this pull request Apr 18, 2023
…zer. (opensearch-project#1516)

Merge paginated plan optimizer into the regular optimizer.
---------

Signed-off-by: MaxKsyunz <[email protected]>
Co-authored-by: Yury-Fridlyand <[email protected]>
Yury-Fridlyand added a commit that referenced this pull request Apr 27, 2023
* Support pagination in V2 engine, phase 1 (#226)

* Fixing integration tests broken during POC

Signed-off-by: MaxKsyunz <[email protected]>

* Comment to clarify an exception.

Signed-off-by: MaxKsyunz <[email protected]>

* Add support for paginated scroll request, first page.

Implement PaginatedPlanCache.convertToPlan for second page to work.

Signed-off-by: MaxKsyunz <[email protected]>

* Progress on paginated scroll request, subsequent page.

Signed-off-by: MaxKsyunz <[email protected]>

* Move `ExpressionSerializer` from `opensearch` to `core`.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Rename `Cursor` `asString` to `toString`.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Disable scroll cleaning.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Add full cursor serialization and deserialization.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Misc fixes.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Further work on pagination.

* Added push down page size from `LogicalPaginate` to `LogicalRelation`.
* Improved cursor encoding and decoding.
* Added cursor compression.
* Fixed issuing `SearchScrollRequest`.
* Fixed returning last empty page.
* Minor code grooming/commenting.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Pagination fix for empty indices.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix error reporting on wrong cursor.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Minor comments and error reporting improvement.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Add an end-to-end integration test.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Add `explain` request handlers.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Add IT for explain.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Address issues flagged by checkstyle build step (#229)

Signed-off-by: MaxKsyunz <[email protected]>

* Pagination, phase 1: Add unit tests for `:core` module with coverage. (#230)

* Add unit tests for `:core` module with coverage. Uncovered: `toCursor`, because it is will be changed soon.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Pagination, phase 1: Add unit tests for SQL module with coverage. (#239)

* Add unit tests for SQL module with coverage.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Update sql/src/main/java/org/opensearch/sql/sql/domain/SQLQueryRequest.java

Signed-off-by: Yury-Fridlyand <[email protected]>

Co-authored-by: GabeFernandez310 <[email protected]>

---------

Signed-off-by: Yury-Fridlyand <[email protected]>
Co-authored-by: GabeFernandez310 <[email protected]>

* Pagination, phase 1: Add unit tests for `:opensearch` module with coverage. (#233)

* Add UT for `:opensearch` module with full coverage, except `toCursor`.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix checkstyle.

Signed-off-by: Yury-Fridlyand <[email protected]>

---------

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix the merges.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix explain.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix scroll cleaning.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Store `TotalHits` and use it to report `total` in response.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Add missing UT for `:protocol` module.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix PPL UTs damaged in f4ea4ad.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Minor checkstyle fixes.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fallback to v1 engine for pagination (#245)

* Pagination fallback integration tests.

Signed-off-by: MaxKsyunz <[email protected]>

* Add UT with coverage for `toCursor` serialization.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix broken tests in `legacy`.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix getting `total` from non-paged requests and from queries without `FROM` clause.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix scroll cleaning.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix cursor request processing.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Update ITs.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix (again) TotalHits feature.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix typo in prometheus config.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Recover commented logging.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Move `test_pagination_blackbox` to a separate class and add logging.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Address some PR feedbacks: rename some classes and revert unnecessary whitespace changed.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Minor commenting.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Address PR comments.

* Add javadocs
* Renames
* Cleaning up some comments
* Remove unused code
* Speed up IT

Signed-off-by: Yury-Fridlyand <[email protected]>

* Minor missing changes.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Integration tests for fetch_size, max_result_window, and query.size_limit (#248)

Signed-off-by: MaxKsyunz <[email protected]>

* Remove `PaginatedQueryService`, extend `QueryService` to hold two planners and use them.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Move push down functions from request builders to a new interface.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Some file moves.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Minor clean-up according to PR review.

Signed-off-by: Yury-Fridlyand <[email protected]>

---------

Signed-off-by: MaxKsyunz <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>
Co-authored-by: MaxKsyunz <[email protected]>
Co-authored-by: GabeFernandez310 <[email protected]>
Co-authored-by: Max Ksyunz <[email protected]>

* Make scroll timeout configurable.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix IT to set cursor keep alive parameter.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Remove `QueryId.None`.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Rename according to PR feedback.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Remove default implementations of `PushDownRequestBuilder`.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Merge paginated plan optimizer into the regular optimizer. (#1516)

Merge paginated plan optimizer into the regular optimizer.
---------

Signed-off-by: MaxKsyunz <[email protected]>
Co-authored-by: Yury-Fridlyand <[email protected]>

* Complete rework on serialization and deserialization. (#1498)

Signed-off-by: Yury-Fridlyand <[email protected]>

* Resolve merge conflicts and fix tests.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Minor cleanup.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Minor cleanup - missing changes for the previous commit.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Remove paginate operator  (#1528)

* Remove PaginateOperator class since it is no longer used.


---------

Signed-off-by: MaxKsyunz <[email protected]>

* Remove `PaginatedPlan` - move logic to `QueryPlan`.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Remove default implementations from `SerializablePlan`.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Add a doc.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Update design graphs.

Signed-off-by: Yury-Fridlyand <[email protected]>

* More fixes for merge from upstream/main.

Signed-off-by: MaxKsyunz <[email protected]>

---------

Signed-off-by: MaxKsyunz <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>
Co-authored-by: MaxKsyunz <[email protected]>
Co-authored-by: GabeFernandez310 <[email protected]>
Co-authored-by: Max Ksyunz <[email protected]>
acarbonetto pushed a commit to Bit-Quill/opensearch-project-sql that referenced this pull request Apr 28, 2023
* Support pagination in V2 engine, phase 1 (#226)

* Fixing integration tests broken during POC

Signed-off-by: MaxKsyunz <[email protected]>

* Comment to clarify an exception.

Signed-off-by: MaxKsyunz <[email protected]>

* Add support for paginated scroll request, first page.

Implement PaginatedPlanCache.convertToPlan for second page to work.

Signed-off-by: MaxKsyunz <[email protected]>

* Progress on paginated scroll request, subsequent page.

Signed-off-by: MaxKsyunz <[email protected]>

* Move `ExpressionSerializer` from `opensearch` to `core`.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Rename `Cursor` `asString` to `toString`.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Disable scroll cleaning.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Add full cursor serialization and deserialization.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Misc fixes.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Further work on pagination.

* Added push down page size from `LogicalPaginate` to `LogicalRelation`.
* Improved cursor encoding and decoding.
* Added cursor compression.
* Fixed issuing `SearchScrollRequest`.
* Fixed returning last empty page.
* Minor code grooming/commenting.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Pagination fix for empty indices.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix error reporting on wrong cursor.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Minor comments and error reporting improvement.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Add an end-to-end integration test.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Add `explain` request handlers.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Add IT for explain.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Address issues flagged by checkstyle build step (#229)

Signed-off-by: MaxKsyunz <[email protected]>

* Pagination, phase 1: Add unit tests for `:core` module with coverage. (#230)

* Add unit tests for `:core` module with coverage. Uncovered: `toCursor`, because it is will be changed soon.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Pagination, phase 1: Add unit tests for SQL module with coverage. (#239)

* Add unit tests for SQL module with coverage.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Update sql/src/main/java/org/opensearch/sql/sql/domain/SQLQueryRequest.java

Signed-off-by: Yury-Fridlyand <[email protected]>

Co-authored-by: GabeFernandez310 <[email protected]>

---------

Signed-off-by: Yury-Fridlyand <[email protected]>
Co-authored-by: GabeFernandez310 <[email protected]>

* Pagination, phase 1: Add unit tests for `:opensearch` module with coverage. (#233)

* Add UT for `:opensearch` module with full coverage, except `toCursor`.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix checkstyle.

Signed-off-by: Yury-Fridlyand <[email protected]>

---------

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix the merges.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix explain.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix scroll cleaning.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Store `TotalHits` and use it to report `total` in response.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Add missing UT for `:protocol` module.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix PPL UTs damaged in f4ea4ad.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Minor checkstyle fixes.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fallback to v1 engine for pagination (#245)

* Pagination fallback integration tests.

Signed-off-by: MaxKsyunz <[email protected]>

* Add UT with coverage for `toCursor` serialization.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix broken tests in `legacy`.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix getting `total` from non-paged requests and from queries without `FROM` clause.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix scroll cleaning.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix cursor request processing.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Update ITs.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix (again) TotalHits feature.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix typo in prometheus config.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Recover commented logging.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Move `test_pagination_blackbox` to a separate class and add logging.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Address some PR feedbacks: rename some classes and revert unnecessary whitespace changed.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Minor commenting.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Address PR comments.

* Add javadocs
* Renames
* Cleaning up some comments
* Remove unused code
* Speed up IT

Signed-off-by: Yury-Fridlyand <[email protected]>

* Minor missing changes.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Integration tests for fetch_size, max_result_window, and query.size_limit (#248)

Signed-off-by: MaxKsyunz <[email protected]>

* Remove `PaginatedQueryService`, extend `QueryService` to hold two planners and use them.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Move push down functions from request builders to a new interface.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Some file moves.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Minor clean-up according to PR review.

Signed-off-by: Yury-Fridlyand <[email protected]>

---------

Signed-off-by: MaxKsyunz <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>
Co-authored-by: MaxKsyunz <[email protected]>
Co-authored-by: GabeFernandez310 <[email protected]>
Co-authored-by: Max Ksyunz <[email protected]>

* Make scroll timeout configurable.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Fix IT to set cursor keep alive parameter.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Remove `QueryId.None`.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Rename according to PR feedback.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Remove default implementations of `PushDownRequestBuilder`.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Merge paginated plan optimizer into the regular optimizer. (opensearch-project#1516)

Merge paginated plan optimizer into the regular optimizer.
---------

Signed-off-by: MaxKsyunz <[email protected]>
Co-authored-by: Yury-Fridlyand <[email protected]>

* Complete rework on serialization and deserialization. (opensearch-project#1498)

Signed-off-by: Yury-Fridlyand <[email protected]>

* Resolve merge conflicts and fix tests.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Minor cleanup.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Minor cleanup - missing changes for the previous commit.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Remove paginate operator  (opensearch-project#1528)

* Remove PaginateOperator class since it is no longer used.

---------

Signed-off-by: MaxKsyunz <[email protected]>

* Remove `PaginatedPlan` - move logic to `QueryPlan`.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Remove default implementations from `SerializablePlan`.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Add a doc.

Signed-off-by: Yury-Fridlyand <[email protected]>

* Update design graphs.

Signed-off-by: Yury-Fridlyand <[email protected]>

* More fixes for merge from upstream/main.

Signed-off-by: MaxKsyunz <[email protected]>

---------

Signed-off-by: MaxKsyunz <[email protected]>
Signed-off-by: Yury-Fridlyand <[email protected]>
Co-authored-by: MaxKsyunz <[email protected]>
Co-authored-by: GabeFernandez310 <[email protected]>
Co-authored-by: Max Ksyunz <[email protected]>
@Yury-Fridlyand Yury-Fridlyand added the pagination Pagination feature, ref #656 label May 1, 2023
@Yury-Fridlyand Yury-Fridlyand deleted the feature/pagination/P1-one-optimizer branch June 16, 2023 22:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pagination Pagination feature, ref #656
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants