Fix Posts List Performance + cursor-based pagination #3872

phiresky · 2023-08-12T00:16:19Z

This PR implements two things:

Cursor-based pagination for the /posts/list API request. A new opaque parameter page_v2 is added that fetches the next page based on the previous one.
The request now looks like curl 'http://domain/api/v3/post/list?page_v2=Pf54e&limit=20&sort=Hot&...
The response now looks like this:
```
{
  "posts": [...],
  "next_page": "Pa46c"
}
```
Cursor-based pagination is optimized so each page only fetches ~LIMIT number of posts from the database. This is not possible with OFFSET-based pagination. Right now this works alongside the page= parameter, but this pagination method should (imo) completely replace the existing OFFSET-based pagination (with the page= parameter being deprecated). There's many articles on the internet about why offset-based pagination is bad, notably this one referencing multiple posts from MariaDB, Slack, Shopify engineering. Right now it only supports forward-paging. It is possible to implement backwards pagination like this but I don't see a reason to right now.
Performant ordering and limiting for the subscribed page (see Slow SQL queries #2877 (comment)).

This works as follows: First, fetch a page for a very active followed community. Then, use that to add a limit to the real suscribed fetch. More detail about why this works is in a code comment. This should hopefully remove all slowness and weird query plans from the subscribed post list query.

Current issues:

fixed ~~Off-by-one-error: page_after actually includes one post from the previous page~~
~~the estimation query currently uses wrong filters (featured_community instead of featured_local)~~
~~code is a bit ugly~~

Future work:

Local view: the "local" view is still slow but could be optimized in the same way. this is relevant for smaller sites (e.g. < 10% of posts are local). otherwise it doesn't matter since the view fetches mostly the same data as the All view.

phiresky · 2023-08-12T12:28:38Z

Fixed the relevant issues. ready for review

phiresky · 2023-08-17T21:12:26Z

In theory, this PR should work with the legacy OFFSET based-pagination (page=X param) without breaking any functionality and while still giving significant perf improvements. I have not really tested that though.

phiresky · 2023-08-18T15:33:39Z

Here is a diagram showing how the ALL fetches should work:

phiresky · 2023-08-18T15:34:15Z

Here's a diagram showing how the Subscribed post fetch will work and why it's currently slow:

phiresky · 2023-08-22T14:57:29Z

But there is always going to be some fringe case where a person has essentially blocked almost everything and runs out of data

With my approach (as detailed above), this shouldn't be an issue.

phiresky · 2023-08-22T15:09:47Z

crates/db_views/src/post_view.rs

@@ -185,6 +186,34 @@ fn queries<'a>() -> Queries<
    };

  let list = move |mut conn: DbConn<'a>, options: PostQuery<'a>| async move {
+    macro_rules! order_and_page_filter_desc {


i've tried to do this without a macro and without copy-pasting this code 10 times, but I couldn't figure out the types for diesel if this was a lambda or function. I'm not sure if it would even be possible, even with BoxedQuery..

I'd almost rather this be copy-pasted around, if even some parts of it could be extracted into functions, rather than using macros. @dullbananas might be able to help with diesel function return types.

The type of query should be inferrable if a closure is used.

The type of the column is tricky because columns with multiple different sql types are used. If only one sql type was needed, then BoxableExpression could be used. And closures can't be generic.

I tried using a closure, but even then you need to be able to name the return type and some arg types and I couldn't manage to figure that out..

Also I think it has to be generic which as you said isn't possible

A function like this could probably be used

fn order_and_page_filter_desc<Q, C, T>( query: &mut Q, column: C, options: &PostQuery, getter: impl Fn(&PostAggregates) -> T, ) where Q: ThenOrderDsl<dsl::Desc<C>, Output = Q> + FilterDsl<dsl::GtEq<C, T>, Output = Q> + FilterDsl<dsl::GtEq<C, T>, Output = Q>, C: Expression, C::SqlType: SingleValue + SqlType, T: AsExpression<C::SqlType>,

Usage:

order_and_page_filter_desc(&mut query, post_aggregates::featured_local, &options, |a| a.featured_local);

The ascending version would use dsl::Asc instead of dsl::Desc

*query = query.then_order_by(column.desc()); seems to not work because of cannot move out of *querywhich is behind a mutable reference move occurs because*queryhas typeQ, which does not implement the Copy` trait

Remove the &mut and make the function return the same type used for the query parameter

dessalines

This'll be a great addition, a few thoughts tho:

Should this just work based on a simple post_id number, rather than easily reverse engineerable hash functions. See comment below.
If cursor based pagination works as simply as where id > cursor and (more filters) LIMIT X, I don't see why specifying upper bounds are necessary.
Needs some tests to make sure pagination is working correctly.
Should just be a breaking change, rather than supporting the old method.

crates/api_common/src/post.rs

crates/db_views/src/post_view.rs

crates/apub/src/api/list_posts.rs

crates/db_views/src/post_view.rs

dessalines · 2023-08-22T16:08:13Z

crates/db_views/src/post_view.rs

@@ -185,6 +186,34 @@ fn queries<'a>() -> Queries<
    };

  let list = move |mut conn: DbConn<'a>, options: PostQuery<'a>| async move {
+    macro_rules! order_and_page_filter_desc {


I'd almost rather this be copy-pasted around, if even some parts of it could be extracted into functions, rather than using macros. @dullbananas might be able to help with diesel function return types.

crates/db_views/src/post_view.rs

crates/apub/src/api/list_posts.rs

crates/db_views/src/post_view.rs

dessalines · 2023-09-08T13:37:02Z

I'll test this with some production data now.

dessalines

Its testing out fine, but the upper-bound stuff is really difficult to wrap my head around, as I'm sure it will be for anyone looking at this code.

We desperately need more DB experts to look at this to make sure that its the best way of doing this, but I won't let that hold up this PR; it'll have to come at a later date. I especially wish @johanndt could have a look, but they don't seem to be around.

I'd also really prefer that the filtering and sorting macro be removed, so @dullbananas if you could attempt that, before we merge, it'd be really helpful. Otherwise, I spose we have no choice.

dessalines · 2023-09-10T12:50:56Z

migrations/2023-09-07-215546_post-queries-efficient/up.sql

+
+CREATE INDEX idx_post_aggregates_community_hot ON post_aggregates (community_id, featured_local DESC, hot_rank DESC, published DESC);
+
+CREATE INDEX idx_post_aggregates_community_scaled ON post_aggregates (community_id, featured_local DESC, scaled_rank DESC, published DESC);


Thx for adding these ones too.

diesel migration redo worked well also.

phiresky · 2023-09-13T13:59:13Z

I've managed to remove the macro with the help of what dullbananas wrote above.

dessalines

Works well with local production data. The only minor issue I see is that the deprecated page doesn't work with subscribed, but that's NBD.

Nutomic · 2023-09-18T10:16:20Z

Is this good to merge?

phiresky · 2023-09-18T10:38:20Z

It is from my side

dessalines · 2023-09-18T22:25:40Z

I'll start to add all these new features into lemmy-ui in the next few days.

phiresky added 3 commits August 11, 2023 22:37

add token-based pagination + fast subscribed post view

6f7a89d

add migrations

6c8623c

fix failing heuristic

a7b1c81

phiresky mentioned this pull request Aug 12, 2023

Slow SQL queries #2877

Closed

revert

bc819a4

MV-GH mentioned this pull request Aug 12, 2023

[Proposal] Concrete backwards compatible API implementation LemmyNet/jerboa#1179

Closed

phiresky added 3 commits August 12, 2023 11:58

output pagination token as next_page, fix off-by-one, restructure

9187418

more cleanup

f23a611

clean

b5733e5

phiresky marked this pull request as ready for review August 12, 2023 12:28

phiresky requested review from dessalines and Nutomic as code owners August 12, 2023 12:28

format sql

caeb302

phiresky mentioned this pull request Aug 13, 2023

Remove left joins and use only one call to select method in post_view.rs #3865

Merged

phiresky added 4 commits August 13, 2023 13:40

fix comment

2e1f9ba

fix tests

e3c7d3e

e

16da0b9

empty

7b45e98

phiresky mentioned this pull request Aug 16, 2023

PostgreSQL EXPLAIN shows faster performance listing posts for community #3839

Closed

This comment was marked as abuse.

Sign in to view

phiresky commented Aug 22, 2023

View reviewed changes

dessalines reviewed Aug 22, 2023

View reviewed changes

move last page thing

6259c7a

phiresky mentioned this pull request Aug 22, 2023

Post list date ranges - establish post/list API parameters when_before when_after targeted published dates #3900

Closed

Merge remote-tracking branch 'origin/main' into fast-post-list

ed74985

phiresky mentioned this pull request Sep 2, 2023

Adding a scaled sort, to boost smaller communities. #3907

Merged

phiresky and others added 9 commits September 3, 2023 12:05

Merge remote-tracking branch 'origin/main' into fast-post-list

9386fd1

restructure a bit for readability

02bde16

rename page_cursor

1445eb2

Merge remote-tracking branch 'origin/main' into fast-post-list

6f51730

update for scaled sort

9baab5f

fix

49f0cf5

sql format

b2bf8de

fix

bbd4876

Merge branch 'main' into fast-post-list

94fb4f4

dessalines reviewed Sep 10, 2023

View reviewed changes

get rid of macros

f3afbb2

phiresky force-pushed the fast-post-list branch from c9b9259 to f3afbb2 Compare September 13, 2023 14:00

Merge branch 'main' into fast-post-list

1675625

dessalines approved these changes Sep 14, 2023

View reviewed changes

Nutomic approved these changes Sep 18, 2023

View reviewed changes

Nutomic merged commit 544d30f into LemmyNet:main Sep 18, 2023

dessalines mentioned this pull request Sep 28, 2023

Add cursor-based pagination LemmyNet/lemmy-ui#2155

Closed

3 tasks

mormaer mentioned this pull request Sep 30, 2023

[0.19.0] - Cursor based pagination mlemgroup/mlem#674

Closed

SrivatsanSenthilkumar mentioned this pull request Sep 30, 2023

Breaking changes to API in lemmy version 0.19 aeharding/voyager#745

Closed

This was referenced Oct 2, 2023

[Bug]: No page limit when sorting by TopAll #3680

Closed

Cache existing post order when loading next page. #3547

Closed

[Bug]: Pagination with "Show Read Posts" disabled leads to skipped pages of content #3659

Closed

potatoleek mentioned this pull request Dec 17, 2023

Browser back from post always returns to first page of default configured sort LemmyNet/lemmy-ui#1794

Open

4 tasks

phiresky mentioned this pull request Dec 21, 2023

Change comparisons in cursor-based pagination #4287

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Posts List Performance + cursor-based pagination #3872

Fix Posts List Performance + cursor-based pagination #3872

phiresky commented Aug 12, 2023 •

edited by dessalines

Loading

phiresky commented Aug 12, 2023

phiresky commented Aug 17, 2023

phiresky commented Aug 18, 2023

phiresky commented Aug 18, 2023

This comment was marked as abuse.

phiresky commented Aug 22, 2023

phiresky Aug 22, 2023

dessalines Aug 22, 2023

dullbananas Aug 23, 2023 •

edited

Loading

phiresky Aug 24, 2023 •

edited

Loading

dullbananas Sep 4, 2023

phiresky Sep 7, 2023

dullbananas Sep 7, 2023

dessalines left a comment

dessalines Aug 22, 2023

dessalines commented Sep 8, 2023

dessalines left a comment

dessalines Sep 10, 2023

phiresky commented Sep 13, 2023

dessalines left a comment

Nutomic commented Sep 18, 2023

phiresky commented Sep 18, 2023

dessalines commented Sep 18, 2023


		CREATE INDEX idx_post_aggregates_community_hot ON post_aggregates (community_id, featured_local DESC, hot_rank DESC, published DESC);

		CREATE INDEX idx_post_aggregates_community_scaled ON post_aggregates (community_id, featured_local DESC, scaled_rank DESC, published DESC);

Fix Posts List Performance + cursor-based pagination #3872

Fix Posts List Performance + cursor-based pagination #3872

Conversation

phiresky commented Aug 12, 2023 • edited by dessalines Loading

phiresky commented Aug 12, 2023

phiresky commented Aug 17, 2023

phiresky commented Aug 18, 2023

phiresky commented Aug 18, 2023

This comment was marked as abuse.

phiresky commented Aug 22, 2023

phiresky Aug 22, 2023

Choose a reason for hiding this comment

dessalines Aug 22, 2023

Choose a reason for hiding this comment

dullbananas Aug 23, 2023 • edited Loading

Choose a reason for hiding this comment

phiresky Aug 24, 2023 • edited Loading

Choose a reason for hiding this comment

dullbananas Sep 4, 2023

Choose a reason for hiding this comment

phiresky Sep 7, 2023

Choose a reason for hiding this comment

dullbananas Sep 7, 2023

Choose a reason for hiding this comment

dessalines left a comment

Choose a reason for hiding this comment

dessalines Aug 22, 2023

Choose a reason for hiding this comment

dessalines commented Sep 8, 2023

dessalines left a comment

Choose a reason for hiding this comment

dessalines Sep 10, 2023

Choose a reason for hiding this comment

phiresky commented Sep 13, 2023

dessalines left a comment

Choose a reason for hiding this comment

Nutomic commented Sep 18, 2023

phiresky commented Sep 18, 2023

dessalines commented Sep 18, 2023

phiresky commented Aug 12, 2023 •

edited by dessalines

Loading

dullbananas Aug 23, 2023 •

edited

Loading

phiresky Aug 24, 2023 •

edited

Loading