Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix offset needing a limit clause (on sqlite) #2029

Merged
merged 21 commits into from
Apr 22, 2020

Conversation

weiznich
Copy link
Member

@weiznich weiznich commented Apr 4, 2019

Just opening this one to get some feedback on the chosen approach

This should ~work but has one downside: Every code that is generic over connection types/backends now requires bounds like SelectStatement<…>: QueryFragment<DB> (For concrete examples see the change in diesel_migrations). That's quite unfortunate in my opinion, but I've found no better way to solve this without falling back to unstable features like specialization

@weiznich weiznich requested a review from a team April 4, 2019 21:35
@@ -33,6 +33,72 @@ use query_source::joins::{AppendSelection, Inner, Join};
use query_source::*;
use result::QueryResult;

#[doc(hidden)]
#[derive(Debug, Clone, Copy, QueryId)]
pub struct LimitOffsetClause<Limit, Offset> {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That needs to be move to somewhere else

Box::new(self.offset),
Box::new(self.group_by),
)
unimplemented!()
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've just want to get the normal SelectStatement working before I will fix that.

@sgrif
Copy link
Member

sgrif commented Apr 4, 2019

Ugh I was hoping LIMIT -1 was specified in ISO SQL so we wouldn't need to special case this...
image

@sgrif
Copy link
Member

sgrif commented Apr 4, 2019

Only other approach I can think of would be this:

impl<DB: Backend + SupportsLonelyOffset> QueryFragment<DB> for NoLimitClause {
    fn walk_ast(...) -> ... {
         Ok(())
    }
}

impl QueryFragment<Sqlite> for NoLimitClause {
    fn walk_ast(...) -> ... {
        out.push_sql("LIMIT -1");
        Ok(())
    }
}

Honestly though, having LimitOffsetClause makes sense. I'm wondering if the right move here is to remove SupportsLonelyOffset, and instead require all backends to provide an explicit impl. This seems like the place that backends diverge the most (can offset be used w/o limit? Do they do limit/offset, or ISO standard offset/fetch first?) Maybe something like this?

pub struct LimitOffsetClause<LimitClause, OffsetClause> {
    pub limit_clause: LimitClause,
    pub offset_clause: OffsetClause,
}

impl<Lim, Off> QueryFragment<Pg> for LimitOffsetClause<Lim, Off>
where
    Lim: QueryFragment<Pg>,
    Off: QueryFragment<Pg>,
{
    fn walk_ast(...) -> ... {
        self.limit_clause.walk_ast(out.reborrow())?;
        self.offset_clause.walk_ast(out.reborrow())?;
        Ok(())
    }
}

// Identical impl for MySQL

impl<Off> QueryFragment<Sqlite> for LimitOffsetClause<NoLimitClause, OffsetClause<Off>>
where
    OffsetClause<Off>: QueryFragment<Sqlite>,
{
    fn walk_ast(...) -> ... {
        out.push_sql(" LIMIT -1 ");
        self.offset_clause.walk_ast(out.reborrow())?;
        Ok(())
    }
}

// explicit impls for `LimitClause<Lim>, AnyOffsetClause` and `NoLimitClause, NoOffsetClause`

@weiznich weiznich force-pushed the fix_offset_for_sqlite branch from ff76f79 to 70f9fc1 Compare April 9, 2019 16:41
@weiznich weiznich marked this pull request as ready for review April 9, 2019 16:41
@weiznich
Copy link
Member Author

weiznich commented Apr 9, 2019

Turns out that mysql does also not support offset without a preceding limit clause. Their "workaround" is even more horrible:

To retrieve all rows from a certain offset up to the end of the result set, you can use some large number for the second parameter.

@sgrif
Copy link
Member

sgrif commented Apr 9, 2019 via email

@weiznich
Copy link
Member Author

weiznich commented Apr 9, 2019

That would be a solution.
Otherwise maybe just add a #[deprecated (note = "MySql is broken, use postgres instead")] to the mysql backend 😋

@weiznich weiznich force-pushed the fix_offset_for_sqlite branch 2 times, most recently from 9d699be to 8cf3422 Compare April 16, 2019 11:50
@weiznich weiznich force-pushed the fix_offset_for_sqlite branch from e80e7c6 to 9bbfa1d Compare August 6, 2019 20:52
@weiznich weiznich force-pushed the fix_offset_for_sqlite branch from 9bbfa1d to 5e9ae34 Compare October 7, 2019 19:42
@weiznich
Copy link
Member Author

weiznich commented Oct 7, 2019

@diesel-rs/contributors This could need a final review.

Copy link
Member Author

@weiznich weiznich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I've tried to port the diesel-oci backend to the latest diesel master branch (including this PR). It turns out that some changes are required to make this work: Basically third party backends need to see things and need to be able to implement certain traits.

where
Self: AsQuery,
DB: Backend,
S: QueryFragment<DB> + SelectableExpression<F> + 'a,
D: QueryFragment<DB> + 'a,
W: Into<BoxedWhereClause<'a, DB>>,
O: Into<Option<Box<dyn QueryFragment<DB> + 'a>>>,
L: QueryFragment<DB> + 'a,
Of: QueryFragment<DB> + 'a,
LOf: IntoBoxedClause<'a, DB, BoxedClause = BoxedLimitOffsetClause<'a, DB>>,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may want to use this trait for the other conversations as well in a future change, but not sure about that

@weiznich weiznich force-pushed the fix_offset_for_sqlite branch 5 times, most recently from da3f459 to 765dbcd Compare January 16, 2020 20:54
MySQL also does not accept an offset clause without limit
clause. Additionally they don't have any workaround for this, instead
they suggest to just use "a large number" as limit clause.
I've choosen `u64::MAX` as "large number". As far as I'm aware that
does not resul in any limitations because mysql only supports up to
64TB of data per table. Assuming 1 bit per row (which is a crazy low
minimum guess) this means 1024 * 1024 * 1024 * 1024 * 8 =
562.949.953.421.312 rows which is smaller than
2^64 = 18.446.744.073.709.551.615
For third party backends it was not possible to implement
`From<LimitOffsetClause<_,_>> for BoxedLimitOffsetClause` and also
`Into<BoxedLimitOffsetClause> for LimitOffsetClause<_,_>` because of
coherence rules. This commit introduces a new trait third party
implementable trait used in place of `From`/`Into`.
* After the rebase the import syntax needed to be changed to rust2018
@weiznich weiznich force-pushed the fix_offset_for_sqlite branch from 765dbcd to 66146da Compare April 20, 2020 15:04
* Clarify changelog about potential breakage
* Add some internal comment why things are implemented as they are
* More tests for different cases
@weiznich weiznich force-pushed the fix_offset_for_sqlite branch from 53dad55 to eb00d5a Compare April 22, 2020 08:22
That's a pg only function, so we can just use a concrete connection
type there and skip those big where clause
@weiznich weiznich merged commit 077c036 into diesel-rs:master Apr 22, 2020
weiznich referenced this pull request in weiznich/wundergraph May 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants