Skip to content
This repository has been archived by the owner on Mar 21, 2024. It is now read-only.

Commit

Permalink
Tasks API - Seek/Keyset based pagination (#115)
Browse files Browse the repository at this point in the history 
* Move cursor based pagination spec to tasks API spec

* remove pagination as a future capability

* Clarify boundaries for limit query parameter

* Update OpenApi specification

* Remove limit field boundaries

* Apply suggestions from code review

Co-authored-by: Clément Renault <[email protected]>

* Update open-api.yml and removes cursor term mentions

* Update text/0060-tasks-api.md

Co-authored-by: Tommy <[email protected]>

* Remove  route mention in seek-keyset pagination section

Co-authored-by: Clément Renault <[email protected]>
Co-authored-by: Tommy <[email protected]>
  • Loading branch information
@gmourier @Kerollmops @dichotommy
3 people committed Jul 7, 2022
1 parent e36c626 commit 1407c3e
Show file tree
Hide file tree
Showing 2 changed files with 175 additions and 3 deletions.
36 changes: 35 additions & 1 deletion open-api.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,18 @@ servers:
description: 'https://example.meilisearch.com:7700'
components:
schemas:
limit:
type: integer
description: Limit given for the query. If limit is not provided as a query parameter, this parameter displays the default limit value.
example: 10
from:
type: integer
description: The first task uid returned.
example: 999
next:
type: integer
description: Represents the value to send in `from` to fetch the next slice of the results. The first item for the next slice starts at this exact number. When the returned value is null, it means that all the data have been browsed in the given order.
example: 989
timestamp:
type: string
description: An `RFC 3339` format for date/time/duration.
Expand Down Expand Up @@ -691,6 +703,12 @@ components:
schema:
type: number
default: 0
from:
name: from
in: query
description: Fetch the next set of results from the given uid.
schema:
type: number
q:
name: q
in: query
Expand Down Expand Up @@ -2882,6 +2900,17 @@ paths:
type: array
items:
$ref: '#/components/schemas/task'
limit:
$ref: '#/components/schemas/limit'
from:
$ref: '#/components/schemas/from'
next:
$ref: '#/components/schemas/next'
required:
- results
- limit
- from
- next
examples:
Example:
value:
Expand Down Expand Up @@ -2915,8 +2944,13 @@ paths:
enqueuedAt: '2021-01-01T09:38:00.000000Z'
startedAt: '2021-01-01T09:38:02.000000Z'
finishedAt: '2021-01-01T09:38:07.000000Z'
limit: 2
from: 1
next: null
operationId: tasks.list
parameters: []
parameters:
- $ref: '#/components/parameters/limit'
- $ref: '#/components/parameters/from'
security:
- apiKey: []
'/tasks/:taskUid':
Expand Down
142 changes: 140 additions & 2 deletions text/0060-tasks-api.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -568,6 +568,145 @@ New task types are also added for these operations. `indexCreation`, `indexUpdat
}
```

#### 9. Paginate `task` resource lists

The API endpoint `/tasks` is browsable using a keyset-based pagination.

##### 9.1 Why a Seek/Keyset based pagination?

Keyset-based pagination is more appropriate when the data can grow or shrink quickly in terms of magnitude.

###### 9.1.1 Pros

The performance is better than the not-so-good but old pagination with `offset`/`limit`.

Seek/Keyset pagination keeps the results consistent between each page as the data evolves. It avoids the [Page Drift effect](https://use-the-index-luke.com/sql/partial-results/fetch-next-page), especially when the data is sorted from the most recent to the oldest.

Moreover, the performance is superior to traditional pagination since the computational complexity remains constant to reach the identifier marking the beginning of the new slice to be returned from a hash table.

###### 9.1.2 Cons

The main drawback of this type of pagination is that it does not navigate within a finite number of pages. It is also limited to a precise sorting criterion on unique identifiers ordered sequentially.

##### 9.2 Response attributes

| field | type | description |
|-------|------|--------------------------------------|
| limit | integer | Default `20`. |
| from | integer | The first task uid returned |
| next | integer - nullable | Represents the value to send in `from` to fetch the next slice of the results. The first item for the next slice starts at this exact number. When the returned value is null, it means that all the data have been browsed in the given order. |

##### 9.3 GET query parameters

| field | type | required | description |
|-------|------|----------|--------------|
| limit | integer | No | Default `20`. Limit on the number of tasks to be returned. |
| from | integer | No | Limit results to tasks with uids equal to and lower than this uid. |

##### 9.4 Usage examples

This part demonstrates keyset paging in action on `/tasks`. The items `uid` remains sorted sequentially and can be used to navigate a list of `tasks` objects.

---

**Initial default slice of `tasks`**

`GET` - `/tasks`

```json
{
"results": [
{
"uid": 1350,
"indexUid": "movies",
"type": "documentAddition",
...,
},
...,
{
"uid": 1330,
"indexUid": "movies_reviews",
"type": "documentAddition",
...,
}
],
"limit": 20,
"from": 1350,
"next": 1329
}
```

**Request the next slice of `tasks` items with a limit of `50` tasks**

`GET` - `/tasks?from=1329&limit=50`

```json
{
"results": [
{
"uid": 1329,
"indexUid": "movies",
"type": "documentAddition",
...,
},
...,
{
"uid": 1279,
"indexUid": "movies",
"type": "settingsUpdate",
...,
}
],
"limit": 50,
"from": 1329,
"next": 1278
}
```

**End of seek/keyset pagination**

`GET` - `/tasks?from=20`

```json
{
"results": [
{
"uid": 19,
"indexUid": "movies",
"type": "documentsAddition",
...,
},
...,
{
"uid": 0,
"indexUid": "movies",
"type": "documentsAddition",
...,
}
],
"limit": 20,
"from": 20,
"next": null
}
```

- 💡 `next` response parameter is null because there are no more `tasks` to fetch. It means that the response represents the last slice of results for the given resource list.

##### 9.5 Behaviors for `limit` and `from` query parameters

###### 9.5.1 `limit`

- If `limit` is not set, the default value is chosen.

###### 9.5.2 `from`

- If `from` is set with an out of bounds task `uid`, the response returns the tasks that are the nearest to the specified uid, the `next` field is set to the next page. It will be equivalent to call the `/tasks` route without any parameter.

###### 9.5.3 Errors

- 🔴 Sending a value with a different type than `Integer` for `limit` returns a [bad_request](0061-error-format-and-definitions.md#bad_request) error.
- 🔴 Sending a value with a different type than `Integer` for `from` returns a [bad_request](0061-error-format-and-definitions.md#bad_request) error.

## 2. Technical details

### I. Measuring
Expand All @@ -579,11 +718,10 @@ New task types are also added for these operations. `indexCreation`, `indexUpdat

## 3. Future Possibilities

- Add a pagination system for on `/tasks` and `indexes/:indexUid/tasks` lists.
- Add enhanced filtering capabilities.
- Simplify `documentAddition` and `documentPartial` type and elaborate on `details` metadata.
- Use Hateoas capability to give direct access to a `task` resource.
- Add dedicated task type names modifying a sub-setting. e.g. `SearchableAttributesUpdate`.
- Reconsider existence of `/indexes/:indexUid/tasks/:taskUid` route since it is similar to `/tasks/:taskUid`.
- Add an archived state for old `tasks`.
- Indicate the `API Key` identity that added a `task`. It should not permits to
- Indicate the `API Key` identity that added a `task`.

0 comments on commit 1407c3e

Please sign in to comment.