account_lines query missing trustlines, with subsequent query returning invalid parameters (Version:1.9.4)

[xrplfaucet]

Issue Description

Two likely related problems seen when querying trustlines for account radT5HJfA6yWP2yvukhG1KxyQq38SybCZw, which at the time of writing has 207 trustlines.

Missing Trustlines
When making an account_lines query/queries covering the first 200 trustlines, only 198 trustlines are found - number 199 and 200 (as seen in https://xrpscan.com/account/radT5HJfA6yWP2yvukhG1KxyQq38SybCZw) are missing.

Using requests with a limit of 10, it is seen that the request for trustlines 171 -180 only returns 171-179. The next request returns 180 - 189 (10 trustlines), and the next request 190 - 198 (again only 9).

Running queries with a limit of 1, a request to obtain trustline 177 (ASC issued by r3qWgpz2ry3BhcRJ8JE6rxM8esrfhuKp4R) returns 0 lines.

Failed subsequent query
In both of these scenarios, when making a subsequent request (using the previous response's marker) to get further trustlines, the query fails with an error of 'Invalid parameters.'

For example, if using a limit of 200:

• the first query returns only 198 trustlines
• second query fails with 'Invalid parameters.'

Using a limit of 100:

• the first query returns 100 trustlines
• the second query returns 98 trustlines
• the third query fails with 'Invalid parameters.'

Using a limit of 1:

• the first 176 requests return 1 trustline each
• the 177 request returns 0 trustlines
• the 178th request fails with 'Invalid paremeters.'

However, this problem is not seen if some other limits are used, such as 5, 60 or 199. For example, using a limit of 199, the first request returns 198 trustlines, and the second request successfully returns 199 to 207.

Steps to Reproduce

1. Make a request (replicated using this tool) containing the following:

{
  "command": "account_lines",
  "account": "radT5HJfA6yWP2yvukhG1KxyQq38SybCZw",
  "limit": 200
}

2. Count the response lines.

3. Using the marker from the above's response, make another query. For example:

{
  "command": "account_lines",
  "account": "radT5HJfA6yWP2yvukhG1KxyQq38SybCZw",
  "limit": 200,
  "marker": "BB87907702F3D12F09DB7CBDA6F00FD14BE1C0D2B889E0D7557C9DA9EE479251,16"
}

Expected Result

The first response should return trustlines 1 - 200 and the second request should return trustlines 201 - 207.

Actual Result

The first request returns trustlines 1 - 198 and the second request returns an invalid params response:

{
  "error": "invalidParams",
  "error_code": 31,
  "error_message": "Invalid parameters.",
  [rest of response omitted]
}

Additional Information

Based on my application logs, I can see an account_lines query (with no specific limit set) successfully retrieved all of the account's trustlines at 7:07pm UTC the 25th November, but failed when attempted at 7pm on the 26th. My code did not change during that time and requests were being made to wss://xrplcluster.com/ via the JavaScript xrpl library.

[ckniffen]

This might be related to #4340

[zgrguric]

It seems to me result set is pulled then artificially filtered after in viewing page.

[ximinez]

I've been able to reproduce this, and should have a fix PR soon.

[ximinez]

#4361 resolves the issue with the marker being invalid. It is expected that fewer than limit lines may be returned along with a marker, because of an optimization released in 1.8.2 to reduce load on rippled servers by counting all objects up to the limit instead of just the requested object type.

Please read the description of that PR for more details.