[1.2.0] Partial results from show field keys with limit set without chunked parameter in some cases

[pkittenis]

Bug report

System info: 1.2.0, Linux

Steps to reproduce:

1. virtualenv test_env
2. source test_env/bin/activate
3. pip install influxdb
4. Run this script to generate test data with python insert_field_keys.py
5. curl -s -G 'http://localhost:8086/query?pretty=true' --data-urlencode "db=integration_test" --data-urlencode "q=show field keys limit 120 offset 0"| grep 'm92' where 'm92' is a measurement in the test data.

Expected behavior:

Field keys for all measurements up to the 120 limit are returned.

Actual behavior:

Measurements after 'm91' are missing in output, plus only a partial list of fields for 'm91' measurement is returned. Field keys output is truncated with partial: true even though limit is set on query and number of rows is below max-row-limit setting.

Querying with chunked: true returns data for the missing keys and output is not partial.

curl -s -G 'http://localhost:8086/query?pretty=true&chunked=true' --data-urlencode "db=integration_test" --data-urlencode "q=show field keys limit 120 offset 0"|grep 'm92'

This returns data for m92 and rest of missing measurements.

Additional info:

Test data creates 150 measurements from m0 to m149 and a linearly increasing number of fields for each measurement starting from one field for m0 measurement to 150 fields for the m149 measurement.

For measurements with a non-identical number of fields, partial results are returned when chunked is not set on the query even though limit is set on query and number of rows has not exceeded max-row-limit. With chunked set, data is returned as expected.

Setting max-row-limit to 0 has no effect on non-chunked show field keys queries.

On smaller values of limit results are not truncated even without chunked: true.

This looks like an issue with calculation of limit/offset on field key lists with a non-identical number of fields on each measurement. This can be see by queries like curl -s -G 'http://localhost:8086/query?pretty=true' --data-urlencode "db=field_keys_test" --data-urlencode "q=show field keys limit 120 offset 1"|grep 'm92' which do contain missing measurements, only to have partial results one measurement further down and so on.

Further queries with offset = limit + offset, for example curl -s -G 'http://localhost:8086/query?pretty=true' --data-urlencode "db=field_keys_test" --data-urlencode "q=show field keys limit 120 offset 120"| also do not contain missing measurements from zero offset query.

Affects all clients that do not set chunked: true, regardless if LIMIT is used or not and regardless of max-row-limit setting.

[jwilder]

@jsternberg

[jsternberg]

Running your test script, I'm getting 10860 rows which seems to exceed the maximum size. That's the number of values that would be returned.

@jwilder which do you view as the row limit? Is it a limit on the number of values or the number of series?

[pkittenis]

Yes, that's what I had assumed. Since number of fields is arbitrary per measurement, it means client cannot be sure that any limit they set will ensure they do not cross max-row-limit.

So either show field keys limit needs to continue to refer to number of measurements/series but ignore max-row-limit or make limit on show field keys apply to number of values instead to get consistent behaviour.

What's interesting is I did not see a change when max-row-limit was set to 0 though. Not true, ignore.

[jsternberg]

This seems to be working as designed. I'm going to close this. Please use chunked or set max-row-limit to zero. We changed the default of max-row-limit to zero in 1.2.2.