Partial series and field keys from InfluxDB 1.2.0

[choffee]

We have ~75k series and when requesting the series list influxdb, 1.1.0, is returning a partial set of results, I think due to memory limits. influxgraph is then not noticing this that just saving the partial results.
I tried increacing the http_row_limit in influxdb to no effect.
Reducing the LOADER_LIMIT in constants.py to 5000 has fixed it for me but seems more of a hack.
It should handle the server setting the partial flag.
I think the partial flag may be returned in a different place in the json returned if it's a server side limit that you have hit. Maybe this is really a bug in python-influxdb?

[choffee]

When I try the query on the cmd line using curl I get:

curl 'http://localhost:8086/query?db=graphite&epoch=s&q=SHOW+SERIES+LIMIT+10000+OFFSET+0&pretty=true' |tail

                    ]
                ],
                "partial": true
            }
        ],
        "partial": true
    }
]
}

Whereas with a limit set to 5000 it just returns the 5000 results.

[pkittenis]

Is there anything in influxdb logs to indicate what is causing partial results to be returned? Influxgraph is regularly used with millions of series in the DB but have not come across this before.

The loader limit should be configurable initially, beyond that it depends on the behaviour of the client if this is something that can be handled in influxgraph code.

[pkittenis]

Hi @choffee, will need some more info per above to see if this can be handled client side - doubt it can but to make sure.

There are changes coming that make loader limit configurable.

As whether or not influxdb returns partial results seems to depend on configuration, available memory, data size and so on it would be up to user to set an appropriate value for loader limit depending on setup.

[pkittenis]

Loader limit now configurable as of 1.2.0. Issue will be closed in a few days if no further info provided.

[pkittenis]

This can be caused by a change in influxdb 1.2.0 which truncates series lists over 10k. Default has been set to 10k accordingly in the 1.2.1 release

[pkittenis]

Looks like field list is also affected, same issue. Needs feedback from influxdb team on expected behaviour. Suggest to stick with influxdb 1.1.x in the mean time.

[pkittenis]

Pagination is now implemented, however, there seems to be a blocking issue with influxdb and non-chunked queries.

[pkittenis]

See also pull request on python client for chunked response support.

[pkittenis]

Currently blocked by new upstream client release with chunked response support.

[pkittenis]

FYI, Influx has reverted max-row-limit change in 1.2.2

Pagination support has been added by myself to influxdb python client, though not released as of now.

Show field keys issue not resolved upstream but also not applicable to this issue.

Pagination for field list is implemented in influxgraph and blocked by both influxdb issue and influxdb python client release with pagination support.

Loader limit in influxgraph is configurable as of 1.2.0.

Therefore closing this.