[minio storage] thanos store: compressed blocks stop working

[wscheicher]

Thanos, Prometheus and Golang version used

thanos build date: 20180925-07:41:17
go version: go1.10.3
prometheus: 2.4.2
local minio storage (version from 20180925)

What happened

As soon as compressed blocks did reach compaction level 6 and ~160MB Size, thanos gives errors like:

No datapoints found.
receive series: rpc error: code = Aborted desc = fetch series for block 01CR98G8ECSA44NMNGKSQ6EJ4T: preload series: invalid remaining size 65536, expected 103779

and the whole range for that block is missing in the graphs.

What you expected to happen

queries and plots working without any errors or gaps, just as it did one comaction level before.

[bwplotka]

Nice. Maybe it's minio limit in size? (:

[wscheicher]

no idea, all i can say is that thanos sidecar and thanos compact didn't have a problem writing to minio.

[bwplotka]

What's the overall size of minio storage? It looks like minio bug rather than Thanos

[wscheicher]

When i filed the bug the tsdb used something around 400MB.
It was about one month of data.
Now size has tripled, filesystem still hast 15GB free space.

[bwplotka]

Looks like a duplicate of #271 (:

This might be caused by sync inconsistency between store and compactor for example.

[bwplotka]

Actually the reason might be similar to this: #146

We fetch fixed size series entries, so for series with an extremely large number of label pairs and chunks, we can have this problem. Looks like in this case 103779 bytes vs max 64*1024.

Let's keep this in our mind, once we have more reports/see this more often we should investigate other options like:

• Retry with fetching more
• Consider adding size info to our index-header or maybe even Posting in TSDB index.