Add S3 storage class configuration

[amitsetty]

Hey,
Right now if I understand correctly Mimir only works with the “Standard” storage class. Due to price concerns we’d want to run Mimir with the “Standard-IA” Storage Class in the same way this ( grafana/tempo#1714 ) pr adds storage class configuration for tempo

[pracucci]

👋 Hi! Can you configure the storage class directly at the bucket level?

That being said, make sure you take in account the read operations cost when comparing "Standard" vs "Standard-IA". Even if "Standard-IA" storage may be cheaper, each read request may be more expensive, and depending on your access pattern you may end up with higher costs at the end. Have you considered it?

[paalkr]

I would recommend using intelligent tiering for the Mimir buckets.
https://aws.amazon.com/s3/storage-classes/intelligent-tiering/

You will absolutely save money, as AWS tracks your objects and only moves infrequent accessed object to other storage tiers. Some very unlikely usage patterns might not be suited for intelligent tiering.

Her is an example from one of our Mimir S3 buckets, using intelligent tiering

[amitsetty]

Intelligent tiering seems useful but due to our on prem solution it seems it won’t work for us

It seems that when uploading blocks to s3 a header must be added in order to tell the s3 what the storage class is.

Thanos objstore supports this since (thanos-io/objstore#25)

The only change that would be necessary would be to make this configurable