metrics(storage): add metrics for block cache

[hzxa21]

No description provided.

[twocode]

Currently sstable_store has a reference to state store metrics and record staff like #block requests/#block fetched remotely and durations.

[Little-Wallace]

But it seems that the block-cache can only record the number of blocks, rather the total memory keep in block cache...

[MrCroxx]

But it seems that the block-cache can only record the number of blocks, rather the total memory keep in block cache...

It's counting on the size by a weigher after #1512

[Little-Wallace]

But it seems that the block-cache can only record the number of blocks, rather the total memory keep in block cache...

It's counting on the size by a weigher after #1512

Ok, I made a mistake. But how could get the accurate current memory usage?

[Little-Wallace]

And as #1773 (comment) mentioned, the block reference in iterator will not be calculate in capacity of Cache if the entry has been evict.

[MrCroxx]

And as #1773 (comment) mentioned, the block reference in iterator will not be calculate in capacity of Cache if the entry has been evict.

Yep. That would be a problem.

[Little-Wallace]

I have ported BlockCache from RocksDB to my own project https://github.com/rust-lib-project/calibur/blob/table-cache/src/util/cache.rs. And it will only reduce memory when iterator release the cache-entry,

Maybe I could copy it to risingwave.

[MrCroxx]

Maybe I could copy it to risingwave.

It seems that the implementation doesn't include an async wait map. We need one to prevent multiple requests on the same block from leading to multiple s3 web requests. 🤔 @Little-Wallace

[Little-Wallace]

Maybe I could copy it to risingwave.

It seems that the implementation doesn't include an async wait map. We need one to prevent multiple requests on the same block from leading to multiple s3 web requests. 🤔 @Little-Wallace

Good catch. I'm thinking solution for this problem