Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support exporting metrics from containerd-stargz-grpc #288

Merged
merged 1 commit into from
Mar 17, 2021

Conversation

ktock
Copy link
Member

@ktock ktock commented Mar 12, 2021

Fixes: #271

This commit adds a metrics endpoint for exporting some useful data generated by the snapshotter.
They can be pulled by prometheus.

This commit exports the following data generated by each layer of the filesystem.
We can add more metrics if needed.

  • stargz_fs_layer_size_bytes (counter) : The size of the layer in bytes
  • stargz_fs_layer_fetched_size_bytes (counter) : The size of the fetched data of the layer in bytes

Each metric is labeled by the following information to distinguish each other.

  • digest : digest of the layer
  • mountpoint : mountpoint where the layer is mounted

How to use

Specify the arbitrary address to export metrics by configuring /etc/containerd-stargz-grpc/config.toml.

metrics_address = "127.0.0.1:8234"

The metrics can get via /metrics API of that endpoint.
For example, when we execute ctr-remote i rpull ghcr.io/stargz-containers/python:3.7-esgz, we get the following metrics:

curl 127.0.0.1:8234/metrics
...(omit)...
# HELP stargz_fs_layer_fetched_size_bytes Total fetched size of the layer
# TYPE stargz_fs_layer_fetched_size_bytes counter
stargz_fs_layer_fetched_size_bytes{digest="sha256:09dcfffe1239fa012cdd72898c3ebe45c6c095f39fd106150187b896a807e982",mountpoint="/var/lib/containerd-stargz-grpc/snapshotter/snapshots/9/fs"} 2.316065e+06
stargz_fs_layer_fetched_size_bytes{digest="sha256:3ef2ba902efb61d82f35b700b184bde123382462f5877451971585ae9e6b31c0",mountpoint="/var/lib/containerd-stargz-grpc/snapshotter/snapshots/3/fs"} 7.199153e+06
stargz_fs_layer_fetched_size_bytes{digest="sha256:5206ddfd1f29b28bdf212e9c32f2bf2f7d53cb4535d3c36c0c009efc2cb4836c",mountpoint="/var/lib/containerd-stargz-grpc/snapshotter/snapshots/7/fs"} 1.6738031e+07
stargz_fs_layer_fetched_size_bytes{digest="sha256:70ad7b0f74ab9fcc8cf624d59b069fd31e61404c4cae74cf67702b2fb1c7ddb9",mountpoint="/var/lib/containerd-stargz-grpc/snapshotter/snapshots/8/fs"} 1254
stargz_fs_layer_fetched_size_bytes{digest="sha256:7999a84dec8ca5cdc77fbf9c6c4790d6cf4c869384ff33c590261704a3eb9b2d",mountpoint="/var/lib/containerd-stargz-grpc/snapshotter/snapshots/2/fs"} 7.501242e+06
stargz_fs_layer_fetched_size_bytes{digest="sha256:80505726a0e2e80ef6336f4976e6a30ed3de572934224603cdc66801bbf195b9",mountpoint="/var/lib/containerd-stargz-grpc/snapshotter/snapshots/1/fs"} 5.0922964e+07
stargz_fs_layer_fetched_size_bytes{digest="sha256:d50ab0bcfb98d6682d284b3863c07a8327f4a40547ba9f88e675269a39614120",mountpoint="/var/lib/containerd-stargz-grpc/snapshotter/snapshots/4/fs"} 5.1242816e+07
stargz_fs_layer_fetched_size_bytes{digest="sha256:dfd38485f3518ed3f34b28bd388be806515bcf575acf8f9fdd405c2287c1af51",mountpoint="/var/lib/containerd-stargz-grpc/snapshotter/snapshots/5/fs"} 1.98436384e+08
stargz_fs_layer_fetched_size_bytes{digest="sha256:e83a4f1702924b39745065c6af11af6662e87e127acee9adb327504027d127f4",mountpoint="/var/lib/containerd-stargz-grpc/snapshotter/snapshots/6/fs"} 6.196871e+06
# HELP stargz_fs_layer_size_bytes Total size of the layer
# TYPE stargz_fs_layer_size_bytes counter
stargz_fs_layer_size_bytes{digest="sha256:09dcfffe1239fa012cdd72898c3ebe45c6c095f39fd106150187b896a807e982",mountpoint="/var/lib/containerd-stargz-grpc/snapshotter/snapshots/9/fs"} 2.366065e+06
stargz_fs_layer_size_bytes{digest="sha256:3ef2ba902efb61d82f35b700b184bde123382462f5877451971585ae9e6b31c0",mountpoint="/var/lib/containerd-stargz-grpc/snapshotter/snapshots/3/fs"} 1.0049153e+07
stargz_fs_layer_size_bytes{digest="sha256:5206ddfd1f29b28bdf212e9c32f2bf2f7d53cb4535d3c36c0c009efc2cb4836c",mountpoint="/var/lib/containerd-stargz-grpc/snapshotter/snapshots/7/fs"} 1.6738031e+07
stargz_fs_layer_size_bytes{digest="sha256:70ad7b0f74ab9fcc8cf624d59b069fd31e61404c4cae74cf67702b2fb1c7ddb9",mountpoint="/var/lib/containerd-stargz-grpc/snapshotter/snapshots/8/fs"} 1254
stargz_fs_layer_size_bytes{digest="sha256:7999a84dec8ca5cdc77fbf9c6c4790d6cf4c869384ff33c590261704a3eb9b2d",mountpoint="/var/lib/containerd-stargz-grpc/snapshotter/snapshots/2/fs"} 7.901242e+06
stargz_fs_layer_size_bytes{digest="sha256:80505726a0e2e80ef6336f4976e6a30ed3de572934224603cdc66801bbf195b9",mountpoint="/var/lib/containerd-stargz-grpc/snapshotter/snapshots/1/fs"} 5.2972964e+07
stargz_fs_layer_size_bytes{digest="sha256:d50ab0bcfb98d6682d284b3863c07a8327f4a40547ba9f88e675269a39614120",mountpoint="/var/lib/containerd-stargz-grpc/snapshotter/snapshots/4/fs"} 5.4442816e+07
stargz_fs_layer_size_bytes{digest="sha256:dfd38485f3518ed3f34b28bd388be806515bcf575acf8f9fdd405c2287c1af51",mountpoint="/var/lib/containerd-stargz-grpc/snapshotter/snapshots/5/fs"} 1.98436384e+08
stargz_fs_layer_size_bytes{digest="sha256:e83a4f1702924b39745065c6af11af6662e87e127acee9adb327504027d127f4",mountpoint="/var/lib/containerd-stargz-grpc/snapshotter/snapshots/6/fs"} 6.346871e+06
...(omit)...

Prometheus can pull metrics from this endpoint.
Make prometheus recognize that endpoint by configuring /etc/prometheus/prometheus.yml:

...
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
...
  - job_name: 'stargz-snapshotter'
    static_configs:
    - targets: ['127.0.0.1:8234']
...

We can see the metrics on the dashboard:

fsmetrics

@ktock
Copy link
Member Author

ktock commented Mar 12, 2021

cc: @imjasonh @AkihiroSuda

}
}()
}
c.layerMu.RUnlock()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be after wg.Wait?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These gorountines don't access c.layer.

@ktock ktock marked this pull request as draft March 15, 2021 06:14
@ktock ktock marked this pull request as ready for review March 15, 2021 06:18
@ktock ktock merged commit 3b3e9cc into containerd:master Mar 17, 2021
@ktock ktock mentioned this pull request Mar 19, 2021
@ktock ktock deleted the prometheus-metrics branch September 3, 2021 09:56
@ktock ktock mentioned this pull request Dec 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Question: performance data and metrics?
2 participants