-
Notifications
You must be signed in to change notification settings - Fork 616
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add runtime stats logger and pprof #3001
Conversation
7dd8386
to
1de4bb4
Compare
6b8786f
to
47c3826
Compare
28ea88c
to
aabeeb9
Compare
96ebb85
to
e850105
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks good one question
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is there any reason why pprof output is through introspection endpoint, while runtime stats is through a file?
why not redirect pprof output to a file, so we can capture those as part of ECS logs collector as well?
The http pprof endpoints are a standard way in which go exposes profiling data. For example, tools like |
faf83eb
to
4df4c51
Compare
Add pprof to agent introspection endpoint fixup! Add runtime stats logger
4df4c51
to
dd3f6a3
Compare
Summary
This change adds a new log file:
runtime-stats.log
, which will contain the go runtime statistics. The intention is to have more information at runtime regarding, memory, CPU, and goroutine usage.In addition, the
ECS_ENABLE_PPROF
configuration is introduced in order to enable pprof.Both of these changes aim to better assess issues such as #2865.
Implementation details
Runtime stats are retrieved by periodically executing
runtime.ReadMemStats
, andruntime.NumGoroutine
. The results of these methods are then logged to the newruntime-stats.log
file. Stats are logged every 5 minutes and the interval is not configurable since there seems to be no justification for it. Also it is not advisable to keep increasing the list of configuration variables if not strictly necessary as the list is already large and might confusing to some.This log is going to be enabled and considered to be just another log file part of the agent.
Pprof is only enabled on demand via the
ECS_ENABLE_PPROF
configuration. In this case, a new configuration was needed since users might not want/need to have pprof enabled in production environments; hence, the configuration is disabled by default.Testing
Manually rested that pprof is enabled and functional when
ECS_ENABLE_PPROF=true
, and not reachable otherwise.Also tested that the runtime stats log logs successfully every 5 minutes. Below is a sample log entry:
UTs were also added/modified as necessary/
New tests cover the changes: yes
Description for the changelog
Licensing
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.