[LiveMetrics] report process metrics CPU Total and Committed Memory

[TimothyMothra]

Changes

• LiveMetrics will now send Process metrics.
  • Memory: Process.PrivateMemorySize64
  • CPU: Process.TotalProcessorTime
    Measured as: ((change in ticks / period) / number of processors)

Testing

I've collected screenshots and detailed description for each counter.
For these metrics, I'm trying to match what Visual Studio's Debugger shows.

Memory (click to expand)

To test I debugged an app in Visual Studio and compared to the Process Memory.

Visual Studio Debugger

This shows roughly 50 MB of Process Memory

OTel uses:

• WorkingSet64
  • "Gets the amount of physical memory, in bytes, allocated for the associated process."
    https://learn.microsoft.com/dotnet/api/system.diagnostics.process.workingset64
• OTel repo link

This gives a value almost double of what Visual Studio Debugger is showing:

AI SDK uses:

• PrivateMemorySize64
  • "Gets the amount of private memory, in bytes, allocated for the associated process."
    https://learn.microsoft.com/dotnet/api/system.diagnostics.process.privatememorysize64
• AI SDK repo link

This value closely matches Visual Studio's Diagnostic Tools:

CPU (click to expand)

To test CPU I compute squareroots in a loop for 20 seconds.

Visual Studio's Debugger:

Here, CPU jumps to 100% and stays high for a period before dropping back to 0%.

Otel uses

• UserProcessorTime
  "A TimeSpan that indicates the amount of time that the associated process has spent running code inside the application portion of the process (not inside the operating system core)."
  https://learn.microsoft.com/dotnet/api/system.diagnostics.process.userprocessortime
• and PrivilegedProcessorTime
  "A TimeSpan that indicates the amount of time that the process has spent running code inside the operating system core."
  https://learn.microsoft.com/dotnet/api/system.diagnostics.process.privilegedprocessortime
• OTel repo link

Note that this grows and never resets to zero. That flat line at the end is when the compute finished.

AI SDK uses

• TotalProcessorTime
  "A TimeSpan that indicates the amount of time that the associated process has spent utilizing the CPU. This value is the sum of the UserProcessorTime and the PrivilegedProcessorTime."
  https://learn.microsoft.com/dotnet/api/system.diagnostics.process.totalprocessortime
• and measures the diff between two different collection timestamps, then divides by the number of processes
• AI SDK repo link1
• AI SDK repo link2

TotalProcessorTime is defined as the sum of UserProcessorTime and PrivilegedProcessorTime. So this is the same metrics used by OTel. But AI SDK has a specific algorithm to calculate the difference.

Note that this more closely matches Visual Studio

[azure-sdk]

API change check

API changes are not detected in this pull request.

[TimothyMothra]

@cijothomas @rajkumar-rangaraj
I've addressed all feedback, please re-review.
I'll be out the rest of the week. If everything looks good feel free to merge. :)

[cijothomas]

The screenshot in the description has one with CPU showing above 100%.. That should never happen, if you are sending normalized values.

[reyang]

The screenshot in the description has one with CPU showing above 100%.. That should never happen, if you are sending normalized values.

I suggest that we evaluate what .NET Extension did and align on the algorithm. At minimum, if we want to introduce our own implementation, I want the algorithm to be carefully designed, reviewed (and backed by Math) and documented (e.g. this PR sets a good example). https://github.com/dotnet/extensions/blob/main/src/Libraries/Microsoft.Extensions.Diagnostics.ResourceMonitoring/Linux/LinuxUtilizationProvider.cs

[cijothomas]

The screenshot in the description has one with CPU showing above 100%.. That should never happen, if you are sending normalized values.

I suggest that we evaluate what .NET Extension did and align on the algorithm. At minimum, if we want to introduce our own implementation, I want the algorithm to be carefully designed, reviewed (and backed by Math) and documented (e.g. this PR sets a good example). https://github.com/dotnet/extensions/blob/main/src/Libraries/Microsoft.Extensions.Diagnostics.ResourceMonitoring/Linux/LinuxUtilizationProvider.cs

Agree mostly. My thoughts are

1. AzMon distro or exporter should not be in the business of producing metrics. It should just bring in existing instrumentation libraries only. And so far, we seem to be sticking with this approach.
2. This particular PR is for the LiveMetrics feature. While it'd be nice to follow above here too, it may not be feasible/worth-the-effort. Considering LiveMetrics feature is not to store metrics anywhere permanent, the efforts to design and maintain an algorithm may not be worth it. (Its just for quick visual indication, not even queryable for later consumption, so carefully find the balance and avoid spending too much effort here)
3. We still need to do the basic accuracy part - i.e do not incorrectly tell what the CPU utilization is that of the machine, while actually showing just the single process cpu utilization. Do not state that the cpu utilization is normalized to take into account the number of cores, but send a non-normalized value. (these requires product changes too). (In fact, ApplicationInsights SDKs are lying in many cases - eg: In Linux or in AppService environmetns, app insights only sends process cpu and memory, but the UI shows it as Total Machine CPU and Memory, which is wrong, and that mistake should not be repeated here.) Again, this needs product change/support (And Mothra is already engaged with Product team to get the necessary changes)

[rajkumar-rangaraj]

Providing a summary of the issue before we exploring into the specifics:

In the Application Insights SDK, for the Windows environment, CPU metrics are collected for the entire operating system. In contrast, for Azure App Service and other non-Windows environments, process-level CPU metrics are utilized. When it comes to memory metrics, Windows uses the performance counter \Memory\Committed Bytes, while other environments rely on process-level private bytes.

Languages other than .NET typically read the OS-level counter to send these metrics, leading to inconsistencies in the data sent across different environments and languages.

Mothra has initiated a collaboration with the services team to establish clear guidelines on what exactly should be sent. Our objective is to ensure that the same metrics are sent from all environments and languages.

The services team has requested additional time to formalize this process. As an interim solution, both service and SDK teams have agreed to use process-level metrics for both CPU and memory. This temporary solution will be implemented in beta versions, and we will not release a Release Candidate (RC) or General Availability (GA) version until the standardization is finalized. This approach allows us to continue our work on filters in the meantime.

AzMon distro or exporter should not be in the business of producing metrics. It should just bring in existing instrumentation libraries only. And so far, we seem to be sticking with this approach.

This is correct, except for standard metrics. For live metrics, we don't want to take the dependency on the OpenTelemetry Metric SDK, as it will make the implementation very complex.

This particular PR is for the LiveMetrics feature. While it'd be nice to follow above here too, it may not be feasible/worth-the-effort.

Correct, it is not worth the effort.

Considering LiveMetrics feature is not to store metrics anywhere permanent, the efforts to design and maintain an algorithm may not be worth it. (Its just for quick visual indication, not even queryable for later consumption, so carefully find the balance and avoid spending too much effort here)

I have to agree with Cijo on this part. The algorithm used is very straightforward, and I don't see any issues with it. The LinuxUtilizationProvider uses the same approach..

The screenshot in the description has one with CPU showing above 100%.. That should never happen, if you are sending normalized values.

This is not true. Mothra is trying to explain that using Process.UserProcessorTime plots an incorrect graph, and a small implementation is needed to solve it. He provided the accurate graph at the end of the description and has also validated the graph by onboarding the runtime metrics to the same app.

Finally, I need to acknowledge that we need to accurately capture the information in the changelogs.

[cijothomas]

The screenshot in the description has one with CPU showing above 100%.. That should never happen, if you are sending normalized values.

This is not true. Mothra is trying to explain that using Process.UserProcessorTime plots an incorrect graph, and a small implementation is needed to solve it. He provided the accurate graph at the end of the description and has also validated the graph by onboarding the runtime metrics to the same app.

Thanks! Yes I can see the description is pretty clear.