How to use EventCounters?

[felipepessoto]

The lack of Performance Counters in .NET Core is blocking me to use ASP.NET Core for Enterprise applications.

I found many issues telling us to use EventCounters, but I couldn't find docs or even code in the repository using it.

There is any docs?

Anyway, ETW events doesn't replace Performance Counters, I use it to monitor Req/sec and I can't see how to do it easily with ETW, or ask the support team to monitor or collect it. Would be nice to have a Windows only implementation.

Thanks

[karelz]

For PerformanceCounters please vote on #15468

@vancem used to have extensive documentation of EventSource, with EventCounters added as well. I can't find the source, but it should be part of the NuGet package. @vancem any pointer?

[felipepessoto]

#15468 is only to read the values. I'm more interested in writing them.

[karelz]

For the producer scenario, why cannot you use EventCounters?
Even PerfCounters (v2) use ETW under the hood, just slightly different events format. Everything should be achievable with just EventCounters.

The problem is that .NET Framework uses PerfCounters v1. It is fragile, old technology. It requires DLL in system32, so admin machine-wide setup and machine-wide registration. It is hard to do multiple products on the same machine (side-by-side). It is buggy.
Machine-wide registration is something we want to avoid in .NET Core. It leads to more and more troubles.

If you still need PerfCounters, I would suggest to use EventCounters and write your own out-of-process consumer of EventCounters which translates the events into PerfCounters (v2). It will be work, it will require you to solve the machine-wide registration, but it could work if you truly believe you need that.

[felipepessoto]

I still need to read more about EventCounters e ETW to say it won't work in my scenario. I'm doing some tests

In particular, it's very easy to use PerfCounter. For devs, we just need to call Increment() and it translates to absolute numbers, rate per second, average, etc. And the support team just need to open Perfmon and watch the numbers.

Another great feature is the MultiInstance support

[karelz]

In particular, it's very easy to use PerfCounter. For devs, we just need to call Increment() and it translates to absolute numbers, rate per second, average, etc.

EventCounters have similar metrics automatically AFAIR.

And the support team just need to open Perfmon and watch the numbers.

That one is tricky - you pay for it by having .NET Framework installed machine wide and registered machine wide. It's a hidden HUGE cost. You can achieve that, but you have to pay the cost of installing & registering.
It shouldn't be that difficult to write the code. The installation in each app will be giant pain IMO.

[felipepessoto]

I find out how to generate the events and read them in other process. But just the events written by EventSource.WriteEvent, I can't read those written by EventCounter.WriteMetric.

I created a project based on this PR: aspnet/Hosting#886. But it's curious nobody else is using EventCounters, even this PR wasn't merged.

[karelz]

EventCounters are based on top of EventSource, so the consumption should be the same.

I let @vancem chime in on how wide-spread EventCounters are these days and where to find the best examples (we had them in pretty extensive internal documentation 2-3 years ago when we designed them, so I hope it is available somewhere - @vancem typically does that, so if it's not available, we should fix that ;)).

[vancem]

• @cshung

Note that EventCounters are work in progress at the moment. Andrew was able to put a certain amount of work to get something working enough so that that direction/architecture was clear, and we could build upon it. The tooling is certainly not there yet. You could imagine building a bridge to windows PerfCounters so that PerfMon works (however it has all the disadvantages of that, including windows-only and admin-install requirements).

Andrew: did we publish your blog on how to use it? I am noticing that none of the docs we wrote up have made it externally. We should massage your blog.html into markdown and at least check in into the CoreFX repo (we can have a blog entry point at it). We should also get the spec that we wrote out so it is visiable. We can use this issue to track that.

The source code for EventCounters is at

https://github.com/dotnet/corefx/blob/master/src/System.Diagnostics.Tracing/src/System/Diagnostics/Tracing/EventCounter.cs

And here are the tests, which can be helpful in actual usage.

https://github.com/dotnet/corefx/blob/master/src/System.Diagnostics.Tracing/tests/BasicEventSourceTest/TestEventCounter.cs

This is no substitute for proper docs (which we will work on), but I can give you a quick sketch of what EventCounters are. Bascially they are a layer on top of EventSource that allow you to have a simple API (WriteMetric) that gets accumulated LOCALLY (calculating sum, average ...). Consumers (e.g. tools), can then ask to turn these counters on and have their values dumped into an event at a given rate (e.g. once every X seconds or on demand).

So from an instrumentation perspective, it is pretty relatively simple to use EventCounters, (you create an EventSource (which defines a 'namespace' for the data), define counters (give the various values names), and then do WriteMetric() calls on it every time one of these values changes. The EventCounter infrastructure calculates stats locally. All of this is off unless some tool asks for it. When the tool turns it on it gives a update rate, and the stats are dumped at that rate as events. As mentioned there is rudimentary support for this in the PerfView tool, but otherwise I know of no other tooling.

To answer the obvious question: why don't you just support PerfCounters? First we are not averse for someone doing this. We just don't think it is the best approach for support in CoreFX because

1. Architecturally Perfcounters are a separate 'pipe' of diagnostic information. This is pretty unfortunate, as there is no real value in having two pipes. Arguably you should pass the counter data through the same 'pipe' as any other telementry. This is in fact what WIndows does under the hood (perfcounters use ETW as a substrate), however from the users's point of view they are not integrated. Fixing this means that you need to change the PerfCounter APIs.

2. PerfCounters are really never enough alone. They are for monitoring, but when things go wrong, you quickly want to get more detailed data (which require events). This is just another argument that counters are just a special kind of low-overhead eventing, and you want the two systems integrated (so you only have to instrument once).

3. The only existing code for PerfCounters is on Windows, and has a fair bit of complexity to it (there are many counter types, and other subtleties) so making this work cross platform is problematic (and it does not fix point (1). Thus making something that is 'drop in compatible' is dubious. Once you decide that you should break the API, then fixing the other architectural problems (it is not integrated with eventing), make even more sense.

4. Existing PerfCounters require admin install. Again this really is a manifestation of point (1), because if you just used the existing telemetry pipe the problem is already solved.

We do understand that we are not at parity with what exists on windows, and it will take some time to get to that parity (we need to make it easy). However even here it is not clear how exactly to do that (arguably we should just be part of a bigger monitoring system like App Insights). We are making progress, but it is slower than we would like.

[felipepessoto]

I tried to use Perview to collect data from EventCounter, but it just logs events from WriteEvent, it seems EventCounter.WriteMetric is doing nothing.

[vancem]

Exactly what did you do to try to turn on the EventCounter using PerfView? You have to do more than just turn on the provider, you have to tell it the sampling frequency. For example to turn on EventCounters associated with the EventSource called 'SomeEventSource' a PerfView command line that would do that is

• PerfView /onlyProviders=*SomeEventSource:EventCounterIntervalSec=1 collect

Did you do something like this?

[felipepessoto]

I only set the provider, without the interval (and I used the GUI).

I will try your suggestion. Thanks

[felipepessoto]

It worked on PerfView.

How can I collect EventCounters using TraceEventSession?

Thanks.

[felipepessoto]

About the admin privileges. Perfmon requires me to be admin to create the definition, what is ok for most people. Does ETW require me to be admin to collect the data?

[karelz]

AFAIK ETW collection requires admin :(

[cshung]

Some time ago I wrote a PerfView extension that allow visualizing the EventCounter data, hope it helps.
microsoft/perfview#146

[vancem]

How can I collect EventCounters using TraceEventSession?

The 'EnableProvider' command has an argument of type 'TraceEventOptions, which has a 'AddArgument(key, value), which you can use to pass key value pairs. Thus

    var options = new TraceEventOptions();
    options.add("EventCounterIntervalSec", 1);
    session.EnableProvider(eventSourceName, TraceEventLevel.Critical, 0, options). 

Should do it.

Does ETW require me to be admin to collect the data?

Yes, (there is request in the OS to fix this, but that is how it is today).

On .NET Core we are building a way to collect this information without using ETW (needed for Linux), and it does not require admin privs @brianrob has more on that. However this very new code, and at the present time is not supported on Windows (thus I mention it only to say 'it is coming').

[vancem]

Some documentation for EventCounters has been checked in here

https://github.com/dotnet/corefx/blob/master/src/System.Diagnostics.Tracing/documentation/EventCounterTutorial.md

[karelz]

Is it enough docs to close this issue or do we need more docs?

[felipepessoto]

I think so

[andreycha]

@fujiy @vancem hi guys. How did you manage to get counter values in PerfView? When I launch it via command line (PerfView /onlyProviders=*SomeEventSource:EventCounterIntervalSec=1 collect) it only shows me events. And I'm only able to see counters (but without events) if I run collection from PerfView UI with the same parameters. I use PerfView 2.0.7.

[andreycha]

@vancem and is it correct to use event counters without calling WriteEvent()? E.g. if I just want to track numbers and don't need events. Should I mark Request() method with [NonEvent] attribute maybe?

[EventSource(Name = "Samples-EventCounterDemos-Minimal")]
public sealed class MinimalEventCounterSource : EventSource
{
    public static MinimalEventCounterSource Log = new MinimalEventCounterSource();

    private EventCounter requestCounter;

    private MinimalEventCounterSource()
    {
        this.requestCounter = new EventCounter("request", this);
    }

    public void Request(string url, float elapsedMSec)
    { 
        this.requestCounter.WriteMetric(elapsedMSec);
    }
}

[vancem]

FYI: There is some tutorial documentation.

https://github.com/dotnet/corefx/blob/master/src/System.Diagnostics.Tracing/documentation/EventCounterTutorial.md

@andreycha - I am very surprised by question above. You seem to have the right PerfView command line and I certainly would not have expected different behavior based on you use the GUI or not.

Your code above will work (and never fire an event) but I would recommend you put the [NonEvent] attribute on your 'Request' method since otherwise the EventSource declares it has a Request event even though it will never log such an event (since its body does not have an WriteEvent call

[andreycha]

@vancem thank you for the answer. Maybe I did smth wrong but after I removed all [Event] methods from my event source and left only "perf counters" methods, it started to work fine.

I have one more question: do I need to register (https://github.com/Microsoft/dotnet-samples/blob/master/Microsoft.Diagnostics.Tracing/EventSource/docs/EventRegister.md) such event source which contains only counters? If so, how to do that for .NET Core application, because Microsoft.Diagnostics.Tracing.EventRegister package targets .NET 4.6? Thanks!

[vancem]

@vancem thank you for the answer. Maybe I did smth wrong but after I removed all [Event] methods from my event source and left only "perf counters" methods, it started to work fine.

That is surprising behavior. It is not how it is supposed to work.

I have one more question: do I need to register (https://github.com/Microsoft/dotnet-samples/blob/master/Microsoft.Diagnostics.Tracing/EventSource/docs/EventRegister.md) such event source which contains only counters?

No. EventRegister only makes sense if you want to use EventSource to log things to things like the windows event viewer.

[andreycha]

@vancem back to the incorrect data I see in PerfView. Here is a simple project created as .NET Core console application. Event source definition is taken from https://github.com/dotnet/corefx/blob/master/src/System.Diagnostics.Tracing/documentation/EventCounterTutorial.md.

using System;
using System.Diagnostics.Tracing;

namespace EventSourceSample
{
    class Program
    {
        private static Random Random = new Random();

        static void Main(string[] args)
        {
            Console.WriteLine(@"Enter anything and press ENTER. Enter ""q"" to exit.");

            string input;
            while ((input = Console.ReadLine()) != "q")
            {
                MinimalEventCounterSource.Log.Request(input, Random.Next(10, 100));
            }
        }
    }

    [EventSource(Name = "Samples-EventCounterDemos-Minimal")]
    public sealed class MinimalEventCounterSource : EventSource
    {
        public static MinimalEventCounterSource Log = new MinimalEventCounterSource();
        private EventCounter requestCounter;

        private MinimalEventCounterSource()
        {
            this.requestCounter = new EventCounter("request", this);
        }

        [Event(1)]
        public void Request(string url, float elapsedMSec)
        {
            WriteEvent(1, url, elapsedMSec);
            this.requestCounter.WriteMetric(elapsedMSec);
        }
    }
}

Here is how I start PerfView:

PerfView collect /OnlyProviders=*Samples-EventCounterDemos-Minimal:EventCounterIntervalSec=1

Here is my input:

aaa
bbb
ccc

First issue is that I don't see separate EventCounter category in the list:

Second issue is that counter name is somehow mixed in into events (I was expecting to see only three events according to my input):

If I remove WriteEvent call and mark Request() method with [NonEvent], I start to see EventCounter category.

If I remove counter instantiation and writing metrics, I start to see only expected events.

Looks like they don't work together. Could you please advise?

[Expecho]

I tried setting up a new TraceEventSession using this code:

var options = new TraceEventProviderOptions();
options.AddArgument("EventCounterIntervalSec", "5");
session.EnableProvider(eventSourceName, options: options);

but I have troubles getting the right values out. PerfView does show the counter values as expected / documented but a complete example of reading them using a TraceEventSession would be more than welcome. For example, should I be able to use session.Source.Dynamic.All to read the EventCounter values?

[aaronhudon]

FYI: There is some tutorial documentation.

https://github.com/dotnet/corefx/blob/master/src/System.Diagnostics.Tracing/documentation/EventCounterTutorial.md

@andreycha - I am very surprised by question above. You seem to have the right PerfView command line and I certainly would not have expected different behavior based on you use the GUI or not.

Your code above will work (and never fire an event) but I would recommend you put the [NonEvent] attribute on your 'Request' method since otherwise the EventSource declares it has a Request event even though it will never log such an event (since its body does not have an WriteEvent call

@vancem The link to this documentation is 404ing

[davidsh]

@noahfalk

[aaronhudon]

How can I collect EventCounters using TraceEventSession?

The 'EnableProvider' command has an argument of type 'TraceEventOptions, which has a 'AddArgument(key, value), which you can use to pass key value pairs. Thus

    var options = new TraceEventOptions();
    options.add("EventCounterIntervalSec", 1);
    session.EnableProvider(eventSourceName, TraceEventLevel.Critical, 0, options). 

Should do it.

Does ETW require me to be admin to collect the data?

Yes, (there is request in the OS to fix this, but that is how it is today).

On .NET Core we are building a way to collect this information without using ETW (needed for Linux), and it does not require admin privs @brianrob has more on that. However this very new code, and at the present time is not supported on Windows (thus I mention it only to say 'it is coming').

@vancem what are some of the other options besides "EventCounterIntervalSec". The EventCounter supposedly has Min/Max,Count/Mean/StdDev

Thanks!

[Expecho]

@vancem what are some of the other options besides "EventCounterIntervalSec". The EventCounter supposedly has Min/Max,Count/Mean/StdDev

Thanks!

Min/Max,Count/Mean/StdDev are not options but part of the output. Given the specified interval values are aggregated and the min, max etc are calculated.

I've written a blog post about eventcounters a while ago, you can find it here

[Expecho]

FYI: There is some tutorial documentation.
https://github.com/dotnet/corefx/blob/master/src/System.Diagnostics.Tracing/documentation/EventCounterTutorial.md
@andreycha - I am very surprised by question above. You seem to have the right PerfView command line and I certainly would not have expected different behavior based on you use the GUI or not.
Your code above will work (and never fire an event) but I would recommend you put the [NonEvent] attribute on your 'Request' method since otherwise the EventSource declares it has a Request event even though it will never log such an event (since its body does not have an WriteEvent call

@vancem The link to this documentation is 404ing

You can find it here