Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: dotnet tracing example #3891

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -80,4 +80,6 @@ Refer to the [Tempo data source configuration documentation](https://grafana.com

## Examples

Check out the [examples](https://github.com/grafana/pyroscope/tree/main/examples/tracing/tempo) directory for a complete demo application of span profiles in multiple languages.
Check out these demo applications for span profiles:
- [.NET example](https://github.com/grafana/pyroscope/tree/main/examples/tracing/dotnet)
- [Other examples](https://github.com/grafana/pyroscope/tree/main/examples/tracing/tempo) in multiple languages
1 change: 1 addition & 0 deletions examples/tracing/dotnet/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
*.so
48 changes: 48 additions & 0 deletions examples/tracing/dotnet/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
ARG SDK_VERSION=8.0
# The build images takes an SDK image of the buildplatform, so the platform the build is running on.
FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:$SDK_VERSION AS build

ARG TARGETPLATFORM
ARG BUILDPLATFORM
ARG SDK_VERSION

WORKDIR /dotnet

ADD example .

# Set the target framework to SDK_VERSION
RUN sed -i -E 's|<TargetFramework>.*</TargetFramework>|<TargetFramework>net'$SDK_VERSION'</TargetFramework>|' Example.csproj

# We hardcode linux-x64 here, as the profiler doesn't support any other platform
RUN dotnet publish -o . --framework net$SDK_VERSION --runtime linux-x64 --no-self-contained

# This fetches the SDK
FROM --platform=linux/amd64 pyroscope/pyroscope-dotnet:0.9.2-glibc AS sdk

# Runtime only image of the targetplatfrom, so the platform the image will be running on.
FROM --platform=linux/amd64 mcr.microsoft.com/dotnet/aspnet:$SDK_VERSION

WORKDIR /dotnet

COPY --from=sdk /Pyroscope.Profiler.Native.so ./Pyroscope.Profiler.Native.so
COPY --from=sdk /Pyroscope.Linux.ApiWrapper.x64.so ./Pyroscope.Linux.ApiWrapper.x64.so
COPY --from=build /dotnet/ ./


ENV CORECLR_ENABLE_PROFILING=1
ENV CORECLR_PROFILER={BD1A650D-AC5D-4896-B64F-D6FA25D6B26A}
ENV CORECLR_PROFILER_PATH=/dotnet/Pyroscope.Profiler.Native.so
ENV LD_PRELOAD=/dotnet/Pyroscope.Linux.ApiWrapper.x64.so

ENV PYROSCOPE_APPLICATION_NAME=rideshare.dotnet.push.app
ENV PYROSCOPE_SERVER_ADDRESS=http://pyroscope:4040
ENV PYROSCOPE_LOG_LEVEL=debug
ENV PYROSCOPE_PROFILING_ENABLED=1
ENV PYROSCOPE_PROFILING_ALLOCATION_ENABLED=true
ENV PYROSCOPE_PROFILING_CONTENTION_ENABLED=true
ENV PYROSCOPE_PROFILING_EXCEPTION_ENABLED=true
ENV PYROSCOPE_PROFILING_HEAP_ENABLED=true
ENV RIDESHARE_LISTEN_PORT=5000


CMD sh -c "ASPNETCORE_URLS=http://*:${RIDESHARE_LISTEN_PORT} exec dotnet /dotnet/example.dll"
10 changes: 10 additions & 0 deletions examples/tracing/dotnet/Dockerfile.load-generator
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
FROM python:3.9

RUN pip3 install requests

COPY load-generator.py ./load-generator.py

ENV PYTHONUNBUFFERED=1

CMD [ "python", "load-generator.py" ]

53 changes: 53 additions & 0 deletions examples/tracing/dotnet/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Span Profiles with Grafana Tempo and Pyroscope

The docker compose consists of:
- The .NET Rideshare App
- Tempo
- Pyroscope
- Grafana

The `rideshare` app generate traces and profiling data that should be available in Grafana.
Datasources for Pyroscope and Tempo are provisioned automatically.

### Build and run

The project can be run locally with the following commands:

```shell
# (optionally) pull latest pyroscope and grafana images:
docker pull grafana/pyroscope:latest
docker pull grafana/grafana:latest

# build and run the example
docker compose up --build
```

Navigate to the [Explore page](http://localhost:3000/explore?schemaVersion=1&panes=%7B%22f36%22:%7B%22datasource%22:%22tempo%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22datasource%22:%7B%22type%22:%22tempo%22,%22uid%22:%22tempo%22%7D,%22queryType%22:%22traceqlSearch%22,%22limit%22:20,%22tableType%22:%22traces%22,%22filters%22:%5B%7B%22id%22:%22e73a615e%22,%22operator%22:%22%3D%22,%22scope%22:%22span%22%7D,%7B%22id%22:%22service-name%22,%22tag%22:%22service.name%22,%22operator%22:%22%3D%22,%22scope%22:%22resource%22,%22value%22:%5B%22rideshare.dotnet.push.app%22%5D,%22valueType%22:%22string%22%7D%5D,%22query%22:%22%7Bresource.service.name%3D%5C%22rideshare.dotnet.push.app%5C%22%7D%22%7D%5D,%22range%22:%7B%22from%22:%22now-15m%22,%22to%22:%22now%22%7D%7D%7D&orgId=1), select a trace and click on a span that has a linked profile:

![image](https://github.com/grafana/otel-profiling-go/assets/12090599/31e33cd1-818b-4116-b952-c9ec7b1fb593)

By default, only the root span gets labeled (the first span created locally): such spans are marked with the _link_ icon
and have the `pyroscope.profile.id` attribute set to the corresponding span ID.
Please note that presence of the attribute does not necessarily
indicate that the span has a profile: stack trace samples might not be collected, if the utilized CPU time is
less than the sample interval (10ms).

### Instrumentation

The `rideshare` demo application is instrumented with Pyroscope: [Pyroscope .NET Agent](https://github.com/grafana/pyroscope-dotnet)

### Grafana Tempo configuration

In order to correlate trace spans with profiling data, the Tempo datasource should have the following configured:
- The profiling data source
- Tags to use when making profiling queries

![image](https://github.com/grafana/pyroscope/assets/12090599/380ac574-a298-440d-acfb-7bc0935a3a7c)

While tags are optional, configuring them is highly recommended for optimizing query performance.
In our example, we configured the `service.name` tag for use in Pyroscope queries as the `service_name` label.
This configuration restricts the data set for lookup, ensuring that queries remain
consistently fast. Note that the tags you configure must be present in the span attributes or resources
for a trace to profiles span link to appear.

Please refer to our [documentation](https://grafana.com/docs/grafana/next/datasources/tempo/configure-tempo-data-source/#trace-to-profiles) for more details.
69 changes: 69 additions & 0 deletions examples/tracing/dotnet/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
services:
pyroscope:
image: grafana/pyroscope
ports:
- "4040:4040"

us-east:
ports:
- "5000"
environment: &env
OTLP_URL: tempo:4318
OTEL_TRACES_EXPORTER: otlp
OTEL_EXPORTER_OTLP_ENDPOINT: http://tempo:4317
OTEL_SERVICE_NAME: rideshare.dotnet.push.app
OTEL_METRICS_EXPORTER: none
OTEL_TRACES_SAMPLER: always_on
OTEL_PROPAGATORS: tracecontext
REGION: us-east
PYROSCOPE_LABELS: region=us-east
PYROSCOPE_SERVER_ADDRESS: http://pyroscope:4040
build:
context: .
eu-north:
ports:
- "5000"
environment:
<<: *env
REGION: eu-north
build:
context: .
ap-south:
ports:
- "5000"
environment:
<<: *env
REGION: ap-south
build:
context: .

load-generator:
build:
context: .
dockerfile: Dockerfile.load-generator

grafana:
image: grafana/grafana:latest
environment:
- GF_INSTALL_PLUGINS=grafana-pyroscope-app
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
- GF_AUTH_DISABLE_LOGIN_FORM=true
- GF_FEATURE_TOGGLES_ENABLE=traceToProfiles tracesEmbeddedFlameGraph
volumes:
- ./grafana-provisioning:/etc/grafana/provisioning
ports:
- "3000:3000"

tempo:
image: grafana/tempo:latest
command: [ "-config.file=/etc/tempo.yml" ]
volumes:
- ./tempo/tempo.yml:/etc/tempo.yml
ports:
- "14268:14268" # jaeger ingest
- "3200:3200" # tempo
- "9095:9095" # tempo grpc
- "4317:4317" # otlp grpc
- "4318:4318" # otlp http
- "9411:9411" # zipkin
2 changes: 2 additions & 0 deletions examples/tracing/dotnet/example/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
bin/
obj/
16 changes: 16 additions & 0 deletions examples/tracing/dotnet/example/BikeService.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
namespace Example;

internal class BikeService
{
private readonly OrderService _orderService;

public BikeService(OrderService orderService)
{
_orderService = orderService;
}

public void Order(int searchRadius)
{
_orderService.FindNearestVehicle(searchRadius, "bike");
}
}
16 changes: 16 additions & 0 deletions examples/tracing/dotnet/example/CarService.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
namespace Example;

internal class CarService
{
private readonly OrderService _orderService;

public CarService(OrderService orderService)
{
_orderService = orderService;
}

public void Order(int searchRadius)
{
_orderService.FindNearestVehicle(searchRadius, "car");
}
}
17 changes: 17 additions & 0 deletions examples/tracing/dotnet/example/Example.csproj
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<AssemblyName>example</AssemblyName>
<OutputType>Exe</OutputType>
<PackageId>example</PackageId>
<Nullable>enable</Nullable>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="OpenTelemetry.Exporter.Console" Version="1.8.0" />
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.8.0" />
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.8.0" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.8.1" />
<PackageReference Include="Pyroscope.OpenTelemetry" Version="0.2.0" />
<PackageReference Include="System.Diagnostics.DiagnosticSource" Version="8.0.1" />
</ItemGroup>
</Project>
4 changes: 4 additions & 0 deletions examples/tracing/dotnet/example/Folder.DotSettings.user
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
<wpf:ResourceDictionary xml:space="preserve" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:s="clr-namespace:System;assembly=mscorlib" xmlns:ss="urn:shemas-jetbrains-com:settings-storage-xaml" xmlns:wpf="http://schemas.microsoft.com/winfx/2006/xaml/presentation">
<s:String x:Key="/Default/Environment/AssemblyExplorer/XmlDocument/@EntryValue">&lt;AssemblyExplorer&gt;
&lt;Assembly Path="/home/korniltsev/.nuget/packages/pyroscope/0.4.0/lib/net6.0/Pyroscope.dll" /&gt;
&lt;/AssemblyExplorer&gt;</s:String></wpf:ResourceDictionary>
57 changes: 57 additions & 0 deletions examples/tracing/dotnet/example/OrderService.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
using System;

namespace Example;

internal class OrderService
{
public void FindNearestVehicle(long searchRadius, string vehicle)
{
lock (_lock)
{
var labels = Pyroscope.LabelSet.Empty.BuildUpon()
.Add("vehicle", vehicle)
.Build();
Pyroscope.LabelsWrapper.Do(labels, () =>
{
for (long i = 0; i < searchRadius * 1000000000; i++)
{
}

if (vehicle.Equals("car"))
{
CheckDriverAvailability(labels, searchRadius);
}
});
}
}

private readonly object _lock = new();

private static void CheckDriverAvailability(Pyroscope.LabelSet ctx, long searchRadius)
{
var region = System.Environment.GetEnvironmentVariable("REGION") ?? "unknown_region";
ctx = ctx.BuildUpon()
.Add("driver_region", region)
.Build();
Pyroscope.LabelsWrapper.Do(ctx, () =>
{
for (long i = 0; i < searchRadius * 1000000000; i++)
{
}

var now = DateTime.Now.Minute % 2 == 0;
var forceMutexLock = DateTime.Now.Minute % 2 == 0;
if ("eu-north".Equals(region) && forceMutexLock)
{
MutexLock(searchRadius);
}
});
}

private static void MutexLock(long searchRadius)
{
for (long i = 0; i < 30 * searchRadius * 1000000000; i++)
{
}
}
}
Loading
Loading