N+1 detection

[jcarres-mdsol]

It would be insanely useful to be able to run a query that detects potential N+1 issues.

I thought this would give me an approximation to that:

({ kind = client  && .db.system = "postgresql" } ~ { kind = client  && .db.system = "postgresql" })
| count() > 10

Trying to mean: 10 siblings all of which are calling a DB.
Something like this would be suspicious enough to take a look.

But I think the result I'm getting for that query is:

({ kind = client  && .db.system = "postgresql" } ~ { kind = client  && .db.system = "postgresql" }) &&
{ kind = client  && .db.system = "postgresql" }  | count() > 10

The very least definitely I do not get traces with 10 calls to DBs from siblings only.
Is this expected?

[joe-elliott]

Here is what I'm seeing:

The query on the left is showing a count of 10 which matches the number of spans in the trace on the right that satisfies the condition. Is this what you were expecting?

[jcarres-mdsol]

That's what I'm seeing too but that's not what I was expecting.
What we see is (I think):

• A trace which contains 10 database spans of which at least there are two in a sibling relationship

What I was expecting:

• A trace which contains 10 database spans as siblings to each other.

The doc about count says "The count of spans in the spanset." and my spanset is a sibling relationship so I was expecting the count on this sibling relationship to be 10.

[joe-elliott]

So all of the spans in a trace are in the same spanset which is why it behaves more like #1 then #2. The query finds a trace where 10 spans satisfy the two conditions:

• I look like this: { kind = client && .db.system = "postgresql" }
• I have at least one sibling that looks like this: { kind = client && .db.system = "postgresql" }

So you could have 5 pairs of 2 siblings in the same trace and it would also show a spanset size of 10.

To get what you're looking for we need to add a "parent span id" intrinsic which we would like to. Then you could write something like the below. This would group the spans into different spansets based on their parent span id.

{ kind = client  && .db.system = "postgresql" } ~ { kind = client  && .db.system = "postgresql" }
| by(parent.span.id)
| count() > 10

A new wave of intrinsics is currently being discussed. We will look at adding this.

Also, if you have a known parent, then I think you can do this now by doing something like:

{ <conditions that specify one parent> } >
({ kind = client  && .db.system = "postgresql" } ~ { kind = client  && .db.system = "postgresql" })
| count() > 10

[jcarres-mdsol]

The only idea I can think of is:
{ .service.name = X && kind=server }

Which does not help when a service is called multiple times in the same trace.
Getting more specific is not very useful in this case, if I'm able to craft such a specific parent, then I can directly search for that and visually verify if there is a N+1 going on.

The possible future where the parent span id is usable would allow for such a trace.
Either this way or some other, any way you can claim Tempo helps finding N+1 will draw attention from users, as it is such a common and easy to understand performance pitfall.

In general I'm sure there are use-cases where it would be interesting to access the span id, parent id , trace id.