Mapping Span Names between Client/Service & Parent/Child spans

[jSwanson99]

Hi! I have a tempo service setup with the metrics-generator enabled, and I have been pretty impressed with what this can do

Rambling
However, I noticed a lot of the instrumentation for outbound HTTP requests typically does not include the request path in the span name, since to the client making that request its just some string (so technically cannot be accurately parameterized). Generally on the server receiving this request, I see those span names do tend to include a parameterized request path in the span name.
I've found it to be extremely valuable to correlate these two spans, so that I can compare the response time as reported by server vs client.

To work around this, I am trying to use some regex/string manipulation to be clever enough to include at least some info in the outbound requests span name, which is OK, but the Tempo-specific question I have is:

Question
Is it possible in Tempo to map the span name of a SERVER type span to the corresponding CLIENT type span, provided the SERVER span is the first child of the CLIENT span? I realize we would need to be pretty opinionated about how we decide if its OK to map these (http.route vs http.path vs http.target etc), but I find these server-server request comparisons to yield some pretty interesting stats

Example
Basically I am trying to make it easier to hone in on problems like these (durations are arbitrary here):
{ kind=client && duration > 500ms } > { kind=server && duration < 500ms}

[jSwanson99]

A follow up here: I was reading this PR: https://github.com/grafana/tempo/pull/3453/files and I am wondering if this is trying to solve a similar problem as I am describing above? It sounds to me like this would introduce a metric representing latency between services?

[joe-elliott]

since to the client making that request its just some string (so technically cannot be accurately parameterized)

This had not occurred to me before, but there are cases where it would be difficult to match up a client and server span.

I do think the PR you linked will give you a tool to answer the questions that you have. I will do my best to give it a final review to day.

cc @adirmatzkin

[jSwanson99]

I didn't realize the service-graph came with metrics, that's pretty useful
Do you think some sort of mapping exercise would be feasible, or are there too many cases where it would be difficult? I am not familiar with what would make this difficult, is why I ask

[joe-elliott]

It depends on your data. If you can expose a label on the client side and add it to metrics that 1:1 matches up with your paths on the server side then I think your issue is solvable.

[adirmatzkin]

As far as I understand the need here, it seems that enabling the enable_client_server_prefix config option together with the adding the http path attribute to the dimensions config (in the service-graph processor) - will give you what you're looking for... isn't it?
You will get a latency histogram from both the client and the server sides (2 separate metrics), and will have the URL path as a label on them.
That way, looking for a dramatic difference between the latency from the client side to the server side, for each server,client,http_path triplet will show you that.

Sounds right? Or am I missing something? @jSwanson99
I don't see it really related to what was needed in #3453

[jSwanson99]

Yeah, using the plain http.route/path would work as I want, its just the cardinality that would be very high. I was hoping to use span names for this, since at least "server" spans have enough knowledge about their service to parameterize the span names (alleviating the metrics cardinality issue). The issue I was having is that client spans do not necessarily have that knowledge, so there is often a mismatch in span names.
Given that I learned there are some metrics in the service-graph to help with this, I think its safe to call my problem "solved" for now, since I have new data to sift through. Thanks to both of you for the input here