[Feature]: Elasticsearch v8 support - LFX Mentorship

[yurishkuro]

For the application process please refer to this issue.

• Description: Jaeger has always supported Elasticsearch (ES) as an official backend. Unfortunately, the Go driver we are using, olivere/elastic, does not support ESv8 and not planning to release any new versions. Despite the licensing changes, Elasticsearch remains a popular choice and v8 support question often comes up. In this project we want to add official support for ESv8. However, we also want to take this opportunity to do a better alignment with OpenTelemetry Collector, which already has an ESv8 exporter, perhaps we can use it directly and minimize the amount of code.
• Expected Outcomes:
  • Use OTEL ESv8 exporter from Jaeger, if possible, otherwise build internal implementation
  • Stretch goal: use ES data as the source for Jaeger SPM views
• Recommended Skills:
  • Go: go.mod, bing able to understand existing code, coding, testing ,refactoring
  • Elasticsearch: understand ES architecture, shards, replicas, inserting data, searching for data, indices, index templates, index aliases, index rotations
• Mentor(s):
  • Yuri Shkuro (@yurishkuro)
• Mentee:
  • @pmuls99

Related: #3571

Resources

• https://medium.com/opentracing/take-opentracing-for-a-hotrod-ride-f6e3141f7941
• current ES implementation https://github.com/jaegertracing/jaeger/tree/main/plugin/storage/es
• https://www.jaegertracing.io/docs/1.47/deployment/#elasticsearch
• https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticsearchexporter

[enesonus]

Hi @yurishkuro
I am planning to apply for this project for the CNCF mentoring program. Are there any resource recommendations you have or approaches that the Jaeger team follows so that I can learn more about the subject and help better? I have some knowledge on Go but did not worked much with Elasticsearch before.

[yurishkuro]

@enesonus it's a good question. Long time ago I read some book on ES, it explained it pretty well, but I don't remember which book, and it's outdated by now. I see lots of tutorials and courses online (some are not free). If you find one that it useful, feel free to post here. In order to be successful in this project one needs to understand the elastic write & query APIs, how it builds indices, etc.

[wasup-yash]

I find this project interesting as it implements the new feature and adds the support for v8 functionality, Looking forward to submitting the Proposal for the same as a part of the mentorship program.

[kanha-gupta]

@yurishkuro API creation is a prominent thing to do in this project right ?

[Sayantan-Ch]

Hi @yurishkuro
Came across this project from CNFC mentorship portal. This is an interesting project to work on. I have some experience of building reverse index based search engine using Apache Lucene ( the native framework behind ES). But I don't have a lot of working idea with Jaegar. Could you point me towards some resources which can help me get upto speed.

[yurishkuro]

@Sayantan-Ch I added some links to the description

[tanayvaswani]

Thanks @yurishkuro, the resources are really helpful.

[anirudhprabhakaran3]

Hello! just wanted to register my interest for this project. Will be going through the provided references and making a proposal, thanks!

[roopeshsn]

@enesonus it's a good question. Long time ago I read some book on ES, it explained it pretty well, but I don't remember which book, and it's outdated by now. I see lots of tutorials and courses online (some are not free). If you find one that it useful, feel free to post here. In order to be successful in this project one needs to understand the elastic write & query APIs, how it builds indices, etc.

This is one of the good resources for understanding indices, and related terminologies that I came across: https://www.youtube.com/live/2WJFMYAri_8?feature=share

[enesonus]

Hi everyone I created this Jaeger Demo that uses Elasticsearch as a backend in order to understand how does the system work. If you are interested in please have a look!

BTW thanks for the resource, @roopeshsn it was very helpful in learning elasticsearch. I am planning to add SPM to my demo and I think this resource will be of help.

Note: I intend to continue improving the demo so if you have any suggestions please contact me at [email protected].

[enesonus]

Hi @yurishkuro

In the demo I shared previously, I tested behaviors of Jaeger Collector's elasticsearch exporter and OTEL's elasticsearch exporter and had span documents that looked like this:

OTEL ESv8 exporter (ES v8.9.1):

{
    "_index": "jaeger-span-otel-v8",
    "_id": "jpNuH4oBZmEi1XTdjwN4",
    "_score": 1.0,
    "_source": {
        "@timestamp": "2023-08-22T22:47:22.508381928Z",
        "Duration": 6040,
        "EndTimestamp": "2023-08-22T22:47:22.514422845Z",
        "Kind": "SPAN_KIND_INTERNAL",
        "Link": "[]",
        "Name": "service-price/album_price",
        "ParentSpanId": "58a11df6fa503b41",
        "Resource": {
            "service": {
                "name": "service-price",
                "version": "0.0.1"
            }
        },
        "SpanId": "388228e050e1c2a6",
        "TraceId": "620c5120e36939d1d073bd62584d619b",
        "TraceStatus": 0
    }
},

Jaeger Collector ES exporter (ES v7.17.12):

{
    "_index": "jaeger-span-2023-08-22",
    "_type": "_doc",
    "_id": "oEeZH4oBgR1Ol3NQjNZv",
    "_score": 1.0,
    "_source": {
        "traceID": "4d41cf4410593189aff875b32f2af308",
        "spanID": "a3cb6594af2d0a79",
        "operationName": "service-price/album_price",
        "references": [
            {
                "refType": "CHILD_OF",
                "traceID": "4d41cf4410593189aff875b32f2af308",
                "spanID": "e1facf589205c00b"
            }
        ],
        "startTime": 1692747268456242,
        "startTimeMillis": 1692747268456,
        "duration": 56,
        "tags": [
            {
                "key": "otel.library.name",
                "type": "string",
                "value": "service-price"
            },
            {
                "key": "span.kind",
                "type": "string",
                "value": "internal"
            },
            {
                "key": "internal.span.format",
                "type": "string",
                "value": "proto"
            }
        ],
        "logs": [],
        "process": {
            "serviceName": "service-price",
            "tags": [
                {
                    "key": "service.version",
                    "type": "string",
                    "value": "0.0.1"
                }
            ]
        }
    }
},

I think that different structure of these traces is a remarkable point for implementation of ESv8 exporter of OTEL collector. I saw in the SPM documentation that Jaeger Collector and OTEL collector can be used together like this:

This combination of both collectors generates the same spans as we only use Jaeger Collector. I think we can mimic reverse of this behavior but inside the Jaeger Collector. Briefly:

I was trying to understand the possible ways for the implementation using my demo and some tools such as microsim and wanted to share my opinion on the issue for your feedback. Thanks in advance.

[yurishkuro]

@enesonus diagram makes sense, but do you have a specific question?

Also, what was your objective of comparing the storage format of spans?

[enesonus]

My objective for comparing storage format spans was that, I was able to see span details on Jaeger UI when I use Jaeger Collector (or OTEL Collector with Jaeger Collector using the Jaeger Collector's ES exporter) but if I use OTEL collector to export spans to ES (both v7 or v8) I could not see the spans on Jaeger UI. So I wanted to investigate this and had spans exported to different indices by collectors and then see if there is a difference in format or data type by querying the spans. I came to the conclusion that the reason why we cannot see the spans from the OTEL Collector in the Jaeger UI may be the document format difference. This means that if we want to use OTEL Collector's ESv8 exporter we need to send the Jaeger-Collector's document format to ES. I think this conversion is already implemented since OTEL Collector --> Jaeger Collector --> ES is usable with Jaeger UI.

My question is:
How do you think I can improve the approach that I have on this possible implementation? Anything that I should do while trying to come up with the solution, anything about the Jaeger's internal mechanism that I understood wrong etc.

With this question, my goal is to be able to improve the approach I have on problem solving and have a bit more grasp on the Jaeger project.

P.S. Sorry for the late reply.

[yurishkuro]

Both Jaeger collector and query use the same SpanStorage implementation that transforms between Jaeger domain model and the database storage format.

flowchart LR
    Collector <-->|domain model| SpanStorage
    Query <-->|domain model| SpanStorage
    SpanStorage <-->|database format| Database

Loading

In the current implementation in Jaeger there is a specific JSON format used to represent spans in ES. That format is different from the format used by OTEL Collector's ESv8 exporter. If we want to use OTEL Collector's ESv8 exporter,
w would need to build a new SpanStorage implementation that uses the exporter on the write part and implement a brand new read path to translate from that new format in ES into Jaeger domain model.

[enesonus]

As I understand it, the main focus will be on SpanStorage implementation if OTEL Collector's ESv8 exporter is used. Also, the difference I had in the span document format was due to the existing SpanStorage running in Jaeger Collector and Jaeger Query.

That helped me understand the system a lot thanks!

[yurishkuro]

@pmuls99's application was selected for this project. Congratulations!