Ability to decode static metadata events #2495

pawel-big-lebowski · 2023-05-30T06:16:30Z

Problem

Static metadata events (known also as runless) where presented within in the proposal:
https://github.com/OpenLineage/OpenLineage/blob/main/proposals/1837/static_lineage.md

and is being introduced to Openlineage specification within the PR: OpenLineage/OpenLineage#1880

Of course, we would like to implement it within Marquez. We would like to achieve this ultimate goal gradually and this PR contains the first step.

The changes introduced within the PR:

Don't affect the way RunEvent events are being collected. Events are saved into database and http status code 201 (created) is returned.
DatasetEvent and JobEvent are distinguished by schemaURL property. If distinction cannot be made, events are treated the default way, as RunEvent.
DatasetEvent and JobEventevents are deserialised by standard Jackso library DatasetEvent/JobEvent classes. The response returns in such cases 200 OK http status code but DOES NOT SAVE data to database (this will be introduced later). The great benefit of that is that it is a good integration test and a demo of of how to do events' distinction on the server side.

Relates to: #1868

Solution

EventTypeResolver is introduced which decodes incoming /create requests to LineageEvent, DatasetEvent or JobEvent based on the schemaURL field.
Nothing changes for RunEvents.
In case of DatasetEvent or JobEvent, they get deserialised and distinguished, however no operation on database is performed.

One-line summary:

Checklist

You've signed-off your work
Your changes are accompanied by tests (if relevant)
Your change contains a small diff and is self-contained
You've updated any relevant documentation (if relevant)
You've included a one-line summary of your change for the CHANGELOG.md (Depending on the change, this may not be necessary).
You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
You've included a header in any source code files (if relevant)

codecov · 2023-05-30T06:21:15Z

Codecov Report

Merging #2495 (24c3240) into main (23de0de) will increase coverage by 0.05%.
The diff coverage is 95.00%.

@@             Coverage Diff              @@
##               main    #2495      +/-   ##
============================================
+ Coverage     83.80%   83.86%   +0.05%     
- Complexity     1233     1245      +12     
============================================
  Files           235      238       +3     
  Lines          5625     5657      +32     
  Branches        270      271       +1     
============================================
+ Hits           4714     4744      +30     
- Misses          767      769       +2     
  Partials        144      144

Impacted Files	Coverage Δ
...main/java/marquez/service/models/LineageEvent.java	`95.23% <ø> (ø)`
...java/marquez/service/models/EventTypeResolver.java	`92.00% <92.00%> (ø)`
...src/main/java/marquez/api/OpenLineageResource.java	`90.00% <100.00%> (+1.11%)`	⬆️
...rc/main/java/marquez/service/models/BaseEvent.java	`100.00% <100.00%> (ø)`
.../main/java/marquez/client/models/LineageEvent.java	`100.00% <100.00%> (ø)`

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

wslulciuc · 2023-06-02T06:22:46Z

api/src/main/java/marquez/api/OpenLineageResource.java

+      log.warn("Unsupported event type {}. Skipping without error", event.getClass().getName());
+
+      // return serialized event
+      asyncResponse.resume(Response.status(200).entity(event).build());


200 status code is correct for the new OL events, and feel we should also return a 200 when accepting OL run events (as outlined by the OL spec). The semantics should be: "Return 200 OK to signify the OL event has been collected, and eventually will be processed." The 201 status code was never changed during the initial PoC phase of OL. More of a thought, and we'll want to have a follow up PR.

The idea here was to distinguish between RunEvent that get saved into database (201 created) and other event types that do not affect application state. At the end, once Marquez will be capable of storing dataset and job events, it should return 201 for all the cases.

should we save them in the lineage_events table to start with?

@julienledem we'll want to write a proposal on how to handle DatasetEvents and JobEvents (see #2544). For now, let's ensure the event can be accepted (but not stored).

api/src/main/java/marquez/service/models/JobEvent.java

wslulciuc

Left a couple questions, otherwise 👍

api/src/test/resources/open_lineage/event_dataset_event.json

julienledem

Overall, this looks good to me. I made a few comments

julienledem · 2023-06-06T23:17:12Z

api/src/main/java/marquez/api/OpenLineageResource.java

+      log.warn("Unsupported event type {}. Skipping without error", event.getClass().getName());
+
+      // return serialized event
+      asyncResponse.resume(Response.status(200).entity(event).build());


should we save them in the lineage_events table to start with?

julienledem · 2023-06-06T23:21:53Z

api/src/main/java/marquez/service/models/EventTypeResolver.java

+
+  @AllArgsConstructor
+  public enum EventSchemaURL {
+    LINEAGE_EVENT(


should this be called RUN_EVENT?

I agree, let's use RUN_EVENT.

julienledem · 2023-06-06T23:22:32Z

api/src/main/java/marquez/service/models/EventTypeResolver.java

+    if (id == null) {
+      return context.constructSpecializedType(superType, LINEAGE_EVENT.subType);
+    }
+
+    int lastSlash = id.lastIndexOf('/');
+
+    if (lastSlash < 0) {
+      return context.constructSpecializedType(superType, LINEAGE_EVENT.subType);
+    }


Maybe add comment that we default to run event for backwards compatibility

Signed-off-by: Pawel Leszczynski <[email protected]>

boring-cyborg bot added api API layer changes client/java labels May 30, 2023

pawel-big-lebowski force-pushed the runless-event branch from 4a17af8 to 836cb03 Compare May 30, 2023 07:03

pawel-big-lebowski changed the title ~~decode static metadata events~~ Ability to decode static metadata events May 30, 2023

pawel-big-lebowski force-pushed the runless-event branch from 836cb03 to dd4aadf Compare May 30, 2023 11:20

boring-cyborg bot added the docs label May 30, 2023

pawel-big-lebowski requested a review from wslulciuc May 30, 2023 11:38

pawel-big-lebowski marked this pull request as ready for review May 30, 2023 11:38

harels requested a review from julienledem June 1, 2023 22:15

wslulciuc reviewed Jun 2, 2023

View reviewed changes

wslulciuc approved these changes Jun 2, 2023

View reviewed changes

wslulciuc reviewed Jun 2, 2023

View reviewed changes

api/src/test/resources/open_lineage/event_dataset_event.json Show resolved Hide resolved

pawel-big-lebowski mentioned this pull request Jun 6, 2023

Emit job and dataset runless metadata OpenLineage/OpenLineage#1880

Merged

10 tasks

julienledem approved these changes Jun 6, 2023

View reviewed changes

wslulciuc mentioned this pull request Jul 17, 2023

Proposal for static lineage support #2544

Closed

decode runless events

24c3240

Signed-off-by: Pawel Leszczynski <[email protected]>

pawel-big-lebowski force-pushed the runless-event branch from dd4aadf to 24c3240 Compare July 17, 2023 10:48

pawel-big-lebowski merged commit 197e1df into main Jul 17, 2023

pawel-big-lebowski deleted the runless-event branch July 17, 2023 13:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ability to decode static metadata events #2495

Ability to decode static metadata events #2495

pawel-big-lebowski commented May 30, 2023 •

edited

Loading

codecov bot commented May 30, 2023 •

edited

Loading

wslulciuc Jun 2, 2023 •

edited

Loading

pawel-big-lebowski Jun 2, 2023

julienledem Jun 6, 2023

wslulciuc Jul 17, 2023

wslulciuc left a comment

julienledem left a comment

julienledem Jun 6, 2023

julienledem Jun 6, 2023

wslulciuc Jul 17, 2023

julienledem Jun 6, 2023

Ability to decode static metadata events #2495

Ability to decode static metadata events #2495

Conversation

pawel-big-lebowski commented May 30, 2023 • edited Loading

Problem

Solution

Checklist

codecov bot commented May 30, 2023 • edited Loading

Codecov Report

wslulciuc Jun 2, 2023 • edited Loading

Choose a reason for hiding this comment

pawel-big-lebowski Jun 2, 2023

Choose a reason for hiding this comment

julienledem Jun 6, 2023

Choose a reason for hiding this comment

wslulciuc Jul 17, 2023

Choose a reason for hiding this comment

wslulciuc left a comment

Choose a reason for hiding this comment

julienledem left a comment

Choose a reason for hiding this comment

julienledem Jun 6, 2023

Choose a reason for hiding this comment

julienledem Jun 6, 2023

Choose a reason for hiding this comment

wslulciuc Jul 17, 2023

Choose a reason for hiding this comment

julienledem Jun 6, 2023

Choose a reason for hiding this comment

pawel-big-lebowski commented May 30, 2023 •

edited

Loading

codecov bot commented May 30, 2023 •

edited

Loading

wslulciuc Jun 2, 2023 •

edited

Loading