feat(lineage) Update Lineage tab and Impact Analysis feature #5121

chriscollins3456 · 2022-06-08T15:06:54Z

Makes a change to the Lineage tab so that we always show the Impact Analysis view but now we can toggle between Upstream and Downstream. We now have Filters default open and we are default selected "Downstream".

The default filter for degree of dependencies is going to be 1 to make load time quick on this tab. They can always change that filter. I've also set that filter at the top of the filters since it will be most important for this tab (I don't believe we show "Degree of Dependencies" elsewhere do we?

This meant doing a filter before query instead of after, so that we can be quicker about things. Therefore we now have a nullable param maxHops that we pass through the backend which will be 1 on load (instead of the default 1000 we have now).

The new look:

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

github-actions · 2022-06-08T15:15:16Z

Unit Test Results (build & test)

382 tests ±0 382 ✔️ ±0 8m 59s ⏱️ -27s
  89 suites ±0     0 💤 ±0
  89 files ±0     0 ❌ ±0

Results for commit 3ded253. ± Comparison against base commit e2f1da00.

♻️ This comment has been updated with latest results.

chriscollins3456 · 2022-06-15T15:01:56Z

datahub-web-react/src/app/entity/EntityRegistry.tsx

@@ -128,6 +128,7 @@ export default class EntityRegistry {
                        entity: relationship.entity as EntityInterface,
                        type: (relationship.entity as EntityInterface).type,
                    })),
+                numDownstreamChildren: genericEntityProperties?.downstream?.total,


this may actually cause a bug.. right now we show the number of upstream and downstream on an entity by getting the length of upstreamChildren and downstreamChildren - however these filter out some things to determine if the entity exists. Now if we just show total then we will probably start saying there are more upstream and downstream children than there actually show be.

Any suggestions on how to make this work in the UI? the point of using total is so that we don't have to query for the actual relationships and entities on entity page load, but just get the number.

yes let's just show the total. I think that makes more sense than showing 100+

gabe-lyons · 2022-06-17T15:42:46Z

datahub-web-react/src/app/entity/shared/containers/profile/nav/EntityProfileNavBar.tsx

+                upstreams={lineage?.numUpstreamChildren || 0}
+                downstreams={lineage?.numDownstreamChildren || 0}


gabe-lyons · 2022-06-17T15:43:47Z

datahub-web-react/src/app/lineage/LineageEntityNode.tsx

@@ -90,12 +91,17 @@ export default function LineageEntityNode({
    const [isExpanding, setIsExpanding] = useState(false);
    const [expandHover, setExpandHover] = useState(false);
    const { getAsyncEntity, asyncData } = useLazyGetEntityQuery();


can we not just replace this line with const [getAsyncEntityLineage, { data: asyncLineageData }] = useGetEntityLineageLazyQuery();? Why do we need both?

well we need both right now because the getAsyncLineage query only gets lineage data and we need this query to get all the other info about entities (like name, platform, etc.) - however I could update the lineage query to get everything we need and then we only have the one query.

Will do now

gabe-lyons

Looks good although I was hoping this change would let us remove useLazyGetEntityQuery entirely... is that not the case?

gabe-lyons · 2022-06-17T15:46:41Z

datahub-web-react/src/app/lineage/LineageExplorer.tsx

+        loading: lineageLoading,
+        error: lineageError,
+        data: lineageData,
+    } = useGetEntityLineageQuery({ variables: { urn } });
    const { loading, error, data } = useGetEntityQuery(urn, type);


same here can we get rid of useGetEntityQuery?

gabe-lyons

nice looks good!

jjoyce0510 · 2022-06-21T14:23:38Z

datahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/GmsGraphQLEngine.java

@@ -633,9 +635,22 @@ private void configureQueryResolvers(final RuntimeWiring.Builder builder) {
            .dataFetcher("getRootGlossaryNodes", new GetRootGlossaryNodesResolver(this.entityClient))
            .dataFetcher("entityExists", new EntityExistsResolver(this.entityService))
            .dataFetcher("getNativeUserInviteToken", new GetNativeUserInviteTokenResolver(this.nativeUserService))
+            .dataFetcher("entity", getEntityResolver())


jjoyce0510 · 2022-06-21T14:27:30Z

datahub-web-react/src/app/lineage/utils/useGetEntityQuery.ts

-import { useGetMlModelQuery } from '../../../graphql/mlModel.generated';
-import { useGetMlModelGroupQuery } from '../../../graphql/mlModelGroup.generated';
-
-export default function useGetEntityQuery(urn: string, entityType?: EntityType) {


Oh thank god

jjoyce0510 · 2022-06-21T14:27:45Z

datahub-web-react/src/app/lineage/utils/useLazyGetEntityQuery.ts

-    const [getAsyncMlModel, { data: asyncMlModel }] = useGetMlModelLazyQuery();
-    const [getAsyncMlModelGroup, { data: asyncMlModelGroup }] = useGetMlModelGroupLazyQuery();
-
-    const getAsyncEntity = useCallback(


Oh thank the lord

jjoyce0510 · 2022-06-21T14:27:51Z

datahub-web-react/src/app/search/SearchFilters.tsx

@@ -4,7 +4,7 @@ import { useEffect, useState } from 'react';
 import { FacetMetadata } from '../../types.generated';
 import { SearchFilter } from './SearchFilter';

-const TOP_FILTERS = ['entity', 'tags', 'glossaryTerms', 'domains', 'owners'];
+const TOP_FILTERS = ['degree', 'entity', 'tags', 'glossaryTerms', 'domains', 'owners'];


jjoyce0510

Awesome PR, @chriscollins3456 !

…-project#5121)

chriscollins3456 requested review from dexter-mh-lee, jjoyce0510 and gabe-lyons June 8, 2022 15:07

chriscollins3456 commented Jun 15, 2022

View reviewed changes

gabe-lyons reviewed Jun 17, 2022

View reviewed changes

gabe-lyons approved these changes Jun 17, 2022

View reviewed changes

Chris Collins and others added 7 commits June 17, 2022 17:52

feat(lineage) Update Lineage tab and Impact Analysis feature

05548e2

fix broken tests

6671395

fix broken cypress test

b3d3e7e

Prevent fetching entities in upstream/downstream gql fields on entities

d7628ca

create generic entity resolver on the backend

69db97c

update the frontend to use generic get entity query to get lineage data

d42ad2d

remove two extra queries and their associated files

3ded253

chriscollins3456 force-pushed the cc--public-update-lineage-tab branch from 33d5ad7 to 3ded253 Compare June 17, 2022 21:54

jjoyce0510 reviewed Jun 21, 2022

View reviewed changes

jjoyce0510 approved these changes Jun 21, 2022

View reviewed changes

jjoyce0510 merged commit 2841f32 into datahub-project:master Jun 21, 2022

shirshanka pushed a commit to shirshanka/datahub that referenced this pull request Jun 22, 2022

feat(lineage) Update Lineage tab and Impact Analysis feature (datahub…

de68b20

…-project#5121)

alexey-kravtsov pushed a commit to infobip/datahub that referenced this pull request Jul 8, 2022

feat(lineage) Update Lineage tab and Impact Analysis feature (datahub…

4b9a3b4

…-project#5121)

maggiehays pushed a commit to maggiehays/datahub that referenced this pull request Aug 1, 2022

feat(lineage) Update Lineage tab and Impact Analysis feature (datahub…

abfb97b

…-project#5121)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(lineage) Update Lineage tab and Impact Analysis feature #5121

feat(lineage) Update Lineage tab and Impact Analysis feature #5121

chriscollins3456 commented Jun 8, 2022

github-actions bot commented Jun 8, 2022 •

edited

Loading

chriscollins3456 Jun 15, 2022

gabe-lyons Jun 17, 2022

gabe-lyons Jun 17, 2022

gabe-lyons Jun 17, 2022

chriscollins3456 Jun 17, 2022

gabe-lyons left a comment

gabe-lyons Jun 17, 2022

gabe-lyons left a comment

jjoyce0510 Jun 21, 2022

jjoyce0510 Jun 21, 2022

jjoyce0510 Jun 21, 2022

jjoyce0510 Jun 21, 2022

jjoyce0510 left a comment

		upstreams={lineage?.numUpstreamChildren \|\| 0}
		downstreams={lineage?.numDownstreamChildren \|\| 0}

feat(lineage) Update Lineage tab and Impact Analysis feature #5121

feat(lineage) Update Lineage tab and Impact Analysis feature #5121

Conversation

chriscollins3456 commented Jun 8, 2022

Checklist

github-actions bot commented Jun 8, 2022 • edited Loading

Unit Test Results (build & test)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gabe-lyons left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gabe-lyons left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jjoyce0510 left a comment

Choose a reason for hiding this comment

github-actions bot commented Jun 8, 2022 •

edited

Loading