feat(aspects): support fetching of versioned aspects #2677

gabe-lyons · 2021-06-11T00:16:33Z

Adds support for fetching a versioned aspect. This is done by:

Adding a new method in EntityService

public abstract AspectWithMetadata getAspectWithMetadata(
  @Nonnull final Urn urn,
  @Nonnull final String aspectName,
  long version
);

Adding a new resource AspectResource that fetches these
Adding a type + resolver AspectType / AspectResolver to fetch these and map them into graphql types.

For now, the SchemaMetadata aspect is the only one to be onboarded. We will onboard more types as needed.

One requirement of using this system is that the name of the field aligns with the aspect types's name in the @aspect annotation. Without that, the field cannot properly be resolved.

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable)

jjoyce0510 · 2021-06-14T02:13:46Z

datahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/AspectLoadKey.java

+import lombok.Data;
+
+@Data
+public class AspectLoadKey {


Can we consider an alternate name:

VersionedAspectKey?

jjoyce0510 · 2021-06-14T02:14:49Z

datahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/GmsGraphQLEngine.java

@@ -196,6 +202,7 @@ public static void configureRuntimeWiring(final RuntimeWiring.Builder builder) {
        return GraphQLEngine.builder()
                .addSchema(schema())
                .addDataLoaders(loaderSuppliers(LOADABLE_TYPES))
+                .addDataLoader("Aspect", (context) -> createAspectLoader(context))


Interesting. We'll have to keep an eye on this. Adding an aspect specific fetcher kind of breaks the existing abstractions huh?

jjoyce0510 · 2021-06-14T02:15:23Z

datahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/GmsGraphQLEngine.java

@@ -293,6 +300,9 @@ private static void configureDatasetResolvers(final RuntimeWiring.Builder builde
                                    UPSTREAM_LINEAGE_TYPE,
                                    (env) -> ((Entity) env.getSource()).getUrn()))
                    )
+                    .dataFetcher("schemaMetadata", new AuthenticatedResolver<>(


So I guess for every aspect we want to support versioned aspect fetching on we'll need this.... Okay

For now, yes. However, when we autogenerate the graph we can also autogenerate these.

jjoyce0510 · 2021-06-14T02:17:08Z

datahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/resolvers/ResolverUtils.java



 public class ResolverUtils {

    private static final ObjectMapper MAPPER = new ObjectMapper();

+    private static final Logger _logger = LoggerFactory.getLogger("ResolverUtils");


Nitpick: You can say ResolverUtils.class.getName() here

jjoyce0510 · 2021-06-14T02:18:47Z

datahub-graphql-core/src/main/java/com/linkedin/datahub/graphql/resolvers/ResolverUtils.java

+        Long version = environment.getArgument("version");
+
+        Object localContext = environment.getLocalContext();
+        // if we have context & the version is 0, we should try to retrieve it from the fetched entity


Interesting ... Do we need this is we are only using for schemaMetadata case, where we are sure we need a specific version?

Not sure I understand this question- this resolver is for all versioned aspect fetchers. If someone fetches version 0, we don't actually need to use the aspect resource since version 0 has already been fetched earlier by the Entity type's batchGet.

Hmmm. Does it make sense to have a versioned aspect fetcher everywhere? I guess that's my main question

jjoyce0510 · 2021-06-14T02:20:26Z

...b-graphql-core/src/main/java/com/linkedin/datahub/graphql/resolvers/load/AspectResolver.java

+
+        String fieldName = environment.getField().getName();
+        Long version = environment.getArgument("version");
+        String urn = ((Entity) environment.getSource()).getUrn();


Will this always be invoked in the context of a broader entity fetch? I guess it must?

We could also have a dedicated "getVersionedAspect" method that requires version, urn, etc to do this

getVersionedAspect is something we could consider- although the type safety of that will be a bit off- the return type will need to be Aspect and we will always need to type check it on the client.

Yes - similar to our entity resource

jjoyce0510 · 2021-06-14T02:20:56Z

...b-graphql-core/src/main/java/com/linkedin/datahub/graphql/resolvers/load/AspectResolver.java

+        String urn = ((Entity) environment.getSource()).getUrn();
+
+        // first, we try fetching the aspect from the local cache
+        AspectWithMetadata aspectFromContext = ResolverUtils.getAspectFromLocalContext(environment);


We are positive this results in no downstream calls in the common case?

jjoyce0510 · 2021-06-14T02:21:38Z

...e/src/main/java/com/linkedin/datahub/graphql/resolvers/type/AspectInterfaceTypeResolver.java

+      // TODO(Gabe): Fill this out. This method is not called today. We will need to fill this
+      // out in the case we ever want to return fields of type Aspect in graphql. Right now
+      // we just use Aspect to define the shared `version` field.
+      return null;


Can we fit this out with something in this PR to set the stage for future extension?

I'm not sure- there are a few ways this could be implemented. Since this method is never called at the moment, I'd rather wait to implement it when we are more clear on requirements.

jjoyce0510 · 2021-06-14T02:24:10Z

datahub-graphql-core/src/main/resources/gms.graphql

+    """
+    Schema metadata of the dataset
+    """
+    schemaMetadata(version: Long): SchemaMetadata


ty for making this consistent. i should have done so in the first place.

btw. what if we want to know the available versions? do we need to support that eventually ?

We can discuss this later- right now if you fetch -1 you get all available versions by checking the version of the result we get back from fetch -1. When we need a more explicit way to fetch this value we can look into it then.

very scary. caller need to know about -1 stuff

Sure, as more use cases arise, lets use those to understand the needs of fetching all version count. We can have another method to do this, or include max version in every versioned aspect.

jjoyce0510 · 2021-06-14T02:26:07Z

metadata-models/src/main/pegasus/com/linkedin/metadata/aspect/AspectWithMetadata.pdl

+
+import com.linkedin.common.AuditStamp
+
+record AspectWithMetadata {


Can we call this "VersionedAspect"?

I prefer names without things like "with" or "and"

VersionedAspect is a bit too specific- we may want to add more metadata later. I can think about a different name

I'm not so sure - I think versioned aspect conveys much more detail and I'm not sure what other metadata we'll want here

fyi renamed to VersionedAspect :)

jjoyce0510 · 2021-06-14T02:26:15Z

metadata-models/src/main/pegasus/com/linkedin/metadata/aspect/Aspect.pdl

+/**
+ * A union of all supported metadata aspects for a Chart
+ */
+typeref Aspect = union[


Thanks for doing this.

jjoyce0510 · 2021-06-15T00:35:53Z

datahub-graphql-core/src/main/resources/gms.graphql

@@ -516,6 +525,53 @@ type InstitutionalMemoryMetadata {
    created: AuditStamp!
 }

+type SchemaMetadata implements Aspect {


Should the base "aspect" have a version? Probably with a default as 0?

jjoyce0510

Overall this is looking good. Really want to start thinking about how we can make this more generalizable as well as support a client understanding of the number of versions of an aspect.

It'd be useful to have an API that allows you to fetch a lsit of aspects of a given type that is paginated.

gabe-lyons · 2021-06-15T00:43:39Z

@jjoyce0510 - I agree on the list of aspects. When a use case arises for that, it should be relatively straightforward to add.

Re: more generalizable, I think the solution is dynamically generating the graph and the data resolvers. Then, each aspect will automatically be wired up like the schemaMetadata field is. This way we will get the ability to fetch a versioned aspect of any aspect.

shirshanka

LGTM. Thanks for addressing the comments @gabe-lyons.

amorskoy · 2021-06-23T08:30:13Z

metadata-io/src/main/java/com/linkedin/metadata/entity/ebean/EbeanEntityService.java

@@ -92,13 +94,47 @@ public EbeanEntityService(
  @Override
  @Nullable
  public RecordTemplate getAspect(@Nonnull final Urn urn, @Nonnull final String aspectName, @Nonnull long version) {
+    if (version < 0) {
+      version = _entityDao.getMaxVersion(urn.toString(), aspectName) - version + 1;


Let's suppose: passed arg version is "-1", getMaxVersion returns 2. In this case line 98 assigns:
version = 2 - (-1) + 1 = 4

Is it correct @shirshanka ?

Thanks for catching @amorskoy - this was fixed in #2723

amorskoy · 2021-06-23T08:41:30Z

metadata-io/src/main/java/com/linkedin/metadata/entity/ebean/EbeanEntityService.java

+    VersionedAspect result = new VersionedAspect();
+
+    if (version < 0) {
+      version = _entityDao.getMaxVersion(urn.toString(), aspectName) + version + 1;


Please compare + version here with - version here https://github.com/linkedin/datahub/pull/2677/files#r656876786 @shirshanka

* fix(react): update ispartofbuilderfromdataflow, update ui in datajob header (datahub-project#2619) * fix(ingestion): improve robustness of glue ingestion source (datahub-project#2626) fixes: datahub-project#2625 Co-authored-by: thomas.larsson <[email protected]> * feat(react): custom properties are now sortable by name in the UI (datahub-project#2622) * fix(ci): update trigger to always generate docker images (datahub-project#2636) * fix(ingest): include urn as key for kafka emitter (datahub-project#2634) * fix(react): url encoding urns and tag profile fix (datahub-project#2623) * feat(react): replace user urn with username (datahub-project#2599) Co-authored-by: shubham.garg <[email protected]> * fix(ingest): improve redshift ingestion performance (datahub-project#2635) * docs: update roadmap with accomplished items (datahub-project#2617) * feat: No Code Metadata Modeling (datahub-project#2629) Co-authored-by: Dexter Lee <[email protected]> Co-authored-by: Gabe Lyons <[email protected]> Co-authored-by: Shirshanka Das <[email protected]> * docs(no-code): removing unused snapshot annotations (datahub-project#2642) * feat(datahub-upgrade): Use the "head" image tag. (datahub-project#2641) * fix(datahub-upgrade): remove debug flag (datahub-project#2640) * fix(build): don't purge ingestion codegen files on gradle clean (datahub-project#2645) * fix(analytics): fix index names (datahub-project#2647) * fix(nocode): Fix docker image tag for run_upgrade script (datahub-project#2648) * docs(nocode): Update migration docs with workaround for stuck upgrade(datahub-project#2649) * fix: use head tag by default in quickstart (datahub-project#2650) * fix(datahub-upgrade): fixing mis-spelled schema registry url (datahub-project#2652) * fix(gms): Return empty snapshots when one does not exist (datahub-project#2646) * docs(nocode): fix links to datahub-upgrade (datahub-project#2651) * feat(datahub-upgrade): improve no code upgrade logging (datahub-project#2653) * fix(docker): pin containers to golden hash for release (datahub-project#2654) * feat(nocode): Add datahub-upgrade job to helm chart and set version to v0.8.1 (datahub-project#2643) * docs(nocode): Adding documentation for no-migration upgrade option (datahub-project#2656) * revert: "fix(docker): pin containers to golden hash for release (datahub-project#2654)" (datahub-project#2659) This reverts commit a483933 and moves us back to using HEAD in quickstart on the master branch. * fix(docker): use debug tag in local dev images (datahub-project#2658) * fix(ingest): support mssql encryption via ODBC (datahub-project#2657) * fix(NoCode): Update snapshot json to latest (datahub-project#2655) * fix(ingest): exclude mssql-odbc from "all" extra (datahub-project#2660) * docs(nocode): adding documentation to handle backwards incompatibility issues (datahub-project#2662) * docs(nocode): fix typo in autocomplete annotation (datahub-project#2664) * build(docker): test docker builds in pull request CI (datahub-project#2661) * fix(ingest): fix MyPy stubs (datahub-project#2666) * feat(ingest): Feast ingestion integration (datahub-project#2605) * Add feast testing setup * Init Feast test script * Add feast to dependencies * Update feast descriptors * Sort integrations * Working feast pytest * Clean up feast docker-compose file * Expand Feast tests * Setup feast classes * Add continuous and bytes data to feature types * Update field type mapping * Add PDLs * Add MLFeatureSetUrn.java * Comment out feast setup * Add snapshot file and update inits * Init Feast golden files generation * Clean up Feast ingest * Feast testing comments * Yield Feature snapshots * Fix Feature URN naming * Update feast MCE * Update Feature URN prefix * Add MLEntity * Update golden files with entities * Specify feast sources * Add feast source configs * Working feast docker ingestion * List entities and features before adding tables * Add featureset names * Remove unused * Rename feast image * Update README * Add env to feast URNs * Fix URN naming * Remove redundant URN names * Fix enum backcompatibility * Move feast testing to docker * Move URN generators to mce_builder * Add source for features * Switch TypeClass -> enum_type * Rename source -> sourceDataset * Add local Feast ingest image builds * Rename Entity -> MLPrimaryKey * Restore features and keys for each featureset * Do not json encode source configs * Remove old source properties from feature sets * Regenerate golden file * Fix race condition with Feast tests * Exclude unknown source * Update feature datatype enum * Update README and fix typos * Fix Entity typo * Fix path to local docker image * Specify feast config and version * Fix feast env variables * PR fixes * Refactor feast ingest constants * Make feature sources optional for back-compatibility * Remove unused GCP files * adding docker publish workflow * Simplify name+namespace in PrimaryKeys * adding docker publish workflow * debug * final attempt * final final attempt * final final final commit * Switch to published ingestion image * Update name and namespace in java files * Rename FeatureSet -> FeatureTable * Regenerate codegen * Fix initial generation errors * Update snapshot jsons * Regenerated schemas * Fix URN formats * Revise builds * Clean up feast URN builders * Fix naming typos * Fix Feature Set -> Feature Table * Fix comments * PR fixes * All you need is Urn * Regenerate snapshots and update validation * Add UNKNOWN data type * URNs for source types * Add note on docker requirement * Fix typo * Reorder aspect unions * Refactor feast ingest functions * Update snapshot jsons * Rebuild Co-authored-by: Shirshanka Das <[email protected]> * fix(no-code): Adding Chart input relationship annotations (datahub-project#2669) * chart input relationship pdl fix * commiting schema.avsc changes * Add get_identifier to hive source in metadata ingestion (datahub-project#2667) * feat(k8s): Add imagePullSecrets to all K8's jobs (datahub-project#2671) * fix(analytics): Support sasl authentication to kafka (datahub-project#2675) * feat(ingest): headers for codegen Python scripts (datahub-project#2637) * fix(ingest): pin to new mypy version (datahub-project#2670) * Fix 2592 neo4j connection options (datahub-project#2632) * fix(ingest): upgrade acryl-pyhive to use sasl3 instead of sasl (datahub-project#2684) * feat(ingest): support Oracle service names (datahub-project#2676) * feat(sql_views): added views as datasets for SQLAlchemy DBs (datahub-project#2663) * removing whitespace from service aspect (datahub-project#2688) * Only work with dbt catalog data if load_catalog is False (datahub-project#2686) * feat(docs): Docs for S3 ingestion with AWS Glue (datahub-project#2672) * feat(datahub cli): DataHub CLI Quickstart (datahub-project#2689) * feat(gms): Merge MAE, MCE consumers into GMS (datahub-project#2690) * fix(NoCode): Fix product analytics (datahub-project#2697) * feat(gms): support basic auth header when connecting to elasticSearch (datahub-project#2701) * Add support for different oidc client authentication methods (datahub-project#2691) * feat(aspects): support fetching of versioned aspects (datahub-project#2677) * feat(entities): add markdown description update/viewer feature in dataset, datajob, dataflow, chart and dashboard, update ui/ux (datahub-project#2707) * feat(ingest): Add test case and docs for SQL view ingestion (datahub-project#2709) * fix(ingest): use `looker` data platform (datahub-project#2708) * test(ingest): simplify docker cleanup commands (datahub-project#2699) * feat: Adding annotation option to frontend service (datahub-project#2674) * feat(ingest): expose additional types to Python via codegen (datahub-project#2712) * Removed key not in catalog filter from dbt and added node filter using AllowDenyPattern (datahub-project#2702) * added node_type_pattern in dbt yaml file (datahub-project#2705) * fix(editable descriptions): adding indexing for editable descriptions (datahub-project#2710) * feat(quickstart): use versioned docker images when on a release tag (datahub-project#2715) * fix(noCode): Improving efficiency of EntityService "listLatestAspects" API (datahub-project#2711) * fix(react): update schema description edit behavior (datahub-project#2718) * Update version for helm (datahub-project#2719) * fixing docs sidebar (datahub-project#2721) * fix(docs): update extending-the-metadata-model.md (datahub-project#2727) * fix(docker): update default tags to head (datahub-project#2730) * fix(looker): fix invalid URN syntax error (datahub-project#2737) * fix(ci): increase Feast docker setup timeout (datahub-project#2722) * fix(ingest): types for dbt (datahub-project#2716) * feat(ingest): add support for Glue ETL jobs (datahub-project#2687) * fix(ingest): fix lookml platform URN (datahub-project#2742) * fix(ingest): update lookml test (datahub-project#2741) * fix(gms): fixes for version aspect fetching (datahub-project#2723) * feat(graph): support using elasticsearch as graph backend. (datahub-project#2726) * feat(ingest): print docker logs on timeout (datahub-project#2743) * fix(ci): increase wait-for-it timeout to fix flaky feast test (datahub-project#2747) * feat(docker): reduce quickstart footprint (datahub-project#2744) * feat(quickstart): remove orphaned docker containers on quickstart through cli (datahub-project#2748) * fix(gms): add rest.li validation in gms (datahub-project#2745) * fix(datahub-upgrade): add support for postgres migration (datahub-project#2740) * fix(docker): modernize docker images and fix vulnerabilities (datahub-project#2746) * feat(ingest): ingest last-modified from dbt sources.json (datahub-project#2729) * feat(ingest): add option to specify source platform database in lookml ingestion (datahub-project#2749) * fix(gms): make get return entity type (datahub-project#2739) * fix(elastic-as-graph): adding elasticsearch setup back in (datahub-project#2752) * fix(browse): sort by doc count descending (datahub-project#2751) * Revert "fix(gms): add rest.li validation in gms (datahub-project#2745)" (datahub-project#2754) This reverts commit f0b4adc. * fix(docker): use head tag for datahub-ingestion (datahub-project#2760) * feat(elastic-as-graph): defaulting to elastic in quickstart (datahub-project#2753) * fix(react): reverse result of topological sort, autofocus in add tag modal (datahub-project#2759) * feat: usage stats (part 1) (datahub-project#2750) Co-authored-by: Gabe Lyons <[email protected]> * feat: usage stats (part 2) (datahub-project#2762) Co-authored-by: Gabe Lyons <[email protected]> * docs: update for Jun townhall (datahub-project#2764) * fix(frontend): auth session ttl specified in hours instead of days. M… (datahub-project#2695) Fixes: datahub-project#2680 Co-authored-by: thomas.larsson <[email protected]> * fix(react): move percent sign after number and update meta tag (datahub-project#2765) * fix(docker): Fix dependency vulnerability (datahub-project#2763) * fix(datahub-upgrade): removing the CleanupStep from datahub upgrade (datahub-project#2728) * docs(ingest): move usage stats docs into the "sources" section (datahub-project#2766) * fix(react): update the query frequency text label (datahub-project#2767) * feat(k8s): add GCP deploy recipe (datahub-project#2768) * fix(react): fix graphql apollo cache update issue cause of usagestats (datahub-project#2769) * feat(logs): improve logging in GMS and datahub-frontend (datahub-project#2761) * fix(nocode): removing service PDL (datahub-project#2772) * fix(react): update platform text in dataset profile header (datahub-project#2771) * feat(logs): add thresholding, misc cleanup (datahub-project#2773) * docs: upgrade docusaurus, minor ingestion updates (datahub-project#2774) * feat(ingest): add non-random sampling for mongo (datahub-project#2778) * feat(react-graphql-mock): mock graphql queries and mutations with miragejs * feat(react): configure Cypress + MirageJS + GraphQL mock for functional testing plus a couple of example tests * fix(react): fix mutation helper for updateentitylink * feat(react): update graphql issues with mock data * fix(docker): Upgrade to 4.1.44 netty-all library (datahub-project#2784) * refactor(ingest): use common get_sys_time method (datahub-project#2782) * fix(docs): links to Feast entities (datahub-project#2780) * feat(k8s): Upgrade to v0.8.4 (datahub-project#2792) * fix(frontend): making nested SSL configs optional (datahub-project#2785) * fix(ingest): use correct platform for MongoDB ingestion (datahub-project#2783) * Update quickstart to include system prune (datahub-project#2794) * docs(website): add releases page (datahub-project#2776) * refactor(ingest): remove deprecated methods and warn on deprecated import (datahub-project#2797) * fix(usage-stats): add indices target for usage stats query (datahub-project#2798) * fix(ingest): handle case when view definition handler is not implemented (datahub-project#2796) * fix(ingest): quote table names in hive (datahub-project#2801) * fix(datahub-upgrade): add runtime dependency on logbackClassic (datahub-project#2799) * fix(search): Filter out "removed" entities from autocomplete and analytics (datahub-project#2781) * feat(ingest): Improve lookml sql derived tables detection, add cascading derived tables to lineage (datahub-project#2770) * feat(ingest): SageMaker feature store ingestion (datahub-project#2758) * fix(ingest): better warnings and error handling for rest sink (datahub-project#2800) * fix(usage-stats): Add usage stats factory to mae processor config (datahub-project#2803) * refactor(ingest): extract sqlalchemy uri generation logic (datahub-project#2786) * fix(quickstart): Fixing manual mysql quickstart (datahub-project#2811) * feat(ingest): support ingesting from multiple snowflake dbs (datahub-project#2793) * feat(backup): Add restore indices and restore backup tasks (datahub-project#2779) * docs(website): hide outdated FAQs page (datahub-project#2809) * feat(ingest): refactor mce comparison and add pytest update golden files option (datahub-project#2812) * fix(k8s): upgrade helm version (datahub-project#2810) * feat(ingest): basic support for complex hive types (datahub-project#2804) * fix(datahub-upgrade): fix vulnerabilities (datahub-project#2813) * chore(helm): upgrade version to v0.8.5 (datahub-project#2814) * feat(datahub-frontend): Adding basic file-based authentication to datahub-frontend (datahub-project#2818) * fix(ingest): view handling resilience for redshift (datahub-project#2816) * docs(ingest): add extra info for Redshift behind a proxy (datahub-project#2817) * fix(docs): update OIDC docs (datahub-project#2823) * docs(docker): fix check command reference (datahub-project#2822) Fixes datahub-project#2820. * docs(elastic-for-graph): Add migrating from neo4j to elastic instructions (datahub-project#2826) * fix(ingest): convert superset timestamps to micros (datahub-project#2827) * Previous values were in seconds, which rendered incorrectly in the UI. * feat(ci): separate metadata-ingestion into a separate workflow (datahub-project#2828) * fix(cli): change docker nuke to also remove stopped containers (datahub-project#2825) * fix(mae-consumer): support standalone mae consumer without neo4j (datahub-project#2824) * docs: update H2 2021 roadmap (datahub-project#2829) * docs(ingest): update links to Kafka docs (datahub-project#2834) * fix(ingest): mask password in info-level logs (datahub-project#2835) * fix(ingest): various BigQuery source fixes (datahub-project#2836) * docs(ingest): clarify that the Kafka options are pass-through (datahub-project#2837) * fix(ingest): do not fail dbt ingestion when encountering missing nodes (datahub-project#2833) * feat (helm datahub-gms): add ingress template to datahub-gms helm chart (datahub-project#2832) Co-authored-by: christopher.coulson <[email protected]> * feat(react): fix apollo client cache issues in entities profile pages (datahub-project#2841) * fix(ingest): avoid setting timestamps unless source system provides it (datahub-project#2843) * fix(docs): fix some broken links in the docs (datahub-project#2846) * fix(ingest): Fix glob pattern and handle possible recursion in lookml (datahub-project#2851) * feat(ingest): prettify stack traces in CLI (datahub-project#2845) * fix(analytics): Fix SSL issue with analytics on frontend (datahub-project#2840) * fix(ingest): check for dbt materialization before proceeding (datahub-project#2842) * feat(docs): throttle and retry requests in doc generation (datahub-project#2847) * feat(ingest): SageMaker jobs and models (datahub-project#2830) * docs(ingest): remove hanging sentence from docs (datahub-project#2853) * fix(ingest): delete pycache files when running clean (datahub-project#2852) * fix(ingest): handle 'fields' list missing in bigquery-usage (datahub-project#2844) * feat(k8s): Extract helm charts into a separate repo (datahub-project#2848) * fix(react): fix searchbar width issue in small screen (datahub-project#2856) * feat(react): implement visualizing historical versions of dataset schema (datahub-project#2855) * feat(docs): carousels for videos and articles (datahub-project#2857) * fix(frontend): handling null aspects in AspectType (datahub-project#2860) * fix: fixing lint issue (datahub-project#2861) * feat(ingest): support dynamic imports for transfomer methods (datahub-project#2858) * feat(docs): swap Medium and videos sections (datahub-project#2859) * feat(ingest): add browse paths + dataplatform for Feast features (datahub-project#2849) * fix(build): increase retries for dependency fetches (datahub-project#2867) * build(ingest): separate metadata-ingestion build workflow fully (datahub-project#2862) * fix(quickstart): update compose spec version (datahub-project#2873) Also update the defaults to point at the head tag instead of latest. * docs(quickstart): add default password to quickstart (datahub-project#2875) * feat(ingest): extract dbt meta fields (datahub-project#2876) * feat(ingest): update golden files only when diff fails (datahub-project#2869) * build(ingestion): add version prompt to release script (datahub-project#2866) * fix(react): fix bug in description update modal (datahub-project#2874) * fix(react): update dataset graphql mock Co-authored-by: Thomas Larsson <[email protected]> Co-authored-by: thomas.larsson <[email protected]> Co-authored-by: Rickard Cardell <[email protected]> Co-authored-by: Dexter Lee <[email protected]> Co-authored-by: Harshal Sheth <[email protected]> Co-authored-by: Gabe Lyons <[email protected]> Co-authored-by: shubham garg <[email protected]> Co-authored-by: shubham.garg <[email protected]> Co-authored-by: Shirshanka Das <[email protected]> Co-authored-by: John Joyce <[email protected]> Co-authored-by: Kevin Hu <[email protected]> Co-authored-by: zack3241 <[email protected]> Co-authored-by: Vincenzo Lavorini <[email protected]> Co-authored-by: Remi <[email protected]> Co-authored-by: JeffSkinner <[email protected]> Co-authored-by: Peter Mortier <[email protected]> Co-authored-by: kuntalkumarbasu <[email protected]> Co-authored-by: vijayan-nallasami-curve <[email protected]> Co-authored-by: wan <[email protected]> Co-authored-by: datascienceChris <[email protected]> Co-authored-by: christopher.coulson <[email protected]> Co-authored-by: Fredrik Sannholm <[email protected]>

gabe-lyons added 7 commits June 7, 2021 11:14

versioned aspects

05093e5

versioned aspects cont

b071987

checkpointing here w/ everything being functional

fd831aa

small fixes & adding tests

ddc1bcb

fixing tests

3a83716

fixing snapshot conflicts

f410866

updating snapshots

e8768a3

jjoyce0510 reviewed Jun 14, 2021

View reviewed changes

renaming aspectloadkey and aspectwithmetadata

5e20b2f

jjoyce0510 reviewed Jun 15, 2021

View reviewed changes

jjoyce0510 approved these changes Jun 15, 2021

View reviewed changes

refactoring to be more clear

9468949

shirshanka approved these changes Jun 16, 2021

View reviewed changes

shirshanka merged commit 523c3bf into datahub-project:master Jun 16, 2021

amorskoy reviewed Jun 23, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(aspects): support fetching of versioned aspects #2677

feat(aspects): support fetching of versioned aspects #2677

gabe-lyons commented Jun 11, 2021

jjoyce0510 Jun 14, 2021

jjoyce0510 Jun 14, 2021

jjoyce0510 Jun 14, 2021

gabe-lyons Jun 14, 2021

jjoyce0510 Jun 14, 2021

jjoyce0510 Jun 14, 2021

gabe-lyons Jun 14, 2021

jjoyce0510 Jun 15, 2021

jjoyce0510 Jun 14, 2021

gabe-lyons Jun 14, 2021

jjoyce0510 Jun 15, 2021

jjoyce0510 Jun 14, 2021

gabe-lyons Jun 14, 2021

jjoyce0510 Jun 14, 2021

gabe-lyons Jun 14, 2021

jjoyce0510 Jun 14, 2021

jjoyce0510 Jun 14, 2021

gabe-lyons Jun 14, 2021

jjoyce0510 Jun 15, 2021

gabe-lyons Jun 15, 2021

jjoyce0510 Jun 14, 2021

jjoyce0510 Jun 14, 2021

gabe-lyons Jun 14, 2021

jjoyce0510 Jun 15, 2021

gabe-lyons Jun 15, 2021

jjoyce0510 Jun 14, 2021

jjoyce0510 Jun 15, 2021

jjoyce0510 left a comment

gabe-lyons commented Jun 15, 2021 •

edited

Loading

shirshanka left a comment

amorskoy Jun 23, 2021

gabe-lyons Jun 23, 2021

amorskoy Jun 23, 2021


		import com.linkedin.common.AuditStamp

		record AspectWithMetadata {

feat(aspects): support fetching of versioned aspects #2677

feat(aspects): support fetching of versioned aspects #2677

Conversation

gabe-lyons commented Jun 11, 2021

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jjoyce0510 left a comment

Choose a reason for hiding this comment

gabe-lyons commented Jun 15, 2021 • edited Loading

shirshanka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gabe-lyons commented Jun 15, 2021 •

edited

Loading