DataHub v0.8.25

Known Issues

• Adding Glossary Terms to schema fields does not work with this version due to a bug. Upgrade to v0.8.26 for the fix.

Release Highlights

Buckle up, folks! v0.8.25 brings some very exciting (and highly-requested!) updates.

Notable UI-Based Features

• UI-based Ingestion - as demoed in December Town Hall, we now support creating, configuring, scheduling, & executing batch metadata ingestion using the DataHub user interface. This makes getting metadata into DataHub easier by minimizing the overhead required to operate custom integration pipelines.
• Data Domains - DataHub now supports grouping data assets into logical collections called Domains. Domains are curated, top-level folders or categories where related assets can be explicitly grouped. Read the guide here!
• Data Containers are now supported! This is the physical grouping of entities, ex. a Schema is a container of 1 or more Datasets; a Dashboard is a container of 1 or more Charts.

Notable Metadata Model & Ingestion-Based Features

• Data Quality test results are now supported in the DataHub metadata model. This is the first milestone toward surfacing Dataset & Column-level Data Quality results in the UI (read full scope of work here). Future releases will include a Great Expectations integration & UI support - we’re on track to complete this in Q1 as planned.
• Avro files are now supported in the Data Lake File ingestion source
• Ingest metadata from multiple instances of the same platform type. This has been a very common use case within the Community - you can now differentiate multiple instances of the same platform type! If you already have pre-existing entries, use the datahub migrate command to migrate them over to platform instances.
• Ignore users from Top Users calculation
  • feat(ingestion): Adding ability to ignore users from top users calculation by @treff7es in #3735
• BigQuery - Data Profiling on only the latest partition/shard
  • feat(ingestion) bigquery: Profiling only the latest partition/shard on bigquery by @treff7es in #3930

• (feat)(Business Glossary) add tabular schema and new UI for business glossary by @saxo-lalrishav in #3813

Notable Fixes

• Fix to support View in Looker * feat(looker): Adding optional Looker external url base url config by @jjoyce0510 in #3985
• fix(graphql): support group display name in ownership by @thomasplarsson in #3979
• fix(profiling): Enabling profiling for low cardinality number columns by @treff7es in #3990
• fix(ingestion): match default username for Azure OIDC and Azure ingestion source by @iasoon in #3926

DataHub Usage Guides

• docs(domains): Adding a User Guide for Domains by @jjoyce0510 in #4038
• docs(ingest): Adding UI ingestion guide by @jjoyce0510 in #4048

What's Changed

• fix(vulnerability): Upgrade gms base image by @dexter-mh-lee in #3962
• logging(frontend): Improve OIDC debug logs by @jjoyce0510 in #3967
• docs(delete): add curl request example to delete entity by @anshbansal in #3928
• fix(ingestion): match default username for Azure OIDC and Azure ingestion source by @iasoon in #3926
• Feature/dynamic platform icons by @RyanHolstien in #3968
• refactor(ingestion): remove duplicate aspect type by @hsheth2 in #3972
• fix(example): fix typo by @anshbansal in #3907
• fix(ingestion): Restrict python to <=3.9.9 by @treff7es in #3961
• feat(build): remove requirement for git directory for builds by @swaroopjagadish in #3977
• fix(ingestion): tighten conditions for restli json transformation by @hsheth2 in #3973
• fix(ingestion): don't dump variables for config errors by @hsheth2 in #3974
• Bugfix/increase socket timeout by @RyanHolstien in #3982
• feat(ingest): support for Avro data lake files by @kevinhu in #3913
• fix(build): exclude old log4j core by @RickardCardell in #3966
• fix(quickstart): Pin Quickstart version to v0.8.23. by @jjoyce0510 in #3983
• feat(looker): Adding optional Looker external url base url config by @jjoyce0510 in #3985
• fix(graphql): support group display name in ownership by @thomasplarsson in #3979
• fix(quickstart): Assign correct mysql-setup container for M1 and remove "head" default version. by @jjoyce0510 in #3987
• feat(embedded search results): support custom endpoints in embedded search result by @gabe-lyons in #3986
• fix(docker): datahub-gms - build in native, copy to target by @swaroopjagadish in #3992
• fix(ci): moving defaults back to head now that docker builds are green by @swaroopjagadish in #3993
• feat(ui): UI-based ingestion (as featured in Dec Townhall) by @jjoyce0510 in #3975
• quickstart: Adding UI ingestion to quickstart YAML by @jjoyce0510 in #3994
• feat(domains): Adding backend for Asset Domains (p1) by @jjoyce0510 in #3952
• Bug: a bug fix to bigquery_to_datahub.yml file by @dipeshmaurya in #3988
• fix(ingest): check if feature data type is present by @maaaikoool in #3932
• feat(platform-instance): a simple client-only change to support platf… by @swaroopjagadish in #3996
• docs(metadata-model): Adding to Metadata model docs by @jjoyce0510 in #3998
• Add Stash Logo & new Source Icons by @maggiehays in #4002
• feat(domains): UI for Asset Domains (p2) by @jjoyce0510 in #3995
• docs: add missing back tick for metadata-ingestion/README.md by @nickwu241 in #4003
• Bugfix/add missing classes by @RyanHolstien in #4000
• fix(superset): fix connection for redshift by @anshbansal in #3944
• fix(setup): fix setup for M1 by @anshbansal in #3958
• docs:add Optum logo by @maggiehays in #4005
• Refining Metadata Model docs further by @jjoyce0510 in #4001
• fix(docker): Alpine based multiplatform docker build for kafka-setup by @treff7es in #3991
• Bugfix/graph concurrency issue by @RyanHolstien in #4007
• feat(ingest): Add additional snowflake auth by @MikeSchlosser16 in #4009
• fix(ci): Reverting unnecessary domain test changes by @jjoyce0510 in #4013
• fix(metrics): Add metrics for mcl hooks by @dexter-mh-lee in #4008
• feat(platform) - Update FabricType enum to represent more fabrics by @aditya-radhakrishnan in #3997
• feat(ingest): emit flags and stats for profiling telemetry by @kevinhu in #3969
• fix(formatting): fix linting lib version requirement by @anshbansal in #3939
• fix(docs): fix business glossary docs by @anshbansal in #3916
• fix(profiling): Enabling profiling for low cardinality number columns by @treff7es in #3990
• fix(docs): update gms link by @lhvubtqn in #3927
• fix(ingest): lint fix a few files by @swaroopjagadish in #4016
• fix(ingest): adding platform instance urn to data platform instance aspects by @swaroopjagadish in #4015
• feat(ingest): use trino python client for sqlalchemy, supports python… by @mayurinehate in #3888
• fix(spark-lineage): select mock server port dynamically for unit test by @MugdhaHardikar-GSLab in #4018
• (feat)(Business Glossary) add tabular schema and new UI for business glossary by @saxo-lalrishav in #3813
• Test/add concurrency issue smoke test by @RyanHolstien in #4014
• feat(glossary-terms): Index glossary term custom properties by @jjoyce0510 in #3960
• feat(ingestion): Adding ability to ignore users from top users calculation by @treff7es in #3735
• Docs/remote deploy and auto render by @RyanHolstien in #4020
• fix(ingest): snowflake - Run authentication validation if default value used by @treff7es in #4024
• feat(nifi): handle provenance api variation for older versions by @mayurinehate in #4022
• feat(ingestion) bigquery: Profiling only the latest partition/shard on bigquery by @treff7es in #3930
• fix(groups): Fix UI encoding of groups with spaces in urns by @jjoyce0510 in #4021
• fix(text): fix confusing text by @anshbansal in #4025
• fix(clean): add missing cleanup by @anshbansal in #4023
• feat(containers): Backend for Asset Containers (as demo'd in townhall) by @jjoyce0510 in #4019
• fix(docs): Adding Initiate login uri to okta docs (Okta OIDC) by @jjoyce0510 in #4030
• fix: docker-compose now persists kafka broker data by @icy in #4031
• feat(ingestion): Support Kafka confluent external schema resolution by name or subject by @rslanka in #4035
• docs(domains): Adding a User Guide for Domains by @jjoyce0510 in #4038
• feat(Stateful Ingestion-3/3): Client side changes for Monitoring/Reporting by @rslanka in #3807
• feat(containers): Adding Containers UI (as demo'd in Jan Townhall) by @jjoyce0510 in #4037
• feat(users): adding user graphql mutation by @gabe-lyons in #4033
• feat(ingest): add tests for platform instance by @swaroopjagadish in #4047
• feat(model): Data quality model by @ksrinath in #3787
• Bugfix/prevent invalid urn by @RyanHolstien in #4045
• refactor(spark-lineage): remove dependency of spark from McpEmitter by @MugdhaHardikar-GSLab in #4042
• feat(analytics): add more analytics for entities by @anshbansal in #4040
• docs(ingest): Adding UI ingestion guide by @jjoyce0510 in #4048
• fix(mae-consumer-docker): Fix condition for skipping elasticsearch check by @dexter-mh-lee in #4052
• feat(ci): pin tox requirements to speed up ci runs, remove airflow-1 … by @swaroopjagadish in #4055
• feat(container): Add domains aspect to container. by @jjoyce0510 in #4059
• feat(profile) - bigquery: Fix for hitting limit with too many partitioned tables by @treff7es in #4056
• [Docs] Mark data lake metadata source as Beta by @pedro93 in #4061
• feat(ingest): log CLI invocations and completions by @kevinhu in #4062
• fix(ingest): Add aws dependencies for data lake by @kevinhu in #4060
• fix(ingest) - add aws_common as a snowflake_common dependency by @aditya-radhakrishnan in #4054
• feat(ui): Add svg datahub loading logo by @eburairu in #4065
• refactor(models): Refactoring new Assertion models by @jjoyce0510 in #4064
• feat(cli): add --force option to ingest rollback subcommand by @danilopeixoto in #4032
• fix(analytics): fix missing events from UI by @anshbansal in #4026
• Data domain containers ingestion by @treff7es in #4051
• docs(ingestion) glue: document required IAM permissions by @iasoon in #3929
• fix(profile):bigquery - Check for every table if it is partitioned to not hit table quota by @treff7es in #4074

New Contributors

• @dipeshmaurya made their first contribution in #3988
• @maaaikoool made their first contribution in #3932
• @icy made their first contribution in #4031
• @ksrinath made their first contribution in #3787
• @eburairu made their first contribution in #4065
• @danilopeixoto made their first contribution in #4032

Full Changelog: v0.8.24...v0.8.25