DataHub v0.7.0

Notable Highlights

• New React Application re-written from the ground up
• Support for GraphQL
• New Metadata Ingestion Framework (Python)
  • Officially-supported Sources: Kafka, MySQL, SQL Server, Hive, Postgres, Snowflake, BigQuery, AWS Athena, Druid, LDAP
• New Homepage and Hosted Docs redesign at datahubproject.io
• Product Features: SSO (OIDC), Tags, Themes, Dashboards
• Metadata Backend Implementations: MLModel ecosystem, DataFlow ecosystem
• Move to Elasticsearch 7. Migration guide from 5.x here

Changelog

• #2263 @jplaisted feat(search) BREAKING Support ElasticSearch 7, drop ES5
• #2260 @gabe-lyons fix(tags): fixing margins on tags for long descriptions
• #2259 @hsheth2 docs: update roadmap progress
• #2258 @dexter-mh-lee refactor(demo): Add empty global tags to BigQuery demo data
• #2255 @jjoyce0510 feat(react): Adding shadow and deeper linear gradient
• #2254 @gabe-lyons feat(tags): improving elastic search templates for tags
• #2253 @gabe-lyons fix(tags): fix ownership on tag create
• #2256 @hsheth2 fix: update slack links
• #2248 @gabe-lyons feat(tags): editing tags from react client on datasets, schemas, charts & dashboards
• #2252 @jjoyce0510 refactor(react): React as the default UI
• #2246 @hsheth2 feat(ingest): various minor fixes
• #2245 @jjoyce0510 feat(react): Adding big query logo
• #2249 @gabe-lyons fix(react): enabling charts and dashboards to be supported by theme config
• #2235 @pedro93 feat(ingest): Add support for druid
• #2244 @gabe-lyons feat(react): moving schema tab to be default
• #2243 @shirshanka docs: adding mar-19 townhall agenda
• #2240 @dexter-mh-lee feat(tags): Enable search for datasets by tags
• #2236 @pedro93 feat(k8s): Add metadata-ingestion as a Helm component
• #2241 @shirshanka docs: Improving architecture docs
• #2239 @hsheth2 feat(docs): use gradle for building docs
• #2232 @hsheth2 fix(ingest): various avro codegen fixes
• #2237 @gabe-lyons fix(dataflow): fixing browse dao access
• #2166 @arunvasudevan feat: MLmodel Graphql Query
• #2197 @frsann feat(datajob): Backend implementation
• #2233 @jjoyce0510 refactor(react): All entity search UI + misc improvements
• #2234 @jjoyce0510 docs(react): Oidc React Doc Updates
• #2231 @dexter-mh-lee fix(docker): start issue when there are multiple kafka brokers in bootstrap config
• #2227 @jjoyce0510 refactor(React): Misc UI improvements
• #2230 @hsheth2 fix(ingest): pin version of avro-gen3
• #2226 @hsheth2 fix(ingest): use python extras in docker image
• #2224 @hsheth2 feat(ingest): use plugin system based on Python extras
• #2190 @jjoyce0510 feat(react): SSO support simple OIDC authentication
• #2223 @dexter-mh-lee Added images to es/kafka-setup
• #2222 @dexter-mh-lee fix(ci): rename file to match git workflow needs
• #2220 @dexter-mh-lee fix(ci): remove paths_ignore from workflow files
• #2219 @thomasplarsson refactor(ingest): improve athena source api and documentation
• #2221 @gabe-lyons fix(ci): setting CI to false for builds
• #2218 @gabe-lyons feat(react): hiding raw schema button when no raw schema exists
• #2216 @dexter-mh-lee fix(es-setup): Add git workflows to upload docker for elasticsearch and kafka setup
• #2213 @thomasplarsson feat(ingest): add aws athena ingestion source
• #2217 @gabe-lyons fix(ci): fail CI on react build errors
• #2215 @gabe-lyons fix(react): fix theming test in react and simplifying api
• #2209 @thomasplarsson feat(ingest): add option for optimized skipping of schemas
• #2212 @hsheth2 fix(ingestion): nullable types and timestamp precision
• #2207 @hsheth2 feat(ingest): standalone metadata emitters
• #2205 @dexter-mh-lee fix(ci): Fix github package path
• #2204 @dexter-mh-lee feat(ci): Add SHA based tagging before pushing to docker registries
• #2203 @gabe-lyons feat(tag): adding search for tags in gms layer
• #2193 @gabe-lyons feat(react): adding ability to support theming of datahub, with two themes included
• #2201 @hsheth2 feat: add date and time types to SQL model
• #2202 @thomasplarsson feat(mae-consumer): enable mae-consumer to use ssl when communicating with elasticsearch
• #2199 @thomasplarsson fix(mae-consumer): mae-consumer needs sslcontext bean
• #2181 @shirshanka chore: renaming business_glossary rfc directory with pull request number
• #2182 @shirshanka chore: renaming graphql_frontend rfc directory with pull request number
• #2183 @shirshanka chore: renaming react-app rfc directory with pull request number
• #2196 @shirshanka docs(roadmap): update project roadmap
• #2195 @jjoyce0510 fix(graphql): Add "fixed" SchemaFieldDataType mapping
• #2194 @gabe-lyons feat(tags): Enriching sample data for tags
• #2191 @hsheth2 feat(docs): automatically populate sidebar with RFCs
• #2192 @jplaisted (feat) Simple python script to carry over ES indices from 5 to 7.
• #2173 @brendansun93 feat(React): Ownership component of user profile
• #2189 @thomasplarsson feat(gms): add elasticsearch SSL support
• #2112 @frsann feat(tags): RFC for tags
• #2187 @gabe-lyons fix(react): fixing test issues that arose from ill-timed merges
• #2164 @gabe-lyons feat(tags): adding support for read/write of tags in gms & read-only in react datahub-frontend.
• #2185 @jjoyce0510 feat(graphql): More forgiving for unknown data platforms during reads
• #2184 @jjoyce0510 test(React): Home page tests
• #2186 @hsheth2 fix(docs): fix broken links
• #2179 @gabe-lyons feat(react): adding raw schema view option for table schemas
• #2178 @hsheth2 feat(ingest): bigquery sample data
• #2176 @hsheth2 docs: point to hosted docs site
• #2177 @hsheth2 docs(ingest): clarify setuptools requirement
• #2175 @hsheth2 build(docs): only deploy docs on main repo
• #2174 @hsheth2 docs: hosted documentation website
• #2167 @jjoyce0510 feat(React): Impl browse UI for Dashboards and Charts
• #2168 @jjoyce0510 fix(React): Fix Browse Pagination Bug
• #2172 @hsheth2 fix(ingest): loosen Kafka broker validation
• #2165 @jjoyce0510 feat(DataPlatform Logos): Adding server driven logos
• #2171 @hsheth2 docs(ingest): clarify Kafka connection config
• #2169 @shirshanka doc(townhall): Add links for Feb 19, upcoming townhall on Mar 19
• #2161 @hsheth2 fix(ingest): bigquery source and dataset naming fixes
• #2163 @jjoyce0510 fix(graphql): Bubbling up exceptions logged in GraphQL resolvers
• #2159 @hsheth2 build(ingest): use multi-stage docker build for datahub-ingestion
• #2157 @hsheth2 feat(ingest): capture table descriptions
• #2158 @hsheth2 feat(ingest): switch quickstart to Python ingestion
• #2156 @pedro93 feat(ingest): support alternative authentication in sql ingestion
• #2152 @gabe-lyons fix(react): fixing format we propagate filters to graphql in
• #2154 @gabe-lyons feat(react): Redirecting /assets to index
• #2151 @hsheth2 build(docker): add large generated directories to dockerignore
• #2150 @hsheth2 ci(ingest): setup docker container for metadata ingestion
• #2145 @RickardCardell feat: neo4j Bolt TLS support (#2100)
• #2143 @dexter-mh-lee feat(dashboards): Add browse end point for charts and dashboards
• #2144 @RickardCardell feat: neo4j https support (#2101)
• #2147 @gabe-lyons docs(frontend): Update docs to clarify running local frontend w/ local react app
• #2148 @jjoyce0510 feat(gms): Add optional data platform display name
• #2149 @jplaisted Switch GMA dep from bintray to artifactory.
• #2146 @jjoyce0510 Fixing required audit stamps bug
• #2140 @jjoyce0510 feat(React): Search page UI improvements, 'all' entity search.
• #2133 @thomasplarsson feat(datahub-dao): enable services to access gms over https
• #2136 @hsheth2 feat(ingest): support Postgres PostGIS extensions
• #2139 @gabe-lyons docs(Ownership): making lack of support for ownergroups in frontend explicit in pdl
• #2137 @dexter-mh-lee refactor(docker-dev): set up elasticsearch using local mapping on docker-compose.dev
• #2135 @hsheth2 ci(ingest): run apt update
• #2134 @hsheth2 refactor(ingest): cleanup configuration models
• #2130 @jjoyce0510 feat(React UI): SearchPage and SearchResultsPage
• #2132 @jjoyce0510 Add URL to dashboard / chart page
• #2131 @gabe-lyons fix(React): Adding test coverage for search page & fixing filter select bug
• #2128 @jjoyce0510 fix(react): Fix authenticated user profile
• #2125 @hsheth2 fix(ingest): gracefully handle unknown types
• #2127 @jjoyce0510 feat: Introducing optional DataPlatform logo url
• #2124 @hsheth2 fix(ingest): update sample MCEs based on MLModel changes
• #2126 @jjoyce0510 fix(gms): fix getAllDataPlatforms bug
• #2123 @hsheth2 docs(ingest): add solutions for common install issues
• #2122 @hsheth2 feat(ingest): add support for LDAP ingestion
• #2120 @hsheth2 test(ingest): verify the output of mssql
• #2119 @jjoyce0510 feat(React): Adding basic chart + dashboard UI
• #2115 @brendansun93 feat(React): Avatar dropdown menu and logout function
• #2121 @hsheth2 feat(ingest): improve error reporting for pipelines
• #2117 @jjoyce0510 feat(GraphQL API): GQL implementation of Charts + Dashboards
• #2118 @jjoyce0510 bug(docker react): Fix react docker image build
• #2116 @hsheth2 docs(ingest): add example DAGs for Airflow
• #1896 @RyanHolstien feat: ML Model Backend Implementation
• #2109 @jjoyce0510 feat(graphql): Add Dashboards + Charts Models
• #2111 @jjoyce0510 bug(docker): Removing datahub-gms-graphql-service from default docker-compose.yml file
• #2108 @hsheth2 docs(ingest): clarify docs for new ingestion framework
• #2107 @shirshanka Update with Feb 19 Agenda
• #2104 @jjoyce0510 refactor(React): Renaming components/ dir to app/
• #2103 @jjoyce0510 docs(React): React README
• #2102 @jjoyce0510 fix(React): Fix Dataset Ownership Role Update Bug
• #2099 @gabe-lyons feat(React): Powering lineage page through graphql endpoint
• #2097 @gabe-lyons feat(React): deprecation status in header
• #2096 @gabe-lyons feat(React): Adding schema tab on dataset entity
• #2093 @jjoyce0510 feat(React): Adding Properties, Documentation tabs, user search, & supporting Dataset mutations
• #2095 @brendansun93 fix (React): Fix broken profile link on ownership + header
• #2094 @hsheth2 fix: Bump gma version
• #2091 @jjoyce0510 feat(GraphQL Mutation): Introducing updateDataset mutation
• #2061 @arpanm96 2056 | Change dataset lineage pendulum animation component
• #2090 @gabe-lyons feat(React Incubation): adding lineage page with sample data
• #2087 @allenhaozi fix(docs): update URN models and DatasetUrn links
• #2083 @gabe-lyons feat(React Incubation): User Profile
• #2084 @jjoyce0510 fix(GraphQL): Add schemaMetadata mapping & correctly form path string
• #2086 @gabe-lyons fix(react): including eslint in CI
• #2085 @jjoyce0510 fix(integ test): Bumping GMA to 0.2.40
• #2082 @hsheth2 fix(SQL ingest): Bump confluent-kafka version
• #2077 @jjoyce0510 refactor(React Incubation): Entity Interface & EntityRegistry
• #2081 @jjoyce0510 Enable mutation only in frontend
• #2076 @jjoyce0510 feat(GraphQL Query): Supporting complete Dataset, CorpUser with Search & Browse in GraphQL API
• #2071 @arunvasudevan feat (graphql): Datahub GMS Graphql Api Application for Querying Dataset
• #2075 @gabe-lyons refactor(React Incubation): Search routing refactor
• #2074 @shirshanka docs: Updating README with upcoming TownHall info
• #2073 @jjoyce0510 Fixing CorpUsers Search Bug
• #2072 @mars-lan Drop obsolete info on mysql-etl
• #2067 @jplaisted feat: update GMA to 0.2.35
• #2069 @gabe-lyons fix (react): fixing browse routing
• #2060 @jjoyce0510 feat(React Browse): Adding Browse Logic and misc refactorings
• #2066 @jjoyce0510 feat(GQL Queries): Productionalizing GraphQL Part 1: Dataset Query support + adding shared GraphQL module
• #2057 @nagarjunakanamarlapudi (refactor): Convert dataPlatforms to GMA aspect models and associated resource to GMA resource.
• #2063 @shirshanka Updating townhall history for Jan 15, and date for next townhall
• #2059 @jjoyce0510 feat(React Incubation): React Playground Proof of Concept
• #2055 @jjoyce0510 React Incubation RFC
• #2058 @jsotelo Support optional initContainers to setup extra initialization
• #2054 @nagarjunakanamarlapudi Update townhall schedule docs to third friday of every month
• #2053 @sattsb Fix for Kafka-UI Connectivity Error with Kafka-Rest Proxy
• #2050 @venkyvb Fix broken link in docs to the Restli resource base classes
• #2049 @shirshanka Townhall Agenda for Jan 15
• #2042 @jjoyce0510 feat(GraphQL support): RFC for GraphQL support in datahub-frontend (Part 1/2) - Queries
• #2045 @shirshanka docs: add links to GMA repo and RFC process from relevant places
• #2013 @mars-lan build(docker): add script to clean up docker environment
• #2041 @shirshanka Update README.md
• #2040 @nagarjunakanamarlapudi fix(search): Fix the rollback unintentional rollback(#2028) of dataset index to search by field paths.
• #2034 @immmus fix: The field connection with type URLConnection is not Serializable.
• #2032 @nagarjunakanamarlapudi Update town hall history for 12/4 and announce the next town hall date
• #2024 @nagarjunakanamarlapudi feat(Search): Inject restli client into index builders
• #2028 @jplaisted feat: add elasticsearch sanity integration tests