From 23e3798aea0835025a857ca6da80ad3642670932 Mon Sep 17 00:00:00 2001 From: Harshal Sheth Date: Wed, 11 Dec 2024 21:36:15 -0500 Subject: [PATCH 1/5] docs(ingest): add docs on the SQL parser --- docs-website/sidebars.js | 1 + docs/lineage/airflow.md | 2 +- docs/lineage/sql_parsing.md | 65 ++++++++++++++++++++++++++++ metadata-ingestion/scripts/docgen.py | 4 +- 4 files changed, 69 insertions(+), 3 deletions(-) create mode 100644 docs/lineage/sql_parsing.md diff --git a/docs-website/sidebars.js b/docs-website/sidebars.js index 2f1ac04772097..c18d8671318f6 100644 --- a/docs-website/sidebars.js +++ b/docs-website/sidebars.js @@ -541,6 +541,7 @@ module.exports = { }, "docs/platform-instances", + "docs/lineage/sql_parsing", "metadata-ingestion/docs/dev_guides/stateful", "metadata-ingestion/docs/dev_guides/classification", "metadata-ingestion/docs/dev_guides/add_stateful_ingestion_to_source", diff --git a/docs/lineage/airflow.md b/docs/lineage/airflow.md index 2bd58334933fb..72b5cbf57592d 100644 --- a/docs/lineage/airflow.md +++ b/docs/lineage/airflow.md @@ -164,7 +164,7 @@ Only the v2 plugin supports automatic lineage extraction. If you're using the v1 To automatically extract lineage information, the v2 plugin builds on top of Airflow's built-in [OpenLineage extractors](https://openlineage.io/docs/integrations/airflow/default-extractors). As such, we support a superset of the default operators that Airflow/OpenLineage supports. -The SQL-related extractors have been updated to use [DataHub's SQL lineage parser](https://blog.datahubproject.io/extracting-column-level-lineage-from-sql-779b8ce17567), which is more robust than the built-in one and uses DataHub's metadata information to generate column-level lineage. +The SQL-related extractors have been updated to use [DataHub's SQL lineage parser](./sql_parsing.md), which is more robust than the built-in one and uses DataHub's metadata information to generate column-level lineage. Supported operators: diff --git a/docs/lineage/sql_parsing.md b/docs/lineage/sql_parsing.md new file mode 100644 index 0000000000000..c8d41bd0f9bb6 --- /dev/null +++ b/docs/lineage/sql_parsing.md @@ -0,0 +1,65 @@ +--- +title: SQL Parsing +--- + +# The DataHub SQL Parser + +Many data platforms are built on top of SQL, which means deeply understanding SQL queries is critical for understanding column-level lineage, usage, and more. + +DataHub's SQL parser is built on top of [sqlglot](https://github.com/tobymao/sqlglot) and adds a number of additional features to improve the accuracy of SQL parsing. + +In our benchmarks, the DataHub SQL parser generates lineage with 97-99% accuracy and outperforms other SQL parsers by a wide margin. + +We've published a blog post on some of the technical details of the parser: [Extracting Column Lineage from SQL Queries](https://blog.datahubproject.io/extracting-column-level-lineage-from-sql-779b8ce17567). + +## Built-in SQL Parsing Support + +If you're using a tool that DataHub already [integrates with](https://datahubproject.io/integrations), check the documentation for that specific integration. +Most of our integrations, including Snowflake, BigQuery, Redshift, dbt, Looker, PowerBI, Airflow, etc, use the SQL parser to generate column-level lineage and usage statistics. + +If you’re using a different database system for which we don’t support column-level lineage out of the box, but you do have a database query log available, the [SQL queries](../generated/ingestion/sources/sql-queries.md) connector can generate column-level lineage and table/column usage statistics from the query log. + +## SDK Support + +Our SDK provides a [`DataHubGraph.parse_sql_lineage()`](../python-sdk/clients/#datahub.ingestion.graph.client.DataHubGraph.parse_sql_lineage) method for programmatically parsing SQL queries. + +The resulting object contains a `sql_parsing_result.debug_info.confidence_score` field, which is a 0-1 value indicating the confidence of the parser. + +There are also a number of utilities in the `datahub.sql_parsing` module. The `SqlParsingAggregator` is particularly useful, as it can also resolve lineage across temp tables and table renames/swaps. +Note that these utilities are not officially part of the DataHub SDK and hence do not have the same level of stability and support as the rest of the SDK. + +## Capabilities and Limitations + +Supported: + +- Table-level lineage for `SELECT`, `CREATE`, `INSERT`, `UPDATE`, `DELETE`, and `MERGE` statements +- Column-level lineage for `SELECT` (including `SELECT INTO`), `CREATE VIEW`, `CREATE TABLE AS SELECT` (CTAS), `INSERT`, and `UPDATE` statements +- Subqueries +- CTEs +- `UNION ALL` constructs - will merge lineage across the clauses of the `UNION` +- `SELECT *` and similar expressions will automatically be expanded with the table schemas registered in DataHub. This includes support for platform instances. +- Automatic handling for systems where table and column names are case insensitive. Generally requires that `convert_urns_to_lowercase` is enabled when the corresponding table schemas were ingested into DataHub. + + - Specifically, we'll do fuzzy matching against the table names and schemas to resolve the correct URNs. We do not support having multiple tables/columns that only differ in casing. + +- For BigQuery, sharded table suffixes will automatically be normalized. For example, `proj.dataset.table_20230616` will be normalized to `proj.dataset.table_yyyymmdd`. This matches the behavior of our BigQuery ingestion connector, and hence will result in lineage linking up correctly. + +Not supported: + +- Scalar `UDFs` - We will generate lineage pointing at the columns that are inputs to the UDF, but will not be able to understand the UDF itself. +- Tabular `UDFs` +- `json_extract` and similar functions +- `UNNEST` - We will do a best-effort job, but cannot reliably generate column-level lineage in the presence of `UNNEST` constructs. +- Structs - We will do a best-effort attempt to resolve struct subfields, but it is not guaranteed. This will only impact column-level lineage. +- Snowflake's multi-table inserts +- Multi-statement SQL / SQL scripting + +Limitations: + +- We only support the 20+ SQL dialects supported by the underlying [sqlglot](https://github.com/tobymao/sqlglot) library. +- There's a few SQL syntaxes that we don't support yet, but intend to support in the future. + - `INSERT INTO (col1_new, col2_new) SELECT col1_old, col2_old FROM ...`. We only support `INSERT INTO` statements that either (1) don't specify a column list, or (2) specify a column list that matches the columns in the `SELECT` clause. + - `MERGE INTO` statements - We don't generate column-level lineage for these. +- In cases where the table schema information in DataHub is outdated or otherwise incorrect, we may not be able to generate accurate column-level lineage. +- We trip over BigQuery queries that use the `_partitiontime` and `_partitiondate` pseudo-columns with a table name prefix e.g. `my_table._partitiontime` fails. However, unqualified references like `_partitiontime` and `_partitiondate` will be fine. +- We do not consider columns referenced in `WHERE`, `GROUP BY`, `ORDER BY`, etc. clauses to be part of lineage. For example, `SELECT col1, col2 FROM upstream_table WHERE col3 = 3` will not generate any lineage related to `col3`. diff --git a/metadata-ingestion/scripts/docgen.py b/metadata-ingestion/scripts/docgen.py index 797a2f698c2f4..402cd8a814199 100644 --- a/metadata-ingestion/scripts/docgen.py +++ b/metadata-ingestion/scripts/docgen.py @@ -918,7 +918,7 @@ def generate(

-By default, The UI shows the latest version of the lineage. The time picker can be used to filter out edges within the latest version to exclude those that were last updated outside of the time window. Selecting time windows in the patch will not show you historical lineages. It will only filter the view of the latest version of the lineage. +By default, the UI shows the latest version of the lineage. The time picker can be used to filter out edges within the latest version to exclude those that were last updated outside of the time window. Selecting time windows in the patch will not show you historical lineages. It will only filter the view of the latest version of the lineage.

@@ -969,7 +969,7 @@ def generate( ## Lineage Support DataHub supports **[automatic table- and column-level lineage detection](#automatic-lineage-extraction-support)** from BigQuery, Snowflake, dbt, Looker, PowerBI, and 20+ modern data tools. -For data tools with limited native lineage tracking, **DataHub's SQL Parser** detects lineage with 97–99% accuracy, ensuring teams will have high quality lineage graphs across all corners of their data stack. +For data tools with limited native lineage tracking, [**DataHub's SQL Parser**](../../lineage/sql_parsing.md) detects lineage with 97-99% accuracy, ensuring teams will have high quality lineage graphs across all corners of their data stack. ### Types of Lineage Connections From b29e5043b04e44721173614eca95be121d99d710 Mon Sep 17 00:00:00 2001 From: Harshal Sheth Date: Wed, 11 Dec 2024 22:13:12 -0500 Subject: [PATCH 2/5] fix link --- docs/lineage/sql_parsing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/lineage/sql_parsing.md b/docs/lineage/sql_parsing.md index c8d41bd0f9bb6..4b907b5f9ffa0 100644 --- a/docs/lineage/sql_parsing.md +++ b/docs/lineage/sql_parsing.md @@ -21,7 +21,7 @@ If you’re using a different database system for which we don’t support colum ## SDK Support -Our SDK provides a [`DataHubGraph.parse_sql_lineage()`](../python-sdk/clients/#datahub.ingestion.graph.client.DataHubGraph.parse_sql_lineage) method for programmatically parsing SQL queries. +Our SDK provides a [`DataHubGraph.parse_sql_lineage()`](../../python-sdk/clients.md#datahub.ingestion.graph.client.DataHubGraph.parse_sql_lineage) method for programmatically parsing SQL queries. The resulting object contains a `sql_parsing_result.debug_info.confidence_score` field, which is a 0-1 value indicating the confidence of the parser. From 8493224641a2d13cd2eb87cd0285eac26f17ff37 Mon Sep 17 00:00:00 2001 From: Harshal Sheth Date: Thu, 12 Dec 2024 13:41:31 -0500 Subject: [PATCH 3/5] fix anchor link parse --- docs-website/generateDocsDir.ts | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs-website/generateDocsDir.ts b/docs-website/generateDocsDir.ts index 032e912c7190e..0f7e347da64eb 100644 --- a/docs-website/generateDocsDir.ts +++ b/docs-website/generateDocsDir.ts @@ -284,6 +284,10 @@ function markdown_add_slug( // ); // } +function trim_anchor_link(url: string): string { + return url.replace(/#.+$/, ""); +} + function new_url(original: string, filepath: string): string { if (original.toLowerCase().startsWith(HOSTED_SITE_URL)) { // For absolute links to the hosted docs site, we transform them into local ones. @@ -313,7 +317,7 @@ function new_url(original: string, filepath: string): string { } // Now we assume this is a local reference. - const suffix = path.extname(original); + const suffix = path.extname(trim_anchor_link(original)); if ( suffix == "" || [ @@ -335,7 +339,7 @@ function new_url(original: string, filepath: string): string { // A reference to a file or directory in the Github repo. const relation = path.dirname(filepath); const updated_path = path.normalize(`${relation}/${original}`); - const check_path = updated_path.replace(/#.+$/, ""); + const check_path = trim_anchor_link(updated_path); if ( !fs.existsSync(`../${check_path}`) && actually_in_sidebar(filepath) && From cda52062148caff5cd4adb797f689cc144386a7f Mon Sep 17 00:00:00 2001 From: Harshal Sheth Date: Thu, 12 Dec 2024 15:40:43 -0500 Subject: [PATCH 4/5] fix sections --- docs/lineage/sql_parsing.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/docs/lineage/sql_parsing.md b/docs/lineage/sql_parsing.md index 4b907b5f9ffa0..532521f496f64 100644 --- a/docs/lineage/sql_parsing.md +++ b/docs/lineage/sql_parsing.md @@ -28,9 +28,9 @@ The resulting object contains a `sql_parsing_result.debug_info.confidence_score` There are also a number of utilities in the `datahub.sql_parsing` module. The `SqlParsingAggregator` is particularly useful, as it can also resolve lineage across temp tables and table renames/swaps. Note that these utilities are not officially part of the DataHub SDK and hence do not have the same level of stability and support as the rest of the SDK. -## Capabilities and Limitations +## Capabilities -Supported: +#### Supported - Table-level lineage for `SELECT`, `CREATE`, `INSERT`, `UPDATE`, `DELETE`, and `MERGE` statements - Column-level lineage for `SELECT` (including `SELECT INTO`), `CREATE VIEW`, `CREATE TABLE AS SELECT` (CTAS), `INSERT`, and `UPDATE` statements @@ -39,12 +39,10 @@ Supported: - `UNION ALL` constructs - will merge lineage across the clauses of the `UNION` - `SELECT *` and similar expressions will automatically be expanded with the table schemas registered in DataHub. This includes support for platform instances. - Automatic handling for systems where table and column names are case insensitive. Generally requires that `convert_urns_to_lowercase` is enabled when the corresponding table schemas were ingested into DataHub. - - Specifically, we'll do fuzzy matching against the table names and schemas to resolve the correct URNs. We do not support having multiple tables/columns that only differ in casing. - - For BigQuery, sharded table suffixes will automatically be normalized. For example, `proj.dataset.table_20230616` will be normalized to `proj.dataset.table_yyyymmdd`. This matches the behavior of our BigQuery ingestion connector, and hence will result in lineage linking up correctly. -Not supported: +#### Not supported - Scalar `UDFs` - We will generate lineage pointing at the columns that are inputs to the UDF, but will not be able to understand the UDF itself. - Tabular `UDFs` @@ -54,7 +52,7 @@ Not supported: - Snowflake's multi-table inserts - Multi-statement SQL / SQL scripting -Limitations: +#### Limitations - We only support the 20+ SQL dialects supported by the underlying [sqlglot](https://github.com/tobymao/sqlglot) library. - There's a few SQL syntaxes that we don't support yet, but intend to support in the future. From f23db0eb8ded6c2dc1af073cdad1631a896347a9 Mon Sep 17 00:00:00 2001 From: Harshal Sheth Date: Thu, 12 Dec 2024 15:41:29 -0500 Subject: [PATCH 5/5] tweak heading level --- docs/lineage/sql_parsing.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/lineage/sql_parsing.md b/docs/lineage/sql_parsing.md index 532521f496f64..76161de0040f4 100644 --- a/docs/lineage/sql_parsing.md +++ b/docs/lineage/sql_parsing.md @@ -30,7 +30,7 @@ Note that these utilities are not officially part of the DataHub SDK and hence d ## Capabilities -#### Supported +### Supported - Table-level lineage for `SELECT`, `CREATE`, `INSERT`, `UPDATE`, `DELETE`, and `MERGE` statements - Column-level lineage for `SELECT` (including `SELECT INTO`), `CREATE VIEW`, `CREATE TABLE AS SELECT` (CTAS), `INSERT`, and `UPDATE` statements @@ -42,7 +42,7 @@ Note that these utilities are not officially part of the DataHub SDK and hence d - Specifically, we'll do fuzzy matching against the table names and schemas to resolve the correct URNs. We do not support having multiple tables/columns that only differ in casing. - For BigQuery, sharded table suffixes will automatically be normalized. For example, `proj.dataset.table_20230616` will be normalized to `proj.dataset.table_yyyymmdd`. This matches the behavior of our BigQuery ingestion connector, and hence will result in lineage linking up correctly. -#### Not supported +### Not supported - Scalar `UDFs` - We will generate lineage pointing at the columns that are inputs to the UDF, but will not be able to understand the UDF itself. - Tabular `UDFs` @@ -52,7 +52,7 @@ Note that these utilities are not officially part of the DataHub SDK and hence d - Snowflake's multi-table inserts - Multi-statement SQL / SQL scripting -#### Limitations +### Limitations - We only support the 20+ SQL dialects supported by the underlying [sqlglot](https://github.com/tobymao/sqlglot) library. - There's a few SQL syntaxes that we don't support yet, but intend to support in the future.