Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ingest/mode): Improve query lineage #10284

Merged
merged 37 commits into from
May 8, 2024
Merged

fix(ingest/mode): Improve query lineage #10284

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
f1645b5
Adding a bit more verbose logging
treff7es Apr 15, 2024
f05a77e
Adding some more debug log
treff7es Apr 15, 2024
1488d80
Adding filter=all to spaces api call
treff7es Apr 15, 2024
82435fa
Adding space filter
treff7es Apr 15, 2024
4b2e3da
Fix test
treff7es Apr 15, 2024
df13fe3
Not throwing error if unable to retrieve user
treff7es Apr 15, 2024
f3f4be1
Resolving forms in queries
treff7es Apr 16, 2024
8ce76f9
Adding jinja2 as dependency
treff7es Apr 16, 2024
5b8ae2e
- Collecting sql parser failures
treff7es Apr 16, 2024
1c385f1
Merge branch 'master' into mode-verbose-log
treff7es Apr 20, 2024
586b6c4
Collecting stats about sql parsing
treff7es Apr 20, 2024
42f2685
Removing token from log
treff7es Apr 20, 2024
40c7d5e
Adding liquid templating parser as dependency
treff7es Apr 21, 2024
38c97c7
Adding report counter for query template rendering
treff7es Apr 21, 2024
46cb4be
Adding query token to debug log
treff7es Apr 21, 2024
6da8ed1
Resolving definition parameters
treff7es Apr 21, 2024
e8a2cc5
A couple of fixes for the lineage extraction
treff7es Apr 22, 2024
d042de1
Handling exceptions in sqlglot
treff7es Apr 22, 2024
9f17176
Fixing invalid cll issue
treff7es Apr 23, 2024
9fe525e
Adding missing file
treff7es Apr 23, 2024
4ab72d4
Fixing table error check
treff7es Apr 24, 2024
375ee75
tweaks to field generation, chart titles, etc
hsheth2 Apr 26, 2024
eff8d26
set chart subtype for mode charts
hsheth2 Apr 28, 2024
ed783e2
capture last refreshed ts
hsheth2 Apr 28, 2024
e62aa57
retain field types + measure/dimension tags on chart fields
hsheth2 Apr 28, 2024
f24a3a1
remove straggling line
hsheth2 Apr 28, 2024
57f8585
report missing imported datasets
hsheth2 Apr 28, 2024
5e6b1cf
add browse path v2
hsheth2 Apr 29, 2024
5a8b1dc
fix lint
hsheth2 Apr 29, 2024
acbab7a
fix bug in lossy list extend
hsheth2 Apr 29, 2024
f98b189
refactor subtypes + generate collection container
hsheth2 Apr 29, 2024
f83409c
add todo
hsheth2 Apr 29, 2024
43f23fc
retain chart type in custom props
hsheth2 Apr 29, 2024
4af8050
fix lint
hsheth2 Apr 30, 2024
9e9a134
dashboards in browse + last refreshed ts for charts
hsheth2 Apr 30, 2024
c36b9de
update tests
hsheth2 Apr 30, 2024
7b7adca
use query / dataset urns in browse
hsheth2 Apr 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion metadata-ingestion/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -369,7 +369,9 @@
# It's technically wrong for packages to depend on setuptools. However, it seems mlflow does it anyways.
"setuptools",
},
"mode": {"requests", "tenacity>=8.0.1"} | sqllineage_lib | sqlglot_lib,
"mode": {"requests", "python-liquid", "tenacity>=8.0.1"}
| sqllineage_lib
| sqlglot_lib,
"mongodb": {"pymongo[srv]>=3.11", "packaging"},
"mssql": sql_common
| {
Expand Down
6 changes: 6 additions & 0 deletions metadata-ingestion/src/datahub/ingestion/source/common/subtypes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ class BIContainerSubTypes(str, Enum):
QLIK_APP = "Qlik App"
SIGMA_WORKSPACE = "Sigma Workspace"
SIGMA_WORKBOOK = "Sigma Workbook"
MODE_COLLECTION = "Collection"


class JobContainerSubTypes(str, Enum):
Expand All @@ -64,3 +65,8 @@ class BIAssetSubTypes(str, Enum):
# PowerBI
POWERBI_TILE = "PowerBI Tile"
POWERBI_PAGE = "PowerBI Page"

# Mode
MODE_REPORT = "Report"
MODE_QUERY = "Query"
MODE_CHART = "Chart"
Loading
Loading