Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ingestion/lookml): liquid template resolution and view-to-view cll #10542

Merged
merged 65 commits into from
Jul 8, 2024
Merged

fix(ingestion/lookml): liquid template resolution and view-to-view cll #10542

Show file tree
Hide file tree
Changes from 63 commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
32a6ab0
wip
sid-acryl May 17, 2024
b308f91
introduce datahub sqlparser for sql parsing
sid-acryl May 20, 2024
d09b296
remove import pdb
sid-acryl May 20, 2024
3010df0
Merge branch 'master' into master+ing-510-lookml-cll
sid-acryl May 24, 2024
4dedd87
syntax1 and syntax2 support of lookml sql view
sid-acryl May 24, 2024
7922cc7
update message
sid-acryl May 24, 2024
86026c6
Merge branch 'master' into master+ing-510-lookml-cll
sid-acryl May 24, 2024
9f9d510
add if else for syntax
sid-acryl May 27, 2024
93add15
Merge branch 'master+ing-510-lookml-cll' of github.com:sid-acryl/data…
sid-acryl May 27, 2024
9dbd32b
liquid variable resolution
sid-acryl May 28, 2024
4f44fd9
liquid variable in config
sid-acryl May 29, 2024
fe96655
Merge branch 'master' into master+ing-510-lookml-cll
sid-acryl May 29, 2024
522d4a5
Merge branch 'master' into master+ing-510-lookml-cll
sid-acryl May 30, 2024
fcd9957
sqlglot_lib
sid-acryl May 30, 2024
808ded1
Merge branch 'master+ing-510-lookml-cll' of github.com:sid-acryl/data…
sid-acryl May 30, 2024
542de95
test fixes
sid-acryl Jun 1, 2024
ab61277
view to view lineage
sid-acryl Jun 5, 2024
734fbbe
Merge branch 'master' into master+ing-510-lookml-cll
sid-acryl Jun 5, 2024
7e75198
drop hive. from urn id
sid-acryl Jun 6, 2024
3d2b13b
Merge branch 'master+ing-510-lookml-cll' of github.com:sid-acryl/data…
sid-acryl Jun 6, 2024
2e6bffd
name.SQL_TABLE_NAME handling
sid-acryl Jun 7, 2024
5f9a582
added debug log
sid-acryl Jun 7, 2024
dc231a6
add stack trace
sid-acryl Jun 7, 2024
4cc0048
fix for select * from something.SQL_TABLE_NAME
sid-acryl Jun 11, 2024
54aff14
fix test cases
sid-acryl Jun 11, 2024
6c79b37
test case
sid-acryl Jun 12, 2024
b77eb3f
Merge branch 'master' into master+ing-510-lookml-cll
sid-acryl Jun 12, 2024
d58600f
rephrased description
sid-acryl Jun 12, 2024
1415b10
Merge branch 'master+ing-510-lookml-cll' of github.com:sid-acryl/data…
sid-acryl Jun 12, 2024
853ca3d
Merge branch 'master' into master+ing-510-lookml-cll
sid-acryl Jun 14, 2024
3cc0487
code refactoring to resolve circular dependency
sid-acryl Jun 14, 2024
27c3861
Merge branch 'master+ing-510-lookml-cll' of github.com:sid-acryl/data…
sid-acryl Jun 14, 2024
85c324b
Added case base logic
sid-acryl Jun 14, 2024
16fc4c9
refactored code
sid-acryl Jun 18, 2024
6cc7c60
refactored code
sid-acryl Jun 18, 2024
2b297c5
WIP
sid-acryl Jun 19, 2024
fd943a4
Merge branch 'master' into master+ing-510-lookml-cll
sid-acryl Jun 19, 2024
93a8413
Merge branch 'master+ing-510-lookml-cll' of github.com:sid-acryl/data…
sid-acryl Jun 19, 2024
2f61c16
WIP
sid-acryl Jun 19, 2024
a368a4b
refactor code
sid-acryl Jun 20, 2024
cfbba07
remove unused py file
sid-acryl Jun 20, 2024
045f769
Merge branch 'master' into master+ing-510-lookml-cll
sid-acryl Jun 21, 2024
84fa6c9
updated test case for custom tag
sid-acryl Jun 21, 2024
31e784b
Merge branch 'master+ing-510-lookml-cll' of github.com:sid-acryl/data…
sid-acryl Jun 21, 2024
7094405
update test case
sid-acryl Jun 21, 2024
ae02aae
Merge branch 'master' into master+ing-510-lookml-cll
sid-acryl Jun 24, 2024
b693251
handle special liquid variable
sid-acryl Jun 24, 2024
853fc25
Add double quotes to cover one customer use case
sid-acryl Jun 24, 2024
17f6204
add lru cache
sid-acryl Jun 24, 2024
6ae99c4
test case for sql_table_name having - in name
sid-acryl Jun 25, 2024
d72f475
Merge branch 'master' into master+ing-510-lookml-cll
sid-acryl Jun 25, 2024
0854446
Merge branch 'master' into master+ing-510-lookml-cll
sid-acryl Jun 26, 2024
12859b0
Merge branch 'master' into master+ing-510-lookml-cll
sid-acryl Jun 27, 2024
659dd3a
Merge branch 'master' into master+ing-510-lookml-cll
sid-acryl Jul 1, 2024
5ad8200
Merge branch 'master' into master+ing-510-lookml-cll
sid-acryl Jul 1, 2024
2e8c14f
refactor code
sid-acryl Jul 1, 2024
b9f8b08
fix existing golden files
sid-acryl Jul 2, 2024
cf0f45b
1. Resolve merge conflict
sid-acryl Jul 2, 2024
33985b5
Merge branch 'master' into master+ing-510-lookml-cll
sid-acryl Jul 3, 2024
519c173
Merge branch 'master' into master+ing-510-lookml-cll
sid-acryl Jul 3, 2024
0b926d0
resolve merge conflict
sid-acryl Jul 8, 2024
d5025d4
Merge branch 'master+ing-510-lookml-cll' of github.com:sid-acryl/data…
sid-acryl Jul 8, 2024
c2d2f6b
address review comments
sid-acryl Jul 8, 2024
e7008d2
Merge branch 'master' into master+ing-510-lookml-cll
sid-acryl Jul 8, 2024
8629f42
Merge branch 'master' into master+ing-510-lookml-cll
sid-acryl Jul 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion metadata-ingestion/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,7 @@
"sql-metadata==2.2.2",
*sqllineage_lib,
"GitPython>2",
"python-liquid",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Can we remove dependencies sql-metadata and sqllineage_lib now that sqlglot parser is used ?
  2. Does sqlglot_lib need to be added to looker_common libraries instead of lookml only, unless the sqlglot imports are strictly separated into lookml only files ?

}

bigquery_common = {
Expand Down Expand Up @@ -370,7 +371,7 @@
"kafka-connect": sql_common | {"requests", "JPype1"},
"ldap": {"python-ldap>=2.4"},
"looker": looker_common,
"lookml": looker_common,
"lookml": looker_common | sqlglot_lib,
"metabase": {"requests"} | sqlglot_lib,
"mlflow": {
"mlflow-skinny>=2.3.0",
Expand Down
Loading
Loading