Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(glue): delete CatalogId parameter from get_jobs api call #4646

Merged
merged 4 commits into from
Apr 21, 2022
Merged

fix(glue): delete CatalogId parameter from get_jobs api call #4646

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a511e13
glue-job-bug-fix
BoyuanZhangDE Apr 12, 2022
6fd9a40
Merge branch 'master' into glue-job-bug-fix
BoyuanZhangDE Apr 14, 2022
13f9045
Merge branch 'master' into glue-job-bug-fix
BoyuanZhangDE Apr 18, 2022
ea37058
re-start test
BoyuanZhangDE Apr 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion metadata-ingestion/source_docs/glue.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,11 @@ Note that a `.` is used to denote nested fields in the YAML recipe.
| `domain.domain_key.allow` | | | List of regex patterns for tables to set domain_key domain key (domain_key can be any string like `sales`. There can be multiple domain key specified. |
| `domain.domain_key.deny` | | | List of regex patterns for tables to not assign domain_key. There can be multiple domain key specified. |
| `domain.domain_key.ignoreCase` | | `True` | Whether to ignore case sensitivity during pattern matching.There can be multiple domain key specified. |
| `catalog_id` | | | The aws account id where the target glue catalog lives. If None, datahub will ingest glue in aws caller's account. |
| `catalog_id` | | None | The aws account id where the target glue catalog lives. If None, datahub will ingest glue catalog in aws caller's account. |

### Cross-account ingestion

To ingest glue catalog from another aws account, use the `catalog_id` field. Note that glue job is not affected by this field and it only ingests from aws caller's account. So if you are ingestion glue catalog from another aws account, you may set `extract_transforms` as `False` to avoid the discrepancy between glue catalog and glue jobs.

## Compatibility

Expand Down
12 changes: 2 additions & 10 deletions metadata-ingestion/src/datahub/ingestion/source/aws/glue.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,22 +156,14 @@ def platform(self) -> str:

def get_all_jobs(self):
"""
List all jobs in Glue.
List all jobs in Glue. boto3 get_jobs api call doesn't support cross account access.
"""

jobs = []

# see https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.get_jobs
paginator = self.glue_client.get_paginator("get_jobs")

if self.source_config.catalog_id:
paginator_response = paginator.paginate(
CatalogId=self.source_config.catalog_id
)
else:
paginator_response = paginator.paginate()

for page in paginator_response:
for page in paginator.paginate():
jobs += page["Jobs"]

return jobs
Expand Down