Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ingestion): ensure source/sink reports are always logged #4592

Merged
merged 10 commits into from
Apr 12, 2022
Merged

fix(ingestion): ensure source/sink reports are always logged #4592

Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
0dc629e
fix(ingestion): add error handling on whole pipeline
anshbansal Apr 6, 2022
209c09b
Update ingest_cli.py
jjoyce0510 Apr 8, 2022
5c04e7c
fix(cli): ensure on exception non-zero exit code
anshbansal Apr 8, 2022
2423d91
merge conflict fix
anshbansal Apr 8, 2022
a77c862
fix(cli): re-throw error always
anshbansal Apr 8, 2022
075f978
fix(cli): get return code always
anshbansal Apr 8, 2022
257b181
fix(cli): ensure there is error on failure
anshbansal Apr 8, 2022
8ca1e24
fix: only log stats when no exception
anshbansal Apr 8, 2022
76f2d89
move stuff to else
anshbansal Apr 8, 2022
0ef35db
remove extra arg
anshbansal Apr 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 15 additions & 5 deletions metadata-ingestion/src/datahub/cli/ingest_cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,11 +102,21 @@ def run(
raise SensitiveError() from e

logger.info("Starting metadata ingestion")
pipeline.run()
logger.info("Finished metadata ingestion")
ret = pipeline.pretty_print_summary(warnings_as_failure=strict_warnings)
pipeline.log_ingestion_stats()
sys.exit(ret)
try:
pipeline.run()
except Exception as e:
logger.info(
f"Source ({pipeline.config.source.type}) report:\n{pipeline.source.get_report().as_string()}"
)
logger.info(
f"Sink ({pipeline.config.sink.type}) report:\n{pipeline.sink.get_report().as_string()}"
)
raise e
else:
logger.info("Finished metadata pipeline")
pipeline.log_ingestion_stats()
ret = pipeline.pretty_print_summary(warnings_as_failure=strict_warnings)
sys.exit(ret)


def get_runs_url(gms_host: str) -> str:
Expand Down