-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(ingest/bigquery): use correct row count in null count profiling c… #9123
fix(ingest/bigquery): use correct row count in null count profiling c… #9123
Conversation
# may be slightly different (more or less) than configured `sample_size`. | ||
# However not doing so to start with, as that adds another query overhead | ||
# plus approximate metrics should work for sampling based profiling. | ||
profile.rowCount = self.config.sample_size |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Imo the row count should still ultimately reflect the full number of rows, and the size of the sample should be stored in the partition spec
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The actual table's rowCount here is always overridden in bigquery source from information_schema, whenever partitionSpec is set. partitionSpec is set whenever limit or sampling is used in profiling or only latest partition is used for partitioned tables. The computed rowCount in ge_data_profiler.py is used primarily for column level profiling computation only. (this is also the case for snowflake and redshift)
Here - https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/sql/sql_generic_profiler.py#L112
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm open to handling this only for null counts computation and not updating local profile aspect. Attempted changing profile aspect to make sure that we don't run into this for any other calculation - maybe in future.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yup that might result in more understandable code
At minimum, we need a comment here explaining why it's ok to override, but ideally we just tweak the null count / unique percentage calculations
…omputation
Checklist