feat(ingest/powerbi): powerbi dataset profiling #9355

looppi · 2023-11-30T11:47:05Z

DAX REST API based profiling implementation to query some details of PowerBI Datasets in the workspace. Does not introduce any breaking changes.

This implementation counts only amount of distinct values, min and max of a column. So it won't completely fulfill the possibilities Datahub UI provides. Extending the implementation to include other metrics isn't very complicated.

Is this something you would like to accept and include in Datahub or is using the DAX API for gathering information considered not a good practice?

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

siddiquebagwan · 2023-12-18T07:19:33Z

metadata-ingestion/src/datahub/ingestion/source/powerbi/rest_api_wrapper/powerbi_profiler.py

+        try:
+            logger.info(f"Column data query for {dataset.name}, {column.name}")
+            query = f"""
+            EVALUATE ROW(


Add query.py in rest_api_wrapper package. In query.py add a DaxQuery class having static method to return the qury

siddiquebagwan · 2023-12-18T08:24:20Z

metadata-ingestion/src/datahub/ingestion/source/powerbi/rest_api_wrapper/powerbi_api.py

@@ -47,6 +50,13 @@ def __init__(self, config: PowerBiDashboardSourceConfig) -> None:
            tenant_id=self.__config.tenant_id,
        )

+        self.__profiling_resolver = PowerBiDatasetProfilingResolver(


There are only two way to get the metadata from PowerBI either using regular API or Admin API. You can clearly see this segregation in PowerBI Rest API documentation: https://learn.microsoft.com/en-us/rest/api/power-bi/.

As per doc it looks like it comes under regular-api, so please add profile_dataset in base resolver and provide implementation in both regular and admin resolver (here it is just pass). Invoke the powerbi_profiler.py from regular api's profile_dataset method

siddiquebagwan · 2023-12-18T08:26:57Z

metadata-ingestion/src/datahub/ingestion/source/powerbi/rest_api_wrapper/data_classes.py

@@ -108,6 +112,10 @@ class Measure:
        BooleanTypeClass, DateTypeClass, NullTypeClass, NumberTypeClass, StringTypeClass
    ] = dataclasses.field(default_factory=NullTypeClass)
    description: Optional[str] = None
+    min: Optional[str] = None


Better to have MeasureProfiling dataclass and add the reference here

siddiquebagwan · 2023-12-18T08:30:02Z

metadata-ingestion/docs/sources/powerbi/powerbi_pre.md

@@ -107,6 +107,10 @@ By default, extracting endorsement information to tags is disabled. The feature

 Please note that the default implementation overwrites tags for the ingested entities, if you need to preserve existing tags, consider using a [transformer](../../../../metadata-ingestion/docs/transformer/dataset_transformer.md#simple-add-dataset-globaltags) with `semantics: PATCH` tags instead of `OVERWRITE`.

+## Profiling
+
+The profiling implementation is done through querying [DAX query endpoint](https://learn.microsoft.com/en-us/rest/api/power-bi/datasets/execute-queries). Therefore the principal needs to have permission to query the datasets to be profiled. Profiling is done with column based queries to be able to handle wide datasets without timeouts.


Additional documentation need to be added:

Please also check this doc: https://datahubproject.io/docs/quick-ingestion-guides/powerbi/setup and add/update any steps if any specific permission is required for profiling. The markdown file is available at docs/quick-ingestion-guides/powerbi/setup.md

On Source class add decorator @capability( SourceCapability.DATA_PROFILING, "Optionally enabled via configuration profiling.enabled", )

looppi · 2024-01-04T07:36:42Z

Thank you for the feedback! I'll start working with this asap.

hsheth2 · 2024-01-18T22:01:56Z

@sid-acryl can you take another look at this PR?

sid-acryl · 2024-02-28T05:31:03Z

metadata-ingestion/src/datahub/ingestion/source/powerbi/powerbi.py

+                f"Table {table.name} in {dataset.name}, not allowed for profiling"
+            )
+            return
+        logger.info(f"Profiling table: {table.name}")


Lets make it debug

sid-acryl · 2024-02-28T05:36:45Z

metadata-ingestion/src/datahub/ingestion/source/powerbi/powerbi.py

+        ] = [*(table.columns or []), *(table.measures or [])]
+        for column in columns:
+            allowed_column = self.__config.profile_pattern.allowed(
+                f"{workspace.name}.{dataset.name}.{table.name}.{column.name}"


Could you please give example value of profile_pattern config to make it clear for user that he needs to specify fully qualified name of table

sid-acryl · 2024-02-28T05:58:58Z

metadata-ingestion/src/datahub/ingestion/source/powerbi/rest_api_wrapper/data_resolver.py

+                }
+            ],
+            "serializerSettings": {
+                "includeNulls": True,


Could you please add comments for Why we need to include Nulls

sid-acryl · 2024-02-28T05:59:51Z

metadata-ingestion/src/datahub/ingestion/source/powerbi/rest_api_wrapper/data_resolver.py

+        query = DaxQuery.row_count_query(table.name)
+        try:
+            data = self._execute_profiling_query(dataset, query)
+            rows = data["results"][0]["tables"][0]["rows"]


add checks here to avoid key error

it looks like they're already catching KeyError below, so it should be ok

sid-acryl · 2024-02-28T06:06:57Z

metadata-ingestion/src/datahub/ingestion/source/powerbi/rest_api_wrapper/profiling_utils.py

+from typing import Dict, List
+
+
+def get_column_name(table_and_col: str) -> str:


Make it -> Optional[str]

looppi · 2024-02-28T18:50:27Z

Thank you for the review and feedback! I'll continue working on this still this week.

sid-acryl · 2024-04-15T18:13:30Z

metadata-ingestion/docs/sources/powerbi/powerbi_pre.md

+Take into account that the profiling implementation exeutes fairly big amount of DAX queries and for big datasets this is substantial load to the PowerBI system.
+
+The `profiling_pattern` setting may be used to limit profiling actions to only a certain set of resources in PowerBI. Both allow and deny rules are matched against following pattern for every table in a PowerBI Dataset: `workspace_name.dataset_name.table_name`. User may limit profiling with these settings at table level, dataset level or workspace level.
+
 ## Admin Ingestion vs. Basic Ingestion
 PowerBI provides two sets of API i.e. [Basic API and Admin API](https://learn.microsoft.com/en-us/rest/api/power-bi/). 



Could you please update the Caveats of setting admin_apis_only to true: and add a bullet point for dataset profiling is not available

Sorry for the delay, I added mention that the dataset profiling is not available through the Admin API

sid-acryl

LGTM

looppi added 2 commits November 30, 2023 10:27

feat(ingest/powerbi): add dax based dataset profiling

1914028

docs(ingest/powerbi): add some documentation about profiling

6c82b50

github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label Nov 30, 2023

vercel bot deployed to Preview November 30, 2023 12:27 View deployment

chriscollins3456 added the poc-marathon-dec-2023 label Dec 6, 2023

hsheth2 requested a review from siddiquebagwan-gslab December 14, 2023 19:12

siddiquebagwan suggested changes Dec 18, 2023

View reviewed changes

looppi added 3 commits January 11, 2024 16:46

fix: move profiling data to a separate dataclass

7f6d96f

refactor: powerbi profiling to existing resolver

4fadf1f

fix: add profiling capability decorator

3edbf3b

vercel bot deployed to Preview January 12, 2024 16:23 View deployment

Merge branch 'master' into powerbi-profiling

d7190fb

vercel bot deployed to Preview January 13, 2024 17:21 View deployment

docs: update docs about required permissions for profiling

71ae650

vercel bot deployed to Preview January 18, 2024 09:33 View deployment

fix: import order

d4bb88f

vercel bot deployed to Preview January 18, 2024 11:16 View deployment

fix: flake8 error

5aefa51

vercel bot deployed to Preview January 18, 2024 15:25 View deployment

hsheth2 added the community-contribution PR or Issue raised by member(s) of DataHub Community label Jan 18, 2024

sid-acryl suggested changes Feb 28, 2024

View reviewed changes

looppi added 4 commits March 2, 2024 13:24

style: code style improvements

9f6d82d

Merge branch 'master' into powerbi-profiling

4f39a14

docs: explain reasoning behind includeNulls

bac8c55

fix: mypy errors

8c0b7f1

vercel bot deployed to Preview March 2, 2024 12:39 View deployment

docs: update docs and parameter definitions

fddb3a4

style: fix formatting

db07420

vercel bot deployed to Preview March 3, 2024 13:56 View deployment

sid-acryl suggested changes Apr 15, 2024

View reviewed changes

looppi added 2 commits May 10, 2024 10:45

Merge branch 'master' into powerbi-profiling

74062c5

docs: dataset profiling is unavailable through admin api

bfdb963

vercel bot deployed to Preview May 10, 2024 08:13 View deployment

lint: fix linting

81c7a31

vercel bot deployed to Preview May 10, 2024 08:41 View deployment

sid-acryl approved these changes May 24, 2024

View reviewed changes

hsheth2 approved these changes May 24, 2024

View reviewed changes

shirshanka merged commit b223281 into datahub-project:master Jun 28, 2024
58 checks passed

yoonhyejin pushed a commit that referenced this pull request Jul 16, 2024

feat(ingest/powerbi): powerbi dataset profiling (#9355)

aa8b508

aviv-julienjehannet pushed a commit to aviv-julienjehannet/datahub that referenced this pull request Jul 17, 2024

feat(ingest/powerbi): powerbi dataset profiling (datahub-project#9355)

067b76f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ingest/powerbi): powerbi dataset profiling #9355

feat(ingest/powerbi): powerbi dataset profiling #9355

looppi commented Nov 30, 2023 •

edited

Loading

siddiquebagwan Dec 18, 2023

siddiquebagwan Dec 18, 2023

siddiquebagwan Dec 18, 2023

siddiquebagwan Dec 18, 2023

looppi commented Jan 4, 2024

hsheth2 commented Jan 18, 2024

sid-acryl Feb 28, 2024

sid-acryl Feb 28, 2024

sid-acryl Feb 28, 2024

sid-acryl Feb 28, 2024

hsheth2 Feb 28, 2024

sid-acryl Feb 28, 2024

looppi commented Feb 28, 2024

sid-acryl Apr 15, 2024

looppi May 10, 2024

sid-acryl left a comment

		from typing import Dict, List


		def get_column_name(table_and_col: str) -> str:

feat(ingest/powerbi): powerbi dataset profiling #9355

feat(ingest/powerbi): powerbi dataset profiling #9355

Conversation

looppi commented Nov 30, 2023 • edited Loading

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

looppi commented Jan 4, 2024

hsheth2 commented Jan 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

looppi commented Feb 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sid-acryl left a comment

Choose a reason for hiding this comment

looppi commented Nov 30, 2023 •

edited

Loading