Skip to content

Commit

Permalink
feat: powerbi - add new source (#4201)
Browse files Browse the repository at this point in the history
  • Loading branch information
siddiquebagwan authored Mar 1, 2022
1 parent a439e1d commit e2f8db7
Show file tree
Hide file tree
Showing 10 changed files with 2,024 additions and 0 deletions.
Binary file added datahub-web-react/src/images/powerbilogo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ We use a plugin architecture so that you can install only the dependencies you a
| [trino](../metadata-ingestion/source_docs/trino.md) | `pip install 'acryl-datahub[trino]` | Trino source |
| [starburst-trino-usage](../metadata-ingestion/source_docs/trino.md) | `pip install 'acryl-datahub[starburst-trino-usage]'` | Starburst Trino usage statistics source |
| [nifi](../metadata-ingestion/source_docs/nifi.md) | `pip install 'acryl-datahub[nifi]` | Nifi source |
| [powerbi](../metadata-ingestion/source_docs/powerbi.md) | `pip install 'acryl-datahub[powerbi]` | Microsoft Power BI source |

### Sinks

Expand Down
20 changes: 20 additions & 0 deletions metadata-ingestion/examples/mce_files/data_platforms.json
Original file line number Diff line number Diff line change
Expand Up @@ -595,5 +595,25 @@
}
},
"proposedDelta": null
},
{
"auditHeader": null,
"proposedSnapshot": {
"com.linkedin.pegasus2avro.metadata.snapshot.DataPlatformSnapshot": {
"urn": "urn:li:dataPlatform:powerbi",
"aspects": [
{
"com.linkedin.pegasus2avro.dataplatform.DataPlatformInfo": {
"datasetNameDelimiter": ".",
"name": "powerbi",
"displayName": "Power BI",
"type": "OTHERS",
"logoUrl": "/assets/platforms/powerbilogo.png"
}
}
]
}
},
"proposedDelta": null
}
]
7 changes: 7 additions & 0 deletions metadata-ingestion/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,10 @@ def get_long_description():
"cryptography",
}

microsoft_common = {
"msal==1.16.0"
}

data_lake_base = {
*aws_common,
"parse>=1.19.0",
Expand Down Expand Up @@ -181,6 +185,7 @@ def get_long_description():
"trino": sql_common | {"trino"},
"starburst-trino-usage": sql_common | {"trino"},
"nifi": {"requests", "packaging"},
"powerbi": {"orderedset"} | microsoft_common
}

all_exclude_plugins: Set[str] = {
Expand Down Expand Up @@ -258,6 +263,7 @@ def get_long_description():
"trino",
"hive",
"starburst-trino-usage",
"powerbi"
# airflow is added below
]
for dependency in plugins[plugin]
Expand Down Expand Up @@ -355,6 +361,7 @@ def get_long_description():
"trino = datahub.ingestion.source.sql.trino:TrinoSource",
"starburst-trino-usage = datahub.ingestion.source.usage.starburst_trino_usage:TrinoUsageSource",
"nifi = datahub.ingestion.source.nifi:NifiSource",
"powerbi = datahub.ingestion.source.powerbi:PowerBiDashboardSource",
],
"datahub.ingestion.sink.plugins": [
"file = datahub.ingestion.sink.file:FileSink",
Expand Down
88 changes: 88 additions & 0 deletions metadata-ingestion/source_docs/powerbi.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Power BI dashboards

For context on getting started with ingestion, check out our [metadata ingestion guide](../README.md).

## Setup

To install this plugin, run `pip install 'acryl-datahub[powerbi]'`.

## Capabilities

This plugin extracts the following:

- Power BI dashboards, tiles, datasets
- Names, descriptions and URLs of dashboard and tile
- Owners of dashboards

## Configuration Notes

See the
1. [Microsoft AD App Creation doc](https://docs.microsoft.com/en-us/power-bi/developer/embedded/embed-service-principal) for the steps to create a app client ID and secret.
2. Login to Power BI as Admin and from `Tenant settings` allow below permissions.
- Allow service principles to use Power BI APIs
- Allow service principals to use read-only Power BI admin APIs
- Enhance admin APIs responses with detailed metadata


## Quickstart recipe

Check out the following recipe to get started with ingestion! See [below](#config-details) for full configuration options.

For general pointers on writing and running a recipe, see our [main recipe guide](../README.md#recipes).

```yml
source:
type: "powerbi"
config:
# Your Power BI tenant identifier
tenant_id: a949d688-67c0-4bf1-a344-e939411c6c0a
# Ingest elements of below PowerBi Workspace into Datahub
workspace_id: 4bd10256-e999-45dd-8e56-571c77153a5f
# Workspace's dataset environments (PROD, DEV, QA, STAGE)
env: DEV
# Azure AD App client identifier
client_id: foo
# Azure AD App client secret
client_secret: bar
# dataset_type_mapping is fixed mapping of Power BI datasources type to equivalent Datahub "data platform" dataset
dataset_type_mapping:
PostgreSql: postgres
Oracle: oracle


sink:
# sink configs
```

## Config details

| Field | Required | Default | Description |
| ------------------------- | -------- | ----------------------- | ------------------------------------------------------------------------------------------------------------ |
| `tenant_id` || | Power BI tenant identifier. |
| `workspace_id` || | Power BI workspace identifier. |
| `env` || | Environment to use in namespace when constructing URNs. |
| `client_id` || | Azure AD App client identifier. |
| `client_secret` | ✅ | | Azure AD App client secret.
| `dataset_type_mapping` | ✅ | | Mapping of Power BI datasource type to Datahub dataset.
| `scan_timeout` | ✅ | 60 | time in seconds to wait for Power BI metadata scan result.

## Concept mapping

| Power BI | Datahub |
| ------------------------- | ------------------- |
| `Dashboard` | `Dashboard` |
| `Dataset, Datasource` | `Dataset` |
| `Tile` | `Chart` |
| `Report.webUrl` | `Chart.externalUrl` |
| `Workspace` | `N/A` |
| `Report` | `N/A` |

If Tile is created from report then Chart.externalUrl is set to Report.webUrl.

## Compatibility

Coming soon!

## Questions

If you've got any questions on configuring this source, feel free to ping us on [our Slack](https://slack.datahubproject.io/)!
Loading

0 comments on commit e2f8db7

Please sign in to comment.