Skip to content

Commit

Permalink
feat(ingest): enable connection string for all sqlalchemy datasources (…
Browse files Browse the repository at this point in the history 
…#4508)

* feat(ingest): enable connection string for all sqlalchemy datasources

* Update sql_common.py

* fix types

* update docs

* rename variable to sqlalchemy_uri

Co-authored-by: Harshal Sheth <[email protected]>
  • Loading branch information
@ms32035 @hsheth2
ms32035 and hsheth2 authored Apr 8, 2022
1 parent 45e09ca commit 7c3ad3d
Show file tree
Hide file tree
Showing 9 changed files with 17 additions and 5 deletions.
1 change: 1 addition & 0 deletions metadata-ingestion/source_docs/clickhouse.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ Note that a `.` is used to denote nested fields in the YAML recipe.
| `password` | | | ClickHouse password. |
| `host_port` | ✅ | | ClickHouse host URL. |
| `database` | | | ClickHouse database to connect. |
| `sqlalchemy_uri` | | | URI of database to connect to. See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls. Takes precedence over other connection parameters. |
| `env` | | `"PROD"` | Environment to use in namespace when constructing URNs. |
| `platform_instance` | | None | The Platform instance to use while constructing URNs. |
| `options.<option>` | | | Any options specified here will be passed to SQLAlchemy's `create_engine` as kwargs.<br />See https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine for details. |
Expand Down
1 change: 1 addition & 0 deletions metadata-ingestion/source_docs/druid.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ As a SQL-based service, the Athena integration is also supported by our SQL prof
| `password` | | | Database password. |
| `host_port` | ✅ | | Host URL and port to connect to. |
| `database` | | | Database to ingest. |
| `sqlalchemy_uri` | | | URI of database to connect to. See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls. Takes precedence over other connection parameters. |
| `database_alias` | | | Alias to apply to database when ingesting. |
| `env` | | `"PROD"` | Environment to use in namespace when constructing URNs. |
| `platform_instance` | | None | The Platform instance to use while constructing URNs. |
Expand Down
1 change: 1 addition & 0 deletions metadata-ingestion/source_docs/mysql.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ Note that a `.` is used to denote nested fields in the YAML recipe.
| `password` | | | MySQL password. |
| `host_port` | | `"localhost:3306"` | MySQL host URL. |
| `database` | | | MySQL database. |
| `sqlalchemy_uri` | | | URI of database to connect to. See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls. Takes precedence over other connection parameters. |
| `database_alias` | | | Alias to apply to database when ingesting. A table with urn `urn:li:dataset:(urn:li:dataPlatform:mysql,<database>.<table>,PROD)` will turn into `urn:li:dataset:(urn:li:dataPlatform:mysql,<database_alias>.<table>,PROD)`. |
| `env` | | `"PROD"` | Environment to use in namespace when constructing URNs. |
| `platform_instance` | | None | The Platform instance to use while constructing URNs. |
Expand Down
1 change: 1 addition & 0 deletions metadata-ingestion/source_docs/oracle.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ As a SQL-based service, the Athena integration is also supported by our SQL prof
| `host_port` | | | Oracle host URL. |
| `database` | If `service_name` is not set | | If using, omit `service_name`. |
| `service_name` | If `database_alias` is not set | | Oracle service name. If using, omit `database`. |
| `sqlalchemy_uri` | | | URI of database to connect to. See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls. Takes precedence over other connection parameters. |
| `database_alias` | | | Alias to apply to database when ingesting. |
| `env` | | `"PROD"` | Environment to use in namespace when constructing URNs. |
| `options.<option>` | | | Any options specified here will be passed to SQLAlchemy's `create_engine` as kwargs.<br />See https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine for details. |
Expand Down
1 change: 1 addition & 0 deletions metadata-ingestion/source_docs/postgres.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ As a SQL-based service, the Athena integration is also supported by our SQL prof
| `password` | | | PostgreSQL password. |
| `host_port` | ✅ | | PostgreSQL host URL. |
| `database` | | | PostgreSQL database. |
| `sqlalchemy_uri` | | | URI of database to connect to. See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls. Takes precedence over other connection parameters. |
| `database_alias` | | | Alias to apply to database when ingesting. |
| `env` | | `"PROD"` | Environment to use in namespace when constructing URNs. |
| `platform_instance` | | None | The Platform instance to use while constructing URNs. |
Expand Down
1 change: 1 addition & 0 deletions metadata-ingestion/source_docs/redshift.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@ Note that a `.` is used to denote nested fields in the YAML recipe.
| `password` | | | Redshift password. |
| `host_port` | ✅ | | Redshift host URL. |
| `database` | | | Redshift database. |
| `sqlalchemy_uri` | | | URI of database to connect to. See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls. Takes precedence over other connection parameters. |
| `database_alias` | | | Alias to apply to database when ingesting. |
| `env` | | `"PROD"` | Environment to use in namespace when constructing URNs. |
| `platform_instance` | | None | The Platform instance to use while constructing URNs. |
Expand Down
1 change: 1 addition & 0 deletions metadata-ingestion/source_docs/snowflake.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,7 @@ Note that a `.` is used to denote nested fields in the YAML recipe.
| `host_port` | ✅ | | Snowflake host URL. |
| `warehouse` | | | Snowflake warehouse. |
| `role` | | | Snowflake role. |
| `sqlalchemy_uri` | | | URI of database to connect to. See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls. Takes precedence over other connection parameters. |
| `env` | | `"PROD"` | Environment to use in namespace when constructing URNs. |
| `platform_instance` | | None | The Platform instance to use while constructing URNs. |
| `options.<option>` | | | Any options specified here will be passed to SQLAlchemy's `create_engine` as kwargs.<br />See https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine for details. |
Expand Down
1 change: 1 addition & 0 deletions metadata-ingestion/source_docs/trino.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ As a SQL-based service, the Trino integration is also supported by our SQL profi
| `password` | | | Trino password. |
| `host_port` | ✅ | | Trino host URL. |
| `database` | ✅ | | Trino database (catalog). |
| `sqlalchemy_uri` | | | URI of database to connect to. See https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls. Takes precedence over other connection parameters.
| `database_alias` | | | Alias to apply to database when ingesting. |
| `env` | | `"PROD"` | Environment to use in namespace when constructing URNs. |
| `options.<option>` | | | Any options specified here will be passed to SQLAlchemy's `create_engine` as kwargs.<br />See https://docs.sqlalchemy.org/en/14/core/engines.html#sqlalchemy.create_engine for details. |
Expand Down
14 changes: 9 additions & 5 deletions metadata-ingestion/src/datahub/ingestion/source/sql/sql_common.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -249,17 +249,21 @@ def get_sql_alchemy_url(self):
class BasicSQLAlchemyConfig(SQLAlchemyConfig):
username: Optional[str] = None
password: Optional[pydantic.SecretStr] = None
host_port: str
host_port: Optional[str] = None
database: Optional[str] = None
database_alias: Optional[str] = None
scheme: str
scheme: Optional[str] = None
sqlalchemy_uri: Optional[str] = None

def get_sql_alchemy_url(self, uri_opts=None):
return make_sqlalchemy_uri(
self.scheme,
if not ((self.host_port and self.scheme) or self.sqlalchemy_uri):
raise ValueError("host_port and schema or connect_uri required.")

return self.sqlalchemy_uri or make_sqlalchemy_uri(
self.scheme, # type: ignore
self.username,
self.password.get_secret_value() if self.password else None,
self.host_port,
self.host_port, # type: ignore
self.database,
uri_opts=uri_opts,
)
Expand Down

0 comments on commit 7c3ad3d

Please sign in to comment.