Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ingest): default values for env #2598

Merged
merged 4 commits into from
May 24, 2021
Merged

fix(ingest): default values for env #2598

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7ed3d58
Update env
kevinhu May 24, 2021
04e27b8
Update README with env
kevinhu May 24, 2021
e8cf7a4
Black
kevinhu May 24, 2021
03fc669
Pin sqlalchemy version
kevinhu May 24, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion metadata-ingestion/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ We use a plugin architecture so that you can install only the dependencies you a
| redshift | `pip install 'acryl-datahub[redshift]'` | Redshift source |
| sqlalchemy | `pip install 'acryl-datahub[sqlalchemy]'` | Generic SQLAlchemy source |
| snowflake | `pip install 'acryl-datahub[snowflake]'` | Snowflake source |
| superset | `pip install 'acryl-datahub[superset]'` | Superset source |
| superset | `pip install 'acryl-datahub[superset]'` | Superset source |
| mongodb | `pip install 'acryl-datahub[mongodb]'` | MongoDB source |
| ldap | `pip install 'acryl-datahub[ldap]'` ([extra requirements]) | LDAP source |
| looker | `pip install 'acryl-datahub[looker]'` | Looker source |
Expand Down Expand Up @@ -340,6 +340,7 @@ source:
password: pass
provider: db | ldap
connect_uri: http://localhost:8088
env: "PROD" # Optional, default is "PROD"
```

See documentation for superset's `/security/login` at https://superset.apache.org/docs/rest-api for more details on superset's login api.
Expand Down Expand Up @@ -496,6 +497,7 @@ source:
connect_uri: "mongodb://localhost"
username: admin
password: password
env: "PROD" # Optional, default is "PROD"
authMechanism: "DEFAULT"
options: {}
database_pattern: {}
Expand Down Expand Up @@ -640,6 +642,7 @@ source:
```

Current limitations:

- Currently works only for Debezium source connectors.

## Sinks
Expand Down
2 changes: 1 addition & 1 deletion metadata-ingestion/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ def get_long_description():

sql_common = {
# Required for all SQL sources.
"sqlalchemy>=1.3.24",
"sqlalchemy==1.3.24",
}

# Note: for all of these, framework_common will be added.
Expand Down
6 changes: 4 additions & 2 deletions metadata-ingestion/src/datahub/ingestion/source/mongodb.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@
# https://stackoverflow.com/a/48273736/5004662.
DENY_DATABASE_LIST = set(["admin", "config", "local"])

DEFAULT_ENV = "PROD"


class MongoDBConfig(ConfigModel):
# See the MongoDB authentication docs for details and examples.
Expand All @@ -52,6 +54,7 @@ class MongoDBConfig(ConfigModel):
options: dict = {}
enableSchemaInference: bool = True
schemaSamplingSize: Optional[PositiveInt] = 1000
env: str = DEFAULT_ENV

database_pattern: AllowDenyPattern = AllowDenyPattern.allow_all()
collection_pattern: AllowDenyPattern = AllowDenyPattern.allow_all()
Expand Down Expand Up @@ -392,7 +395,6 @@ def get_field_type(
return SchemaFieldDataType(type=TypeClass())

def get_workunits(self) -> Iterable[MetadataWorkUnit]:
env = "PROD"
platform = "mongodb"

database_names: List[str] = self.mongo_client.list_database_names()
Expand All @@ -417,7 +419,7 @@ def get_workunits(self) -> Iterable[MetadataWorkUnit]:
continue

dataset_snapshot = DatasetSnapshot(
urn=f"urn:li:dataset:(urn:li:dataPlatform:{platform},{dataset_name},{env})",
urn=f"urn:li:dataset:(urn:li:dataPlatform:{platform},{dataset_name},{self.config.env})",
aspects=[],
)

Expand Down
6 changes: 4 additions & 2 deletions metadata-ingestion/src/datahub/ingestion/source/superset.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,8 @@ def get_platform_from_sqlalchemy_uri(sqlalchemy_uri: str) -> str:
"box_plot": ChartTypeClass.BAR,
}

DEFAULT_ENV = "PROD"


class SupersetConfig(ConfigModel):
# See the Superset /security/login endpoint for details
Expand All @@ -83,6 +85,7 @@ class SupersetConfig(ConfigModel):
password: Optional[str] = None
provider: str = "db"
options: dict = {}
env: str = DEFAULT_ENV


def get_metric_name(metric):
Expand Down Expand Up @@ -110,7 +113,6 @@ def get_filter_name(filter_obj):
class SupersetSource(Source):
config: SupersetConfig
report: SourceReport
env = "PROD"
platform = "superset"

def __hash__(self):
Expand Down Expand Up @@ -181,7 +183,7 @@ def get_datasource_urn_from_id(self, datasource_id):
f"urn:li:dataset:("
f"{platform_urn},{database_name + '.' if database_name else ''}"
f"{schema_name + '.' if schema_name else ''}"
f"{table_name},{self.env})"
f"{table_name},{self.config.env})"
)
return dataset_urn
return None
Expand Down