Skip to content

Commit

Permalink
feat(bigquery): Add configurations for integration tests for BigQuery
Browse files Browse the repository at this point in the history
  • Loading branch information
charles-astrafy committed Aug 12, 2022
1 parent 40cc3ef commit 628f680
Show file tree
Hide file tree
Showing 4 changed files with 25 additions and 0 deletions.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,7 @@ pipx install tox
```
tox -e integration_snowflake # For the Snowflake tests
tox -e integration_databricks # For the Databricks tests
tox -e integration_bigquery # For the BigQuery tests
```
The Spark tests require installing the [ODBC driver](https://www.databricks.com/spark/odbc-drivers-download). On a Mac, DBT_ENV_SPARK_DRIVER_PATH should be set to `/Library/simba/spark/lib/libsparkodbc_sbu.dylib`. Spark tests have not yet been added to the integration tests.
Expand Down
2 changes: 2 additions & 0 deletions integration_test_project/example-env.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ export DBT_ENV_SECRET_DATABRICKS_HTTP_PATH=
export DBT_ENV_SECRET_DATABRICKS_TOKEN=
export DBT_ENV_SPARK_DRIVER_PATH= # /Library/simba/spark/lib/libsparkodbc_sbu.dylib on a Mac
export DBT_ENV_SPARK_ENDPOINT= # The endpoint ID from the Databricks HTTP path
export DBT_ENV_SECRET_BIGQUERY_TEST_DATABASE=
export DBT_ENV_SECRET_BIGQUERY_TEST_DATASET=

# dbt environment variables, change these
export DBT_CLOUD_PROJECT_ID=
Expand Down
11 changes: 11 additions & 0 deletions integration_test_project/profiles.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,3 +34,14 @@ dbt_artifacts:
endpoint: "{{ env_var('DBT_ENV_SPARK_ENDPOINT') }}"
token: "{{ env_var('DBT_ENV_SECRET_DATABRICKS_TOKEN') }}"
threads: 8
bigquery:
type: bigquery
method: oauth
project: "{{ env_var('DBT_ENV_SECRET_BIGQUERY_TEST_DATABASE') }}"
execution_project: "{{ env_var('DBT_ENV_SECRET_BIGQUERY_TEST_DATABASE') }}"
dataset: "{{ env_var('DBT_ENV_SECRET_BIGQUERY_TEST_DATASET') }}"
threads: 8
timeout_seconds: 300
location: EU
priority: interactive
retries: 1
11 changes: 11 additions & 0 deletions tox.ini
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,8 @@ passenv =
DBT_ENV_SECRET_DATABRICKS_HOST
DBT_ENV_SECRET_DATABRICKS_HTTP_PATH
DBT_ENV_SECRET_DATABRICKS_TOKEN
DBT_ENV_SECRET_BIGQUERY_TEST_DATABASE
DBT_ENV_SECRET_BIGQUERY_TEST_DATASET
DBT_ENV_SPARK_DRIVER_PATH
DBT_ENV_SPARK_ENDPOINT
DBT_CLOUD_PROJECT_ID
Expand Down Expand Up @@ -131,6 +133,15 @@ commands =
dbt build --exclude dbt_artifacts --target databricks
dbt build -s dbt_artifacts --target databricks

[testenv:integration_bigquery]
changedir = integration_test_project
deps = dbt-bigquery~=1.2.0
commands =
dbt deps
dbt run-operation create_dbt_artifacts_tables --target bigquery
dbt build --exclude dbt_artifacts --target bigquery
dbt build -s dbt_artifacts --target bigquery

[testenv:integration_spark]
changedir = integration_test_project
deps = dbt-spark[ODBC]~=1.1.0
Expand Down

0 comments on commit 628f680

Please sign in to comment.