Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement initial adapter tests on bigquery #142

Merged
merged 3 commits into from
Mar 29, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
## dbt-bigquery 1.1.0 (TBD)

### Under the hood
- Use dbt.tests.adapter.basic in tests (new test framework) ([#135](https://github.com/dbt-labs/dbt-bigquery/issues/135), [#142](https://github.com/dbt-labs/dbt-bigquery/pull/142))

## dbt-bigquery 1.1.0b1 (March 23, 2022)
### Features
- Provide a fine-grained control of the timeout and retry of BigQuery query with four new dbt profile configs: `job_creation_timeout_seconds`, `job_execution_timeout_seconds`, `job_retry_deadline_seconds`, and `job_retries` ([#45](https://github.com/dbt-labs/dbt-bigquery/issues/45), [#50](https://github.com/dbt-labs/dbt-bigquery/pull/50))
Expand Down
15 changes: 15 additions & 0 deletions dbt/adapters/bigquery/impl.py
Original file line number Diff line number Diff line change
Expand Up @@ -849,3 +849,18 @@ def string_add_sql(
raise dbt.exceptions.RuntimeException(
f'Got an unexpected location value of "{location}"'
)

# This is used by the test suite
def run_sql_for_tests(self, sql, fetch, conn=None):
""" For the testing framework.
Run an SQL query on a bigquery adapter. No cursors, transactions,
etc. to worry about"""

do_fetch = fetch != 'None'
_, res = self.execute(sql, fetch=do_fetch)

# convert dataframe to matrix-ish repr
if fetch == 'one':
return res[0]
else:
return list(res)
3 changes: 2 additions & 1 deletion dev_requirements.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# install latest changes in dbt-core
# TODO: how to automate switching from develop to version branches?
git+https://github.com/dbt-labs/dbt.git#egg=dbt-core&subdirectory=core
git+https://github.com/dbt-labs/dbt-core.git#egg=dbt-core&subdirectory=core
git+https://github.com/dbt-labs/dbt-core.git#egg=dbt-tests-adapter&subdirectory=tests/adapter

bumpversion
flake8
Expand Down
10 changes: 10 additions & 0 deletions pytest.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
[pytest]
filterwarnings =
ignore:.*'soft_unicode' has been renamed to 'soft_str'*:DeprecationWarning
ignore:unclosed file .*:ResourceWarning
env_files =
test.env
testpaths =
tests/unit
tests/integration
tests/functional
22 changes: 22 additions & 0 deletions tests/conftest.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
import pytest
import os
import json

# Import the fuctional fixtures as a plugin
# Note: fixtures with session scope need to be local

pytest_plugins = ["dbt.tests.fixtures.project"]

# The profile dictionary, used to write out profiles.yml
@pytest.fixture(scope="class")
def dbt_profile_target():
credentials_json_str = os.getenv('BIGQUERY_TEST_SERVICE_ACCOUNT_JSON').replace("'", '')
credentials = json.loads(credentials_json_str)
project_id = credentials.get('project_id')
return {
'type': 'bigquery',
'method': 'service-account-json',
'threads': 1,
'project': project_id,
'keyfile_json': credentials,
}
52 changes: 52 additions & 0 deletions tests/functional/adapter/test_basic.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
import pytest

from dbt.tests.adapter.basic.test_base import BaseSimpleMaterializations
from dbt.tests.adapter.basic.test_singular_tests import BaseSingularTests
from dbt.tests.adapter.basic.test_singular_tests_ephemeral import (
BaseSingularTestsEphemeral,
)
from dbt.tests.adapter.basic.test_empty import BaseEmpty
from dbt.tests.adapter.basic.test_ephemeral import BaseEphemeral
from dbt.tests.adapter.basic.test_incremental import BaseIncremental
from dbt.tests.adapter.basic.test_generic_tests import BaseGenericTests
from dbt.tests.adapter.basic.test_snapshot_check_cols import BaseSnapshotCheckCols
from dbt.tests.adapter.basic.test_snapshot_timestamp import BaseSnapshotTimestamp


class TestSimpleMaterializationsBigQuery(BaseSimpleMaterializations):
# This test requires a full-refresh to replace a table with a view
Copy link
Contributor

@ChenyuLInx ChenyuLInx Mar 29, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider add this kind of logic to the dbt-core tests later on so that we don't need to do it anywhere in bigquery?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test_config sets a flag which does initiate different behavior in the dbt-core tests. I think we don't want to always do a full refresh in the adapter tests, because then we wouldn't ever be testing behavior without the full refresh, which ought to work for the other adapters.

@pytest.fixture(scope="class")
def test_config(self):
return {"require_full_refresh": True}


class TestSingularTestsBigQuery(BaseSingularTests):
pass


class TestSingularTestsEphemeralBigQuery(BaseSingularTestsEphemeral):
pass


class TestEmptyBigQuery(BaseEmpty):
pass


class TestEphemeralBigQuery(BaseEphemeral):
pass


class TestIncrementalBigQuery(BaseIncremental):
pass


class TestGenericTestsBigQuery(BaseGenericTests):
pass


class TestSnapshotCheckColsBigQuery(BaseSnapshotCheckCols):
pass


class TestSnapshotTimestampBigQuery(BaseSnapshotTimestamp):
pass
7 changes: 0 additions & 7 deletions tox.ini
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,3 @@ commands =
deps =
-rdev_requirements.txt
-e.

[pytest]
env_files =
test.env
testpaths =
tests/unit
tests/integration