Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added unique_key to the test_docs_generate places relevant to changes… #114

Merged
merged 23 commits into from
Feb 22, 2022
Merged

added unique_key to the test_docs_generate places relevant to changes… #114

Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
ecf59ff
added unique_key to the test_docs_generate places relevant to changes…
McKnight-42 Feb 4, 2022
0424a9e
Merge branch 'main' of https://github.com/dbt-labs/dbt-bigquery into …
McKnight-42 Feb 7, 2022
25ca18f
init attempt at test, for unique key, getting type error for hashable…
McKnight-42 Feb 7, 2022
6a03501
moved to using basic hash method, and changing up calls and saving of…
McKnight-42 Feb 8, 2022
f984df2
fixed minor errors have all models building properly including increm…
McKnight-42 Feb 8, 2022
bc83866
remove items from docs_generate.py objects to move to secondary pr
McKnight-42 Feb 9, 2022
df08072
using assertTableEqual
McKnight-42 Feb 9, 2022
2de84a2
added new models and test for single unique key
McKnight-42 Feb 9, 2022
55a8d22
updating to main
McKnight-42 Feb 9, 2022
57657db
adding tests for single unique key, and removing unused import
McKnight-42 Feb 11, 2022
0b9492f
removing import
McKnight-42 Feb 11, 2022
f4625f6
add first draft for changelog which links to bigquery issue which poi…
McKnight-42 Feb 11, 2022
063a49d
Merge branch 'main' of https://github.com/dbt-labs/dbt-bigquery into …
McKnight-42 Feb 11, 2022
a519c2c
Merge branch 'main' into mcknight/CT-116
McKnight-42 Feb 11, 2022
1cef0c6
working on new version of unique_key as list test in incremental_sche…
McKnight-42 Feb 15, 2022
03a2e2a
Merge branch 'mcknight/CT-116' of https://github.com/dbt-labs/dbt-big…
McKnight-42 Feb 15, 2022
4688b90
comments and docstring addition
McKnight-42 Feb 16, 2022
e3c5bcb
beginning transition to unify tests between adapters, only issue seem…
McKnight-42 Feb 17, 2022
86c9373
applied castings to expected tables as bigquery auto detects certain …
McKnight-42 Feb 18, 2022
81cb8ca
minor fixes
McKnight-42 Feb 18, 2022
feddfda
minor naming changes and removal of file from older tests
McKnight-42 Feb 22, 2022
5561b35
updates implementing mila final versions to remain similar
McKnight-42 Feb 22, 2022
ac33d1b
removing unnecessary line
McKnight-42 Feb 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
## dbt-bigquery 1.1.0 (Release TBD)
### Features
- Provide a fine-grained control of the timeout and retry of BigQuery query with four new dbt profile configs: `job_creation_timeout_seconds`, `job_execution_timeout_seconds`, `job_retry_deadline_seconds`, and `job_retries` ([#45](https://github.com/dbt-labs/dbt-bigquery/issues/45), [#50](https://github.com/dbt-labs/dbt-bigquery/pull/50))
- Adds new integration test to check against new ability to allow unique_key to be a list. [#112](https://github.com/dbt-labs/dbt-bigquery/issues/112)

### Fixes
- Fix test related to preventing coercion of boolean values (True, False) to numeric values (0, 1) in query results ([#93](https://github.com/dbt-labs/dbt-bigquery/issues/93))
Expand Down
1 change: 1 addition & 0 deletions tests/integration/incremental_schema_tests/test_incremental_schema.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,3 +140,4 @@ def test__bigquery__run_incremental_sync_all_columns(self):
@use_profile('bigquery')
def test__bigquery__run_incremental_fail_on_schema_change(self):
self.run_incremental_fail_on_schema_change()

McKnight-42 marked this conversation as resolved.
Show resolved Hide resolved
17 changes: 17 additions & 0 deletions tests/integration/incremental_unique_id_test/models/duplicated_unary_unique_key_list.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{{
config(
materialized='incremental',
unique_key=['state', 'state']
)
}}

select
state as state,
county as county,
city as city,
last_visit_date as last_visit_date
from {{ ref('seed') }}

{% if is_incremental() %}
where last_visit_date > (select max(last_visit_date) from {{ this }})
{% endif %}
14 changes: 14 additions & 0 deletions tests/integration/incremental_unique_id_test/models/empty_str_unique_key.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{{
config(
materialized='incremental',
unique_key=''
)
}}

select
*
from {{ ref('seed') }}

{% if is_incremental() %}
where last_visit_date > (select max(last_visit_date) from {{ this }})
{% endif %}
12 changes: 12 additions & 0 deletions tests/integration/incremental_unique_id_test/models/empty_unique_key_list.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{{
config(
materialized='incremental',
unique_key=[]
)
}}

select * from {{ ref('seed') }}

{% if is_incremental() %}
where last_visit_date > (select max(last_visit_date) from {{ this }})
{% endif %}
21 changes: 21 additions & 0 deletions tests/integration/incremental_unique_id_test/models/expected/one_str__overwrite.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{{
config(
materialized='table'
)
}}

select
'CT' as state,
'Hartford' as county,
'Hartford' as city,
cast('2022-02-14' as date) as last_visit_date
union all
select 'MA','Suffolk','Boston','2020-02-12'
union all
select 'NJ','Mercer','Trenton','2022-01-01'
union all
select 'NY','Kings','Brooklyn','2021-04-02'
union all
select 'NY','New York','Manhattan','2021-04-01'
union all
select 'PA','Philadelphia','Philadelphia','2021-05-21'
21 changes: 21 additions & 0 deletions ...gration/incremental_unique_id_test/models/expected/unique_key_list__inplace_overwrite.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{{
config(
materialized='table'
)
}}

select
'CT' as state,
'Hartford' as county,
'Hartford' as city,
cast('2022-02-14' as date) as last_visit_date
union all
select 'MA','Suffolk','Boston','2020-02-12'
union all
select 'NJ','Mercer','Trenton','2022-01-01'
union all
select 'NY','Kings','Brooklyn','2021-04-02'
union all
select 'NY','New York','Manhattan','2021-04-01'
union all
select 'PA','Philadelphia','Philadelphia','2021-05-21'
13 changes: 13 additions & 0 deletions tests/integration/incremental_unique_id_test/models/no_unique_key.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{{
config(
materialized='incremental'
)
}}

select
*
from {{ ref('seed') }}

{% if is_incremental() %}
where last_visit_date > (select max(last_visit_date) from {{ this }})
{% endif %}
19 changes: 19 additions & 0 deletions tests/integration/incremental_unique_id_test/models/nontyped_trinary_unique_key_list.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
-- for comparing against auto-typed seeds

{{
config(
materialized='incremental',
unique_key=['state', 'county', 'city']
)
}}

select
state as state,
county as county,
city as city,
last_visit_date as last_visit_date
from {{ ref('seed') }}

{% if is_incremental() %}
where last_visit_date > (select max(last_visit_date) from {{ this }})
{% endif %}
14 changes: 14 additions & 0 deletions tests/integration/incremental_unique_id_test/models/not_found_unique_key.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{{
config(
materialized='incremental',
unique_key='asdf'
)
}}

select
*
from {{ ref('seed') }}

{% if is_incremental() %}
where last_visit_date > (select max(last_visit_date) from {{ this }})
{% endif %}
8 changes: 8 additions & 0 deletions tests/integration/incremental_unique_id_test/models/not_found_unique_key_list.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{{
config(
materialized='incremental',
unique_key=['state', 'asdf']
)
}}

select * from {{ ref('seed') }}
17 changes: 17 additions & 0 deletions tests/integration/incremental_unique_id_test/models/str_unique_key.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{{
config(
materialized='incremental',
unique_key='state'
)
}}

select
state as state,
county as county,
city as city,
last_visit_date as last_visit_date
from {{ ref('seed') }}

{% if is_incremental() %}
where last_visit_date > (select max(last_visit_date) from {{ this }})
{% endif %}
19 changes: 19 additions & 0 deletions tests/integration/incremental_unique_id_test/models/trinary_unique_key_list.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
-- types needed to compare against expected model reliably

{{
config(
materialized='incremental',
unique_key=['state', 'county', 'city']
)
}}

select
state as state,
county as county,
city as city,
last_visit_date as last_visit_date
from {{ ref('seed') }}

{% if is_incremental() %}
where last_visit_date > (select max(last_visit_date) from {{ this }})
{% endif %}
17 changes: 17 additions & 0 deletions tests/integration/incremental_unique_id_test/models/unary_unique_key_list.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{{
config(
materialized='incremental',
unique_key=['state']
)
}}

select
state as state,
county as county,
city as city,
last_visit_date as last_visit_date
from {{ ref('seed') }}

{% if is_incremental() %}
where last_visit_date > (select max(last_visit_date) from {{ this }})
{% endif %}
21 changes: 21 additions & 0 deletions tests/integration/incremental_unique_id_test/models/unique_key_list__no_update.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{{
config(
materialized='table'
)
}}

select
'CT' as state,
'Hartford' as county,
'Hartford' as city,
'2022-02-14' as last_visit_date
union all
select 'MA','Suffolk','Boston','2020-02-12'
union all
select 'NJ','Mercer','Trenton','2022-01-01'
union all
select 'NY','Kings','Brooklyn','2021-04-02'
union all
select 'NY','New York','Manhattan','2021-04-01'
union all
select 'PA','Philadelphia','Philadelphia','2021-05-21'
9 changes: 9 additions & 0 deletions tests/integration/incremental_unique_id_test/seeds/add_new_rows.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
-- insert two new rows, both of which should be in incremental model
-- with any unique columns
insert into {schema}.seed
(state, county, city, last_visit_date)
values ('WA','King','Seattle','2022-02-01');

insert into {schema}.seed
(state, county, city, last_visit_date)
values ('CA','Los Angeles','Los Angeles','2022-02-01');
5 changes: 5 additions & 0 deletions tests/integration/incremental_unique_id_test/seeds/duplicate_insert.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
-- insert new row, which should not be in incremental model
-- with primary or first three columns unique
insert into {schema}.seed
(state, county, city, last_visit_date)
values ('CT','Hartford','Hartford','2022-02-14');
7 changes: 7 additions & 0 deletions tests/integration/incremental_unique_id_test/seeds/seed.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
state,county,city,last_visit_date
CT,Hartford,Hartford,2020-09-23
MA,Suffolk,Boston,2020-02-12
NJ,Mercer,Trenton,2022-01-01
NY,Kings,Brooklyn,2021-04-02
NY,New York,Manhattan,2021-04-01
PA,Philadelphia,Philadelphia,2021-05-21
Loading