Support for BigQuery adapter #172
Conversation
…BigQuery dispatcher macros
|
Thank you @charles-astrafy ! Would you mind adding a description with any key design decisions or things to be aware of in this? That would be super helpful! |
|
Main design decisions:
All those changes are in commit 0d5e732. Other commits details: 8f8286b: Feature for incremental logic in staging tables. Nice one in case you want to materialize your staging tables; the logic is present in the SQL but won't be applied if you keep those models as views. Let me know if you need any further explanation and we can jump in a call if you prefer. |
README.md
Outdated
| You can also implement your own logic by overriding the default [generate_database_name](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/using-custom-databases) | ||
| and [generate_schema_name](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/using-custom-schemas) macros. | ||
|
|
||
| Note that the model materializations are defined in this package's `dbt_project.yml` as views. Incremental logic is set up in the | ||
| staging models and if you want to leverage this materialization, you need to specify the following configuration in the `dbt_project.yml` | ||
| of your project: | ||
|
|
||
| ```yml | ||
| models: | ||
| ... | ||
| dbt_artifacts: | ||
| staging: | ||
| +materialized: incremental | ||
| ... | ||
| ``` |
There was a problem hiding this comment.
Let's remove this as it's not within scope of the PR if ok :)
There was a problem hiding this comment.
sure ;) and will put it in a separate PR.
There was a problem hiding this comment.
@NiallRees : removed from the PR
|
Hey @charles-astrafy left a couple of comments and got the BQ integration tests running now. Let me know if I can be of any help to getting this over the line and I appreciate your patience! |
|
Thank you for your comments. Will block some time this weekend to work on it and will keep you posted as soon as those small changes are done. Great job for the BQ integration tests. |
|
|
||
| {% set insert_into_table_query %} | ||
| insert into {{database_name}}.{{ schema_name }}.{{ table_name }} | ||
| VALUES |
There was a problem hiding this comment.
Hi @charles-astrafy I was just playing around with your branch, and I got this query from from the statement that stores the tests (I've generalized the info yo you can play with it in your BQ instance):
insert into project.dbt_artifacts.tests
VALUES
select
1,
2,
3,
4,
parse_json(5),
6,
7,
parse_json(8)
from values
(
'061f741c-c350-4774-9343-c409f0518d48',
'test.project.test_name',
'2022-08-24 20:46:12.337589+00:00',
'test_name',
'["model.project.model_name"]',
'project',
'path/to/model.sql',
'[]'
)
However this query produces an error, saying that
Syntax error: INSERT target cannot have an alias at [2:1]
Is there something I'm doing wrong?
There was a problem hiding this comment.
@adrpino : I think you are doing smth wrong because the DML that insert values in the raw table for BigQuery does not use the syntax
insert into {{database_name}}.{{ schema_name }}.{{ table_name }} select ...... from values
but it uses instead the syntax:
insert into {{database_name}}.{{ schema_name }}.{{ table_name }} VALUES
So it seems you are not running the code with the BigQuery adapter as it is taking the code from the default adapter.
There was a problem hiding this comment.
Hi @charles-astrafy thanks!
is there any way in which I have to enforce the BQ adapter? isn't this done by default?
There was a problem hiding this comment.
Hi @adrpino ,
The adapter that dbt will use is defined via your profiles.yaml file. You define it in the "type" parameter of the target definition within that profiles.yaml file (see screenshot in attachment).
Let me know if you still have issues.
…deactivate on-run-end macro by providing a variable updon running a dbt command
… type is BigQuery. Remove as well unncessary helper logs
README.md
Outdated
| 4. In case you don't want the artifacts to be uploaded for certain dbt commands (for instance 'dbt compile'), add the following conditional: | ||
| `on-run-end: "{% if not var('dbt_artifacts_upload_results', False) %}{{ dbt_artifacts.upload_results(results) }}{% endif %}"` | ||
| If you then run the following command by providing a variable to the command, the 'on-run-end' will not execute: | ||
| `dbt compile --vars 'artifact: disable'` |
There was a problem hiding this comment.
Going to remove this for now but feel free to add to a separate PR @charles-astrafy
There was a problem hiding this comment.
Sure, will put it in a separate PR.
No description provided.