Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pass region param on to redshift conn as is #485

Merged
merged 15 commits into from
Jun 14, 2023
Merged

Conversation

colin-rogers-dbt
Copy link
Contributor

@colin-rogers-dbt colin-rogers-dbt commented Jun 7, 2023

resolves #484 resolves #483

Description

Region is a non-obvious semi-requirement post 1.5 (you don't need to specify it if it's in your hostname), this has been a headache for various users and it's not clear that it's strictly speaking necessary.

This change makes it a truly optional requirement for dbt-redshift, this removes duplication with the underlying redshift-connector to properly manage and validate this parameter.

Checklist

@cla-bot cla-bot bot added the cla:yes label Jun 7, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Jun 7, 2023

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-redshift contributing guide.

@colin-rogers-dbt colin-rogers-dbt self-assigned this Jun 9, 2023
@colin-rogers-dbt colin-rogers-dbt marked this pull request as ready for review June 12, 2023 18:52
@colin-rogers-dbt colin-rogers-dbt requested a review from a team as a code owner June 12, 2023 18:52
@colin-rogers-dbt colin-rogers-dbt requested a review from Fleid June 12, 2023 18:52
@colin-rogers-dbt colin-rogers-dbt changed the title Explore not requiring a region Do not require region in the hostname Jun 12, 2023
@colin-rogers-dbt colin-rogers-dbt changed the title Do not require region in the hostname Pass region param on to redshift conn as is Jun 13, 2023
@dbeatty10
Copy link
Contributor

Loving how this is looking @colin-rogers-dbt 🤩

Since we're removing the region logic entirely in 2dfa187, we'll need to open an issue to update docs for this page:
https://docs.getdbt.com/docs/core/connect-data-platform/redshift-setup

Specifically, to make this change in a couple different places:

-      region: # optional, if not provided, will be determined from host (e.g. host.123.us-east-1.redshift-serverless.amazonaws.com)
+      region: # optional

@dbeatty10
Copy link
Contributor

As a follow-up to #485 (comment), opened up a PR to update the docs here:
dbt-labs/docs.getdbt.com#3519

@alison985
Copy link

👋 Hi. A few questions:

  1. Will this work for Cloud IDE customers? Both [ADAP-500] [Bug] dbt-redshift 1.5 does not work with non-standard redshift hostnames #419 and [ADAP-609] [CT-2617] [Bug] Database Error Invalid region provided: net #483 mention profiles.yml, but I don't have one since they don't work with dbt Cloud. I am facing the Invalid region provided: 0 error with Redshift when moving from dbt v1.2 to dbt v1.5 (latest).
  2. Where would I provide region within the dbt Cloud interface?
  3. My read of the code in this PR says it's removing inferring the region from the host. If region is a required parameter for the new redshift connection library, redshift-connector, then why remove a fallback safety?
  4. If region is not provided to the connection library then what are the risks?

@colin-rogers-dbt
Copy link
Contributor Author

colin-rogers-dbt commented Jun 14, 2023

  1. Will this work for Cloud IDE customers?

This change is specifically for customers (like those using the Cloud IDE) who cannot pass the region parameter at this time.

  1. Where would I provide region within the dbt Cloud interface?

For now you will not provide this parameter.

  1. My read of the code in this PR says it's removing inferring the region from the host. If region is a required parameter for the new redshift connection library, redshift-connector, then why remove a fallback safety? 4. If region is not provided to the connection library then what are the risks?

The region parameter is only required if using IAM auth so for Cloud IDE users this poses no risk. This change has been made after discussion with the maintainers of redshift-connector (where they are also doing this fallback inference in some places) with the decision that we don't want to maintain logic that should be handled by that package.

dataders added a commit to dbt-labs/docs.getdbt.com that referenced this pull request Jun 14, 2023
Previews:
- [Password-based
authentication](https://deploy-preview-3519--docs-getdbt-com.netlify.app/docs/core/connect-data-platform/redshift-setup#password-based-authentication)
- [IAM
Authentication](https://deploy-preview-3519--docs-getdbt-com.netlify.app/docs/core/connect-data-platform/redshift-setup#iam-authentication)

## What are you changing in this pull request and why?
dbt-labs/dbt-redshift#485 is updating handling
of the `region` parameter for dbt-redshift to be completely optional
without additional side-effects (like attempting to parse it from the
host name).

## Checklist
- [x] Review the [Content style
guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md)
and [About
versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version)
so my content adheres to these guidelines.
@colin-rogers-dbt colin-rogers-dbt merged commit cbb45ca into main Jun 14, 2023
@colin-rogers-dbt colin-rogers-dbt deleted the stopRequiringRegion branch June 14, 2023 19:34
Copy link
Contributor

@VersusFacit VersusFacit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Late approval :p

github-actions bot pushed a commit that referenced this pull request Jun 15, 2023
* convert test_store_test_failures to functional test

* test not requiring a region in hostname

* remove dev-requirements.txt change

* add unit test and clean up logic

* add changie

* simplify region inference logic

* remove region logic entirely

(cherry picked from commit cbb45ca)
colin-rogers-dbt added a commit that referenced this pull request Jun 15, 2023
* convert test_store_test_failures to functional test

* test not requiring a region in hostname

* remove dev-requirements.txt change

* add unit test and clean up logic

* add changie

* simplify region inference logic

* remove region logic entirely

(cherry picked from commit cbb45ca)

Co-authored-by: colin-rogers-dbt <[email protected]>
@colin-rogers-dbt colin-rogers-dbt mentioned this pull request Jun 20, 2023
6 tasks
abbywh pushed a commit to abbywh/dbt-redshift that referenced this pull request Oct 11, 2023
* convert test_store_test_failures to functional test

* test not requiring a region in hostname

* remove dev-requirements.txt change

* add unit test and clean up logic

* add changie

* simplify region inference logic

* remove region logic entirely
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
6 participants