Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dbt docs generate --select "source:athena" --profile athena fails with An error occurred (InvalidRequestException) when calling the GetDataCatalog operation: DataCatalog xxxxxx_xxxx was not found. #748

Open
2 tasks done
soham-dasgupta opened this issue Jan 17, 2025 · 0 comments
Labels
pkg:dbt-athena Issue affects dbt-athena

Comments

@soham-dasgupta
Copy link

Is this a new bug in dbt-athena?

  • I believe this is a new bug in dbt-athena
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I have two major databases in my dbt project sources - Athena and Trino and two targets in projects profiles as shown below

version: 2

sources:
  - name: trino
    database: trino
    schema: trino_prod
    tables:
      - name: xxxxxx
  - name: athena
    database: AwsDataCatalog
    schema: athena
    tables:
      - name: xxxxx
      - 
trino:
  target: prod
  outputs:
    prod:
      type: trino
      host: xxx.xxx.xxx.xxx
      user: xxx
      password: xxx
      catalog: xxx
      port: 8080
      schema: xxx
      threads: 1
      method: ldap
      http_scheme: https
athena:
  target: prod
  outputs:
    prod:
      type: athena
      region_name: us-east-1
      schema: xxx
      database: awsdatacatalog
      aws_profile_name: xxx-xxx-xxx
      work_group: xxx-xx-xxx
      s3_staging_dir: s3://xxx
      s3_data_dir: s3://xxxx/xxx
      s3_data_naming: table
      s3_tmp_table_dir: s3://xxxxxx
      poll_interval: 10

I can use dbt docs generate --select "source:trino" --profile trino to limit generating documentation for trino but I can't do the same for dbt docs generate --select "source:athena" --profile athena as this leads to generating documentation for source:trino ultimately leading to failure

13:26:16  Encountered an error while generating catalog: An error occurred (InvalidRequestException) when calling the GetDataCatalog operation: DataCatalog xxxxxx_xxxx was not found.
13:26:16  Encountered an error while generating catalog: An error occurred (InvalidRequestException) when calling the GetDataCatalog operation: DataCatalog xxxxxxxx_xxxx_xxxxxx was not found.

Expected Behavior

When I run dbt docs generate --select "source:athena" --profile athena I would like to generate the documentation for objects that depend on "source:athena"

Steps To Reproduce

  1. Create a sources.yml that has Athena and Trino sources
  2. Create a profiles.yml that has Athena and Trino targets
  3. Run dbt docs generate --select "source:athena" --profile athena and it tries to generate the metadata for databases accessible via Trino as well.

Relevant log output

13:26:16 Encountered an error while generating catalog: An error occurred (InvalidRequestException) when calling the GetDataCatalog operation: DataCatalog xxxxxx_xxxx was not found.
13:26:16 Encountered an error while generating catalog: An error occurred (InvalidRequestException) when calling the GetDataCatalog operation: DataCatalog xxxxxxxx_xxxx_xxxxxx was not found.

Environment

- OS: AL2
- Python: Python 3.11.11
- dbt-core: 1.7.15
- dbt-athena: 1.7.2
- dbt-trino: 1.7.1

Additional Context

No response

@mikealfare mikealfare added the pkg:dbt-athena Issue affects dbt-athena label Jan 27, 2025
@mikealfare mikealfare transferred this issue from dbt-labs/dbt-athena Jan 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pkg:dbt-athena Issue affects dbt-athena
Projects
None yet
Development

No branches or pull requests

2 participants