Skip to content

Commit

Permalink
Add spill bucket to workflow and edit readme
Browse files Browse the repository at this point in the history
  • Loading branch information
ejeffrli committed Oct 5, 2023
1 parent 6130304 commit 1da60e0
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 2 deletions.
1 change: 1 addition & 0 deletions .github/workflows/run_release_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,4 @@ jobs:
DATABASE_PASSWORD: ${{ secrets.DATABASE_PASSWORD }}
S3_DATA_PATH: ${{ secrets.S3_DATA_PATH }}
S3_JARS_BUCKET: ${{ secrets.S3_JARS_BUCKET }}
SPILL_BUCKET: ${{ secrets.SPILL_BUCKET }}
4 changes: 2 additions & 2 deletions validation_testing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,8 @@ export REPOSITORY_ROOT=<base dir of repository>
export SPILL_BUCKET=<spill bucket name (not full s3 path)>
```

Then, run `python3 main.py`. If you want to change what connectors are tested, modify the array of connectors in `main.py`, but you can only use ones that already have a corresponding CDK stack and glue job ready.
Then, run `python3 main.py` to test all connectors. If you want to change what connectors are tested, modify the array of connectors in `main.py`, but you can only use ones that already have a corresponding CDK stack and glue job ready.

### Local Development

Since `python3 main.py` runs all available connectors, `sh run_release_tests.sh {CONNECTOR_NAME}` was split up to make it more accessible for local development. Within ./scripts directory, first run `sh deploy_infra.sh {CONNECTOR_NAME}`, then run `sh run_glue.sh {CONNECTOR_NAME}` to get sample data. Then you can run `sh run_tests.sh {CONNECTOR_NAME}` repeatedly for development. When you are finished, clean up by running `sh cleanup.sh {CONNECTOR_NAME}`. To add/remove tests, modify ./python_scripts/exec_release_test_queries.py
`sh run_release_tests.sh {CONNECTOR_NAME}` was split up to make it more accessible for local development. First, run `sh deploy_infra.sh {CONNECTOR_NAME}` to deploy database, connector, and etl jobs, then run `sh run_glue.sh {CONNECTOR_NAME}` to load test data (tpcds customer and customer_address tables). Then, run `sh run_tests.sh {CONNECTOR_NAME}` repeatedly for development. Run `sh cleanup.sh {CONNECTOR_NAME}` to cleanup resources. To add/remove tests, modify ./scripts/exec_release_test_queries.py

0 comments on commit 1da60e0

Please sign in to comment.