Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Echo Scale Test VDS Updates [VS-1095] [VS-1112] #8602

Merged
merged 56 commits into from
Dec 15, 2023
Merged

Echo Scale Test VDS Updates [VS-1095] [VS-1112] #8602

merged 56 commits into from
Dec 15, 2023

Conversation

mcovarr
Copy link
Collaborator

@mcovarr mcovarr commented Dec 6, 2023 •
edited
Loading

Lessons learned in VDS creation during Echo Scale Testing.

Successful integration run here.

rsasch and others added 30 commits November 15, 2023 15:19
@rsasch
doc updates
26865ea
@rsasch
try to enable logging on hail dataproc cluster
54c1bdd
@rsasch
attempt #2 at parameters
a8f46ad
@rsasch
comment out cluster stopping code
71a1beb
@rsasch
Merge branch 'ah_var_store' into rsa_vs_1095
295ddc3
@rsasch
parameterize whether to kill the cluster
4a70e11
@rsasch
update docker
434ee75
@rsasch
tweaks after failed scale run
26ef66f
@rsasch
Merge branch 'ah_var_store' into rsa_vs_1095
e1d33ba
@rsasch
add property to reduce the frequency of cluster output
cb97808
@rsasch
grab the Env package
0443ef7
@rsasch
replaced False with empty string
f9fe1ad
@rsasch
different package import
d4f7f30
@rsasch
fix use of local_hail_log_path
076ca64
@rsasch
try import a slightly different way
7f7736a
@rsasch
put log path in different part of the code
c9810ac
@rsasch
RouterFS fix
fd04daa
@rsasch
Merge branch 'ah_var_store' into rsa_vs_1095
0090296
@rsasch
set shuffle flag
d0f1873
@mcovarr
highmem the driver
20910fd
@mcovarr
sanify cluster termination
5c3321e
@mcovarr
Intermediate start point support
2610d11
@mcovarr
whoops
fe91737
@mcovarr
docker
9fb4ac9
@mcovarr
empty
1aa3aa5
@mcovarr
master memory, message cluster non-shutdown
8b0f652
@mcovarr
docker
657cd48
@mcovarr
Docker
ff12cde
@mcovarr
remove idle limit
38c55c6
@mcovarr
Docker
b9aff88
@mcovarr mcovarr changed the title Rsa vs 1095 Echo Scale Test VDS Update [VS-1095] [VS-1112] Dec 6, 2023
@mcovarr mcovarr changed the title Echo Scale Test VDS Update [VS-1095] [VS-1112] Echo Scale Test VDS Updates [VS-1095] [VS-1112] Dec 6, 2023
mcovarr and others added 9 commits December 6, 2023 12:33
@mcovarr
Merge remote-tracking branch 'origin/ah_var_store' into rsa_vs_1095
9e6b19a
@chrisvittal
Use combiner's calculate_new_intervals for target_final_intervals (#8603
2734df5 
)

This should work around a rare error in `_calculate_new_intervals` that
could generate invalid partitioners in a way that
`calculate_new_intervals` cannot.
@chrisvittal
Do not delete raw combiner output (#8604)
d1cb2a2 
If the combiner finishes, but import_gvs fails, and this script is run
again, the combiner output will be deleted and the finished plan will be
used causing no new output to be generated causing the script to fail.
@mcovarr
make things optional for validation
c4ec6e4
@mcovarr
docker
6c22481
@danking
Update import_gvs.py (#8617)
f9d2342
@mcovarr
update Docker
083dea0
@mcovarr
Merge remote-tracking branch 'origin/ah_var_store' into rsa_vs_1095
2476d08
@mcovarr
docker
bb8ce1d
@mcovarr mcovarr marked this pull request as ready for review December 14, 2023 23:09
gbggrant
gbggrant approved these changes Dec 15, 2023
View reviewed changes
Copy link
Collaborator

@gbggrant gbggrant left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

danking
danking approved these changes Dec 15, 2023
View reviewed changes
Copy link

@danking danking left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This all looks right to me. The big fixes are in:

  1. No rmtree before vds combiner.
  2. Print exit code rather than wait status.
  3. Make max age and max idle configurable with a default of infinity.
  4. Support restarting half-way through Avro importing by way of intermediate_resume_point.
  5. Copy Hail log off driver to well-known location.
  6. Use VDS's calculate_new_intervals which correctly constructs intervals for the variants data (key: locus, alleles) based on the density of reference data (key: locus).

@mcovarr mcovarr merged commit f8d5f2d into ah_var_store Dec 15, 2023
20 checks passed
@mcovarr mcovarr deleted the rsa_vs_1095 branch December 15, 2023 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@danking danking danking approved these changes

@gbggrant gbggrant gbggrant approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@mcovarr @danking @gbggrant @rsasch @chrisvittal