Skip to content

Commit

Permalink
better support for MongoDB Atlas with cloud workflows
Browse files Browse the repository at this point in the history
[I'll pause the MongoDB server in Google Cloud. For Vivarium uses we can resume it when needed, and for Fireworks workflows we can use Atlas servers.]

* The updated borealis-fireworks pip supports MongoDB Atlas servers via `uri_mode` and `=` chars in the host metadata strings, e.g. `host=mongodb+srv://.../ecoli1?retryWrites=true&w=majority`.
* Distinguish Fireworker VM names in case the DB name is not unique (`ecoli1` in this example).
* Allocate more RAM to the Fireworker VMs after Parca crashed once in `cvx`. RAM use must've spiked quickly since GCE Monitoring showed it ≈30%.
  ```
  terminate called after throwing an instance of 'std::bad_alloc'
    what():  std::bad_alloc
  Fatal Python error: Aborted
  ...
  Fatal Python error: Segmentation fault
  ```
  • Loading branch information
1fish2 authored May 21, 2021
1 parent e8b0a2c commit 768d8d7
Show file tree
Hide file tree
Showing 4 changed files with 12 additions and 13 deletions.
8 changes: 0 additions & 8 deletions .idea/codeStyles/Project.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ scipy==1.5.4
aesara==2.0.8
biopython==1.77
bokeh==1.4.0
borealis-fireworks==0.9.1
borealis-fireworks==0.11.0
confluent-kafka==1.4.2
cvxpy==1.1.10
Cython==0.29.21
Expand Down
6 changes: 5 additions & 1 deletion runscripts/cloud/util/workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -471,8 +471,12 @@ def copy_key(src, key, dest):
if val is not None:
dest[key] = val

# Construct a name prefix that distinguishes this group of fireworker
# VMs from groups that run workflows in other LaunchPad DBs so VM
# creation won't hit a name conflict.
db_name = config['name']
prefix = 'fireworker-{}'.format(db_name)
worker_key = f'{self.owner_id}-{db_name}' if self.owner_id != db_name else db_name
prefix = f'fireworker-{worker_key}'
options = {
'image-family': 'fireworker',
'network-interface': 'no-address', # no External IP
Expand Down
9 changes: 6 additions & 3 deletions runscripts/cloud/util/workflow_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,12 @@
from runscripts.cloud.util.workflow import (DEFAULT_LPAD_YAML,
STORAGE_ROOT_ENV_VAR, Task, Workflow)

#: 'custom-1-5120' selects a GCE N1 VM with 1 CPU and 5 GB RAM.
#: N1's max RAM per CPU is 6656 MB = 6.5 GB.
GCE_VM_MACHINE_TYPE = 'custom-1-5120'
#: custom-1-6656: N1 VM with 1 vCPU + 6.5 GB RAM (N1's max RAM per vCPU).
#: N2, N2D: Should be faster but they charge for 2+ vCPUs while our firetasks
#: peg 1 core at 100% and rarely use multiple cores. Worth testing.
#: e2-medium: 50% of each of 2 shared vCPUs, 100% in bursts. Lower cost. Does it
#: sustain 100% of 1 CPU? No control over processor type. Worth testing.
GCE_VM_MACHINE_TYPE = 'custom-1-6656'


class WorkflowCLI(scriptBase.ScriptBase):
Expand Down

0 comments on commit 768d8d7

Please sign in to comment.