anerobic nightly build fails in SimulationDaughterTask, also warnings

[1fish2]

The anerobic nightly build is failing in SimulationDaughterTask.

The build prints lots of warnings like this from SimulationTask and SimulationDaughterTask:

Warning: basis matrix is ill-conditioned (cond = 4.53e+12)

also a few times:

Warning: GLP_NOFEAS: no feasible error while solving FBA - repeating FBA solve

and a few times:

Warning: could not find solution with primal method, switching to dual

then the sim daughter fails in modular_fba:

/scratch/groups/mcovert/jenkins/workspace/wholecell/utils/polymerize.py:85: RuntimeWarning: invalid value encountered in true_divide
  self.elongation_rates = elongation_rates / np.max(elongation_rates)
Traceback (most recent call last):
  File "/home/groups/mcovert/pyenv/versions/wcEcoli3/lib/python3.8/site-packages/fireworks/core/rocket.py", line 262, in run
    m_action = t.run_task(my_spec)
  File "/scratch/groups/mcovert/jenkins/workspace/wholecell/fireworks/firetasks/simulationDaughter.py", line 89, in run_task
    sim.run()
  File "/scratch/groups/mcovert/jenkins/workspace/wholecell/sim/simulation.py", line 244, in run
    self.run_incremental(self._lengthSec + self.initialTime())
  File "/scratch/groups/mcovert/jenkins/workspace/wholecell/sim/simulation.py", line 276, in run_incremental
    self._evolveState(processes)
  File "/scratch/groups/mcovert/jenkins/workspace/wholecell/sim/simulation.py", line 345, in _evolveState
    process.evolveState()
  File "/scratch/groups/mcovert/jenkins/workspace/models/ecoli/processes/metabolism.py", line 159, in evolveState
    fba.solve(n_retries)
  File "/scratch/groups/mcovert/jenkins/workspace/wholecell/utils/modular_fba.py", line 1479, in solve
    self.solve(iterations - 1)
  File "/scratch/groups/mcovert/jenkins/workspace/wholecell/utils/modular_fba.py", line 1479, in solve
    self.solve(iterations - 1)
  File "/scratch/groups/mcovert/jenkins/workspace/wholecell/utils/modular_fba.py", line 1479, in solve
    self.solve(iterations - 1)
  File "/scratch/groups/mcovert/jenkins/workspace/wholecell/utils/modular_fba.py", line 1472, in solve
    self._solver._solve()
  File "/scratch/groups/mcovert/jenkins/workspace/wholecell/utils/_netflow/nf_glpk.py", line 478, in _solve
    raise RuntimeError(self.status_string)
RuntimeError: GLP_UNBND: unbounded

[1fish2]

The Anerobic build last passed on May 31 and started failing on June 1, implicating #1078 and #1079.

[tahorst]

The Anerobic build last passed on May 31 and started failing on June 1, implicating #1078 and #1079.

I think it's actually been failing since before that and was just masked by #1075. There are a bunch of directories in $PI_SCRATCH/wc_ecoli/failed/ with the last successful one being 3/1/21 with a bunch of failures before that as well.

In the failed gen, the metabolites plot shows Arg being depleted:

I used the metabolism debug tool to see which enzyme was missing to get Arg:

$ runscripts/debug/metabolism.py out/20210607.005825__Anaerobic./ -g4 --prod ARG[c] -t 10
Checking for production of ARG[c]:                                                                                                           
        N-ACETYLTRANSFER-CPLX[c] causes positive flux

N-ACETYLTRANSFER-CPLX is ArgA.

[tahorst]

It looks like the simulation part is completing now after merging in #1082. I'm not sure if it really solved the problem or if it just shifted the stochastic portion of the sim to produce enough ArgA. Even though the sims now complete, the last generation does not appear to be producing RNA - likely due to another enzyme limitation. Anaerobic conditions are often troubled by lack of enzyme production because the cells are so small and have fewer proteins leading to a greater chance of diluting out enzymes at division.

Output from last generation:

========  ========  ===========  ===========  ===========  ===========  ===========
Time (s)  Dry mass     Dry mass      Protein          RNA    Small mol     Expected
              (fg)  fold change  fold change  fold change  fold change  fold change
========  ========  ===========  ===========  ===========  ===========  ===========
54784.06    251.02        1.993        2.284        1.056        1.842        2.801
54786.06    251.06        1.993        2.285        1.056        1.842        2.801
54788.06    251.09        1.993        2.285        1.056        1.842        2.802
54790.06    251.13        1.994        2.285        1.056        1.843        2.803
54792.06    251.16        1.994        2.286        1.056        1.843        2.803
54794.06    251.20        1.994        2.286        1.056        1.843        2.804
54796.06    251.24        1.994        2.286        1.056        1.844        2.805
54798.06    251.27        1.995        2.287        1.056        1.844        2.805
54800.06    251.31        1.995        2.287        1.056        1.844        2.806
54802.06    251.34        1.995        2.287        1.056        1.845        2.807
54804.06    251.38        1.996        2.288        1.056        1.845        2.807
54806.06    251.42        1.996        2.288        1.056        1.845        2.808
54808.06    251.45        1.996        2.289        1.056        1.846        2.808
54810.06    251.49        1.996        2.289        1.056        1.846        2.809
54812.06    251.52        1.997        2.289        1.056        1.846        2.810
54814.06    251.56        1.997        2.290        1.056        1.846        2.810
54816.06    251.59        1.997        2.290        1.056        1.847        2.811
54818.06    251.63        1.997        2.290        1.056        1.847        2.812
54820.06    251.67        1.998        2.291        1.056        1.847        2.812
54822.06    251.70        1.998        2.291        1.056        1.848        2.813
54824.06    251.74        1.998        2.291        1.056        1.848        2.814
54826.06    251.77        1.999        2.292        1.056        1.848        2.814
54828.06    251.81        1.999        2.292        1.056        1.849        2.815
54830.06    251.85        1.999        2.292        1.056        1.849        2.816
54832.06    251.88        1.999        2.293        1.056        1.849        2.816
54834.06    251.92        2.000        2.293        1.056        1.850        2.817
54836.06    251.95        2.000        2.293        1.056        1.850        2.818
54838.06    251.99        2.000        2.294        1.056        1.850        2.818
54840.06    252.02        2.001        2.294        1.056        1.851        2.819
54842.06    252.06        2.001        2.294        1.056        1.851        2.820
54844.06    252.10        2.001        2.295        1.056        1.851        2.820
54846.06    252.13        2.001        2.295        1.056        1.852        2.821
54848.06    252.17        2.002        2.295        1.056        1.852        2.821
54850.06    252.20        2.002        2.296        1.056        1.852        2.822
54852.06    252.24        2.002        2.296        1.056        1.853        2.823
54854.06    252.28        2.003        2.297        1.056        1.853        2.823
54856.06    252.31        2.003        2.297        1.056        1.853        2.824
54858.06    252.35        2.003        2.297        1.056        1.853        2.825
54860.06    252.38        2.003        2.298        1.056        1.854        2.825
54862.06    252.42        2.004        2.298        1.056        1.854        2.826
54864.06    252.45        2.004        2.298        1.056        1.854        2.827
54866.06    252.49        2.004        2.299        1.056        1.855        2.827

Once #1073 is fixed, the build should be back to completing "successfully" even if we aren't getting RNA production.

[tahorst]

These issues could also suggest our doubling time for anaerobic growth is off which would lead to lower cell volumes and fewer enzymes.

From the paper that Markus recently shared, they have growth rates for aerobic and anaerobic growth:

Anaerobic growth
We next compared the proteome of NCM3722 cells growing exponentially in aerobic and anaerobic conditions in glucose minimal medium (Fig 7C). Anaerobic growth is expected to induce significant proteome rearrangements due to the much lower efficiency in energy generation. Indeed, we observed a total variation of about 25% of the proteome (Appendix Fig S11). However, the growth rates, 0.68/h and 0.91/h, respectively, were not very different.

NCM3722 is a K-12 derived strain. 0.68/h = 61 min doubling time, 0.91/h = 46 min doubling time. I remember KC also mentioned at one of our meetings that when their lab grew in anaerobic conditions, the growth rate was not significantly slower.

I think the 100 min doubling time (0.6 doublings/h) for anaerobic conditions used in our model was just selected as the slowest growing cell from Dennis and Bremer data but that was for succinate minimal media:

The media used for the data in Table 2 and Table 3 below include, with increasing growth rate, succinate minimal medium, glycerol minimal medium, glucose minimal medium, glucose-amino acids medium, and glucose-supplemented Luria-Bertani broth (LB medium). The growth rates observed in these media range between 0.6 and 3.0 doublings/h.

Maybe we should just update anaerobic conditions to use a 60 minute doubling time?