gatk-sv-single-sample workflow (v.0.9.1-beta-hotfix) failing at RunWhamgIncludelist subtask

[riazgillani]

Bug Report

Affected module(s) or script(s)

gatk-sv-single-sample workflow, Module00a, Whamg task, RunWhamgIncludelist subtask

Affected version(s)

v.0.9.1-beta-hotfix

Description

I've successfully run this version of the workflow to call germline SVs on WGS CRAMs using a modified version of "GATKSVPipelineSingleSample.ref_panel_1kg.na12878.no_melt.json" for inputs. Over the last week, I've encountered recurrent failures in this workflow on multiple samples (including those on which this has previously run successfully), with the following error in the RunWhamgIncludelist subtask: "cat: /sys/fs/cgroup/memory/memory.stat: No such file or directory". I've attached the full log for broader context on this error.

Expected behavior

The gatk-sv-single-sample workflow should run to completion.

Actual behavior

The gatk-sv-single-sample workflow fails at this step. I've confirmed that this fails on samples on which the workflow has run successfully before, as well as new samples. I am providing a cram and matching crai for input, and I have not changed any other inputs to this workflow, so am having a hard time deciphering what may be contributing to this failure. Is it possible that the docker for this task has changed in some way?

gatk_sv_single_sample_whamg_log_4_11_22.txt

[riazgillani]

From what I am finding online, this error seems to occur when there is a change to the virtual environment. This may be an error inherent to the docker image used for this task ("gatksv/wham:8645aa"). Thank you for any direction you are able to provide here

[cwhelan]

It seems like something might have changed in the way some GCE VMs are configured such that this memory.stat file is no longer in the same location. The line in question is only used for debugging memory issues occasionally encountered by Wham so it should be safe to remove from the WDL file, if you are able to modify it. We will remove the line in question and test out the change as well.

[riazgillani]

Thanks Chris - the WDL I have is encapsulated, so I don't believe I can access and modify the command line. I appreciate your help looking into this

[epiercehoffman]

Fixed as part of #333 by deleting the cat /sys/fs/cgroup/memory/memory.stat line in RunWhamgIncludeList and RunWhamg. You can use the latest release v0.21-beta along with the updated docker images associated with that release (see inputs/values/dockers.json, or follow instructions in the README to build the input JSONs from templates), or if you need to keep the version consistent, you can edit the Whamg.wdl file to delete the problematic line.

[riazgillani]

Thank you for your help. I have a few questions:

1. Where would I find the v0.21-beta release for import into Terra? I can't find it in the dockstore here https://dockstore.org/workflows/github.com/broadinstitute/gatk-sv/gatk-sv-single-sample

2. If I did want to use my existing version, how would I access the Whamg.wdl? From looking at the gatk-sv-single-sample wdl, I see that Module00a.wdl is imported, and I think the Whamg.wdl would be part of that. Is there a way to access the Module00a.wdl directly for modification?

[riazgillani]

I've created a pull request for the v.0.9.1-beta-hotfix branch to modify the Whamg.wdl as such. If you are able to accept this, I believe it would allow me to continue using the same version of the workflow, which would be ideal

[epiercehoffman]

I have refreshed the workflow in Dockstore so v0.21-beta should now be available.

You will not be able to modify the WDL or its dependencies directly in Dockstore - only via GitHub.

The release tag v0.9.1-beta-hotfix is stable and cannot be modified. So your PR is to update the branch with the same name, but we don't keep feature branches around indefinitely. So I think the solution here is that I will merge your change into the branch, then create a new release tag v0.9.1-beta-hotfix2 off of that branch, then delete the branch, then refresh Dockstore, and then you should be able to select that new release tag v0.9.1-beta-hotfix2. I'll comment again when all of those steps are complete.

[epiercehoffman]

Should be ready now

[riazgillani]

This is working well so far - thank you for your help!