Merge pull request #388 from latchbio/rahuldesai1/snakemake/raise-on-…

…registration-error Rahuldesai1/snakemake/raise on registration error
latchbio · Jan 31, 2024 · f8fcd3b · f8fcd3b
2 parents e726fe8 + 9faf35c
commit f8fcd3b
Show file tree

Hide file tree

Showing 11 changed files with 153 additions and 260 deletions.
diff --git a/docs/source/index.md b/docs/source/index.md
@@ -157,13 +157,11 @@ cli/mv
 :hidden:
 :maxdepth: 2
 :caption: Snakemake Integration
+snakemake/overview.md
 snakemake/quickstart.md
-snakemake/tutorial.md
-snakemake/metadata.md
-snakemake/environments.md
-snakemake/lifecycle.md
-snakemake/resources.md
 snakemake/cloud.md
+snakemake/configuration.md
+snakemake/tutorial.md
 snakemake/debugging.md
 snakemake/troubleshooting.md
 ```

diff --git a/docs/source/snakemake/configuration.md b/docs/source/snakemake/configuration.md
@@ -0,0 +1,7 @@
+# Advanced Configuration
+
+```{toctree}
+:maxdepth: 2
+configuration/resources.md
+configuration/registries.md
+```
diff --git a/docs/source/snakemake/configuration/registries.md b/docs/source/snakemake/configuration/registries.md
@@ -0,0 +1,32 @@
+# Private Container Registries
+
+When executing Snakemake workflows in containers, the container images may exist in a private registry that the Latch cloud cannot access. Downloading images from private registries at runtime requires two steps:
+
+1. Upload your private container registry's password/access token to the Latch platform. See [Storing and using Secrets](../basics/adding_secrets.md).
+2. Add the `docker_metadata` field to your workflow's `SnakemakeMetadata` object so the workflow engine knows where to pull your credentials. For example:
+
+```
+# latch_metadata.py
+from latch.types.metadata import SnakemakeMetadata, SnakemakeFileParameter, DockerMetadata
+from latch.types.directory import LatchDir
+from latch.types.metadata import LatchAuthor, LatchMetadata, LatchParameter
+from pathlib import Path
+
+SnakemakeMetadata(
+    display_name="snakemake_tutorial_workflow",
+    author=LatchAuthor(
+            name="latchbio",
+    ),
+    env_config=EnvironmentConfig(
+      use_conda=False,
+      use_container=True,
+    ),
+    docker_metadata=DockerMetadata(
+      username="user0",
+      secret_name="LATCH_SECRET_NAME",
+    ),
+    ...
+)
+```
+
+**Note**: the `secret_name` field specifies the name of the Latch Secret uploaded in step #1, NOT the actual registry password.
diff --git a/docs/source/snakemake/resources.md → ...urce/snakemake/configuration/resources.md b/docs/source/snakemake/resources.md → ...urce/snakemake/configuration/resources.md
@@ -28,7 +28,7 @@ SnakemakeMetadata(
         name="Your Name",
     ),
     parameters=generated_parameters,
-    cores=8, # added
+    cores=8, # updated
 )
 ```
 

diff --git a/docs/source/snakemake/environments.md b/docs/source/snakemake/environments.md
diff --git a/docs/source/snakemake/metadata.md b/docs/source/snakemake/metadata.md
diff --git a/docs/source/snakemake/lifecycle.md → docs/source/snakemake/overview.md b/docs/source/snakemake/lifecycle.md → docs/source/snakemake/overview.md
@@ -1,4 +1,10 @@
-# Snakemake Execution Lifecycle
+# Overview
+
+Latch's Snakemake integration allows developers to build graphical interfaces to expose their Snakemake workflows to wet lab teams. It also provides managed cloud infrastructure for executing the workflow's jobs.
+
+A primary goal for the Snakemake integration is to allow developers to register existing Snakemake projects with minimal added boilerplate and modifications to code.
+
+## Snakemake Execution on Latch
 
 There are two stages to every Snakemake execution:
 
@@ -12,15 +18,10 @@ The first ("JIT") workflow does the following:
 1. Create empty input files; this enables the JIT task to mock the file structure at runtime without using unnecessary network bandwidth from downloading the entire file
 2. Import the Snakefile, calculate the dependency graph, and determine which jobs need to be run
 3. Generate a Latch SDK workflow Python script for the second ("runtime") workflow and register it
-4. Run the runtime workflow using the same inputs
+4. Run the runtime workflow
 
 ![JIT task execution](../assets/snakemake/jit-task-with-logs.jpg)
 
-Debugging:
-
-- The generated runtime workflow entrypoint is uploaded to `latch:///.snakemake_latch/workflows/<workflow_name>/entrypoint.py`
-- Internal workflow specifications are uploaded to `latch:///.snakemake_latch/workflows/<workflow_name>/spec`
-
 ### Runtime Workflow
 
 The runtime workflow will spawn a task per each Snakemake job. This means there will be a separate task per each wildcard instantiation of each rule. This can lead to workflows with hundreds of tasks. Note that the execution graph can be filtered by task status.
@@ -33,7 +34,7 @@ When a task executes, it will:
 
 ![Runtime execution](../assets/snakemake/snakemake-execution.jpg)
 
-### Limitations
+## Limitations
 
 1. The workflow will execute the first rule defined in the Snakefile (matching standard Snakemake behavior). There is no way to change the default rule other than by moving the desired rule up in the file
 1. Rules only download their inputs, which can be a subset of the input files. If the Snakefile tries to read input files outside of the ones explicitly defined in the rule, it will usually fail at runtime