diff --git a/assets/euryale_logo.png b/assets/euryale_logo.png
new file mode 100644
index 0000000..386a105
Binary files /dev/null and b/assets/euryale_logo.png differ
diff --git a/assets/euryale_workflow.drawio.png b/assets/euryale_workflow.drawio.png
new file mode 100644
index 0000000..eb27043
Binary files /dev/null and b/assets/euryale_workflow.drawio.png differ
diff --git a/assets/euryale_workflow.light.png b/assets/euryale_workflow.light.png
new file mode 100644
index 0000000..6e330d5
Binary files /dev/null and b/assets/euryale_workflow.light.png differ
diff --git a/images/mqc_fastqc_adapter.png b/assets/mqc_fastqc_adapter.png
similarity index 100%
rename from images/mqc_fastqc_adapter.png
rename to assets/mqc_fastqc_adapter.png
diff --git a/images/mqc_fastqc_counts.png b/assets/mqc_fastqc_counts.png
similarity index 100%
rename from images/mqc_fastqc_counts.png
rename to assets/mqc_fastqc_counts.png
diff --git a/images/mqc_fastqc_quality.png b/assets/mqc_fastqc_quality.png
similarity index 100%
rename from images/mqc_fastqc_quality.png
rename to assets/mqc_fastqc_quality.png
diff --git a/index.html b/index.html
index 1856b31..4f706e6 100644
--- a/index.html
+++ b/index.html
@@ -86,10 +86,20 @@
           <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
             <div class="section" itemprop="articleBody">
               
-                <h2 id="introduction">Introduction</h2>
+                <p><a href="https://github.com/dalmolingroup/euryale/actions/workflows/ci.yml"><img alt="nf-core CI" src="https://github.com/dalmolingroup/euryale/actions/workflows/ci.yml/badge.svg" /></a>
+<a href="https://github.com/dalmolingroup/euryale/actions/workflows/docs.yml"><img alt="docs" src="https://github.com/dalmolingroup/euryale/actions/workflows/docs.yml/badge.svg" /></a></p>
+<p><a href="https://www.nextflow.io/"><img alt="Nextflow" src="https://img.shields.io/badge/nextflow%20DSL2-%E2%89%A522.10.1-4bc7e2a.svg?labelColor=000000" /></a>
+<a href="https://www.docker.com/"><img alt="run with docker" src="https://img.shields.io/badge/run%20with-docker-0db7ed?labelColor=000000&amp;logo=docker" /></a>
+<a href="https://sylabs.io/docs/"><img alt="run with singularity" src="https://img.shields.io/badge/run%20with-singularity-1d355c.svg?labelColor=000000" /></a></p>
+<p><img alt="EURYALE Logo" src="assets/euryale_logo.png" /></p>
+<h2 id="introduction">Introduction</h2>
 <p><strong>dalmolingroup/euryale</strong> is a pipeline for taxonomic classification and functional annotation of metagenomic reads. Based on <a href="https://github.com/dalmolingroup/medusa">MEDUSA</a>.</p>
 <p>The pipeline is built using <a href="https://www.nextflow.io">Nextflow</a>, a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. The <a href="https://www.nextflow.io/docs/latest/dsl2.html">Nextflow DSL2</a> implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. Where possible, these processes have been submitted to and installed from <a href="https://github.com/nf-core/modules">nf-core/modules</a> in order to make them available to all nf-core pipelines, and to everyone within the Nextflow community!</p>
 <h2 id="pipeline-summary">Pipeline summary</h2>
+<p><picture></p>
+<p><source media="(prefers-color-scheme: dark)" srcset="./assets/euryale_workflow.drawio.png">
+  <img alt="EURYALE diagram" src="./assets/euryale_workflow.light.png">
+</picture></p>
 <h3 id="pre-processing">Pre-processing</h3>
 <ul>
 <li>Read QC (<a href="https://www.bioinformatics.babraham.ac.uk/projects/fastqc/"><code>FastQC</code></a>)</li>
@@ -105,6 +115,7 @@ <h3 id="assembly">Assembly</h3>
 <h3 id="taxonomic-classification">Taxonomic classification</h3>
 <ul>
 <li>Sequence classification (<a href="https://github.com/bioinformatics-centre/kaiju/"><code>Kaiju</code></a>)</li>
+<li>(<strong>WIP</strong>) Sequence classification (<a href="https://github.com/DerrickWood/kraken2"><code>Kraken2</code></a>)</li>
 <li>Visualization (<a href="https://github.com/marbl/Krona/wiki"><code>Krona</code></a>)</li>
 </ul>
 <h3 id="functional-annotation">Functional annotation</h3>
@@ -235,5 +246,5 @@ <h2 id="citations">Citations</h2>
 
 <!--
 MkDocs version : 1.5.3
-Build Date UTC : 2023-10-18 13:29:05.988264+00:00
+Build Date UTC : 2023-11-02 12:48:07.722070+00:00
 -->
diff --git a/search/search_index.json b/search/search_index.json
index c11b941..adc72e0 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -1 +1 @@
-{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Introduction dalmolingroup/euryale is a pipeline for taxonomic classification and functional annotation of metagenomic reads. Based on MEDUSA . The pipeline is built using Nextflow , a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. The Nextflow DSL2 implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. Where possible, these processes have been submitted to and installed from nf-core/modules in order to make them available to all nf-core pipelines, and to everyone within the Nextflow community! Pipeline summary Pre-processing Read QC ( FastQC ) Read trimming and merging ( fastp ) ( optionally ) Host read removal ( BowTie2 ) Duplicated sequence removal ( fastx collapser ) Present QC and other data ( MultiQC ) Assembly ( optionally ) Read assembly ( MEGAHIT ) Taxonomic classification Sequence classification ( Kaiju ) Visualization ( Krona ) Functional annotation Sequence alignment ( DIAMOND ) Map alignment matches to functional database ( annotate ) Quick Start Install Nextflow ( >=22.10.1 ) Install any of Docker , Singularity (you can follow this tutorial ), Podman , Shifter or Charliecloud for full pipeline reproducibility (you can use Conda both to install Nextflow itself and also to manage software within pipelines. Please only use it within pipelines as a last resort; see docs ) . Download the pipeline and test it on a minimal dataset with a single command: nextflow run dalmolingroup/euryale -profile test,YOURPROFILE --outdir <OUTDIR> Note that some form of configuration will be needed so that Nextflow knows how to fetch the required software. This is usually done in the form of a config profile ( YOURPROFILE in the example command above). You can chain multiple config profiles in a comma-separated string. The pipeline comes with config profiles called docker , singularity , podman , shifter , charliecloud and conda which instruct the pipeline to use the named tool for software management. For example, -profile test,docker . Please check nf-core/configs to see if a custom config file to run nf-core pipelines already exists for your Institute. If so, you can simply use -profile <institute> in your command. This will enable either docker or singularity and set the appropriate execution settings for your local compute environment. If you are using singularity , please use the nf-core download command to download images first, before running the pipeline. Setting the NXF_SINGULARITY_CACHEDIR or singularity.cacheDir Nextflow options enables you to store and re-use the images from a central location for future pipeline runs. If you are using conda , it is highly recommended to use the NXF_CONDA_CACHEDIR or conda.cacheDir settings to store the environments in a central location for future pipeline runs. Start running your own analysis! nextflow run dalmolingroup/euryale \\ --input samplesheet.csv \\ --outdir <OUTDIR> \\ --kaiju_db kaiju_reference \\ --diamond_db diamond_db \\ --reference_fasta diamond_fasta \\ --host_fasta host_reference_fasta \\ --id_mapping id_mapping_file \\ -profile <docker/singularity/podman/shifter/charliecloud/conda/institute> Documentation The dalmolingroup/euryale documentation is split into the following pages: Usage An overview of how the pipeline works, how to run it and a description of all of the different command-line flags. Output An overview of the different results produced by the pipeline and how to interpret them. Credits dalmolingroup/euryale was originally written by Jo\u00e3o Cavalcante. We thank the following people for their extensive assistance in the development of this pipeline: Diego Morais (for developing the original MEDUSA pipeline) Citations Morais DAA, Cavalcante JVF, Monteiro SS, Pasquali MAB and Dalmolin RJS (2022) MEDUSA: A Pipeline for Sensitive Taxonomic Classification and Flexible Functional Annotation of Metagenomic Shotgun Sequences. Front. Genet. 13:814437. doi: 10.3389/fgene.2022.814437 This pipeline uses code and infrastructure developed and maintained by the nf-core community, reused here under the MIT license . The nf-core framework for community-curated bioinformatics pipelines. Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen. Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x .","title":"Home"},{"location":"#introduction","text":"dalmolingroup/euryale is a pipeline for taxonomic classification and functional annotation of metagenomic reads. Based on MEDUSA . The pipeline is built using Nextflow , a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. The Nextflow DSL2 implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. Where possible, these processes have been submitted to and installed from nf-core/modules in order to make them available to all nf-core pipelines, and to everyone within the Nextflow community!","title":"Introduction"},{"location":"#pipeline-summary","text":"","title":"Pipeline summary"},{"location":"#pre-processing","text":"Read QC ( FastQC ) Read trimming and merging ( fastp ) ( optionally ) Host read removal ( BowTie2 ) Duplicated sequence removal ( fastx collapser ) Present QC and other data ( MultiQC )","title":"Pre-processing"},{"location":"#assembly","text":"( optionally ) Read assembly ( MEGAHIT )","title":"Assembly"},{"location":"#taxonomic-classification","text":"Sequence classification ( Kaiju ) Visualization ( Krona )","title":"Taxonomic classification"},{"location":"#functional-annotation","text":"Sequence alignment ( DIAMOND ) Map alignment matches to functional database ( annotate )","title":"Functional annotation"},{"location":"#quick-start","text":"Install Nextflow ( >=22.10.1 ) Install any of Docker , Singularity (you can follow this tutorial ), Podman , Shifter or Charliecloud for full pipeline reproducibility (you can use Conda both to install Nextflow itself and also to manage software within pipelines. Please only use it within pipelines as a last resort; see docs ) . Download the pipeline and test it on a minimal dataset with a single command: nextflow run dalmolingroup/euryale -profile test,YOURPROFILE --outdir <OUTDIR> Note that some form of configuration will be needed so that Nextflow knows how to fetch the required software. This is usually done in the form of a config profile ( YOURPROFILE in the example command above). You can chain multiple config profiles in a comma-separated string. The pipeline comes with config profiles called docker , singularity , podman , shifter , charliecloud and conda which instruct the pipeline to use the named tool for software management. For example, -profile test,docker . Please check nf-core/configs to see if a custom config file to run nf-core pipelines already exists for your Institute. If so, you can simply use -profile <institute> in your command. This will enable either docker or singularity and set the appropriate execution settings for your local compute environment. If you are using singularity , please use the nf-core download command to download images first, before running the pipeline. Setting the NXF_SINGULARITY_CACHEDIR or singularity.cacheDir Nextflow options enables you to store and re-use the images from a central location for future pipeline runs. If you are using conda , it is highly recommended to use the NXF_CONDA_CACHEDIR or conda.cacheDir settings to store the environments in a central location for future pipeline runs. Start running your own analysis! nextflow run dalmolingroup/euryale \\ --input samplesheet.csv \\ --outdir <OUTDIR> \\ --kaiju_db kaiju_reference \\ --diamond_db diamond_db \\ --reference_fasta diamond_fasta \\ --host_fasta host_reference_fasta \\ --id_mapping id_mapping_file \\ -profile <docker/singularity/podman/shifter/charliecloud/conda/institute>","title":"Quick Start"},{"location":"#documentation","text":"The dalmolingroup/euryale documentation is split into the following pages: Usage An overview of how the pipeline works, how to run it and a description of all of the different command-line flags. Output An overview of the different results produced by the pipeline and how to interpret them.","title":"Documentation"},{"location":"#credits","text":"dalmolingroup/euryale was originally written by Jo\u00e3o Cavalcante. We thank the following people for their extensive assistance in the development of this pipeline: Diego Morais (for developing the original MEDUSA pipeline)","title":"Credits"},{"location":"#citations","text":"Morais DAA, Cavalcante JVF, Monteiro SS, Pasquali MAB and Dalmolin RJS (2022) MEDUSA: A Pipeline for Sensitive Taxonomic Classification and Flexible Functional Annotation of Metagenomic Shotgun Sequences. Front. Genet. 13:814437. doi: 10.3389/fgene.2022.814437 This pipeline uses code and infrastructure developed and maintained by the nf-core community, reused here under the MIT license . The nf-core framework for community-curated bioinformatics pipelines. Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen. Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x .","title":"Citations"},{"location":"CITATIONS/","text":"dalmolingroup/euryale: Citations nf-core Ewels PA, Peltzer A, Fillinger S, Patel H, Alneberg J, Wilm A, Garcia MU, Di Tommaso P, Nahnsen S. The nf-core framework for community-curated bioinformatics pipelines. Nat Biotechnol. 2020 Mar;38(3):276-278. doi: 10.1038/s41587-020-0439-x. PubMed PMID: 32055031. Nextflow Di Tommaso P, Chatzou M, Floden EW, Barja PP, Palumbo E, Notredame C. Nextflow enables reproducible computational workflows. Nat Biotechnol. 2017 Apr 11;35(4):316-319. doi: 10.1038/nbt.3820. PubMed PMID: 28398311. Pipeline tools FastQC MultiQC Ewels P, Magnusson M, Lundin S, K\u00e4ller M. MultiQC: summarize analysis results for multiple tools and samples in a single report. Bioinformatics. 2016 Oct 1;32(19):3047-8. doi: 10.1093/bioinformatics/btw354. Epub 2016 Jun 16. PubMed PMID: 27312411; PubMed Central PMCID: PMC5039924. Software packaging/containerisation tools Anaconda Anaconda Software Distribution. Computer software. Vers. 2-2.4.0. Anaconda, Nov. 2016. Web. Bioconda Gr\u00fcning B, Dale R, Sj\u00f6din A, Chapman BA, Rowe J, Tomkins-Tinch CH, Valieris R, K\u00f6ster J; Bioconda Team. Bioconda: sustainable and comprehensive software distribution for the life sciences. Nat Methods. 2018 Jul;15(7):475-476. doi: 10.1038/s41592-018-0046-7. PubMed PMID: 29967506. BioContainers da Veiga Leprevost F, Gr\u00fcning B, Aflitos SA, R\u00f6st HL, Uszkoreit J, Barsnes H, Vaudel M, Moreno P, Gatto L, Weber J, Bai M, Jimenez RC, Sachsenberg T, Pfeuffer J, Alvarez RV, Griss J, Nesvizhskii AI, Perez-Riverol Y. BioContainers: an open-source and community-driven framework for software standardization. Bioinformatics. 2017 Aug 15;33(16):2580-2582. doi: 10.1093/bioinformatics/btx192. PubMed PMID: 28379341; PubMed Central PMCID: PMC5870671. Docker Singularity Kurtzer GM, Sochat V, Bauer MW. Singularity: Scientific containers for mobility of compute. PLoS One. 2017 May 11;12(5):e0177459. doi: 10.1371/journal.pone.0177459. eCollection 2017. PubMed PMID: 28494014; PubMed Central PMCID: PMC5426675.","title":"Citations"},{"location":"CITATIONS/#dalmolingroupeuryale-citations","text":"","title":"dalmolingroup/euryale: Citations"},{"location":"CITATIONS/#nf-core","text":"Ewels PA, Peltzer A, Fillinger S, Patel H, Alneberg J, Wilm A, Garcia MU, Di Tommaso P, Nahnsen S. The nf-core framework for community-curated bioinformatics pipelines. Nat Biotechnol. 2020 Mar;38(3):276-278. doi: 10.1038/s41587-020-0439-x. PubMed PMID: 32055031.","title":"nf-core"},{"location":"CITATIONS/#nextflow","text":"Di Tommaso P, Chatzou M, Floden EW, Barja PP, Palumbo E, Notredame C. Nextflow enables reproducible computational workflows. Nat Biotechnol. 2017 Apr 11;35(4):316-319. doi: 10.1038/nbt.3820. PubMed PMID: 28398311.","title":"Nextflow"},{"location":"CITATIONS/#pipeline-tools","text":"FastQC MultiQC Ewels P, Magnusson M, Lundin S, K\u00e4ller M. MultiQC: summarize analysis results for multiple tools and samples in a single report. Bioinformatics. 2016 Oct 1;32(19):3047-8. doi: 10.1093/bioinformatics/btw354. Epub 2016 Jun 16. PubMed PMID: 27312411; PubMed Central PMCID: PMC5039924.","title":"Pipeline tools"},{"location":"CITATIONS/#software-packagingcontainerisation-tools","text":"Anaconda Anaconda Software Distribution. Computer software. Vers. 2-2.4.0. Anaconda, Nov. 2016. Web. Bioconda Gr\u00fcning B, Dale R, Sj\u00f6din A, Chapman BA, Rowe J, Tomkins-Tinch CH, Valieris R, K\u00f6ster J; Bioconda Team. Bioconda: sustainable and comprehensive software distribution for the life sciences. Nat Methods. 2018 Jul;15(7):475-476. doi: 10.1038/s41592-018-0046-7. PubMed PMID: 29967506. BioContainers da Veiga Leprevost F, Gr\u00fcning B, Aflitos SA, R\u00f6st HL, Uszkoreit J, Barsnes H, Vaudel M, Moreno P, Gatto L, Weber J, Bai M, Jimenez RC, Sachsenberg T, Pfeuffer J, Alvarez RV, Griss J, Nesvizhskii AI, Perez-Riverol Y. BioContainers: an open-source and community-driven framework for software standardization. Bioinformatics. 2017 Aug 15;33(16):2580-2582. doi: 10.1093/bioinformatics/btx192. PubMed PMID: 28379341; PubMed Central PMCID: PMC5870671. Docker Singularity Kurtzer GM, Sochat V, Bauer MW. Singularity: Scientific containers for mobility of compute. PLoS One. 2017 May 11;12(5):e0177459. doi: 10.1371/journal.pone.0177459. eCollection 2017. PubMed PMID: 28494014; PubMed Central PMCID: PMC5426675.","title":"Software packaging/containerisation tools"},{"location":"output/","text":"dalmolingroup/euryale: Output Introduction This document describes the output produced by the pipeline. Most of the plots are taken from the MultiQC report, which summarises results at the end of the pipeline. The directories listed below will be created in the results directory after the pipeline has finished. All paths are relative to the top-level results directory. Pipeline overview The pipeline is built using Nextflow and processes data using the following steps (steps in italics don't run by default): Kaiju - Taxonomically classify reads or contigs Krona - Visualize the taxonomic classification for each sample. Diamond - Alignment reads and contigs against a reference database (such as NCBI-nr). Annotate - Functional annotation of alignment matches. MEGAHIT - Assembled contigs. MultiQC - Aggregate report describing results and QC from the whole pipeline Pipeline information - Report metrics generated during the workflow execution Kaiju Output files - `taxonomy/` - `${sample}.tsv`: Kaiju classification output. - `${sample}.txt`: Kaiju2Table TSV output. Kaiju is a software to perform fast taxonomic classification of metagenomic sequencing reads using a protein reference database. Krona Output files - `taxonomy/` - `${sample}.html`: Krona visualization for the sample. Krona is a tool to interactively explore metagenomes and more from a web browser. Diamond Output files - `alignment/${sample}/` - `${sample}.txt`: Alignment matches in blast tabular output format. - `${sample}.log`: DIAMOND execution log. DIAMOND is an accelerated BLAST compatible local sequence aligner. Annotate Output files - `functional/${sample}/` - `${sample}_annotated.txt`: Alignment matches annotated to chosen functional database (e.g. GO). Annotate is a tool to annotate each query using the best alignment for which a mapping is known. MEGAHIT Output files - `assembly/${sample}/` - `${sample}.contigs.fa.gz`: Contigs assembled for the sample. MEGAHIT is an ultra-fast and memory-efficient (meta-)genome assembler DIAMOND database Output files - `diamond_db/` - `${database_name}.dmnd`: DIAMOND database for the reference fasta file. This output is present if you add the --save_db parameter. DIAMOND is an accelerated BLAST compatible local sequence aligner. MultiQC Output files - `multiqc/` - `multiqc_report.html`: a standalone HTML file that can be viewed in your web browser. - `multiqc_data/`: directory containing parsed statistics from the different tools used in the pipeline. - `multiqc_plots/`: directory containing static images from the report in various formats. MultiQC is a visualization tool that generates a single HTML report summarising all samples in your project. Most of the pipeline QC results are visualised in the report and further statistics are available in the report data directory. Results generated by MultiQC collate pipeline QC from supported tools e.g. FastQC. The pipeline has special steps which also allow the software versions to be reported in the MultiQC output for future traceability. For more information about how to use MultiQC reports, see http://multiqc.info . Pipeline information Output files - `pipeline_info/` - Reports generated by Nextflow: `execution_report.html`, `execution_timeline.html`, `execution_trace.txt` and `pipeline_dag.dot`/`pipeline_dag.svg`. - Reports generated by the pipeline: `pipeline_report.html`, `pipeline_report.txt` and `software_versions.yml`. The `pipeline_report*` files will only be present if the `--email` / `--email_on_fail` parameter's are used when running the pipeline. - Reformatted samplesheet files used as input to the pipeline: `samplesheet.valid.csv`. Nextflow provides excellent functionality for generating various reports relevant to the running and execution of the pipeline. This will allow you to troubleshoot errors with the running of the pipeline, and also provide you with other information such as launch commands, run times and resource usage.","title":"Output"},{"location":"output/#dalmolingroupeuryale-output","text":"","title":"dalmolingroup/euryale: Output"},{"location":"output/#introduction","text":"This document describes the output produced by the pipeline. Most of the plots are taken from the MultiQC report, which summarises results at the end of the pipeline. The directories listed below will be created in the results directory after the pipeline has finished. All paths are relative to the top-level results directory.","title":"Introduction"},{"location":"output/#pipeline-overview","text":"The pipeline is built using Nextflow and processes data using the following steps (steps in italics don't run by default): Kaiju - Taxonomically classify reads or contigs Krona - Visualize the taxonomic classification for each sample. Diamond - Alignment reads and contigs against a reference database (such as NCBI-nr). Annotate - Functional annotation of alignment matches. MEGAHIT - Assembled contigs. MultiQC - Aggregate report describing results and QC from the whole pipeline Pipeline information - Report metrics generated during the workflow execution","title":"Pipeline overview"},{"location":"output/#kaiju","text":"Output files - `taxonomy/` - `${sample}.tsv`: Kaiju classification output. - `${sample}.txt`: Kaiju2Table TSV output. Kaiju is a software to perform fast taxonomic classification of metagenomic sequencing reads using a protein reference database.","title":"Kaiju"},{"location":"output/#krona","text":"Output files - `taxonomy/` - `${sample}.html`: Krona visualization for the sample. Krona is a tool to interactively explore metagenomes and more from a web browser.","title":"Krona"},{"location":"output/#diamond","text":"Output files - `alignment/${sample}/` - `${sample}.txt`: Alignment matches in blast tabular output format. - `${sample}.log`: DIAMOND execution log. DIAMOND is an accelerated BLAST compatible local sequence aligner.","title":"Diamond"},{"location":"output/#annotate","text":"Output files - `functional/${sample}/` - `${sample}_annotated.txt`: Alignment matches annotated to chosen functional database (e.g. GO). Annotate is a tool to annotate each query using the best alignment for which a mapping is known.","title":"Annotate"},{"location":"output/#megahit","text":"Output files - `assembly/${sample}/` - `${sample}.contigs.fa.gz`: Contigs assembled for the sample. MEGAHIT is an ultra-fast and memory-efficient (meta-)genome assembler","title":"MEGAHIT"},{"location":"output/#diamond-database","text":"Output files - `diamond_db/` - `${database_name}.dmnd`: DIAMOND database for the reference fasta file. This output is present if you add the --save_db parameter. DIAMOND is an accelerated BLAST compatible local sequence aligner.","title":"DIAMOND database"},{"location":"output/#multiqc","text":"Output files - `multiqc/` - `multiqc_report.html`: a standalone HTML file that can be viewed in your web browser. - `multiqc_data/`: directory containing parsed statistics from the different tools used in the pipeline. - `multiqc_plots/`: directory containing static images from the report in various formats. MultiQC is a visualization tool that generates a single HTML report summarising all samples in your project. Most of the pipeline QC results are visualised in the report and further statistics are available in the report data directory. Results generated by MultiQC collate pipeline QC from supported tools e.g. FastQC. The pipeline has special steps which also allow the software versions to be reported in the MultiQC output for future traceability. For more information about how to use MultiQC reports, see http://multiqc.info .","title":"MultiQC"},{"location":"output/#pipeline-information","text":"Output files - `pipeline_info/` - Reports generated by Nextflow: `execution_report.html`, `execution_timeline.html`, `execution_trace.txt` and `pipeline_dag.dot`/`pipeline_dag.svg`. - Reports generated by the pipeline: `pipeline_report.html`, `pipeline_report.txt` and `software_versions.yml`. The `pipeline_report*` files will only be present if the `--email` / `--email_on_fail` parameter's are used when running the pipeline. - Reformatted samplesheet files used as input to the pipeline: `samplesheet.valid.csv`. Nextflow provides excellent functionality for generating various reports relevant to the running and execution of the pipeline. This will allow you to troubleshoot errors with the running of the pipeline, and also provide you with other information such as launch commands, run times and resource usage.","title":"Pipeline information"},{"location":"params/","text":"dalmolingroup/euryale pipeline parameters A pipeline for metagenomic taxonomic classification and functional annotation. Based on MEDUSA. Input/output options Define where the pipeline should find input data and save output data. Parameter Description Type Default Required Hidden input Path to comma-separated file containing information about the samples in the experiment. Help You will need to create a design file with information about the samples in your experiment before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row. string True outdir The output directory where the results will be saved. You have to use absolute paths to storage on Cloud infrastructure. string True email Email address for completion summary. Help Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file ( ~/.nextflow/config ) then you don't need to specify this on the command line for every run. string multiqc_title MultiQC report title. Printed as page header, used for filename if not otherwise specified. string save_dbs Save DIAMOND db to results directory after construction boolean Skip Steps Choose to skip pipeline steps Parameter Description Type Default Required Hidden skip_classification Skip taxonomic classification boolean skip_alignment Skip alignment boolean skip_functional Skip functional annotation boolean Decontamination Parameter Description Type Default Required Hidden host_fasta string None Alignment Parameter Description Type Default Required Hidden reference_fasta Path to FASTA genome file. string None diamond_db Path to pre-built DIAMOND db. string None Taxonomy Parameter Description Type Default Required Hidden kaiju_db string None True Functional Parameter Description Type Default Required Hidden id_mapping Path to ID mapping file to be used for the Functional annotation string None True minimum_bitscore Minimum bitscore of a match to be used for annotation integer 50 minimum_pident Minimum identity of a match to be used for annotation integer 80 minimum_alen Minimum alignment length of a match to be used for annotation integer 50 maximum_evalue Maximum evalue of a match to be used for annotation number 0.0001 Assembly Parameter Description Type Default Required Hidden assembly_based boolean Reference genome options Reference genome related files and options required for the workflow. Parameter Description Type Default Required Hidden genome Name of iGenomes reference. Help If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. --genome GRCh38 . See the nf-core website docs for more details. string igenomes_base Directory / URL base for iGenomes references. string s3://ngi-igenomes/igenomes True igenomes_ignore Do not load the iGenomes reference config. Help Do not load igenomes.config when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in igenomes.config . boolean True fasta string None Max job request options Set the top limit for requested resources for any single job. Parameter Description Type Default Required Hidden max_cpus Maximum number of CPUs that can be requested for any single job. Help Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. --max_cpus 1 integer 16 True max_memory Maximum amount of memory that can be requested for any single job. Help Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. --max_memory '8.GB' string 128.GB True max_time Maximum amount of time that can be requested for any single job. Help Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. --max_time '2.h' string 240.h True Generic options Less common options for the pipeline, typically set in a config file. Parameter Description Type Default Required Hidden help Display help text. boolean True version Display version and exit. boolean True publish_dir_mode Method used to save pipeline results to output directory. Help The Nextflow publishDir option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See Nextflow docs for details. string copy True email_on_fail Email address for completion summary, only when pipeline fails. Help An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully. string True plaintext_email Send plain-text email instead of HTML. boolean True max_multiqc_email_size File size limit when attaching MultiQC reports to summary emails. string 25.MB True monochrome_logs Do not use coloured log outputs. boolean True hook_url Incoming hook URL for messaging service Help Incoming hook URL for messaging service. Currently, MS Teams and Slack are supported. string True multiqc_config Custom config file to supply to MultiQC. string True multiqc_logo Custom logo file to supply to MultiQC. File name must also be set in the MultiQC config file string True multiqc_methods_description Custom MultiQC yaml file containing HTML including a methods description. string tracedir Directory to keep pipeline Nextflow logs and reports. string ${params.outdir}/pipeline_info True validate_params Boolean whether to validate parameters against the schema at runtime boolean True True show_hidden_params Show all params when using --help Help By default, parameters set as hidden in the schema are not shown on the command line when a user runs with --help . Specifying this option will tell the pipeline to show all parameters. boolean True Other parameters Parameter Description Type Default Required Hidden schema_ignore_params string genomes","title":"Parameters"},{"location":"params/#dalmolingroupeuryale-pipeline-parameters","text":"A pipeline for metagenomic taxonomic classification and functional annotation. Based on MEDUSA.","title":"dalmolingroup/euryale pipeline parameters"},{"location":"params/#inputoutput-options","text":"Define where the pipeline should find input data and save output data. Parameter Description Type Default Required Hidden input Path to comma-separated file containing information about the samples in the experiment. Help You will need to create a design file with information about the samples in your experiment before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row. string True outdir The output directory where the results will be saved. You have to use absolute paths to storage on Cloud infrastructure. string True email Email address for completion summary. Help Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file ( ~/.nextflow/config ) then you don't need to specify this on the command line for every run. string multiqc_title MultiQC report title. Printed as page header, used for filename if not otherwise specified. string save_dbs Save DIAMOND db to results directory after construction boolean","title":"Input/output options"},{"location":"params/#skip-steps","text":"Choose to skip pipeline steps Parameter Description Type Default Required Hidden skip_classification Skip taxonomic classification boolean skip_alignment Skip alignment boolean skip_functional Skip functional annotation boolean","title":"Skip Steps"},{"location":"params/#decontamination","text":"Parameter Description Type Default Required Hidden host_fasta string None","title":"Decontamination"},{"location":"params/#alignment","text":"Parameter Description Type Default Required Hidden reference_fasta Path to FASTA genome file. string None diamond_db Path to pre-built DIAMOND db. string None","title":"Alignment"},{"location":"params/#taxonomy","text":"Parameter Description Type Default Required Hidden kaiju_db string None True","title":"Taxonomy"},{"location":"params/#functional","text":"Parameter Description Type Default Required Hidden id_mapping Path to ID mapping file to be used for the Functional annotation string None True minimum_bitscore Minimum bitscore of a match to be used for annotation integer 50 minimum_pident Minimum identity of a match to be used for annotation integer 80 minimum_alen Minimum alignment length of a match to be used for annotation integer 50 maximum_evalue Maximum evalue of a match to be used for annotation number 0.0001","title":"Functional"},{"location":"params/#assembly","text":"Parameter Description Type Default Required Hidden assembly_based boolean","title":"Assembly"},{"location":"params/#reference-genome-options","text":"Reference genome related files and options required for the workflow. Parameter Description Type Default Required Hidden genome Name of iGenomes reference. Help If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. --genome GRCh38 . See the nf-core website docs for more details. string igenomes_base Directory / URL base for iGenomes references. string s3://ngi-igenomes/igenomes True igenomes_ignore Do not load the iGenomes reference config. Help Do not load igenomes.config when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in igenomes.config . boolean True fasta string None","title":"Reference genome options"},{"location":"params/#max-job-request-options","text":"Set the top limit for requested resources for any single job. Parameter Description Type Default Required Hidden max_cpus Maximum number of CPUs that can be requested for any single job. Help Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. --max_cpus 1 integer 16 True max_memory Maximum amount of memory that can be requested for any single job. Help Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. --max_memory '8.GB' string 128.GB True max_time Maximum amount of time that can be requested for any single job. Help Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. --max_time '2.h' string 240.h True","title":"Max job request options"},{"location":"params/#generic-options","text":"Less common options for the pipeline, typically set in a config file. Parameter Description Type Default Required Hidden help Display help text. boolean True version Display version and exit. boolean True publish_dir_mode Method used to save pipeline results to output directory. Help The Nextflow publishDir option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See Nextflow docs for details. string copy True email_on_fail Email address for completion summary, only when pipeline fails. Help An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully. string True plaintext_email Send plain-text email instead of HTML. boolean True max_multiqc_email_size File size limit when attaching MultiQC reports to summary emails. string 25.MB True monochrome_logs Do not use coloured log outputs. boolean True hook_url Incoming hook URL for messaging service Help Incoming hook URL for messaging service. Currently, MS Teams and Slack are supported. string True multiqc_config Custom config file to supply to MultiQC. string True multiqc_logo Custom logo file to supply to MultiQC. File name must also be set in the MultiQC config file string True multiqc_methods_description Custom MultiQC yaml file containing HTML including a methods description. string tracedir Directory to keep pipeline Nextflow logs and reports. string ${params.outdir}/pipeline_info True validate_params Boolean whether to validate parameters against the schema at runtime boolean True True show_hidden_params Show all params when using --help Help By default, parameters set as hidden in the schema are not shown on the command line when a user runs with --help . Specifying this option will tell the pipeline to show all parameters. boolean True","title":"Generic options"},{"location":"params/#other-parameters","text":"Parameter Description Type Default Required Hidden schema_ignore_params string genomes","title":"Other parameters"},{"location":"usage/","text":"dalmolingroup/euryale: Usage Documentation of pipeline parameters is generated automatically from the pipeline schema and can no longer be found in markdown files. Introduction Samplesheet input You will need to create a samplesheet with information about the samples you would like to analyse before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row as shown in the examples below. --input '[path to samplesheet file]' Multiple runs of the same sample The sample identifiers have to be the same when you have re-sequenced the same sample more than once e.g. to increase sequencing depth. The pipeline will concatenate the raw reads before performing any downstream analysis. Below is an example for the same sample sequenced across 3 lanes: sample,fastq_1,fastq_2 CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz CONTROL_REP1,AEG588A1_S1_L003_R1_001.fastq.gz,AEG588A1_S1_L003_R2_001.fastq.gz CONTROL_REP1,AEG588A1_S1_L004_R1_001.fastq.gz,AEG588A1_S1_L004_R2_001.fastq.gz Full samplesheet The pipeline will auto-detect whether a sample is single- or paired-end using the information provided in the samplesheet. The samplesheet can have as many columns as you desire, however, there is a strict requirement for the first 3 columns to match those defined in the table below. A final samplesheet file consisting of both single- and paired-end data may look something like the one below. This is for 6 samples, where TREATMENT_REP3 has been sequenced twice. sample,fastq_1,fastq_2 CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz CONTROL_REP2,AEG588A2_S2_L002_R1_001.fastq.gz,AEG588A2_S2_L002_R2_001.fastq.gz CONTROL_REP3,AEG588A3_S3_L002_R1_001.fastq.gz,AEG588A3_S3_L002_R2_001.fastq.gz TREATMENT_REP1,AEG588A4_S4_L003_R1_001.fastq.gz, TREATMENT_REP2,AEG588A5_S5_L003_R1_001.fastq.gz, TREATMENT_REP3,AEG588A6_S6_L003_R1_001.fastq.gz, TREATMENT_REP3,AEG588A6_S6_L004_R1_001.fastq.gz, Column Description sample Custom sample name. This entry will be identical for multiple sequencing libraries/runs from the same sample. Spaces in sample names are automatically converted to underscores ( _ ). fastq_1 Full path to FastQ file for Illumina short reads 1. File has to be gzipped and have the extension \".fastq.gz\" or \".fq.gz\". fastq_2 Full path to FastQ file for Illumina short reads 2. File has to be gzipped and have the extension \".fastq.gz\" or \".fq.gz\". An example samplesheet has been provided with the pipeline. Running the pipeline The typical command for running the pipeline is as follows: nextflow run dalmolingroup/euryale --input samplesheet.csv --outdir <OUTDIR> --genome GRCh37 -profile docker This will launch the pipeline with the docker configuration profile. See below for more information about profiles. Note that the pipeline will create the following files in your working directory: work # Directory containing the nextflow working files <OUTDIR> # Finished results in specified location (defined with --outdir) .nextflow_log # Log file from Nextflow # Other nextflow hidden files, eg. history of pipeline runs and old logs. Updating the pipeline When you run the above command, Nextflow automatically pulls the pipeline code from GitHub and stores it as a cached version. When running the pipeline after this, it will always use the cached version if available - even if the pipeline has been updated since. To make sure that you're running the latest version of the pipeline, make sure that you regularly update the cached version of the pipeline: nextflow pull dalmolingroup/euryale Reproducibility It is a good idea to specify a pipeline version when running the pipeline on your data. This ensures that a specific version of the pipeline code and software are used when you run your pipeline. If you keep using the same tag, you'll be running the same version of the pipeline, even if there have been changes to the code since. First, go to the dalmolingroup/euryale releases page and find the latest pipeline version - numeric only (eg. 1.3.1 ). Then specify this when running the pipeline with -r (one hyphen) - eg. -r 1.3.1 . Of course, you can switch to another version by changing the number after the -r flag. This version number will be logged in reports when you run the pipeline, so that you'll know what you used when you look back in the future. For example, at the bottom of the MultiQC reports. Core Nextflow arguments NB: These options are part of Nextflow and use a single hyphen (pipeline parameters use a double-hyphen). -profile Use this parameter to choose a configuration profile. Profiles can give configuration presets for different compute environments. Several generic profiles are bundled with the pipeline which instruct the pipeline to use software packaged using different methods (Docker, Singularity, Podman, Shifter, Charliecloud, Conda) - see below. We highly recommend the use of Docker or Singularity containers for full pipeline reproducibility, however when this is not possible, Conda is also supported. Note that multiple profiles can be loaded, for example: -profile test,docker - the order of arguments is important! They are loaded in sequence, so later profiles can overwrite earlier profiles. If -profile is not specified, the pipeline will run locally and expect all software to be installed and available on the PATH . This is not recommended, since it can lead to different results on different machines dependent on the computer enviroment. test A profile with a complete configuration for automated testing Includes links to test data so needs no other parameters docker A generic configuration profile to be used with Docker singularity A generic configuration profile to be used with Singularity podman A generic configuration profile to be used with Podman shifter A generic configuration profile to be used with Shifter charliecloud A generic configuration profile to be used with Charliecloud conda A generic configuration profile to be used with Conda . Please only use Conda as a last resort i.e. when it's not possible to run the pipeline with Docker, Singularity, Podman, Shifter or Charliecloud. -resume Specify this when restarting a pipeline. Nextflow will use cached results from any pipeline steps where the inputs are the same, continuing from where it got to previously. For input to be considered the same, not only the names must be identical but the files' contents as well. For more info about this parameter, see this blog post . You can also supply a run name to resume a specific run: -resume [run-name] . Use the nextflow log command to show previous run names. -c Specify the path to a specific config file (this is a core Nextflow command). See the nf-core website documentation for more information. Custom configuration Resource requests Whilst the default requirements set within the pipeline will hopefully work for most people and with most input data, you may find that you want to customise the compute resources that the pipeline requests. Each step in the pipeline has a default set of requirements for number of CPUs, memory and time. For most of the steps in the pipeline, if the job exits with any of the error codes specified here it will automatically be resubmitted with higher requests (2 x original, then 3 x original). If it still fails after the third attempt then the pipeline execution is stopped. For example, if the nf-core/rnaseq pipeline is failing after multiple re-submissions of the STAR_ALIGN process due to an exit code of 137 this would indicate that there is an out of memory issue: [62/149eb0] NOTE: Process `NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)` terminated with an error exit status (137) -- Execution is retried (1) Error executing process > 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)' Caused by: Process `NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)` terminated with an error exit status (137) Command executed: STAR \\ --genomeDir star \\ --readFilesIn WT_REP1_trimmed.fq.gz \\ --runThreadN 2 \\ --outFileNamePrefix WT_REP1. \\ <TRUNCATED> Command exit status: 137 Command output: (empty) Command error: .command.sh: line 9: 30 Killed STAR --genomeDir star --readFilesIn WT_REP1_trimmed.fq.gz --runThreadN 2 --outFileNamePrefix WT_REP1. <TRUNCATED> Work dir: /home/pipelinetest/work/9d/172ca5881234073e8d76f2a19c88fb Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run` For beginners A first step to bypass this error, you could try to increase the amount of CPUs, memory, and time for the whole pipeline. Therefor you can try to increase the resource for the parameters --max_cpus , --max_memory , and --max_time . Based on the error above, you have to increase the amount of memory. Therefore you can go to the parameter documentation of rnaseq and scroll down to the show hidden parameter button to get the default value for --max_memory . In this case 128GB, you than can try to run your pipeline again with --max_memory 200GB -resume to skip all process, that were already calculated. If you can not increase the resource of the complete pipeline, you can try to adapt the resource for a single process as mentioned below. Advanced option on process level To bypass this error you would need to find exactly which resources are set by the STAR_ALIGN process. The quickest way is to search for process STAR_ALIGN in the nf-core/rnaseq Github repo . We have standardised the structure of Nextflow DSL2 pipelines such that all module files will be present in the modules/ directory and so, based on the search results, the file we want is modules/nf-core/star/align/main.nf . If you click on the link to that file you will notice that there is a label directive at the top of the module that is set to label process_high . The Nextflow label directive allows us to organise workflow processes in separate groups which can be referenced in a configuration file to select and configure subset of processes having similar computing requirements. The default values for the process_high label are set in the pipeline's base.config which in this case is defined as 72GB. Providing you haven't set any other standard nf-core parameters to cap the maximum resources used by the pipeline then we can try and bypass the STAR_ALIGN process failure by creating a custom config file that sets at least 72GB of memory, in this case increased to 100GB. The custom config below can then be provided to the pipeline via the -c parameter as highlighted in previous sections. process { withName: 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN' { memory = 100.GB } } NB: We specify the full process name i.e. NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN in the config file because this takes priority over the short name ( STAR_ALIGN ) and allows existing configuration using the full process name to be correctly overridden. If you get a warning suggesting that the process selector isn't recognised check that the process name has been specified correctly. Updating containers (advanced users) The Nextflow DSL2 implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. If for some reason you need to use a different version of a particular tool with the pipeline then you just need to identify the process name and override the Nextflow container definition for that process using the withName declaration. For example, in the nf-core/viralrecon pipeline a tool called Pangolin has been used during the COVID-19 pandemic to assign lineages to SARS-CoV-2 genome sequenced samples. Given that the lineage assignments change quite frequently it doesn't make sense to re-release the nf-core/viralrecon everytime a new version of Pangolin has been released. However, you can override the default container used by the pipeline by creating a custom config file and passing it as a command-line argument via -c custom.config . Check the default version used by the pipeline in the module file for Pangolin Find the latest version of the Biocontainer available on Quay.io Create the custom config accordingly: For Docker: nextflow process { withName: PANGOLIN { container = 'quay.io/biocontainers/pangolin:3.0.5--pyhdfd78af_0' } } For Singularity: nextflow process { withName: PANGOLIN { container = 'https://depot.galaxyproject.org/singularity/pangolin:3.0.5--pyhdfd78af_0' } } For Conda: nextflow process { withName: PANGOLIN { conda = 'bioconda::pangolin=3.0.5' } } NB: If you wish to periodically update individual tool-specific results (e.g. Pangolin) generated by the pipeline then you must ensure to keep the work/ directory otherwise the -resume ability of the pipeline will be compromised and it will restart from scratch. Running in the background Nextflow handles job submissions and supervises the running jobs. The Nextflow process must run until the pipeline is finished. The Nextflow -bg flag launches Nextflow in the background, detached from your terminal so that the workflow does not stop if you log out of your session. The logs are saved to a file. Alternatively, you can use screen / tmux or similar tool to create a detached session which you can log back into at a later time. Some HPC setups also allow you to run nextflow within a cluster job submitted your job scheduler (from where it submits more jobs). Nextflow memory requirements In some cases, the Nextflow Java virtual machines can start to request a large amount of memory. We recommend adding the following line to your environment to limit this (typically in ~/.bashrc or ~./bash_profile ): NXF_OPTS='-Xms1g -Xmx4g'","title":"Usage"},{"location":"usage/#dalmolingroupeuryale-usage","text":"Documentation of pipeline parameters is generated automatically from the pipeline schema and can no longer be found in markdown files.","title":"dalmolingroup/euryale: Usage"},{"location":"usage/#introduction","text":"","title":"Introduction"},{"location":"usage/#samplesheet-input","text":"You will need to create a samplesheet with information about the samples you would like to analyse before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row as shown in the examples below. --input '[path to samplesheet file]'","title":"Samplesheet input"},{"location":"usage/#multiple-runs-of-the-same-sample","text":"The sample identifiers have to be the same when you have re-sequenced the same sample more than once e.g. to increase sequencing depth. The pipeline will concatenate the raw reads before performing any downstream analysis. Below is an example for the same sample sequenced across 3 lanes: sample,fastq_1,fastq_2 CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz CONTROL_REP1,AEG588A1_S1_L003_R1_001.fastq.gz,AEG588A1_S1_L003_R2_001.fastq.gz CONTROL_REP1,AEG588A1_S1_L004_R1_001.fastq.gz,AEG588A1_S1_L004_R2_001.fastq.gz","title":"Multiple runs of the same sample"},{"location":"usage/#full-samplesheet","text":"The pipeline will auto-detect whether a sample is single- or paired-end using the information provided in the samplesheet. The samplesheet can have as many columns as you desire, however, there is a strict requirement for the first 3 columns to match those defined in the table below. A final samplesheet file consisting of both single- and paired-end data may look something like the one below. This is for 6 samples, where TREATMENT_REP3 has been sequenced twice. sample,fastq_1,fastq_2 CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz CONTROL_REP2,AEG588A2_S2_L002_R1_001.fastq.gz,AEG588A2_S2_L002_R2_001.fastq.gz CONTROL_REP3,AEG588A3_S3_L002_R1_001.fastq.gz,AEG588A3_S3_L002_R2_001.fastq.gz TREATMENT_REP1,AEG588A4_S4_L003_R1_001.fastq.gz, TREATMENT_REP2,AEG588A5_S5_L003_R1_001.fastq.gz, TREATMENT_REP3,AEG588A6_S6_L003_R1_001.fastq.gz, TREATMENT_REP3,AEG588A6_S6_L004_R1_001.fastq.gz, Column Description sample Custom sample name. This entry will be identical for multiple sequencing libraries/runs from the same sample. Spaces in sample names are automatically converted to underscores ( _ ). fastq_1 Full path to FastQ file for Illumina short reads 1. File has to be gzipped and have the extension \".fastq.gz\" or \".fq.gz\". fastq_2 Full path to FastQ file for Illumina short reads 2. File has to be gzipped and have the extension \".fastq.gz\" or \".fq.gz\". An example samplesheet has been provided with the pipeline.","title":"Full samplesheet"},{"location":"usage/#running-the-pipeline","text":"The typical command for running the pipeline is as follows: nextflow run dalmolingroup/euryale --input samplesheet.csv --outdir <OUTDIR> --genome GRCh37 -profile docker This will launch the pipeline with the docker configuration profile. See below for more information about profiles. Note that the pipeline will create the following files in your working directory: work # Directory containing the nextflow working files <OUTDIR> # Finished results in specified location (defined with --outdir) .nextflow_log # Log file from Nextflow # Other nextflow hidden files, eg. history of pipeline runs and old logs.","title":"Running the pipeline"},{"location":"usage/#updating-the-pipeline","text":"When you run the above command, Nextflow automatically pulls the pipeline code from GitHub and stores it as a cached version. When running the pipeline after this, it will always use the cached version if available - even if the pipeline has been updated since. To make sure that you're running the latest version of the pipeline, make sure that you regularly update the cached version of the pipeline: nextflow pull dalmolingroup/euryale","title":"Updating the pipeline"},{"location":"usage/#reproducibility","text":"It is a good idea to specify a pipeline version when running the pipeline on your data. This ensures that a specific version of the pipeline code and software are used when you run your pipeline. If you keep using the same tag, you'll be running the same version of the pipeline, even if there have been changes to the code since. First, go to the dalmolingroup/euryale releases page and find the latest pipeline version - numeric only (eg. 1.3.1 ). Then specify this when running the pipeline with -r (one hyphen) - eg. -r 1.3.1 . Of course, you can switch to another version by changing the number after the -r flag. This version number will be logged in reports when you run the pipeline, so that you'll know what you used when you look back in the future. For example, at the bottom of the MultiQC reports.","title":"Reproducibility"},{"location":"usage/#core-nextflow-arguments","text":"NB: These options are part of Nextflow and use a single hyphen (pipeline parameters use a double-hyphen).","title":"Core Nextflow arguments"},{"location":"usage/#-profile","text":"Use this parameter to choose a configuration profile. Profiles can give configuration presets for different compute environments. Several generic profiles are bundled with the pipeline which instruct the pipeline to use software packaged using different methods (Docker, Singularity, Podman, Shifter, Charliecloud, Conda) - see below. We highly recommend the use of Docker or Singularity containers for full pipeline reproducibility, however when this is not possible, Conda is also supported. Note that multiple profiles can be loaded, for example: -profile test,docker - the order of arguments is important! They are loaded in sequence, so later profiles can overwrite earlier profiles. If -profile is not specified, the pipeline will run locally and expect all software to be installed and available on the PATH . This is not recommended, since it can lead to different results on different machines dependent on the computer enviroment. test A profile with a complete configuration for automated testing Includes links to test data so needs no other parameters docker A generic configuration profile to be used with Docker singularity A generic configuration profile to be used with Singularity podman A generic configuration profile to be used with Podman shifter A generic configuration profile to be used with Shifter charliecloud A generic configuration profile to be used with Charliecloud conda A generic configuration profile to be used with Conda . Please only use Conda as a last resort i.e. when it's not possible to run the pipeline with Docker, Singularity, Podman, Shifter or Charliecloud.","title":"-profile"},{"location":"usage/#-resume","text":"Specify this when restarting a pipeline. Nextflow will use cached results from any pipeline steps where the inputs are the same, continuing from where it got to previously. For input to be considered the same, not only the names must be identical but the files' contents as well. For more info about this parameter, see this blog post . You can also supply a run name to resume a specific run: -resume [run-name] . Use the nextflow log command to show previous run names.","title":"-resume"},{"location":"usage/#-c","text":"Specify the path to a specific config file (this is a core Nextflow command). See the nf-core website documentation for more information.","title":"-c"},{"location":"usage/#custom-configuration","text":"","title":"Custom configuration"},{"location":"usage/#resource-requests","text":"Whilst the default requirements set within the pipeline will hopefully work for most people and with most input data, you may find that you want to customise the compute resources that the pipeline requests. Each step in the pipeline has a default set of requirements for number of CPUs, memory and time. For most of the steps in the pipeline, if the job exits with any of the error codes specified here it will automatically be resubmitted with higher requests (2 x original, then 3 x original). If it still fails after the third attempt then the pipeline execution is stopped. For example, if the nf-core/rnaseq pipeline is failing after multiple re-submissions of the STAR_ALIGN process due to an exit code of 137 this would indicate that there is an out of memory issue: [62/149eb0] NOTE: Process `NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)` terminated with an error exit status (137) -- Execution is retried (1) Error executing process > 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)' Caused by: Process `NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)` terminated with an error exit status (137) Command executed: STAR \\ --genomeDir star \\ --readFilesIn WT_REP1_trimmed.fq.gz \\ --runThreadN 2 \\ --outFileNamePrefix WT_REP1. \\ <TRUNCATED> Command exit status: 137 Command output: (empty) Command error: .command.sh: line 9: 30 Killed STAR --genomeDir star --readFilesIn WT_REP1_trimmed.fq.gz --runThreadN 2 --outFileNamePrefix WT_REP1. <TRUNCATED> Work dir: /home/pipelinetest/work/9d/172ca5881234073e8d76f2a19c88fb Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`","title":"Resource requests"},{"location":"usage/#for-beginners","text":"A first step to bypass this error, you could try to increase the amount of CPUs, memory, and time for the whole pipeline. Therefor you can try to increase the resource for the parameters --max_cpus , --max_memory , and --max_time . Based on the error above, you have to increase the amount of memory. Therefore you can go to the parameter documentation of rnaseq and scroll down to the show hidden parameter button to get the default value for --max_memory . In this case 128GB, you than can try to run your pipeline again with --max_memory 200GB -resume to skip all process, that were already calculated. If you can not increase the resource of the complete pipeline, you can try to adapt the resource for a single process as mentioned below.","title":"For beginners"},{"location":"usage/#advanced-option-on-process-level","text":"To bypass this error you would need to find exactly which resources are set by the STAR_ALIGN process. The quickest way is to search for process STAR_ALIGN in the nf-core/rnaseq Github repo . We have standardised the structure of Nextflow DSL2 pipelines such that all module files will be present in the modules/ directory and so, based on the search results, the file we want is modules/nf-core/star/align/main.nf . If you click on the link to that file you will notice that there is a label directive at the top of the module that is set to label process_high . The Nextflow label directive allows us to organise workflow processes in separate groups which can be referenced in a configuration file to select and configure subset of processes having similar computing requirements. The default values for the process_high label are set in the pipeline's base.config which in this case is defined as 72GB. Providing you haven't set any other standard nf-core parameters to cap the maximum resources used by the pipeline then we can try and bypass the STAR_ALIGN process failure by creating a custom config file that sets at least 72GB of memory, in this case increased to 100GB. The custom config below can then be provided to the pipeline via the -c parameter as highlighted in previous sections. process { withName: 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN' { memory = 100.GB } } NB: We specify the full process name i.e. NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN in the config file because this takes priority over the short name ( STAR_ALIGN ) and allows existing configuration using the full process name to be correctly overridden. If you get a warning suggesting that the process selector isn't recognised check that the process name has been specified correctly.","title":"Advanced option on process level"},{"location":"usage/#updating-containers-advanced-users","text":"The Nextflow DSL2 implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. If for some reason you need to use a different version of a particular tool with the pipeline then you just need to identify the process name and override the Nextflow container definition for that process using the withName declaration. For example, in the nf-core/viralrecon pipeline a tool called Pangolin has been used during the COVID-19 pandemic to assign lineages to SARS-CoV-2 genome sequenced samples. Given that the lineage assignments change quite frequently it doesn't make sense to re-release the nf-core/viralrecon everytime a new version of Pangolin has been released. However, you can override the default container used by the pipeline by creating a custom config file and passing it as a command-line argument via -c custom.config . Check the default version used by the pipeline in the module file for Pangolin Find the latest version of the Biocontainer available on Quay.io Create the custom config accordingly: For Docker: nextflow process { withName: PANGOLIN { container = 'quay.io/biocontainers/pangolin:3.0.5--pyhdfd78af_0' } } For Singularity: nextflow process { withName: PANGOLIN { container = 'https://depot.galaxyproject.org/singularity/pangolin:3.0.5--pyhdfd78af_0' } } For Conda: nextflow process { withName: PANGOLIN { conda = 'bioconda::pangolin=3.0.5' } } NB: If you wish to periodically update individual tool-specific results (e.g. Pangolin) generated by the pipeline then you must ensure to keep the work/ directory otherwise the -resume ability of the pipeline will be compromised and it will restart from scratch.","title":"Updating containers (advanced users)"},{"location":"usage/#running-in-the-background","text":"Nextflow handles job submissions and supervises the running jobs. The Nextflow process must run until the pipeline is finished. The Nextflow -bg flag launches Nextflow in the background, detached from your terminal so that the workflow does not stop if you log out of your session. The logs are saved to a file. Alternatively, you can use screen / tmux or similar tool to create a detached session which you can log back into at a later time. Some HPC setups also allow you to run nextflow within a cluster job submitted your job scheduler (from where it submits more jobs).","title":"Running in the background"},{"location":"usage/#nextflow-memory-requirements","text":"In some cases, the Nextflow Java virtual machines can start to request a large amount of memory. We recommend adding the following line to your environment to limit this (typically in ~/.bashrc or ~./bash_profile ): NXF_OPTS='-Xms1g -Xmx4g'","title":"Nextflow memory requirements"}]}
\ No newline at end of file
+{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Introduction dalmolingroup/euryale is a pipeline for taxonomic classification and functional annotation of metagenomic reads. Based on MEDUSA . The pipeline is built using Nextflow , a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. The Nextflow DSL2 implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. Where possible, these processes have been submitted to and installed from nf-core/modules in order to make them available to all nf-core pipelines, and to everyone within the Nextflow community! Pipeline summary Pre-processing Read QC ( FastQC ) Read trimming and merging ( fastp ) ( optionally ) Host read removal ( BowTie2 ) Duplicated sequence removal ( fastx collapser ) Present QC and other data ( MultiQC ) Assembly ( optionally ) Read assembly ( MEGAHIT ) Taxonomic classification Sequence classification ( Kaiju ) ( WIP ) Sequence classification ( Kraken2 ) Visualization ( Krona ) Functional annotation Sequence alignment ( DIAMOND ) Map alignment matches to functional database ( annotate ) Quick Start Install Nextflow ( >=22.10.1 ) Install any of Docker , Singularity (you can follow this tutorial ), Podman , Shifter or Charliecloud for full pipeline reproducibility (you can use Conda both to install Nextflow itself and also to manage software within pipelines. Please only use it within pipelines as a last resort; see docs ) . Download the pipeline and test it on a minimal dataset with a single command: nextflow run dalmolingroup/euryale -profile test,YOURPROFILE --outdir <OUTDIR> Note that some form of configuration will be needed so that Nextflow knows how to fetch the required software. This is usually done in the form of a config profile ( YOURPROFILE in the example command above). You can chain multiple config profiles in a comma-separated string. The pipeline comes with config profiles called docker , singularity , podman , shifter , charliecloud and conda which instruct the pipeline to use the named tool for software management. For example, -profile test,docker . Please check nf-core/configs to see if a custom config file to run nf-core pipelines already exists for your Institute. If so, you can simply use -profile <institute> in your command. This will enable either docker or singularity and set the appropriate execution settings for your local compute environment. If you are using singularity , please use the nf-core download command to download images first, before running the pipeline. Setting the NXF_SINGULARITY_CACHEDIR or singularity.cacheDir Nextflow options enables you to store and re-use the images from a central location for future pipeline runs. If you are using conda , it is highly recommended to use the NXF_CONDA_CACHEDIR or conda.cacheDir settings to store the environments in a central location for future pipeline runs. Start running your own analysis! nextflow run dalmolingroup/euryale \\ --input samplesheet.csv \\ --outdir <OUTDIR> \\ --kaiju_db kaiju_reference \\ --diamond_db diamond_db \\ --reference_fasta diamond_fasta \\ --host_fasta host_reference_fasta \\ --id_mapping id_mapping_file \\ -profile <docker/singularity/podman/shifter/charliecloud/conda/institute> Documentation The dalmolingroup/euryale documentation is split into the following pages: Usage An overview of how the pipeline works, how to run it and a description of all of the different command-line flags. Output An overview of the different results produced by the pipeline and how to interpret them. Credits dalmolingroup/euryale was originally written by Jo\u00e3o Cavalcante. We thank the following people for their extensive assistance in the development of this pipeline: Diego Morais (for developing the original MEDUSA pipeline) Citations Morais DAA, Cavalcante JVF, Monteiro SS, Pasquali MAB and Dalmolin RJS (2022) MEDUSA: A Pipeline for Sensitive Taxonomic Classification and Flexible Functional Annotation of Metagenomic Shotgun Sequences. Front. Genet. 13:814437. doi: 10.3389/fgene.2022.814437 This pipeline uses code and infrastructure developed and maintained by the nf-core community, reused here under the MIT license . The nf-core framework for community-curated bioinformatics pipelines. Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen. Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x .","title":"Home"},{"location":"#introduction","text":"dalmolingroup/euryale is a pipeline for taxonomic classification and functional annotation of metagenomic reads. Based on MEDUSA . The pipeline is built using Nextflow , a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. The Nextflow DSL2 implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. Where possible, these processes have been submitted to and installed from nf-core/modules in order to make them available to all nf-core pipelines, and to everyone within the Nextflow community!","title":"Introduction"},{"location":"#pipeline-summary","text":"","title":"Pipeline summary"},{"location":"#pre-processing","text":"Read QC ( FastQC ) Read trimming and merging ( fastp ) ( optionally ) Host read removal ( BowTie2 ) Duplicated sequence removal ( fastx collapser ) Present QC and other data ( MultiQC )","title":"Pre-processing"},{"location":"#assembly","text":"( optionally ) Read assembly ( MEGAHIT )","title":"Assembly"},{"location":"#taxonomic-classification","text":"Sequence classification ( Kaiju ) ( WIP ) Sequence classification ( Kraken2 ) Visualization ( Krona )","title":"Taxonomic classification"},{"location":"#functional-annotation","text":"Sequence alignment ( DIAMOND ) Map alignment matches to functional database ( annotate )","title":"Functional annotation"},{"location":"#quick-start","text":"Install Nextflow ( >=22.10.1 ) Install any of Docker , Singularity (you can follow this tutorial ), Podman , Shifter or Charliecloud for full pipeline reproducibility (you can use Conda both to install Nextflow itself and also to manage software within pipelines. Please only use it within pipelines as a last resort; see docs ) . Download the pipeline and test it on a minimal dataset with a single command: nextflow run dalmolingroup/euryale -profile test,YOURPROFILE --outdir <OUTDIR> Note that some form of configuration will be needed so that Nextflow knows how to fetch the required software. This is usually done in the form of a config profile ( YOURPROFILE in the example command above). You can chain multiple config profiles in a comma-separated string. The pipeline comes with config profiles called docker , singularity , podman , shifter , charliecloud and conda which instruct the pipeline to use the named tool for software management. For example, -profile test,docker . Please check nf-core/configs to see if a custom config file to run nf-core pipelines already exists for your Institute. If so, you can simply use -profile <institute> in your command. This will enable either docker or singularity and set the appropriate execution settings for your local compute environment. If you are using singularity , please use the nf-core download command to download images first, before running the pipeline. Setting the NXF_SINGULARITY_CACHEDIR or singularity.cacheDir Nextflow options enables you to store and re-use the images from a central location for future pipeline runs. If you are using conda , it is highly recommended to use the NXF_CONDA_CACHEDIR or conda.cacheDir settings to store the environments in a central location for future pipeline runs. Start running your own analysis! nextflow run dalmolingroup/euryale \\ --input samplesheet.csv \\ --outdir <OUTDIR> \\ --kaiju_db kaiju_reference \\ --diamond_db diamond_db \\ --reference_fasta diamond_fasta \\ --host_fasta host_reference_fasta \\ --id_mapping id_mapping_file \\ -profile <docker/singularity/podman/shifter/charliecloud/conda/institute>","title":"Quick Start"},{"location":"#documentation","text":"The dalmolingroup/euryale documentation is split into the following pages: Usage An overview of how the pipeline works, how to run it and a description of all of the different command-line flags. Output An overview of the different results produced by the pipeline and how to interpret them.","title":"Documentation"},{"location":"#credits","text":"dalmolingroup/euryale was originally written by Jo\u00e3o Cavalcante. We thank the following people for their extensive assistance in the development of this pipeline: Diego Morais (for developing the original MEDUSA pipeline)","title":"Credits"},{"location":"#citations","text":"Morais DAA, Cavalcante JVF, Monteiro SS, Pasquali MAB and Dalmolin RJS (2022) MEDUSA: A Pipeline for Sensitive Taxonomic Classification and Flexible Functional Annotation of Metagenomic Shotgun Sequences. Front. Genet. 13:814437. doi: 10.3389/fgene.2022.814437 This pipeline uses code and infrastructure developed and maintained by the nf-core community, reused here under the MIT license . The nf-core framework for community-curated bioinformatics pipelines. Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen. Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x .","title":"Citations"},{"location":"CITATIONS/","text":"dalmolingroup/euryale: Citations nf-core Ewels PA, Peltzer A, Fillinger S, Patel H, Alneberg J, Wilm A, Garcia MU, Di Tommaso P, Nahnsen S. The nf-core framework for community-curated bioinformatics pipelines. Nat Biotechnol. 2020 Mar;38(3):276-278. doi: 10.1038/s41587-020-0439-x. PubMed PMID: 32055031. Nextflow Di Tommaso P, Chatzou M, Floden EW, Barja PP, Palumbo E, Notredame C. Nextflow enables reproducible computational workflows. Nat Biotechnol. 2017 Apr 11;35(4):316-319. doi: 10.1038/nbt.3820. PubMed PMID: 28398311. Pipeline tools FastQC MultiQC Ewels P, Magnusson M, Lundin S, K\u00e4ller M. MultiQC: summarize analysis results for multiple tools and samples in a single report. Bioinformatics. 2016 Oct 1;32(19):3047-8. doi: 10.1093/bioinformatics/btw354. Epub 2016 Jun 16. PubMed PMID: 27312411; PubMed Central PMCID: PMC5039924. Software packaging/containerisation tools Anaconda Anaconda Software Distribution. Computer software. Vers. 2-2.4.0. Anaconda, Nov. 2016. Web. Bioconda Gr\u00fcning B, Dale R, Sj\u00f6din A, Chapman BA, Rowe J, Tomkins-Tinch CH, Valieris R, K\u00f6ster J; Bioconda Team. Bioconda: sustainable and comprehensive software distribution for the life sciences. Nat Methods. 2018 Jul;15(7):475-476. doi: 10.1038/s41592-018-0046-7. PubMed PMID: 29967506. BioContainers da Veiga Leprevost F, Gr\u00fcning B, Aflitos SA, R\u00f6st HL, Uszkoreit J, Barsnes H, Vaudel M, Moreno P, Gatto L, Weber J, Bai M, Jimenez RC, Sachsenberg T, Pfeuffer J, Alvarez RV, Griss J, Nesvizhskii AI, Perez-Riverol Y. BioContainers: an open-source and community-driven framework for software standardization. Bioinformatics. 2017 Aug 15;33(16):2580-2582. doi: 10.1093/bioinformatics/btx192. PubMed PMID: 28379341; PubMed Central PMCID: PMC5870671. Docker Singularity Kurtzer GM, Sochat V, Bauer MW. Singularity: Scientific containers for mobility of compute. PLoS One. 2017 May 11;12(5):e0177459. doi: 10.1371/journal.pone.0177459. eCollection 2017. PubMed PMID: 28494014; PubMed Central PMCID: PMC5426675.","title":"Citations"},{"location":"CITATIONS/#dalmolingroupeuryale-citations","text":"","title":"dalmolingroup/euryale: Citations"},{"location":"CITATIONS/#nf-core","text":"Ewels PA, Peltzer A, Fillinger S, Patel H, Alneberg J, Wilm A, Garcia MU, Di Tommaso P, Nahnsen S. The nf-core framework for community-curated bioinformatics pipelines. Nat Biotechnol. 2020 Mar;38(3):276-278. doi: 10.1038/s41587-020-0439-x. PubMed PMID: 32055031.","title":"nf-core"},{"location":"CITATIONS/#nextflow","text":"Di Tommaso P, Chatzou M, Floden EW, Barja PP, Palumbo E, Notredame C. Nextflow enables reproducible computational workflows. Nat Biotechnol. 2017 Apr 11;35(4):316-319. doi: 10.1038/nbt.3820. PubMed PMID: 28398311.","title":"Nextflow"},{"location":"CITATIONS/#pipeline-tools","text":"FastQC MultiQC Ewels P, Magnusson M, Lundin S, K\u00e4ller M. MultiQC: summarize analysis results for multiple tools and samples in a single report. Bioinformatics. 2016 Oct 1;32(19):3047-8. doi: 10.1093/bioinformatics/btw354. Epub 2016 Jun 16. PubMed PMID: 27312411; PubMed Central PMCID: PMC5039924.","title":"Pipeline tools"},{"location":"CITATIONS/#software-packagingcontainerisation-tools","text":"Anaconda Anaconda Software Distribution. Computer software. Vers. 2-2.4.0. Anaconda, Nov. 2016. Web. Bioconda Gr\u00fcning B, Dale R, Sj\u00f6din A, Chapman BA, Rowe J, Tomkins-Tinch CH, Valieris R, K\u00f6ster J; Bioconda Team. Bioconda: sustainable and comprehensive software distribution for the life sciences. Nat Methods. 2018 Jul;15(7):475-476. doi: 10.1038/s41592-018-0046-7. PubMed PMID: 29967506. BioContainers da Veiga Leprevost F, Gr\u00fcning B, Aflitos SA, R\u00f6st HL, Uszkoreit J, Barsnes H, Vaudel M, Moreno P, Gatto L, Weber J, Bai M, Jimenez RC, Sachsenberg T, Pfeuffer J, Alvarez RV, Griss J, Nesvizhskii AI, Perez-Riverol Y. BioContainers: an open-source and community-driven framework for software standardization. Bioinformatics. 2017 Aug 15;33(16):2580-2582. doi: 10.1093/bioinformatics/btx192. PubMed PMID: 28379341; PubMed Central PMCID: PMC5870671. Docker Singularity Kurtzer GM, Sochat V, Bauer MW. Singularity: Scientific containers for mobility of compute. PLoS One. 2017 May 11;12(5):e0177459. doi: 10.1371/journal.pone.0177459. eCollection 2017. PubMed PMID: 28494014; PubMed Central PMCID: PMC5426675.","title":"Software packaging/containerisation tools"},{"location":"output/","text":"dalmolingroup/euryale: Output Introduction This document describes the output produced by the pipeline. Most of the plots are taken from the MultiQC report, which summarises results at the end of the pipeline. The directories listed below will be created in the results directory after the pipeline has finished. All paths are relative to the top-level results directory. Pipeline overview The pipeline is built using Nextflow and processes data using the following steps (steps in italics don't run by default): Kaiju - Taxonomically classify reads or contigs Krona - Visualize the taxonomic classification for each sample. Diamond - Alignment reads and contigs against a reference database (such as NCBI-nr). Annotate - Functional annotation of alignment matches. MEGAHIT - Assembled contigs. MultiQC - Aggregate report describing results and QC from the whole pipeline Pipeline information - Report metrics generated during the workflow execution Kaiju Output files - `taxonomy/` - `${sample}.tsv`: Kaiju classification output. - `${sample}.txt`: Kaiju2Table TSV output. Kaiju is a software to perform fast taxonomic classification of metagenomic sequencing reads using a protein reference database. Krona Output files - `taxonomy/` - `${sample}.html`: Krona visualization for the sample. Krona is a tool to interactively explore metagenomes and more from a web browser. Diamond Output files - `alignment/${sample}/` - `${sample}.txt`: Alignment matches in blast tabular output format. - `${sample}.log`: DIAMOND execution log. DIAMOND is an accelerated BLAST compatible local sequence aligner. Annotate Output files - `functional/${sample}/` - `${sample}_annotated.txt`: Alignment matches annotated to chosen functional database (e.g. GO). Annotate is a tool to annotate each query using the best alignment for which a mapping is known. MEGAHIT Output files - `assembly/${sample}/` - `${sample}.contigs.fa.gz`: Contigs assembled for the sample. MEGAHIT is an ultra-fast and memory-efficient (meta-)genome assembler DIAMOND database Output files - `diamond_db/` - `${database_name}.dmnd`: DIAMOND database for the reference fasta file. This output is present if you add the --save_db parameter. DIAMOND is an accelerated BLAST compatible local sequence aligner. MultiQC Output files - `multiqc/` - `multiqc_report.html`: a standalone HTML file that can be viewed in your web browser. - `multiqc_data/`: directory containing parsed statistics from the different tools used in the pipeline. - `multiqc_plots/`: directory containing static images from the report in various formats. MultiQC is a visualization tool that generates a single HTML report summarising all samples in your project. Most of the pipeline QC results are visualised in the report and further statistics are available in the report data directory. Results generated by MultiQC collate pipeline QC from supported tools e.g. FastQC. The pipeline has special steps which also allow the software versions to be reported in the MultiQC output for future traceability. For more information about how to use MultiQC reports, see http://multiqc.info . Pipeline information Output files - `pipeline_info/` - Reports generated by Nextflow: `execution_report.html`, `execution_timeline.html`, `execution_trace.txt` and `pipeline_dag.dot`/`pipeline_dag.svg`. - Reports generated by the pipeline: `pipeline_report.html`, `pipeline_report.txt` and `software_versions.yml`. The `pipeline_report*` files will only be present if the `--email` / `--email_on_fail` parameter's are used when running the pipeline. - Reformatted samplesheet files used as input to the pipeline: `samplesheet.valid.csv`. Nextflow provides excellent functionality for generating various reports relevant to the running and execution of the pipeline. This will allow you to troubleshoot errors with the running of the pipeline, and also provide you with other information such as launch commands, run times and resource usage.","title":"Output"},{"location":"output/#dalmolingroupeuryale-output","text":"","title":"dalmolingroup/euryale: Output"},{"location":"output/#introduction","text":"This document describes the output produced by the pipeline. Most of the plots are taken from the MultiQC report, which summarises results at the end of the pipeline. The directories listed below will be created in the results directory after the pipeline has finished. All paths are relative to the top-level results directory.","title":"Introduction"},{"location":"output/#pipeline-overview","text":"The pipeline is built using Nextflow and processes data using the following steps (steps in italics don't run by default): Kaiju - Taxonomically classify reads or contigs Krona - Visualize the taxonomic classification for each sample. Diamond - Alignment reads and contigs against a reference database (such as NCBI-nr). Annotate - Functional annotation of alignment matches. MEGAHIT - Assembled contigs. MultiQC - Aggregate report describing results and QC from the whole pipeline Pipeline information - Report metrics generated during the workflow execution","title":"Pipeline overview"},{"location":"output/#kaiju","text":"Output files - `taxonomy/` - `${sample}.tsv`: Kaiju classification output. - `${sample}.txt`: Kaiju2Table TSV output. Kaiju is a software to perform fast taxonomic classification of metagenomic sequencing reads using a protein reference database.","title":"Kaiju"},{"location":"output/#krona","text":"Output files - `taxonomy/` - `${sample}.html`: Krona visualization for the sample. Krona is a tool to interactively explore metagenomes and more from a web browser.","title":"Krona"},{"location":"output/#diamond","text":"Output files - `alignment/${sample}/` - `${sample}.txt`: Alignment matches in blast tabular output format. - `${sample}.log`: DIAMOND execution log. DIAMOND is an accelerated BLAST compatible local sequence aligner.","title":"Diamond"},{"location":"output/#annotate","text":"Output files - `functional/${sample}/` - `${sample}_annotated.txt`: Alignment matches annotated to chosen functional database (e.g. GO). Annotate is a tool to annotate each query using the best alignment for which a mapping is known.","title":"Annotate"},{"location":"output/#megahit","text":"Output files - `assembly/${sample}/` - `${sample}.contigs.fa.gz`: Contigs assembled for the sample. MEGAHIT is an ultra-fast and memory-efficient (meta-)genome assembler","title":"MEGAHIT"},{"location":"output/#diamond-database","text":"Output files - `diamond_db/` - `${database_name}.dmnd`: DIAMOND database for the reference fasta file. This output is present if you add the --save_db parameter. DIAMOND is an accelerated BLAST compatible local sequence aligner.","title":"DIAMOND database"},{"location":"output/#multiqc","text":"Output files - `multiqc/` - `multiqc_report.html`: a standalone HTML file that can be viewed in your web browser. - `multiqc_data/`: directory containing parsed statistics from the different tools used in the pipeline. - `multiqc_plots/`: directory containing static images from the report in various formats. MultiQC is a visualization tool that generates a single HTML report summarising all samples in your project. Most of the pipeline QC results are visualised in the report and further statistics are available in the report data directory. Results generated by MultiQC collate pipeline QC from supported tools e.g. FastQC. The pipeline has special steps which also allow the software versions to be reported in the MultiQC output for future traceability. For more information about how to use MultiQC reports, see http://multiqc.info .","title":"MultiQC"},{"location":"output/#pipeline-information","text":"Output files - `pipeline_info/` - Reports generated by Nextflow: `execution_report.html`, `execution_timeline.html`, `execution_trace.txt` and `pipeline_dag.dot`/`pipeline_dag.svg`. - Reports generated by the pipeline: `pipeline_report.html`, `pipeline_report.txt` and `software_versions.yml`. The `pipeline_report*` files will only be present if the `--email` / `--email_on_fail` parameter's are used when running the pipeline. - Reformatted samplesheet files used as input to the pipeline: `samplesheet.valid.csv`. Nextflow provides excellent functionality for generating various reports relevant to the running and execution of the pipeline. This will allow you to troubleshoot errors with the running of the pipeline, and also provide you with other information such as launch commands, run times and resource usage.","title":"Pipeline information"},{"location":"params/","text":"dalmolingroup/euryale pipeline parameters A pipeline for metagenomic taxonomic classification and functional annotation. Based on MEDUSA. Input/output options Define where the pipeline should find input data and save output data. Parameter Description Type Default Required Hidden input Path to comma-separated file containing information about the samples in the experiment. Help You will need to create a design file with information about the samples in your experiment before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row. string True outdir The output directory where the results will be saved. You have to use absolute paths to storage on Cloud infrastructure. string True email Email address for completion summary. Help Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file ( ~/.nextflow/config ) then you don't need to specify this on the command line for every run. string multiqc_title MultiQC report title. Printed as page header, used for filename if not otherwise specified. string save_dbs Save DIAMOND db to results directory after construction boolean Skip Steps Choose to skip pipeline steps Parameter Description Type Default Required Hidden skip_classification Skip taxonomic classification boolean skip_alignment Skip alignment boolean skip_functional Skip functional annotation boolean Decontamination Parameter Description Type Default Required Hidden host_fasta string None Alignment Parameter Description Type Default Required Hidden reference_fasta Path to FASTA genome file. string None diamond_db Path to pre-built DIAMOND db. string None Taxonomy Parameter Description Type Default Required Hidden kaiju_db string None True Functional Parameter Description Type Default Required Hidden id_mapping Path to ID mapping file to be used for the Functional annotation string None True minimum_bitscore Minimum bitscore of a match to be used for annotation integer 50 minimum_pident Minimum identity of a match to be used for annotation integer 80 minimum_alen Minimum alignment length of a match to be used for annotation integer 50 maximum_evalue Maximum evalue of a match to be used for annotation number 0.0001 Assembly Parameter Description Type Default Required Hidden assembly_based boolean Reference genome options Reference genome related files and options required for the workflow. Parameter Description Type Default Required Hidden genome Name of iGenomes reference. Help If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. --genome GRCh38 . See the nf-core website docs for more details. string igenomes_base Directory / URL base for iGenomes references. string s3://ngi-igenomes/igenomes True igenomes_ignore Do not load the iGenomes reference config. Help Do not load igenomes.config when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in igenomes.config . boolean True fasta string None Max job request options Set the top limit for requested resources for any single job. Parameter Description Type Default Required Hidden max_cpus Maximum number of CPUs that can be requested for any single job. Help Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. --max_cpus 1 integer 16 True max_memory Maximum amount of memory that can be requested for any single job. Help Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. --max_memory '8.GB' string 128.GB True max_time Maximum amount of time that can be requested for any single job. Help Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. --max_time '2.h' string 240.h True Generic options Less common options for the pipeline, typically set in a config file. Parameter Description Type Default Required Hidden help Display help text. boolean True version Display version and exit. boolean True publish_dir_mode Method used to save pipeline results to output directory. Help The Nextflow publishDir option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See Nextflow docs for details. string copy True email_on_fail Email address for completion summary, only when pipeline fails. Help An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully. string True plaintext_email Send plain-text email instead of HTML. boolean True max_multiqc_email_size File size limit when attaching MultiQC reports to summary emails. string 25.MB True monochrome_logs Do not use coloured log outputs. boolean True hook_url Incoming hook URL for messaging service Help Incoming hook URL for messaging service. Currently, MS Teams and Slack are supported. string True multiqc_config Custom config file to supply to MultiQC. string True multiqc_logo Custom logo file to supply to MultiQC. File name must also be set in the MultiQC config file string True multiqc_methods_description Custom MultiQC yaml file containing HTML including a methods description. string tracedir Directory to keep pipeline Nextflow logs and reports. string ${params.outdir}/pipeline_info True validate_params Boolean whether to validate parameters against the schema at runtime boolean True True show_hidden_params Show all params when using --help Help By default, parameters set as hidden in the schema are not shown on the command line when a user runs with --help . Specifying this option will tell the pipeline to show all parameters. boolean True Other parameters Parameter Description Type Default Required Hidden schema_ignore_params string genomes","title":"Parameters"},{"location":"params/#dalmolingroupeuryale-pipeline-parameters","text":"A pipeline for metagenomic taxonomic classification and functional annotation. Based on MEDUSA.","title":"dalmolingroup/euryale pipeline parameters"},{"location":"params/#inputoutput-options","text":"Define where the pipeline should find input data and save output data. Parameter Description Type Default Required Hidden input Path to comma-separated file containing information about the samples in the experiment. Help You will need to create a design file with information about the samples in your experiment before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row. string True outdir The output directory where the results will be saved. You have to use absolute paths to storage on Cloud infrastructure. string True email Email address for completion summary. Help Set this parameter to your e-mail address to get a summary e-mail with details of the run sent to you when the workflow exits. If set in your user config file ( ~/.nextflow/config ) then you don't need to specify this on the command line for every run. string multiqc_title MultiQC report title. Printed as page header, used for filename if not otherwise specified. string save_dbs Save DIAMOND db to results directory after construction boolean","title":"Input/output options"},{"location":"params/#skip-steps","text":"Choose to skip pipeline steps Parameter Description Type Default Required Hidden skip_classification Skip taxonomic classification boolean skip_alignment Skip alignment boolean skip_functional Skip functional annotation boolean","title":"Skip Steps"},{"location":"params/#decontamination","text":"Parameter Description Type Default Required Hidden host_fasta string None","title":"Decontamination"},{"location":"params/#alignment","text":"Parameter Description Type Default Required Hidden reference_fasta Path to FASTA genome file. string None diamond_db Path to pre-built DIAMOND db. string None","title":"Alignment"},{"location":"params/#taxonomy","text":"Parameter Description Type Default Required Hidden kaiju_db string None True","title":"Taxonomy"},{"location":"params/#functional","text":"Parameter Description Type Default Required Hidden id_mapping Path to ID mapping file to be used for the Functional annotation string None True minimum_bitscore Minimum bitscore of a match to be used for annotation integer 50 minimum_pident Minimum identity of a match to be used for annotation integer 80 minimum_alen Minimum alignment length of a match to be used for annotation integer 50 maximum_evalue Maximum evalue of a match to be used for annotation number 0.0001","title":"Functional"},{"location":"params/#assembly","text":"Parameter Description Type Default Required Hidden assembly_based boolean","title":"Assembly"},{"location":"params/#reference-genome-options","text":"Reference genome related files and options required for the workflow. Parameter Description Type Default Required Hidden genome Name of iGenomes reference. Help If using a reference genome configured in the pipeline using iGenomes, use this parameter to give the ID for the reference. This is then used to build the full paths for all required reference genome files e.g. --genome GRCh38 . See the nf-core website docs for more details. string igenomes_base Directory / URL base for iGenomes references. string s3://ngi-igenomes/igenomes True igenomes_ignore Do not load the iGenomes reference config. Help Do not load igenomes.config when running the pipeline. You may choose this option if you observe clashes between custom parameters and those supplied in igenomes.config . boolean True fasta string None","title":"Reference genome options"},{"location":"params/#max-job-request-options","text":"Set the top limit for requested resources for any single job. Parameter Description Type Default Required Hidden max_cpus Maximum number of CPUs that can be requested for any single job. Help Use to set an upper-limit for the CPU requirement for each process. Should be an integer e.g. --max_cpus 1 integer 16 True max_memory Maximum amount of memory that can be requested for any single job. Help Use to set an upper-limit for the memory requirement for each process. Should be a string in the format integer-unit e.g. --max_memory '8.GB' string 128.GB True max_time Maximum amount of time that can be requested for any single job. Help Use to set an upper-limit for the time requirement for each process. Should be a string in the format integer-unit e.g. --max_time '2.h' string 240.h True","title":"Max job request options"},{"location":"params/#generic-options","text":"Less common options for the pipeline, typically set in a config file. Parameter Description Type Default Required Hidden help Display help text. boolean True version Display version and exit. boolean True publish_dir_mode Method used to save pipeline results to output directory. Help The Nextflow publishDir option specifies which intermediate files should be saved to the output directory. This option tells the pipeline what method should be used to move these files. See Nextflow docs for details. string copy True email_on_fail Email address for completion summary, only when pipeline fails. Help An email address to send a summary email to when the pipeline is completed - ONLY sent if the pipeline does not exit successfully. string True plaintext_email Send plain-text email instead of HTML. boolean True max_multiqc_email_size File size limit when attaching MultiQC reports to summary emails. string 25.MB True monochrome_logs Do not use coloured log outputs. boolean True hook_url Incoming hook URL for messaging service Help Incoming hook URL for messaging service. Currently, MS Teams and Slack are supported. string True multiqc_config Custom config file to supply to MultiQC. string True multiqc_logo Custom logo file to supply to MultiQC. File name must also be set in the MultiQC config file string True multiqc_methods_description Custom MultiQC yaml file containing HTML including a methods description. string tracedir Directory to keep pipeline Nextflow logs and reports. string ${params.outdir}/pipeline_info True validate_params Boolean whether to validate parameters against the schema at runtime boolean True True show_hidden_params Show all params when using --help Help By default, parameters set as hidden in the schema are not shown on the command line when a user runs with --help . Specifying this option will tell the pipeline to show all parameters. boolean True","title":"Generic options"},{"location":"params/#other-parameters","text":"Parameter Description Type Default Required Hidden schema_ignore_params string genomes","title":"Other parameters"},{"location":"usage/","text":"dalmolingroup/euryale: Usage Documentation of pipeline parameters is generated automatically from the pipeline schema and can no longer be found in markdown files. Introduction Samplesheet input You will need to create a samplesheet with information about the samples you would like to analyse before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row as shown in the examples below. --input '[path to samplesheet file]' Multiple runs of the same sample The sample identifiers have to be the same when you have re-sequenced the same sample more than once e.g. to increase sequencing depth. The pipeline will concatenate the raw reads before performing any downstream analysis. Below is an example for the same sample sequenced across 3 lanes: sample,fastq_1,fastq_2 CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz CONTROL_REP1,AEG588A1_S1_L003_R1_001.fastq.gz,AEG588A1_S1_L003_R2_001.fastq.gz CONTROL_REP1,AEG588A1_S1_L004_R1_001.fastq.gz,AEG588A1_S1_L004_R2_001.fastq.gz Full samplesheet The pipeline will auto-detect whether a sample is single- or paired-end using the information provided in the samplesheet. The samplesheet can have as many columns as you desire, however, there is a strict requirement for the first 3 columns to match those defined in the table below. A final samplesheet file consisting of both single- and paired-end data may look something like the one below. This is for 6 samples, where TREATMENT_REP3 has been sequenced twice. sample,fastq_1,fastq_2 CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz CONTROL_REP2,AEG588A2_S2_L002_R1_001.fastq.gz,AEG588A2_S2_L002_R2_001.fastq.gz CONTROL_REP3,AEG588A3_S3_L002_R1_001.fastq.gz,AEG588A3_S3_L002_R2_001.fastq.gz TREATMENT_REP1,AEG588A4_S4_L003_R1_001.fastq.gz, TREATMENT_REP2,AEG588A5_S5_L003_R1_001.fastq.gz, TREATMENT_REP3,AEG588A6_S6_L003_R1_001.fastq.gz, TREATMENT_REP3,AEG588A6_S6_L004_R1_001.fastq.gz, Column Description sample Custom sample name. This entry will be identical for multiple sequencing libraries/runs from the same sample. Spaces in sample names are automatically converted to underscores ( _ ). fastq_1 Full path to FastQ file for Illumina short reads 1. File has to be gzipped and have the extension \".fastq.gz\" or \".fq.gz\". fastq_2 Full path to FastQ file for Illumina short reads 2. File has to be gzipped and have the extension \".fastq.gz\" or \".fq.gz\". An example samplesheet has been provided with the pipeline. Running the pipeline The typical command for running the pipeline is as follows: nextflow run dalmolingroup/euryale --input samplesheet.csv --outdir <OUTDIR> --genome GRCh37 -profile docker This will launch the pipeline with the docker configuration profile. See below for more information about profiles. Note that the pipeline will create the following files in your working directory: work # Directory containing the nextflow working files <OUTDIR> # Finished results in specified location (defined with --outdir) .nextflow_log # Log file from Nextflow # Other nextflow hidden files, eg. history of pipeline runs and old logs. Updating the pipeline When you run the above command, Nextflow automatically pulls the pipeline code from GitHub and stores it as a cached version. When running the pipeline after this, it will always use the cached version if available - even if the pipeline has been updated since. To make sure that you're running the latest version of the pipeline, make sure that you regularly update the cached version of the pipeline: nextflow pull dalmolingroup/euryale Reproducibility It is a good idea to specify a pipeline version when running the pipeline on your data. This ensures that a specific version of the pipeline code and software are used when you run your pipeline. If you keep using the same tag, you'll be running the same version of the pipeline, even if there have been changes to the code since. First, go to the dalmolingroup/euryale releases page and find the latest pipeline version - numeric only (eg. 1.3.1 ). Then specify this when running the pipeline with -r (one hyphen) - eg. -r 1.3.1 . Of course, you can switch to another version by changing the number after the -r flag. This version number will be logged in reports when you run the pipeline, so that you'll know what you used when you look back in the future. For example, at the bottom of the MultiQC reports. Core Nextflow arguments NB: These options are part of Nextflow and use a single hyphen (pipeline parameters use a double-hyphen). -profile Use this parameter to choose a configuration profile. Profiles can give configuration presets for different compute environments. Several generic profiles are bundled with the pipeline which instruct the pipeline to use software packaged using different methods (Docker, Singularity, Podman, Shifter, Charliecloud, Conda) - see below. We highly recommend the use of Docker or Singularity containers for full pipeline reproducibility, however when this is not possible, Conda is also supported. Note that multiple profiles can be loaded, for example: -profile test,docker - the order of arguments is important! They are loaded in sequence, so later profiles can overwrite earlier profiles. If -profile is not specified, the pipeline will run locally and expect all software to be installed and available on the PATH . This is not recommended, since it can lead to different results on different machines dependent on the computer enviroment. test A profile with a complete configuration for automated testing Includes links to test data so needs no other parameters docker A generic configuration profile to be used with Docker singularity A generic configuration profile to be used with Singularity podman A generic configuration profile to be used with Podman shifter A generic configuration profile to be used with Shifter charliecloud A generic configuration profile to be used with Charliecloud conda A generic configuration profile to be used with Conda . Please only use Conda as a last resort i.e. when it's not possible to run the pipeline with Docker, Singularity, Podman, Shifter or Charliecloud. -resume Specify this when restarting a pipeline. Nextflow will use cached results from any pipeline steps where the inputs are the same, continuing from where it got to previously. For input to be considered the same, not only the names must be identical but the files' contents as well. For more info about this parameter, see this blog post . You can also supply a run name to resume a specific run: -resume [run-name] . Use the nextflow log command to show previous run names. -c Specify the path to a specific config file (this is a core Nextflow command). See the nf-core website documentation for more information. Custom configuration Resource requests Whilst the default requirements set within the pipeline will hopefully work for most people and with most input data, you may find that you want to customise the compute resources that the pipeline requests. Each step in the pipeline has a default set of requirements for number of CPUs, memory and time. For most of the steps in the pipeline, if the job exits with any of the error codes specified here it will automatically be resubmitted with higher requests (2 x original, then 3 x original). If it still fails after the third attempt then the pipeline execution is stopped. For example, if the nf-core/rnaseq pipeline is failing after multiple re-submissions of the STAR_ALIGN process due to an exit code of 137 this would indicate that there is an out of memory issue: [62/149eb0] NOTE: Process `NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)` terminated with an error exit status (137) -- Execution is retried (1) Error executing process > 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)' Caused by: Process `NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)` terminated with an error exit status (137) Command executed: STAR \\ --genomeDir star \\ --readFilesIn WT_REP1_trimmed.fq.gz \\ --runThreadN 2 \\ --outFileNamePrefix WT_REP1. \\ <TRUNCATED> Command exit status: 137 Command output: (empty) Command error: .command.sh: line 9: 30 Killed STAR --genomeDir star --readFilesIn WT_REP1_trimmed.fq.gz --runThreadN 2 --outFileNamePrefix WT_REP1. <TRUNCATED> Work dir: /home/pipelinetest/work/9d/172ca5881234073e8d76f2a19c88fb Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run` For beginners A first step to bypass this error, you could try to increase the amount of CPUs, memory, and time for the whole pipeline. Therefor you can try to increase the resource for the parameters --max_cpus , --max_memory , and --max_time . Based on the error above, you have to increase the amount of memory. Therefore you can go to the parameter documentation of rnaseq and scroll down to the show hidden parameter button to get the default value for --max_memory . In this case 128GB, you than can try to run your pipeline again with --max_memory 200GB -resume to skip all process, that were already calculated. If you can not increase the resource of the complete pipeline, you can try to adapt the resource for a single process as mentioned below. Advanced option on process level To bypass this error you would need to find exactly which resources are set by the STAR_ALIGN process. The quickest way is to search for process STAR_ALIGN in the nf-core/rnaseq Github repo . We have standardised the structure of Nextflow DSL2 pipelines such that all module files will be present in the modules/ directory and so, based on the search results, the file we want is modules/nf-core/star/align/main.nf . If you click on the link to that file you will notice that there is a label directive at the top of the module that is set to label process_high . The Nextflow label directive allows us to organise workflow processes in separate groups which can be referenced in a configuration file to select and configure subset of processes having similar computing requirements. The default values for the process_high label are set in the pipeline's base.config which in this case is defined as 72GB. Providing you haven't set any other standard nf-core parameters to cap the maximum resources used by the pipeline then we can try and bypass the STAR_ALIGN process failure by creating a custom config file that sets at least 72GB of memory, in this case increased to 100GB. The custom config below can then be provided to the pipeline via the -c parameter as highlighted in previous sections. process { withName: 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN' { memory = 100.GB } } NB: We specify the full process name i.e. NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN in the config file because this takes priority over the short name ( STAR_ALIGN ) and allows existing configuration using the full process name to be correctly overridden. If you get a warning suggesting that the process selector isn't recognised check that the process name has been specified correctly. Updating containers (advanced users) The Nextflow DSL2 implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. If for some reason you need to use a different version of a particular tool with the pipeline then you just need to identify the process name and override the Nextflow container definition for that process using the withName declaration. For example, in the nf-core/viralrecon pipeline a tool called Pangolin has been used during the COVID-19 pandemic to assign lineages to SARS-CoV-2 genome sequenced samples. Given that the lineage assignments change quite frequently it doesn't make sense to re-release the nf-core/viralrecon everytime a new version of Pangolin has been released. However, you can override the default container used by the pipeline by creating a custom config file and passing it as a command-line argument via -c custom.config . Check the default version used by the pipeline in the module file for Pangolin Find the latest version of the Biocontainer available on Quay.io Create the custom config accordingly: For Docker: nextflow process { withName: PANGOLIN { container = 'quay.io/biocontainers/pangolin:3.0.5--pyhdfd78af_0' } } For Singularity: nextflow process { withName: PANGOLIN { container = 'https://depot.galaxyproject.org/singularity/pangolin:3.0.5--pyhdfd78af_0' } } For Conda: nextflow process { withName: PANGOLIN { conda = 'bioconda::pangolin=3.0.5' } } NB: If you wish to periodically update individual tool-specific results (e.g. Pangolin) generated by the pipeline then you must ensure to keep the work/ directory otherwise the -resume ability of the pipeline will be compromised and it will restart from scratch. Running in the background Nextflow handles job submissions and supervises the running jobs. The Nextflow process must run until the pipeline is finished. The Nextflow -bg flag launches Nextflow in the background, detached from your terminal so that the workflow does not stop if you log out of your session. The logs are saved to a file. Alternatively, you can use screen / tmux or similar tool to create a detached session which you can log back into at a later time. Some HPC setups also allow you to run nextflow within a cluster job submitted your job scheduler (from where it submits more jobs). Nextflow memory requirements In some cases, the Nextflow Java virtual machines can start to request a large amount of memory. We recommend adding the following line to your environment to limit this (typically in ~/.bashrc or ~./bash_profile ): NXF_OPTS='-Xms1g -Xmx4g'","title":"Usage"},{"location":"usage/#dalmolingroupeuryale-usage","text":"Documentation of pipeline parameters is generated automatically from the pipeline schema and can no longer be found in markdown files.","title":"dalmolingroup/euryale: Usage"},{"location":"usage/#introduction","text":"","title":"Introduction"},{"location":"usage/#samplesheet-input","text":"You will need to create a samplesheet with information about the samples you would like to analyse before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row as shown in the examples below. --input '[path to samplesheet file]'","title":"Samplesheet input"},{"location":"usage/#multiple-runs-of-the-same-sample","text":"The sample identifiers have to be the same when you have re-sequenced the same sample more than once e.g. to increase sequencing depth. The pipeline will concatenate the raw reads before performing any downstream analysis. Below is an example for the same sample sequenced across 3 lanes: sample,fastq_1,fastq_2 CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz CONTROL_REP1,AEG588A1_S1_L003_R1_001.fastq.gz,AEG588A1_S1_L003_R2_001.fastq.gz CONTROL_REP1,AEG588A1_S1_L004_R1_001.fastq.gz,AEG588A1_S1_L004_R2_001.fastq.gz","title":"Multiple runs of the same sample"},{"location":"usage/#full-samplesheet","text":"The pipeline will auto-detect whether a sample is single- or paired-end using the information provided in the samplesheet. The samplesheet can have as many columns as you desire, however, there is a strict requirement for the first 3 columns to match those defined in the table below. A final samplesheet file consisting of both single- and paired-end data may look something like the one below. This is for 6 samples, where TREATMENT_REP3 has been sequenced twice. sample,fastq_1,fastq_2 CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz CONTROL_REP2,AEG588A2_S2_L002_R1_001.fastq.gz,AEG588A2_S2_L002_R2_001.fastq.gz CONTROL_REP3,AEG588A3_S3_L002_R1_001.fastq.gz,AEG588A3_S3_L002_R2_001.fastq.gz TREATMENT_REP1,AEG588A4_S4_L003_R1_001.fastq.gz, TREATMENT_REP2,AEG588A5_S5_L003_R1_001.fastq.gz, TREATMENT_REP3,AEG588A6_S6_L003_R1_001.fastq.gz, TREATMENT_REP3,AEG588A6_S6_L004_R1_001.fastq.gz, Column Description sample Custom sample name. This entry will be identical for multiple sequencing libraries/runs from the same sample. Spaces in sample names are automatically converted to underscores ( _ ). fastq_1 Full path to FastQ file for Illumina short reads 1. File has to be gzipped and have the extension \".fastq.gz\" or \".fq.gz\". fastq_2 Full path to FastQ file for Illumina short reads 2. File has to be gzipped and have the extension \".fastq.gz\" or \".fq.gz\". An example samplesheet has been provided with the pipeline.","title":"Full samplesheet"},{"location":"usage/#running-the-pipeline","text":"The typical command for running the pipeline is as follows: nextflow run dalmolingroup/euryale --input samplesheet.csv --outdir <OUTDIR> --genome GRCh37 -profile docker This will launch the pipeline with the docker configuration profile. See below for more information about profiles. Note that the pipeline will create the following files in your working directory: work # Directory containing the nextflow working files <OUTDIR> # Finished results in specified location (defined with --outdir) .nextflow_log # Log file from Nextflow # Other nextflow hidden files, eg. history of pipeline runs and old logs.","title":"Running the pipeline"},{"location":"usage/#updating-the-pipeline","text":"When you run the above command, Nextflow automatically pulls the pipeline code from GitHub and stores it as a cached version. When running the pipeline after this, it will always use the cached version if available - even if the pipeline has been updated since. To make sure that you're running the latest version of the pipeline, make sure that you regularly update the cached version of the pipeline: nextflow pull dalmolingroup/euryale","title":"Updating the pipeline"},{"location":"usage/#reproducibility","text":"It is a good idea to specify a pipeline version when running the pipeline on your data. This ensures that a specific version of the pipeline code and software are used when you run your pipeline. If you keep using the same tag, you'll be running the same version of the pipeline, even if there have been changes to the code since. First, go to the dalmolingroup/euryale releases page and find the latest pipeline version - numeric only (eg. 1.3.1 ). Then specify this when running the pipeline with -r (one hyphen) - eg. -r 1.3.1 . Of course, you can switch to another version by changing the number after the -r flag. This version number will be logged in reports when you run the pipeline, so that you'll know what you used when you look back in the future. For example, at the bottom of the MultiQC reports.","title":"Reproducibility"},{"location":"usage/#core-nextflow-arguments","text":"NB: These options are part of Nextflow and use a single hyphen (pipeline parameters use a double-hyphen).","title":"Core Nextflow arguments"},{"location":"usage/#-profile","text":"Use this parameter to choose a configuration profile. Profiles can give configuration presets for different compute environments. Several generic profiles are bundled with the pipeline which instruct the pipeline to use software packaged using different methods (Docker, Singularity, Podman, Shifter, Charliecloud, Conda) - see below. We highly recommend the use of Docker or Singularity containers for full pipeline reproducibility, however when this is not possible, Conda is also supported. Note that multiple profiles can be loaded, for example: -profile test,docker - the order of arguments is important! They are loaded in sequence, so later profiles can overwrite earlier profiles. If -profile is not specified, the pipeline will run locally and expect all software to be installed and available on the PATH . This is not recommended, since it can lead to different results on different machines dependent on the computer enviroment. test A profile with a complete configuration for automated testing Includes links to test data so needs no other parameters docker A generic configuration profile to be used with Docker singularity A generic configuration profile to be used with Singularity podman A generic configuration profile to be used with Podman shifter A generic configuration profile to be used with Shifter charliecloud A generic configuration profile to be used with Charliecloud conda A generic configuration profile to be used with Conda . Please only use Conda as a last resort i.e. when it's not possible to run the pipeline with Docker, Singularity, Podman, Shifter or Charliecloud.","title":"-profile"},{"location":"usage/#-resume","text":"Specify this when restarting a pipeline. Nextflow will use cached results from any pipeline steps where the inputs are the same, continuing from where it got to previously. For input to be considered the same, not only the names must be identical but the files' contents as well. For more info about this parameter, see this blog post . You can also supply a run name to resume a specific run: -resume [run-name] . Use the nextflow log command to show previous run names.","title":"-resume"},{"location":"usage/#-c","text":"Specify the path to a specific config file (this is a core Nextflow command). See the nf-core website documentation for more information.","title":"-c"},{"location":"usage/#custom-configuration","text":"","title":"Custom configuration"},{"location":"usage/#resource-requests","text":"Whilst the default requirements set within the pipeline will hopefully work for most people and with most input data, you may find that you want to customise the compute resources that the pipeline requests. Each step in the pipeline has a default set of requirements for number of CPUs, memory and time. For most of the steps in the pipeline, if the job exits with any of the error codes specified here it will automatically be resubmitted with higher requests (2 x original, then 3 x original). If it still fails after the third attempt then the pipeline execution is stopped. For example, if the nf-core/rnaseq pipeline is failing after multiple re-submissions of the STAR_ALIGN process due to an exit code of 137 this would indicate that there is an out of memory issue: [62/149eb0] NOTE: Process `NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)` terminated with an error exit status (137) -- Execution is retried (1) Error executing process > 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)' Caused by: Process `NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN (WT_REP1)` terminated with an error exit status (137) Command executed: STAR \\ --genomeDir star \\ --readFilesIn WT_REP1_trimmed.fq.gz \\ --runThreadN 2 \\ --outFileNamePrefix WT_REP1. \\ <TRUNCATED> Command exit status: 137 Command output: (empty) Command error: .command.sh: line 9: 30 Killed STAR --genomeDir star --readFilesIn WT_REP1_trimmed.fq.gz --runThreadN 2 --outFileNamePrefix WT_REP1. <TRUNCATED> Work dir: /home/pipelinetest/work/9d/172ca5881234073e8d76f2a19c88fb Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`","title":"Resource requests"},{"location":"usage/#for-beginners","text":"A first step to bypass this error, you could try to increase the amount of CPUs, memory, and time for the whole pipeline. Therefor you can try to increase the resource for the parameters --max_cpus , --max_memory , and --max_time . Based on the error above, you have to increase the amount of memory. Therefore you can go to the parameter documentation of rnaseq and scroll down to the show hidden parameter button to get the default value for --max_memory . In this case 128GB, you than can try to run your pipeline again with --max_memory 200GB -resume to skip all process, that were already calculated. If you can not increase the resource of the complete pipeline, you can try to adapt the resource for a single process as mentioned below.","title":"For beginners"},{"location":"usage/#advanced-option-on-process-level","text":"To bypass this error you would need to find exactly which resources are set by the STAR_ALIGN process. The quickest way is to search for process STAR_ALIGN in the nf-core/rnaseq Github repo . We have standardised the structure of Nextflow DSL2 pipelines such that all module files will be present in the modules/ directory and so, based on the search results, the file we want is modules/nf-core/star/align/main.nf . If you click on the link to that file you will notice that there is a label directive at the top of the module that is set to label process_high . The Nextflow label directive allows us to organise workflow processes in separate groups which can be referenced in a configuration file to select and configure subset of processes having similar computing requirements. The default values for the process_high label are set in the pipeline's base.config which in this case is defined as 72GB. Providing you haven't set any other standard nf-core parameters to cap the maximum resources used by the pipeline then we can try and bypass the STAR_ALIGN process failure by creating a custom config file that sets at least 72GB of memory, in this case increased to 100GB. The custom config below can then be provided to the pipeline via the -c parameter as highlighted in previous sections. process { withName: 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN' { memory = 100.GB } } NB: We specify the full process name i.e. NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN in the config file because this takes priority over the short name ( STAR_ALIGN ) and allows existing configuration using the full process name to be correctly overridden. If you get a warning suggesting that the process selector isn't recognised check that the process name has been specified correctly.","title":"Advanced option on process level"},{"location":"usage/#updating-containers-advanced-users","text":"The Nextflow DSL2 implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. If for some reason you need to use a different version of a particular tool with the pipeline then you just need to identify the process name and override the Nextflow container definition for that process using the withName declaration. For example, in the nf-core/viralrecon pipeline a tool called Pangolin has been used during the COVID-19 pandemic to assign lineages to SARS-CoV-2 genome sequenced samples. Given that the lineage assignments change quite frequently it doesn't make sense to re-release the nf-core/viralrecon everytime a new version of Pangolin has been released. However, you can override the default container used by the pipeline by creating a custom config file and passing it as a command-line argument via -c custom.config . Check the default version used by the pipeline in the module file for Pangolin Find the latest version of the Biocontainer available on Quay.io Create the custom config accordingly: For Docker: nextflow process { withName: PANGOLIN { container = 'quay.io/biocontainers/pangolin:3.0.5--pyhdfd78af_0' } } For Singularity: nextflow process { withName: PANGOLIN { container = 'https://depot.galaxyproject.org/singularity/pangolin:3.0.5--pyhdfd78af_0' } } For Conda: nextflow process { withName: PANGOLIN { conda = 'bioconda::pangolin=3.0.5' } } NB: If you wish to periodically update individual tool-specific results (e.g. Pangolin) generated by the pipeline then you must ensure to keep the work/ directory otherwise the -resume ability of the pipeline will be compromised and it will restart from scratch.","title":"Updating containers (advanced users)"},{"location":"usage/#running-in-the-background","text":"Nextflow handles job submissions and supervises the running jobs. The Nextflow process must run until the pipeline is finished. The Nextflow -bg flag launches Nextflow in the background, detached from your terminal so that the workflow does not stop if you log out of your session. The logs are saved to a file. Alternatively, you can use screen / tmux or similar tool to create a detached session which you can log back into at a later time. Some HPC setups also allow you to run nextflow within a cluster job submitted your job scheduler (from where it submits more jobs).","title":"Running in the background"},{"location":"usage/#nextflow-memory-requirements","text":"In some cases, the Nextflow Java virtual machines can start to request a large amount of memory. We recommend adding the following line to your environment to limit this (typically in ~/.bashrc or ~./bash_profile ): NXF_OPTS='-Xms1g -Xmx4g'","title":"Nextflow memory requirements"}]}
\ No newline at end of file
diff --git a/sitemap.xml.gz b/sitemap.xml.gz
index 1210c04..abccb6e 100644
Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ