Building multiple package Containers

Creating containers is easy! All you need to do is add a combination of packages to the hash.tsv file.

To assemble the required string of package combinations you can use the little helper service which we provide for you. This service will also tell you the name of the container that will be created.

Currently, this will create you Docker and rkt containers hosted at https://quay.io/organization/biocontainers and Singularity images hosted at https://depot.galaxyproject.org/singularity/ (Hosting sponsors and mirrors welcome!).

Associated command line tools

You can list and search for existing conda packages and container images as well as build multi-package container images locally using the command line mulled-* tools available as part of galaxy-tool-util, which source code is hosted as part of the Galaxy repo. These can be installed using conda

conda create -n mulled galaxy-tool-util -c conda-forge -c bioconda
conda activate mulled

Finding mulled hash names for containers

(planemo) bag@bag:~/projects/code/$ mulled-hash  r-shiny=1.8.1.1,bioconductor-phyloseq=1.46.0,r-curl=5.1.0,r-biocmanager=1.30.23
mulled-v2-3f22c1adbbead1a8888120ab6f59758c0a05e86b:e77384d3aca3277e7caf46a60e0eb848aec72912

Searching for containers

Multi-tool containers have hashed names to make them uniquely identifiable. You usually don't search for containers you construct the hash and pull them down. However, if you have the need to search for them, we have a CLI tool that helps you.

(planemo) bag@bag:~$ mulled-search --destination quay singularity -s deeptools samtools
The query returned the following result(s).
LOCATION     NAME                                                VERSION                                     COMMAND
quay         samtools                                            1.3.1--4                                    docker pull quay.io/biocontainers/samtools:1.3.1--4
quay         samtools                                            1.3.1--3                                    docker pull quay.io/biocontainers/samtools:1.3.1--3
quay         samtools                                            1.3.1--2                                    docker pull quay.io/biocontainers/samtools:1.3.1--2
quay         samtools                                            1.0--0                                      docker pull quay.io/biocontainers/samtools:1.0--0
quay         samtools                                            1.3--1                                      docker pull quay.io/biocontainers/samtools:1.3--1
...
singularity  deeptools                                           3.4.2--py_0                                 wget https://depot.galaxyproject.org/singularity/deeptools:3.4.2--py_0
singularity  deeptools                                           3.4.3--py_0                                 wget https://depot.galaxyproject.org/singularity/deeptools:3.4.3--py_0
singularity  deeptools                                           3.5.0--py_0                                 wget https://depot.galaxyproject.org/singularity/deeptools:3.5.0--py_0
singularity  deeptools                                           3.5.1--py_0                                 wget https://depot.galaxyproject.org/singularity/deeptools:3.5.1--py_0
singularity  deeptools                                           3.5.1--pyhdfd78af_1                         wget https://depot.galaxyproject.org/singularity/deeptools:3.5.1--pyhdfd78af_1
singularity  deeptools                                           3.5.2--pyhdfd78af_0                         wget https://depot.galaxyproject.org/singularity/deeptools:3.5.2--pyhdfd78af_0
...
singularity  samtools                                            1.9--h8571acd_10                            wget https://depot.galaxyproject.org/singularity/samtools:1.9--h8571acd_10
singularity  samtools                                            1.9--h8571acd_11                            wget https://depot.galaxyproject.org/singularity/samtools:1.9--h8571acd_11
singularity  samtools                                            1.9--h8ee4bcc_1                             wget https://depot.galaxyproject.org/singularity/samtools:1.9--h8ee4bcc_1
singularity  samtools                                            1.9--h91753b0_2                             wget https://depot.galaxyproject.org/singularity/samtools:1.9--h91753b0_2
singularity  samtools                                            1.9--h91753b0_3                             wget https://depot.galaxyproject.org/singularity/samtools:1.9--h91753b0_3
singularity  samtools                                            1.9--h91753b0_4                             wget https://depot.galaxyproject.org/singularity/samtools:1.9--h91753b0_4
singularity  samtools                                            1.9--h91753b0_5                             wget https://depot.galaxyproject.org/singularity/samtools:1.9--h91753b0_5
singularity  samtools                                            1.9--h91753b0_8                             wget https://depot.galaxyproject.org/singularity/samtools:1.9--h91753b0_8
...
singularity  mulled-v2-eb9e7907c7a753917c1e4d7a64384c047429618a  9796fae7f3dc33539425911572b1af41da23d71c-0  wget https://depot.galaxyproject.org/singularity/mulled-v2-eb9e7907c7a753917c1e4d7a64384c047429618a:9796fae7f3dc33539425911572b1af41da23d71c-0
singularity  mulled-v2-eb9e7907c7a753917c1e4d7a64384c047429618a  b7b2dac42861f5ad3a6dcbf1626378de27694869-0  wget https://depot.galaxyproject.org/singularity/mulled-v2-eb9e7907c7a753917c1e4d7a64384c047429618a:b7b2dac42861f5ad3a6dcbf1626378de27694869-0
singularity  mulled-v2-eb9e7907c7a753917c1e4d7a64384c047429618a  d4ca6cb58c17558e45d789d552f8d20c2f0cf912-0  wget https://depot.galaxyproject.org/singularity/mulled-v2-eb9e7907c7a753917c1e4d7a64384c047429618a:d4ca6cb58c17558e45d789d552f8d20c2f0cf912-0
singularity  mulled-v2-eb9e7907c7a753917c1e4d7a64384c047429618a  d67d021ff194d58c15cca934dd3dfe948d1c7145-0  wget https://depot.galaxyproject.org/singularity/mulled-v2-eb9e7907c7a753917c1e4d7a64384c047429618a:d67d021ff194d58c15cca934dd3dfe948d1c7145-0
singularity  mulled-v2-eb9e7907c7a753917c1e4d7a64384c047429618a  e4be5f66213daef5a53e2a179b7e0c1d22016afe-0  wget https://depot.galaxyproject.org/singularity/mulled-v2-eb9e7907c7a753917c1e4d7a64384c047429618a:e4be5f66213daef5a53e2a179b7e0c1d22016afe-0

Setting up such a repository

• Fork this repository or copy the github workflow.
• Create quay.io oauth token in an organization.
• Change the MULLED_NAMESPACE variable to point to your quay.io organization.
• Add QUAY_OAUTH_TOKEN, MY_USER MY_PASSWORD and MY_EMAIL to the repository secrets.

Broader context

Bioconda and Conda-Forge are collaborative communities focused on creating and maintaining Conda recipes. These recipes facilitate the installation and management of scientific software, providing researchers and developers with streamlined workflows. Packages from both Bioconda and Conda-Forge are hosted on anaconda.org, a central repository that ensures reliable distribution.

The process for managing these packages is highly automated. Both Bioconda and Conda-Forge build and upload pre-compiled packages directly to anaconda.org. Bioconda goes a step further by testing all packages in minimal containerized environments to verify their compatibility and reliability. These containerized versions of the packages are subsequently pushed to quay.io/biocontainers, creating a robust and accessible ecosystem of ready-to-use environments.

For every package and every version within Bioconda, a corresponding pre-built container is available. This approach ensures that users have access to specific versions of tools in isolated and consistent environments. Additionally, the multi-package-containers repository builds containers that include multiple Conda packages in a single container, known as mulled containers. The naming convention for these containers relies on hashed package names and versions, providing unique identifiers for each. Docker containers are directly pushed to quay.io/biocontainers, while Singularity images are uploaded to depot.galaxyproject.org/singularity.

To ensure seamless synchronization between these platforms, the singularity-build-bot repository monitors quay.io for updates and uploads the corresponding containers to depot.galaxyproject.org. This ensures that all containers available on quay.io are also accessible via depot.galaxyproject.org/singularity.

Singularity containers hosted on depot.galaxyproject.org are further mirrored and geographically distributed using the CernVM File System (CVMFS). This infrastructure optimizes access to containers globally, providing end-users with a reliable and efficient way to consume these resources. The CVMFS distribution model parallels the accessibility of quay.io/biocontainers, ensuring that users can seamlessly integrate these resources into their workflows.