TCIA_Aspera_CLI_Downloads.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyMWKKgzKF0HWDCt83t5owv2"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "You can download and run this notebook locally, or you can run it for free in a cloud environment using Colab or Sagemaker Studio Lab:\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/kirbyju/TCIA_Notebooks/blob/main/TCIA_Aspera_CLI_Downloads.ipynb)\n",
        "\n",
        "[![Open In SageMaker Studio Lab](https://studiolab.sagemaker.aws/studiolab.svg)](https://studiolab.sagemaker.aws/import/github.com/kirbyju/TCIA_Notebooks/blob/main/TCIA_Aspera_CLI_Downloads.ipynb)"
      ],
      "metadata": {
        "id": "s0EsIhNbmDdK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Summary\n",
        "Much of non-DICOM content in [The Cancer Imaging Archive (TCIA)](https://www.cancerimagingarchive.net/) is provided via links to IBM Aspera Faspex packages.  Aspera’s FASP protocol is designed to move data rapidly across networks with minimal disruption to other traffic.  Aspera’s Faspex application bundles data into packages that can be referenced via a web link (i.e., a URI).  When an Aspera Faspex link resolves in a browser, it presents a GUI that guides a user through the process of installing a browser extension and a local Aspera Connect client, if not already there, that moves the data using FASP between TCIA servers and the computer the browser is running on.\n",
        "\n",
        "We frequently get requests from researchers for an option to download TCIA Faspex packages using a command line interface that bypasses the GUI.  While not a part of the standard Aspera distributions, the IBM Aspera developers have provided an open source tool (Apache 2.0 license) called [ascli (aspera-cli)](https://github.com/IBM/aspera-cli) that allows a client to download an Aspera Faspex package using its URI.  **This notebook is focused on demonstrating how to download TCIA data from Aspera packages via the command line on a linux system.**\n",
        "\n",
        "**Note:** The performance of ascli appears to be significantly worse when using the free tier of Google Colab.  \n"
      ],
      "metadata": {
        "id": "r9j_S5oaJ4nl"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Setup\n",
        "\n",
        "First we'll install some necessary prerequisite software.  First you need to install Ruby, then the aspera-cli gem, and afterward you can use that to install ascli.  The steps to [install Ruby](https://www.ruby-lang.org/en/downloads/) vary by operating system but the gem and ascli commands should be the same as the last two lines below."
      ],
      "metadata": {
        "id": "y3eFafyJJ9Lt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# consult the link above if you're not on an OS that uses apt\n",
        "!apt install -y ruby ruby-dev rubygems ruby-json\n",
        "\n",
        "# these should work in all environments after ruby is installed\n",
        "!gem install aspera-cli\n",
        "!ascli conf ascp install"
      ],
      "metadata": {
        "id": "Cbfvmj_FbEnS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "If the ascli command above fails you can try this alternate 'local SDK installation' method described on https://github.com/IBM/aspera-cli:\n",
        "\n",
        "```\n",
        "curl -Lso sdk.zip https://ibm.biz/aspera_transfer_sdk\n",
        "ascli config ascp install --sdk-url=file:///sdk.zip\n",
        "```\n",
        "\n",
        "The format is: file:///<path>, where <path> can be either a relative path (not starting with /), or an absolute path."
      ],
      "metadata": {
        "id": "p-77Y1yu54ts"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Download an entire Aspera package"
      ],
      "metadata": {
        "id": "KgL7C-R_KCIq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now that the prerequisite installations are complete, you will be able to use the ascli **receive** command to transfer a TCIA Faspex package using its link by following these steps:\n",
        "1.\tBrowse to the dataset landing page that describes the package,\n",
        "2.\tFind the “Download” button that one normally would use to download the package using the Faspex GUI, but do not click it,\n",
        "3.\tRight click that button icon and select “copy link address” or “copy link” or similar (depends on your browser) to extract the package URI into the clipboard or copy/paste buffer.\n",
        "\n",
        "Replace the **url** parameter in the following cell with the link of the package you want to download.  \n",
        "\n",
        "**Note:** If you don't change the package URL below it will only take a few seconds to download an example package (~100MB) from the [Cancer Moonshot Biobank - Gastroesophageal Cancer Collection (CMB-GEC)](https://doi.org/10.7937/E7KH-R486).\n"
      ],
      "metadata": {
        "id": "y0f2sW2QHf7t"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!ascli faspex5 packages receive --url='https://faspex.cancerimagingarchive.net/aspera/faspex/public/package?context=eyJyZXNvdXJjZSI6InBhY2thZ2VzIiwidHlwZSI6ImV4dGVybmFsX2Rvd25sb2FkX3BhY2thZ2UiLCJpZCI6Ijc2MiIsInBhc3Njb2RlIjoiZDg0NmZlM2Q5ZjZjNzliYjUxYWU2MWMzNjJkNmE1ODJmMTc0YmVkYSIsInBhY2thZ2VfaWQiOiI3NjIiLCJlbWFpbCI6ImhlbHBAY2FuY2VyaW1hZ2luZ2FyY2hpdmUubmV0In0='"
      ],
      "metadata": {
        "id": "260sFaZam_SN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Optional: Verify your download\n",
        "\n",
        "Aspera has built in checksum capabilities and asserts that any download that completes successfully had a checksum verification performed so this is generally not necessary.  \n",
        "\n",
        "However, each package TCIA maintains includes a sums file at the root level of the package.  Not only can this help with troubleshooting issues, it can also be used to quickly understand the contents of a package in case you only want to download a subset of it as shown in the next section.\n",
        "\n",
        "**Note:** Performing this verification on large datasets may take a very long time depending on how much compute you have available.\n",
        "\n",
        "Here's an example python function that will perform the checksum verification."
      ],
      "metadata": {
        "id": "DcT2KuaTUecm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def verify_sums_file(sums_file_path, data_directory):\n",
        "    \"\"\"\n",
        "    Function to assist with verifying the checksum\n",
        "    files that TCIA publishes with Aspera packages. Specify\n",
        "    the path to your *.sums file and the directory where\n",
        "    you've saved the Aspera package.\n",
        "    \"\"\"\n",
        "    import hashlib\n",
        "    import os\n",
        "    if not os.path.exists(sums_file_path):\n",
        "        print(\"Sums file not found at the specified path.\")\n",
        "        return\n",
        "\n",
        "    with open(sums_file_path, 'r') as file:\n",
        "        for line in file:\n",
        "            file_hash, file_name = line.strip().split(' ')\n",
        "            file_path = os.path.join(data_directory, file_name)\n",
        "            if os.path.exists(file_path):\n",
        "                with open(file_path, 'rb') as f:\n",
        "                    file_contents = f.read()\n",
        "                    calculated_hash = hashlib.md5(file_contents).hexdigest()\n",
        "                    if calculated_hash == file_hash:\n",
        "                        print(f\"{file_name}: OK - Hashes match\")\n",
        "                    else:\n",
        "                        print(f\"{file_name}: MISMATCH - Hashes do not match\")\n",
        "            else:\n",
        "                print(f\"{file_name}: NOT FOUND\")"
      ],
      "metadata": {
        "id": "Wyzs8j8aUwLf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You'll need to update the file names/paths for the sums file and data directory if you downloaded a different package."
      ],
      "metadata": {
        "id": "pFOTWa88nZAb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sums_file_path = '/content/PKG - Biobank_CMB-GEC_v1/Biobank_CMB-GEC_v1.sums'\n",
        "data_directory = '/content/PKG - Biobank_CMB-GEC_v1'\n",
        "verify_sums_file(sums_file_path, data_directory)"
      ],
      "metadata": {
        "id": "E8NJIWn5UkAw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Download individual files or directories from an Aspera package\n",
        "\n",
        "In order to grab specific parts of a package you can use the **browse** parameter to look around inside the package.  Running the command without any path specified will show you the root folder of the package.  \n",
        "\n",
        "Let's take a look at the [HER2 tumor ROIs](https://doi.org/10.7937/NVA3-N783) collection as an example. Here's the root folder of the package."
      ],
      "metadata": {
        "id": "XP6unlXT1Dzu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!ascli faspex5 packages browse --url=\"https://faspex.cancerimagingarchive.net/aspera/faspex/public/package?context=eyJyZXNvdXJjZSI6InBhY2thZ2VzIiwidHlwZSI6ImV4dGVybmFsX2Rvd25sb2FkX3BhY2thZ2UiLCJpZCI6IjczOSIsInBhc3Njb2RlIjoiNzEwNmUzNDFjMDY4MjljNjBkMmM0ZjcxYTBhMTE1ODcxNGIzZWNjNSIsInBhY2thZ2VfaWQiOiI3MzkiLCJlbWFpbCI6ImhlbHBAY2FuY2VyaW1hZ2luZ2FyY2hpdmUubmV0In0=\""
      ],
      "metadata": {
        "id": "aJY_3Xwrlv7o"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "If you want to look at the contents of one of the directories you just add the directory path to the end of this command.  Let's look at the Yale_HER2_cohort folder."
      ],
      "metadata": {
        "id": "P2yGf6w2Pg1m"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!ascli faspex5 packages browse --url=\"https://faspex.cancerimagingarchive.net/aspera/faspex/public/package?context=eyJyZXNvdXJjZSI6InBhY2thZ2VzIiwidHlwZSI6ImV4dGVybmFsX2Rvd25sb2FkX3BhY2thZ2UiLCJpZCI6IjczOSIsInBhc3Njb2RlIjoiNzEwNmUzNDFjMDY4MjljNjBkMmM0ZjcxYTBhMTE1ODcxNGIzZWNjNSIsInBhY2thZ2VfaWQiOiI3MzkiLCJlbWFpbCI6ImhlbHBAY2FuY2VyaW1hZ2luZ2FyY2hpdmUubmV0In0=\" Yale_trastuzumab_response_cohort"
      ],
      "metadata": {
        "id": "OxsvzgCrPna0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "When you want to download a file, you can once again use the **recieve** command.  Let's pretend that we're interested in grabbing the entire annotation folder.  You can do that by simply appending the path of the directory to the end of the **receive** command."
      ],
      "metadata": {
        "id": "w4mDYN3Xfdkz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!ascli faspex5 packages receive --url=\"https://faspex.cancerimagingarchive.net/aspera/faspex/public/package?context=eyJyZXNvdXJjZSI6InBhY2thZ2VzIiwidHlwZSI6ImV4dGVybmFsX2Rvd25sb2FkX3BhY2thZ2UiLCJpZCI6IjczOSIsInBhc3Njb2RlIjoiNzEwNmUzNDFjMDY4MjljNjBkMmM0ZjcxYTBhMTE1ODcxNGIzZWNjNSIsInBhY2thZ2VfaWQiOiI3MzkiLCJlbWFpbCI6ImhlbHBAY2FuY2VyaW1hZ2luZ2FyY2hpdmUubmV0In0=\" Yale_trastuzumab_response_cohort/Annotations"
      ],
      "metadata": {
        "id": "8DssIjsEfeV6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can also download individual files.  We'll demonstrate that by retrieving the included **sums** file, which will provide a full list of the files in the package.  This can help us more easily figure out which specific data we may want to download without having to dig through every folder one at a time using the previously shown **browse** commands."
      ],
      "metadata": {
        "id": "0bqxJU9OPyUF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!ascli faspex5 packages receive --url=\"https://faspex.cancerimagingarchive.net/aspera/faspex/public/package?context=eyJyZXNvdXJjZSI6InBhY2thZ2VzIiwidHlwZSI6ImV4dGVybmFsX2Rvd25sb2FkX3BhY2thZ2UiLCJpZCI6IjczOSIsInBhc3Njb2RlIjoiNzEwNmUzNDFjMDY4MjljNjBkMmM0ZjcxYTBhMTE1ODcxNGIzZWNjNSIsInBhY2thZ2VfaWQiOiI3MzkiLCJlbWFpbCI6ImhlbHBAY2FuY2VyaW1hZ2luZ2FyY2hpdmUubmV0In0=\" HER2_tumor_ROIs_v3.sums"
      ],
      "metadata": {
        "id": "JpRzGbeTQcum"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Sometimes the sums files are not sorted in a way that makes them easy to read.  In this case you may want to sort them alphabetically by file path/name.  If you're using a different sums file you just need to update the original and sorted file names accordingly in this next step."
      ],
      "metadata": {
        "id": "o_X65J00ZEXf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!sort -k2,2 -o HER2_tumor_ROIs_v3_sorted.sums HER2_tumor_ROIs_v3.sums"
      ],
      "metadata": {
        "id": "1eQXIBaKYX_k"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's say that after reviewing the sums file you're interested in obtaining just the first HER2 positive image and annotation file from the Yale HER2 cohort.  We can pull both files at the same time by appending their paths to the end of the previous **receive** command as shown here."
      ],
      "metadata": {
        "id": "ldP2wNWzZ7BU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!ascli faspex5 packages receive --url=\"https://faspex.cancerimagingarchive.net/aspera/faspex/public/package?context=eyJyZXNvdXJjZSI6InBhY2thZ2VzIiwidHlwZSI6ImV4dGVybmFsX2Rvd25sb2FkX3BhY2thZ2UiLCJpZCI6IjczOSIsInBhc3Njb2RlIjoiNzEwNmUzNDFjMDY4MjljNjBkMmM0ZjcxYTBhMTE1ODcxNGIzZWNjNSIsInBhY2thZ2VfaWQiOiI3MzkiLCJlbWFpbCI6ImhlbHBAY2FuY2VyaW1hZ2luZ2FyY2hpdmUubmV0In0=\" Yale_HER2_cohort/Annotations/Her2Pos_Case_01.xml Yale_HER2_cohort/SVS/Her2Pos_Case_01.svs"
      ],
      "metadata": {
        "id": "A6YEw5p5aJSK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Acknowledgements\n",
        "TCIA is funded by the [Cancer Imaging Program (CIP)](https://imaging.cancer.gov/), a part of the United States [National Cancer Institute (NCI)](https://www.cancer.gov/).  It is managed by the [Frederick National Laboratory for Cancer Research (FNLCR)](https://frederick.cancer.gov/) and hosted by the [University of Arkansas for Medical Sciences (UAMS)](https://www.uams.edu/)\n",
        "\n",
        "This notebook was created by [Justin Kirby](https://www.linkedin.com/in/justinkirby82/), [Quasar Jarosz](https://www.linkedin.com/in/quasarjarosz/) and [Lawrence Tarbox](https://www.linkedin.com/in/lawrence-tarbox-088335/). Also, big thanks to [@VolodymyrChapman](https://github.com/VolodymyrChapman) for pointing out we can download individual files/directories from a package and to [Laurent Martin](https://www.linkedin.com/in/laurentmartinjp/) for assisting with a variety of our ASCLI questions!\n",
        "\n",
        "If you leverage this notebook or any TCIA datasets in your work, please be sure to comply with the [TCIA Data Usage Policy](https://wiki.cancerimagingarchive.net/x/c4hF). In particular, make sure to cite the DOI(s) for the specific TCIA datasets you used in addition to the following paper!\n",
        "\n",
        "## TCIA Citation\n",
        "\n",
        "Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., & Prior, F. (2013). The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository. Journal of Digital Imaging, 26(6), 1045–1057. https://doi.org/10.1007/s10278-013-9622-7"
      ],
      "metadata": {
        "id": "nah09wnJ825e"
      }
    }
  ]
}