Pipe-Answers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4d75367b-9bd0-4956-9e39-c340dfcae575",
   "metadata": {},
   "source": [
    "# Pipe the output of one script to the input of another\n",
    "<br>There are two things we need to do to make this happen. \n",
    "1. We need to set the second Python script up to take in piped input.\n",
    "2. We need to use the pipe character `|` on the command line\n",
    "\n",
    "The second part is easy, but the first part is a little tricky."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6721aa6-1287-4212-8dea-77d61f473438",
   "metadata": {},
   "source": [
    "### `sys.stdin`\n",
    "This Python command will pull an input file from your pipe into your Python script.\n",
    "\n",
    "**`sys.stdin` always pulls data in as a file object, which by default is a list of lines. You can change it to a string (which is easy to convert to a number, if that's what you need) with `sys.stdin.read()` - just like `f.read()`.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "103dcb84-bf75-47c7-8a17-e3cecf3a59ed",
   "metadata": {},
   "source": [
    "You can also take in a file from the command line using `sys.stdin`, without needing to receive that file object from another script, so that is how we will practice first. Let's use the script `PIPEargv2.py` to see how I've set it up.\n",
    "<br><br>After we look at the script, we can run it on the command line by directing a file in to the script. We'll use the file `number.txt` which only contains the number 3.\n",
    "<br><br>When we use sys.stdin, we don't need to represent that data on the command line with {} or another placeholder. Instead, we will **direct** the file in with the **<** sign.\n",
    "<br><br>`python PIPEargv2.py kiwi < number.txt`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc07dbcd-31da-47fb-8cf9-efddae87f959",
   "metadata": {},
   "source": [
    "<br><br>Now that the script is ready to take input in through sys.stdin, we can pipe the output from another script into it.\n",
    "<br><br>Remember `add100.py` that adds 100 to a number? We can call that script with the argument 8, and then pipe the output (108) into the `PIPEargv2.py` script. This means that instead of printing the number to the screen, bash is taking whatever is printed, saving it as a file, and then giving it the next script as `sys.stdin`.\n",
    "<br><br>`python add100.py 8 | python PIPEargv2.py kiwi`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b96c81c5-3da6-4e12-8dfc-0b2373ff4b43",
   "metadata": {},
   "source": [
    "### <br><br>`sys.stdout`\n",
    "Let's think about how the output of the first script is getting to the second script."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7279ed7-4de6-4c12-bb23-1cc8415a454a",
   "metadata": {},
   "source": [
    "### <br>Redirect output from `print()` with > on the command line\n",
    "This will take anything that gets printed to the screen and **instead** print it to a file."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d96e4ba-d8e5-492b-9dad-14f513eb2fbc",
   "metadata": {},
   "source": [
    "`python add100.py 3 > number.txt`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2340ba85-28e8-4852-85b8-e7f610c0d7ed",
   "metadata": {},
   "source": [
    "### Print to the screen and save to a file by piping and using tee\n",
    "If we want the output to appear on the screen and also get saved in a file, we can also pipe the output of a script to the tee command.\n",
    "<br><br>`python add100.py 3 | tee numberT.txt`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2f521d1-7c50-4047-a105-17b91fc5264c",
   "metadata": {},
   "source": [
    "<br>The PowerShell equivalent is Tee-Object.\n",
    "<br>`python add100.py 3 | Tee-Object numberT.txt`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10e7675d-c83c-46e0-abb8-eb2b3d8ef2d6",
   "metadata": {},
   "source": [
    "<br>The print function is, by default, always going to print to what is called standard out. We can also use `sys.stdout.write()` to write to standard out, but anything that gets printed will also go there. There are ways to explicitly print to another file, but we're not going to cover that today. Let's look at a reworked version of the add100 script, called `PIPEadd100.py`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04762ce6-9d71-428c-9324-64f0f2a24478",
   "metadata": {},
   "source": [
    "<br>And we can run it the exact same way on the command line to see that it works the same as the version with `print()`:\n",
    "<br><br>`python PIPEadd100.py 3 | tee numberT.txt`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2100301-a249-472c-bedd-a731bf8dabf7",
   "metadata": {},
   "source": [
    "### <br><br>Pipe with or without saving the intermediate outputs.\n",
    "Let's revisit our full pipeline, but use the number 60 as the input for the first script.\n",
    "<br><br>`python add100.py 60 | python PIPEargv2.py kiwi`\n",
    "<br><br>This will not save the number 160 to any file - it only gets passed to the next piece of the pipeline. If we want to save it, we have to tee it up in the middle:\n",
    "<br><br>`python add100.py 60 | tee number.txt | python PIPEargv2.py kiwi`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5fae92b-67ba-4b85-9367-f2c2cdf6b891",
   "metadata": {},
   "source": [
    "### <br><br>Exercise\n",
    "1. Copy and paste the last pipeline we wrote and have the final output print to a file called kiwis.txt instead of printing to the screen.\n",
    "2. Copy and paste the last pipeline we wrote and have the final output print to a file called kiwis.txt AND print to the screen."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1375c38b-f416-4c86-8bc5-de25b7a834f8",
   "metadata": {},
   "source": [
    "ANSWERS:\n",
    "<br><br>`python add100.py 60 | tee number.txt | python PIPEargv2.py kiwi > kiwi.txt`\n",
    "<br><br>`python add100.py 60 | tee number.txt | python PIPEargv2.py kiwi | tee kiwi.txt`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a13ca19-fa00-479e-ac0b-1fb2066543b2",
   "metadata": {},
   "source": [
    "### <br><br>A more complicated example\n",
    "We can also include command line software programs in our pipeline. Each program may have a different way that you have to tell it to use standard in or standard out, or that may be the default. \n",
    "<br><br>I'm going to walk through a pipeline that starts with a pdf, uses a command line tool to convert it to text, feeds that into a Python script that reformats the text into a csv file, then passes the csv to a version of our sortEmails.py script. The pipeline looks like this:\n",
    "<br><br>`pdftotext email_addresses4.pdf - | python PIPEemailsTxtToCsv.py | python PIPEsortEmails.py > final.csv`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31fa86d9-1b0c-478d-ab6d-22129c3c99ca",
   "metadata": {},
   "source": [
    "And in parallel!\n",
    "<br><br>`parallel \"pdftotext {.}.pdf - | python PIPEemailsTxtToCsv.py | python PIPEsortEmails.py > {.}.csv\" ::: *.pdf`\n",
    "<br><br>Or append both outputs to one file:\n",
    "<br><br>`parallel \"pdftotext {.}.pdf - | python PIPEemailsTxtToCsv.py | python PIPEsortEmails.py >> final.csv\" ::: *.pdf`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6aead6d5-c051-4f6d-b8bf-0ae690147909",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}