notebooks-v1/eods-api-example-wps.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Web Processing Service (WPS) with the Defra Earth Observation Data Service API\n",
    "This notebook introduces the concept of the Web Processing Service (WPS) which enables users to submit instructions to the EO Data Service to return outputs. The instructions are contained in the XML files provided.  Please ensure these are saved in the XML folder within the Scripts folder.\n",
    "\n",
    "## Instructions:\n",
    "1. Select one of the eight WPS processes to run by typing its name in the relevant cell.\n",
    "2. Select 'Restart and Clear Output' from the Kernel menu at the top of this page if necessary.\n",
    "3. Select 'Run All' from the Cell menu at the top of this page.\n",
    "4. Look at the outputs beneath the final cell.  You should see the following, preceded by a date-time stamp:\n",
    "        -WPS request submitted\n",
    "        -WPS request response code (200) and the full WPS response including executionId number\n",
    "        -exececutionId extracted from the WPS request response code \n",
    "        -The status check URL including the executionId. This will run automatically to check the status of your request.\n",
    "        -Status check results every 15 seconds: process is still running or result is ready for download. \n",
    "        -Download URL\n",
    "5. Click on the download URL or copy and paste it into your browser.\n",
    "6. The downloaded file should be saved to your default download location.\n",
    "7. Check the downloaded file, e.g. open spatial datasets in a GIS application. \n",
    "8. Now choose one of the other WPS processes (step 1) and repeat the above steps until you have tested them all.\n",
    "        \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import os\n",
    "from IPython.display import Image\n",
    "import time\n",
    "from datetime import datetime\n",
    "import importlib\n",
    "import urllib3\n",
    "import config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "importlib.reload(config)\n",
    "urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_fmt='%Y%m%dT%H%M%S'\n",
    "pretty_fmt='%Y-%m-%d %H:%M:%S'\n",
    "wps_server = config.URL + config.WPS_SUF + '?access_token=' + config.ACCESS_TOKEN\n",
    "headers = {'Content-type': 'application/xml','User-Agent': 'curl/7.65.1'}\n",
    "\n",
    "wps_test_config = {\n",
    "    'bandselect':['bandselect_EODS-sp14.xml','image/tiff','result.tiff'],\n",
    "    'rasterclip':['rasterclip_EODS-sp14.xml','image/tiff','result.tiff'],\n",
    "    'zonalstats':['zonalstats_EODS.xml','text/csv','statistics.csv'],\n",
    "    'rastertopoints':['raster-to-points_EODS-sp14.xml','application/zip','result.zip'],\n",
    "    'coverageclassstats':['coverageclassstats_EODS-sp14.xml','text/xml','result.xml'],\n",
    "    'reproject':['reproject_EODS-sp14.xml','application/zip','result.zip'],\n",
    "    'gsdownload-small':['gsdownload-EODS-small-s2.xml','application/zip','result.zip'],\n",
    "    'gsdownload-large':['gsdownload-EODS-large-s1.xml','application/zip','result.zip'],\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Select the WPS \"Tool\" to Run\n",
    "\n",
    "### The options are:\n",
    "\n",
    "* 'bandselect' (WPS tool name: ras:BandSelect) - selects a single band from a raster\n",
    "* 'rasterclip' (WPS tool name: ras:CropCoverage) - clips an area of a raster defined by geometry provided in well-known text (WKT) format.\n",
    "* 'zonalstats' (WPS tool name: ras:RasterZonalStatistics) - generates zonal statistics from a raster using geometry supplied as a shapefile.\n",
    "* 'rastertopoints' (WPS tool name: gs:RasterAsPointCollection) - generates a point for each valid pixel in a raster dataset.  Band values are stored as attributes of the points.\n",
    "* 'coverageclassstats' (WPS tool name: ras:CoverageClassStats) - NOT CURRENTLY WORKING but should calculate statistics from raster values classified into bins/classes (i.e. a histogram).\n",
    "* 'reproject' (WPS tool name: gs:reproject) - reprojects a vector dataset into a supplied coordinate reference system.\n",
    "* 'gsdownload-small' (WPS tool name: gs:download) - downloads a single layer, in this case a small S2 granule.\n",
    "* 'gsdownload-large' (WPS tool name: gs:download) - downloads a single layer, in this case a large S1 scene.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wps_tool = 'bandselect'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# STEP 1 of 3. Submit WPS \"Process submission\" request\n",
    "\n",
    "# get the configuration for the wps tool to run based in user input\n",
    "xml_payload_file = open(os.path.join(os.path.join(os.getcwd(),'xml'),wps_test_config.get(wps_tool)[0]),'r')\n",
    "mime_type = wps_test_config.get(wps_tool)[1]\n",
    "output_id = wps_test_config.get(wps_tool)[2]\n",
    "\n",
    "# set file extension based on mime type\n",
    "print('\\n' + datetime.utcnow().strftime(pretty_fmt) + ' :: the wps request was submitted to the address :: ' + wps_server)\n",
    "wps_submit_response = requests.post(wps_server, data=xml_payload_file.read(), headers=headers, verify=True)\n",
    "status_code = wps_submit_response.status_code\n",
    "print('\\n' + datetime.utcnow().strftime(pretty_fmt)\n",
    "      + ' :: the wps request response code is: \"' + str(status_code) + '\" and the wps request response is \\n'\n",
    "      + wps_submit_response.text)\n",
    "\n",
    "# if connection to the WPS server was successfully, check progress and download result\n",
    "if status_code == 200 and not wps_submit_response.text.find('ExceptionReport') > 0:\n",
    "    execution_id = wps_submit_response.text.split('executionId=')[1].split('&')[0]\n",
    "    print('\\n' + datetime.utcnow().strftime(pretty_fmt) + ' :: the WPS execution_id is ' + str(execution_id))\n",
    "       \n",
    "    for ii in range(99):\n",
    "\n",
    "        # STEP 2 of 3. Submit WPS \"STATUS CHECK\" request, depending on the size of the wps job, this could take time        \n",
    "        status_check_url = wps_server + '&request=GetExecutionStatus&executionid=' + execution_id\n",
    "        if ii == 0:\n",
    "            print('\\n' + datetime.utcnow().strftime(pretty_fmt) + ' :: status check url is ' + status_check_url)\n",
    "        status_response = requests.get(status_check_url,headers=headers)\n",
    "        print('\\n' + datetime.utcnow().strftime(pretty_fmt) \n",
    "              + ' :: Status Check/Download Attempt No: ' + str(ii) \n",
    "              + ' :: process currently running on wps server, please wait')\n",
    "\n",
    "        if status_response.text.find('wps:ProcessFailed') != -1:\n",
    "            print('\\n' + datetime.utcnow().strftime(pretty_fmt) \n",
    "                  + ' :: WPS Processed Failed, check the status URL for the specific error = ' + status_check_url)\n",
    "            break\n",
    "        \n",
    "        # STEP 3 of 3. if the download is ready with 'Process succeeded' message, then print the download URL to the output\n",
    "        if status_response.text.find('wps:ProcessSucceeded') != -1:\n",
    "            result_url = wps_server + '&request=GetExecutionResult&executionid=' + execution_id + '&outputId=' + output_id +'&mimetype=' + mime_type\n",
    "            print('\\n' + datetime.utcnow().strftime(pretty_fmt) \n",
    "                  + ' :: Result is ready for download on URL = ' + result_url)\n",
    "            break\n",
    "\n",
    "        # wait 15 seconds between checks \n",
    "        time.sleep(15)\n",
    "\n",
    "else:        \n",
    "    # if WPS request was not successful, quit workflow\n",
    "    print('\\n' + datetime.utcnow().strftime(pretty_fmt) + ' the WPS request submission was not successful, quitting workflow')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Thank you for your help with testing the Earth Observation Data Service API and WPS."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}