EODS_API_Best_pixel.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this Notebook to work correctly, the virtual memory on the Windows 10 machine where this Notebook is being run must be increased. This is because the notebook will download and process satellite images that are, generally, very large. In order to achieve this, there are some system modifications that need to be done before running the Notebook. To increase the virtual memory of the machine:\n",
    "\n",
    "Go to Control panel---> System---> Advanced System Settings.\n",
    "\n",
    "In the window that appears select the Advance tab, then settings, in the new window select the advanced tab, the click on change. Here, untick the \"Automatically manage paging file for all drives\" Select then the Custom size and in the initial size (MB) box. Enter 4096 and in the Maximum size (MB) box enter 7000 or 8000. (The maximum size can be increased if the mosaic to be created is very large)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Importing the neccesary libraries for downloading data \n",
    "import requests\n",
    "import time\n",
    "from datetime import datetime\n",
    "import pandas as pd\n",
    "from pathlib import Path\n",
    "import xmltodict\n",
    "from zipfile import ZipFile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell will create a folder, where the notebook is located, and all\n",
    "the downloaded images will be stored there "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "APITEST130721\n"
     ]
    }
   ],
   "source": [
    "# get input for \"test name\" eg 'PRD_STANDARDISED_TEST_WPS_S1x10_RUN1'\n",
    "RUNNAME = input()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_fmt='%Y%m%dT%H%M%S'\n",
    "pretty_fmt='%Y-%m-%d %H:%M:%S'\n",
    "headers = {'Content-type': 'application/xml','User-Agent': 'curl/7.65.1'}\n",
    "# need to download gsdownload-template.xml to scripts/xml folder.\n",
    "wps_test_config = {\n",
    "        'gs:Download':{\n",
    "            'template_xml':'gsdownload-template.xml',\n",
    "            'mimetype':'application/zip',\n",
    "            'output':'result.zip',\n",
    "        }}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point it must be noted that this script in order to work, the input images must be of the same granule from different time stamps. This script will not create a mosaic of diferent granules but a mosaic of the same granule without any cloud coverage using diferent sensing dates of the same granule.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load list of granules from dowloaded csv \n",
    "#(Those are the resulting csv from eods-api-example-generate-cloudless-mosaic Notebook)\n",
    "col_list = [\"typename\"]\n",
    "min_cloud_per_granule = pd.read_csv(r\"C:\\Users\\Charlie.Howarth\\Miniconda3\\Scripts\\data\\min_cloud_per_granule.csv\")\n",
    "\n",
    "min_cloud_per_granule_per_orbit = pd.read_csv(r\"C:\\Users\\Charlie.Howarth\\Miniconda3\\Scripts\\data\\min-cloud-per-granule-per-orbit.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "### USER INPUT\n",
    "### Enter your token in PRD_DM = \n",
    "\n",
    "wps_job_request_list = [\n",
    "     {'tool':'gs:Download','layer_list':min_cloud_per_granule_per_orbit[\"typename\"],'dl_bool':True}\n",
    "]\n",
    "\n",
    "PRD_DM = \"Zz1Q3arH4NroQAYKixNTw6ik6G0Reg\"\n",
    "ACCESS_TOKEN = PRD_DM\n",
    "\n",
    "ENV_PRD = 'https://earthobs.defra.gov.uk'\n",
    "wps_env = ENV_PRD\n",
    "\n",
    "wps_server = wps_env + '/geoserver/ows/?access_token=' + ACCESS_TOKEN + '&SERVICE=WPS&VERSION=1.0.0'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "2021-07-13T14:44:54.129353 :: tool=gs:Download, lyr=geonode:S2A_20210404_lat54lon066_T30UXE_ORB037_utm30n_osgb_vmsk_sharp_rad_srefdem_stdsref, download=True\n",
      "\n",
      "2021-07-13T14:44:55.576114 :: request POST url =https://earthobs.defra.gov.uk/geoserver/ows/?access_token=Zz1Q3arH4NroQAYKixNTw6ik6G0Reg&SERVICE=WPS&VERSION=1.0.0&REQUEST=EXECUTE\n",
      "\n",
      "2021-07-13T14:44:55.576688:: WPS job ae1cb4d6-5698-4d21-b5d3-4d7d26190dde was successfully submitted\n",
      "\n",
      "2021-07-13T14:44:55.576688:: b'<?xml version=\"1.0\" encoding=\"UTF-8\"?><wps:ExecuteResponse xmlns:xs=\"http://www.w3.org/2001/XMLSchema\" xmlns:ows=\"http://www.opengis.net/ows/1.1\" xmlns:wps=\"http://www.opengis.net/wps/1.0.0\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" service=\"WPS\" serviceInstance=\"https://earthobs.defra.gov.uk/geoserver/ows?access_token=Zz1Q3arH4NroQAYKixNTw6ik6G0Reg&amp;\" statusLocation=\"https://earthobs.defra.gov.uk/geoserver/ows?service=WPS&amp;version=1.0.0&amp;request=GetExecutionStatus&amp;executionId=ae1cb4d6-5698-4d21-b5d3-4d7d26190dde&amp;access_token=Zz1Q3arH4NroQAYKixNTw6ik6G0Reg\" version=\"1.0.0\"><wps:Process wps:processVersion=\"1.0.0\"><ows:Identifier>gs:Download</ows:Identifier><ows:Title>Enterprise Download Process</ows:Title><ows:Abstract>Downloads Layer Stream and provides a ZIP.</ows:Abstract></wps:Process><wps:Status creationTime=\"2021-07-13T14:44:56.703Z\"><wps:ProcessAccepted>Process accepted.</wps:ProcessAccepted></wps:Status></wps:ExecuteResponse>'\n",
      "\n",
      "2021-07-13T14:44:55.576688 :: tool=gs:Download, lyr=geonode:S2A_20210531_lat54lon066_T30UXE_ORB137_utm30n_osgb_vmsk_sharp_rad_srefdem_stdsref, download=True\n",
      "\n",
      "2021-07-13T14:44:56.659692 :: request POST url =https://earthobs.defra.gov.uk/geoserver/ows/?access_token=Zz1Q3arH4NroQAYKixNTw6ik6G0Reg&SERVICE=WPS&VERSION=1.0.0&REQUEST=EXECUTE\n",
      "\n",
      "2021-07-13T14:44:56.661696:: WPS job 0c884b62-1769-4e3c-9f01-1392313858aa was successfully submitted\n",
      "\n",
      "2021-07-13T14:44:56.661696:: b'<?xml version=\"1.0\" encoding=\"UTF-8\"?><wps:ExecuteResponse xmlns:xs=\"http://www.w3.org/2001/XMLSchema\" xmlns:ows=\"http://www.opengis.net/ows/1.1\" xmlns:wps=\"http://www.opengis.net/wps/1.0.0\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" service=\"WPS\" serviceInstance=\"https://earthobs.defra.gov.uk/geoserver/ows?access_token=Zz1Q3arH4NroQAYKixNTw6ik6G0Reg&amp;\" statusLocation=\"https://earthobs.defra.gov.uk/geoserver/ows?service=WPS&amp;version=1.0.0&amp;request=GetExecutionStatus&amp;executionId=0c884b62-1769-4e3c-9f01-1392313858aa&amp;access_token=Zz1Q3arH4NroQAYKixNTw6ik6G0Reg\" version=\"1.0.0\"><wps:Process wps:processVersion=\"1.0.0\"><ows:Identifier>gs:Download</ows:Identifier><ows:Title>Enterprise Download Process</ows:Title><ows:Abstract>Downloads Layer Stream and provides a ZIP.</ows:Abstract></wps:Process><wps:Status creationTime=\"2021-07-13T14:44:57.793Z\"><wps:ProcessAccepted>Process accepted.</wps:ProcessAccepted></wps:Status></wps:ExecuteResponse>'\n",
      "\n",
      "2021-07-13T14:44:56.661696 :: A TOTAL OF 2 WPS REQUESTS SUBMITTED\n",
      "\n",
      "2021-07-13T14:44:56.661696 :: The ExecutionIDs are :: dict_keys(['ae1cb4d6-5698-4d21-b5d3-4d7d26190dde', '0c884b62-1769-4e3c-9f01-1392313858aa'])\n",
      "\n",
      "2021-07-13T14:44:56.661696 :: WPS Check Executions URL is https://earthobs.defra.gov.uk/geoserver/ows/?access_token=Zz1Q3arH4NroQAYKixNTw6ik6G0Reg&SERVICE=WPS&VERSION=1.0.0&REQUEST=GetExecutions\n"
     ]
    }
   ],
   "source": [
    "# submitting the WPS request(s)\n",
    "\n",
    "def mod_the_xml(xml_name,layer_name):\n",
    "    \"\"\"\n",
    "    function read xml payload template and modify the payload with the config\n",
    "    \"\"\"\n",
    "    \n",
    "    with open(Path.cwd() / 'xml' / xml_name,'r') as template_xml:\n",
    "        file_data = template_xml.read()\n",
    "    modded_xml = file_data.replace('template_layer_name', layer_name)\n",
    "    \n",
    "    return modded_xml\n",
    "\n",
    "def submit_wps_request(payload):\n",
    "    \"\"\"\n",
    "    function submit the wps POST request\n",
    "    \"\"\"\n",
    "\n",
    "    try:\n",
    "        wps_submit_response = requests.post(wps_server + '&REQUEST=EXECUTE', data=payload, headers=headers, verify=True)\n",
    "        print('\\n' + datetime.utcnow().isoformat() + ' :: request POST url =' + wps_server + '&REQUEST=EXECUTE')\n",
    "    except:\n",
    "        print('wps POST was not successful')\n",
    "    \n",
    "    # receive status code\n",
    "    timestamp_submission = datetime.utcnow()\n",
    "    status_code = wps_submit_response.status_code\n",
    "        \n",
    "    if status_code == 200 and not wps_submit_response.text.find('ExceptionReport') > 0:\n",
    "        execution_id = wps_submit_response.text.split('executionId=')[1].split('&')[0]        \n",
    "        print('\\n' + datetime.utcnow().isoformat() + ':: WPS job ' + execution_id + ' was successfully submitted')\n",
    "        print('\\n' + datetime.utcnow().isoformat() + ':: ' + str(wps_submit_response.content))\n",
    "        return execution_id\n",
    "    else:\n",
    "        print(datetime.utcnow().isoformat() + ' :: wps request was not successful :: STATUS = ' + str(status_code))\n",
    "        print(datetime.utcnow().isoformat() + ' :: response = ' + str(wps_submit_response.content))\n",
    "        return None\n",
    "\n",
    "execution_id_dict={}\n",
    "\n",
    "for item in wps_job_request_list:\n",
    "        \n",
    "    for lyr in item['layer_list']:\n",
    "        print('\\n' + datetime.utcnow().isoformat() + ' :: tool=' + item['tool'] + ', lyr=' + lyr + ', download=' + str(item['dl_bool']))\n",
    "        modded_xml = mod_the_xml(wps_test_config[item['tool']]['template_xml'],lyr)\n",
    "        execution_id = submit_wps_request(modded_xml)\n",
    "        execution_id_dict.update({execution_id:lyr})\n",
    "        submission_time = datetime.utcnow()\n",
    "\n",
    "print('\\n' + datetime.utcnow().isoformat() + ' :: A TOTAL OF ' + str(len(execution_id_dict)) +  ' WPS REQUESTS SUBMITTED')\n",
    "\n",
    "print('\\n' + datetime.utcnow().isoformat() + ' :: The ExecutionIDs are :: ' + str(execution_id_dict.keys()))\n",
    "\n",
    "print('\\n' + datetime.utcnow().isoformat() + ' :: WPS Check Executions URL is ' + wps_server + '&REQUEST=GetExecutions')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "#001 of #n   :: 2021-07-13T14:44:56.661696 :: Logging file: C:\\Users\\Charlie.Howarth\\Miniconda3\\Scripts\\output\\20210713T144456-APITEST130721\\20210713T144456-APITEST130721-wps-log.csv\n",
      "\n",
      "#001 of #n   :: 2021-07-13T14:45:11.124166 :: Poll Time\n",
      "#001 of #n   :: 2021-07-13T14:45:14.593390 :: 3 xml pages parsed in 3.0 seconds\n",
      "#001 of #n   :: 2021-07-13T14:45:14.623392 :: Jobs are still \"STATUS=RUNNING\"...script will poll again in 60sec time ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Charlie.Howarth\\Miniconda3\\envs\\eods\\lib\\site-packages\\pandas\\core\\indexing.py:1597: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self.obj[key] = value\n",
      "C:\\Users\\Charlie.Howarth\\Miniconda3\\envs\\eods\\lib\\site-packages\\pandas\\core\\indexing.py:1720: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self._setitem_single_column(loc, value, pi)\n",
      "C:\\Users\\Charlie.Howarth\\Miniconda3\\envs\\eods\\lib\\site-packages\\pandas\\core\\indexing.py:1599: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self.obj[key] = infer_fill_value(value)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "#002 of #n   :: 2021-07-13T14:46:14.627219 :: Poll Time\n",
      "#002 of #n   :: 2021-07-13T14:46:18.131404 :: 3 xml pages parsed in 4.0 seconds\n",
      "#002 of #n   :: 2021-07-13T14:46:18.154407 :: Jobs are still \"STATUS=RUNNING\"...script will poll again in 60sec time ...\n",
      "\n",
      "#003 of #n   :: 2021-07-13T14:47:18.158603 :: Poll Time\n",
      "#003 of #n   :: 2021-07-13T14:47:21.313780 :: 3 xml pages parsed in 3.0 seconds\n",
      "#003 of #n   :: 2021-07-13T14:47:21.335779 :: Jobs are still \"STATUS=RUNNING\"...script will poll again in 60sec time ...\n",
      "\n",
      "#004 of #n   :: 2021-07-13T14:48:21.340999 :: Poll Time\n",
      "#004 of #n   :: 2021-07-13T14:48:24.516187 :: 3 xml pages parsed in 3.0 seconds\n",
      "#004 of #n   :: 2021-07-13T14:48:24.538155 :: Jobs are still \"STATUS=RUNNING\"...script will poll again in 60sec time ...\n",
      "\n",
      "#005 of #n   :: 2021-07-13T14:49:24.541347 :: Poll Time\n",
      "#005 of #n   :: 2021-07-13T14:49:27.910521 :: 3 xml pages parsed in 3.0 seconds\n",
      "\n",
      "#005 of #005 :: 2021-07-13T14:49:27.939509 :: Script has finished\n"
     ]
    }
   ],
   "source": [
    "# poll the status page and wait until all jobs are completed\n",
    "\n",
    "# create output directory and log file\n",
    "Path(Path.cwd() / 'output').mkdir(parents=True, exist_ok=True)\n",
    "my_path = Path.cwd() / 'output' / str(submission_time.strftime(output_fmt) + '-' + RUNNAME)\n",
    "my_path.mkdir(parents=True, exist_ok=True)\n",
    "log_file_name = my_path / str(submission_time.strftime(output_fmt) + '-' + RUNNAME + '-wps-log.csv')\n",
    "poll_frequency_sec = 60\n",
    "frames = []\n",
    "\n",
    "# poll api for 1 hour unless all jobs submitted are NOT STATUS=RUNNING\n",
    "for i in range(1,3600):\n",
    "    \n",
    "    check_time = datetime.utcnow()\n",
    "    if i == 1:\n",
    "        print('\\n#' + format(i, '03') + ' of #n   :: ' + submission_time.isoformat() + ' :: Logging file: ' + str(log_file_name))\n",
    "    \n",
    "    print('\\n#' + format(i, '03') + ' of #n   :: ' + check_time.isoformat() + ' :: Poll Time')\n",
    "    \n",
    "    start_index = 0\n",
    "    matching_of_dicts = []\n",
    "    for page_counter in range(0,1000,10):\n",
    "\n",
    "        # constrcut the paging URL and make the request\n",
    "        all_status_url = wps_server + '&REQUEST=GetExecutions&startIndex=' + str(page_counter)\n",
    "\n",
    "        r = requests.get(all_status_url,headers=headers)\n",
    "\n",
    "        # parse the xml to an ordered dict using 3rd party imported module xmltodict\n",
    "        d = xmltodict.parse(r.content)\n",
    "\n",
    "        # grab the dicts of 'wps:ExecuteResponse' and add to a list\n",
    "        jobs_list = [value for value in d['wps:GetExecutionsResponse']['wps:ExecuteResponse']]\n",
    "\n",
    "        # add this to a master list\n",
    "        matching_of_dicts.append(jobs_list)\n",
    "\n",
    "        # if there is no 'next' attribute, then you're on the last xml page so break out the loop\n",
    "        if not any('@next' == key for key in list(d['wps:GetExecutionsResponse'].keys())):\n",
    "            break\n",
    "\n",
    "    num_of_xml_pages = int(page_counter / 10) + 1\n",
    "    duration_to_parse_xml_dt = datetime.utcnow() - check_time\n",
    "    duration_to_parse_xml_sec = round(duration_to_parse_xml_dt.total_seconds(),0)\n",
    "    print('#' + format(i, '03') + ' of #n   :: ' + datetime.utcnow().isoformat() + ' :: ' + str(num_of_xml_pages) + ' xml pages parsed in ' + str(duration_to_parse_xml_sec) + ' seconds')\n",
    "            \n",
    "    # now flatten the list into all responses\n",
    "    response_list = [item['wps:Status'] for sublist in matching_of_dicts for item in sublist]\n",
    "\n",
    "    # parse dict to pandas dataframe and set the index\n",
    "    df_temp = pd.DataFrame.from_dict(response_list)\n",
    "    df = df_temp.set_index('wps:JobID')\n",
    "\n",
    "    # filter df on current job_ids\n",
    "    filter_df  = df[df.index.isin(execution_id_dict.keys())]\n",
    "            \n",
    "    # add \"check time\" as column\n",
    "    filter_df.loc[:,'check_time'] = check_time.isoformat()\n",
    "    \n",
    "    # append on some extra info to the dataframe using the index\n",
    "    for index, row in filter_df.iterrows():\n",
    "        filter_df.loc[index,'layer_name'] = execution_id_dict[index]\n",
    "        filter_df.loc[index,'dl_url'] = wps_server + '&REQUEST=GetExecutionResult&EXECUTIONID='  + index + '&outputId=' + wps_test_config['gs:Download']['output'] + '&mimetype=' + wps_test_config['gs:Download']['mimetype']\n",
    "    \n",
    "    # sort out the indices and concatenate dataframes together and output to csv\n",
    "    filter_df.reset_index(inplace=True)\n",
    "    filter_df.set_index(['wps:JobID','check_time'],inplace=True)\n",
    "    frames.append(filter_df)\n",
    "    rolling_merged_df = pd.concat(frames)\n",
    "    rolling_merged_df.to_csv(log_file_name)\n",
    "        \n",
    "    # if NO running processes, then break, otherwise loop again\n",
    "    if not any(filter_df['wps:Status'] == 'RUNNING'):\n",
    "        break\n",
    "    print('#' + format(i, '03') + ' of #n   :: ' + datetime.utcnow().isoformat() + ' :: Jobs are still \"STATUS=RUNNING\"...script will poll again in ' + str(poll_frequency_sec) + 'sec time ...')\n",
    "    \n",
    "    time.sleep(poll_frequency_sec)\n",
    "    \n",
    "print('\\n#' + format(i, '03') + ' of #' + format(i, '03') + ' :: ' + datetime.utcnow().isoformat() + ' :: Script has finished')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# export summary wps jobs with \"runtimes\"\n",
    "sumdf = rolling_merged_df.reset_index()\n",
    "grouped = sumdf.groupby(['wps:JobID','wps:Status']).first()\n",
    "grouped.to_csv(my_path / str(submission_time.strftime(output_fmt) + '-' + RUNNAME + '-summary-wps-log.csv'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "2021-07-13T14:51:18.429881 :: DOWNLOAD START :: URL = https://earthobs.defra.gov.uk/geoserver/ows/?access_token=Zz1Q3arH4NroQAYKixNTw6ik6G0Reg&SERVICE=WPS&VERSION=1.0.0&REQUEST=GetExecutionResult&EXECUTIONID=0c884b62-1769-4e3c-9f01-1392313858aa&outputId=result.zip&mimetype=application/zip\n",
      "2021-07-13T14:52:54.213211 :: DOWNLOAD COMPLETE\n",
      "2021-07-13T14:52:54.213211 :: FILE WRITE START :: FILE = C:\\Users\\Charlie.Howarth\\Miniconda3\\Scripts\\output\\20210713T144456-APITEST130721\\S2A_20210531_lat54lon066_T30UXE_ORB137_utm30n_osgb_vmsk_sharp_rad_srefdem_stdsref.zip\n",
      "2021-07-13T14:52:56.900330 :: FILE WRITE COMPLETE\n",
      "2021-07-13T14:52:56.901324 :: EXTRACT ARCHIVE FILE STARTED\n",
      "2021-07-13T14:53:37.442475 :: EXTRACT ARCHIVE FILE COMPLETE\n",
      "2021-07-13T14:53:37.442475 :: RENAME/CLEAN UP DIRECTORY STARTED\n",
      "2021-07-13T14:53:37.561479 :: RENAME/CLEAN UP DIRECTORY COMPLETE\n",
      "\n",
      "\n",
      "2021-07-13T14:53:37.562479 :: DOWNLOAD START :: URL = https://earthobs.defra.gov.uk/geoserver/ows/?access_token=Zz1Q3arH4NroQAYKixNTw6ik6G0Reg&SERVICE=WPS&VERSION=1.0.0&REQUEST=GetExecutionResult&EXECUTIONID=ae1cb4d6-5698-4d21-b5d3-4d7d26190dde&outputId=result.zip&mimetype=application/zip\n",
      "2021-07-13T14:54:39.568829 :: DOWNLOAD COMPLETE\n",
      "2021-07-13T14:54:39.569829 :: FILE WRITE START :: FILE = C:\\Users\\Charlie.Howarth\\Miniconda3\\Scripts\\output\\20210713T144456-APITEST130721\\S2A_20210404_lat54lon066_T30UXE_ORB037_utm30n_osgb_vmsk_sharp_rad_srefdem_stdsref.zip\n",
      "2021-07-13T14:54:41.827912 :: FILE WRITE COMPLETE\n",
      "2021-07-13T14:54:41.828912 :: EXTRACT ARCHIVE FILE STARTED\n",
      "2021-07-13T14:55:14.462078 :: EXTRACT ARCHIVE FILE COMPLETE\n",
      "2021-07-13T14:55:14.463078 :: RENAME/CLEAN UP DIRECTORY STARTED\n",
      "2021-07-13T14:55:14.642081 :: RENAME/CLEAN UP DIRECTORY COMPLETE\n",
      "\n",
      "\n",
      "2021-07-13T14:55:14.643083 :: #### SCRIPT FINISHED\n"
     ]
    }
   ],
   "source": [
    "# DOWNLOAD, WRITE FILE AND RENAME FILE\n",
    "for index, row in filter_df.iterrows():\n",
    "    if row['wps:Status'] == 'SUCCEEDED':\n",
    "        print('\\n\\n' + datetime.utcnow().isoformat() + ' :: DOWNLOAD START :: URL = ' + row['dl_url'])        \n",
    "        response = requests.get(row['dl_url'],headers=headers);\n",
    "        print(datetime.utcnow().isoformat() + ' :: DOWNLOAD COMPLETE')\n",
    "        local_file_name = my_path / str(row['layer_name'].split(':')[-1] + '.zip')\n",
    "        print(datetime.utcnow().isoformat() + ' :: FILE WRITE START :: FILE = ' + str(local_file_name))        \n",
    "        with open(local_file_name, 'wb') as f:\n",
    "            f.write(response.content);\n",
    "            f.close();        \n",
    "        print(datetime.utcnow().isoformat() + ' :: FILE WRITE COMPLETE')\n",
    "        print(datetime.utcnow().isoformat() + ' :: EXTRACT ARCHIVE FILE STARTED')\n",
    "                \n",
    "        # TODO. Check if downloaded file is actually a zip file type. Handle exception for an xml error report?\n",
    "        zip_ref = ZipFile(local_file_name, 'r')\n",
    "        zip_ref.extractall(my_path)\n",
    "        zip_ref.close()\n",
    "        print(datetime.utcnow().isoformat() + ' :: EXTRACT ARCHIVE FILE COMPLETE')\n",
    "        print(datetime.utcnow().isoformat() + ' :: RENAME/CLEAN UP DIRECTORY STARTED')\n",
    "        for f in zip_ref.filelist:\n",
    "            if f.filename.endswith(\".sld\"):\n",
    "                Path(my_path / f.filename).unlink()\n",
    "            \n",
    "            # rename the uuid.tiff to layer_name.tif\n",
    "            elif f.filename.endswith(\".tif\") or f.filename.endswith(\".tiff\"):\n",
    "                Path(my_path / f.filename).rename(my_path / str(row['layer_name'].split(':')[-1] + '.tif'))\n",
    "            else:\n",
    "                raise ValueError(\"ERROR - some unknown file has been given:\", f)\n",
    "        \n",
    "        # remove the zip file from disk\n",
    "        Path(local_file_name).unlink()\n",
    "        print(datetime.utcnow().isoformat() + ' :: RENAME/CLEAN UP DIRECTORY COMPLETE')\n",
    "\n",
    "print('\\n\\n' + datetime.utcnow().isoformat() + ' :: #### SCRIPT FINISHED')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Importing libraries for Mosaic creation\n",
    "import rsgislib\n",
    "import rsgislib.imageutils\n",
    "import rsgislib.rastergis\n",
    "import rsgislib.imagecalc\n",
    "import rsgislib.imagecalc.calcindices\n",
    "import rsgislib.imageutils.imagecomp\n",
    "import os\n",
    "import glob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "### uesr needs to download or create refimg which is a raster with the same projection and extent as the full extent \n",
    "### of image to be combined. If using a single granule the files can be downloaded from CEDA archive.\n",
    "#Setting up the folder and the tif files for the mosaic\n",
    "dirpath= my_path\n",
    "search_critiria = \"*.tif\"\n",
    "q = os.path.join(dirpath,search_critiria)\n",
    "S2_fps = glob.glob(q)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "##Importing libraries for converting .tif to .kea for the Mosaic creation\n",
    "import os.path\n",
    "import sys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Setting up the convertion \n",
    "def replaceGTIFF_kea(inputtext):\n",
    "    outputtext = \"\"    \n",
    "    for w in inputtext:\n",
    "        w = w.replace(\"GEOTIFF\",\"KEA\")\n",
    "        w = w.replace(\".TIF\",\".kea\")\n",
    "        # this line should be unnecessary since Landsat MTL files use capital letters       \n",
    "        # but just in case you have one that doesn't\n",
    "        w = w.replace(\".tif\",\".kea\")\n",
    "        outputtext += w\n",
    "    return outputtex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find all *.TIF files and *MTL.txt files in the current directory\n",
    "directory = os.chdir(my_path)\n",
    "dirFileList = os.listdir(directory)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print dirFileList\n",
    "tifFileList = [f for f in dirFileList if ((f[-4:]=='.TIF')or(f[-4:]=='.tif'))]\n",
    "MTLFileList = [f for f in dirFileList if (f[-7:]=='MTL.txt')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "#output format (GDAL code)\n",
    "outFormat = 'KEA'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gdal_translate -of KEA S2A_20210404_lat54lon066_T30UXE_ORB037_utm30n_osgb_vmsk_sharp_rad_srefdem_stdsref.tif S2A_20210404_lat54lon066_T30UXE_ORB037_utm30n_osgb_vmsk_sharp_rad_srefdem_stdsref.kea\n",
      "gdal_translate -of KEA S2A_20210531_lat54lon066_T30UXE_ORB137_utm30n_osgb_valid.tif S2A_20210531_lat54lon066_T30UXE_ORB137_utm30n_osgb_valid.kea\n",
      "gdal_translate -of KEA S2A_20210531_lat54lon066_T30UXE_ORB137_utm30n_osgb_vmsk_sharp_rad_srefdem_stdsref.tif S2A_20210531_lat54lon066_T30UXE_ORB137_utm30n_osgb_vmsk_sharp_rad_srefdem_stdsref.kea\n"
     ]
    }
   ],
   "source": [
    "# run gdal_translate on all TIFs to convert to KEA\n",
    "for t in tifFileList:\n",
    "    gdaltranscmd = \"gdal_translate -of \"+outFormat+\" \"+t+\" \"+t[:-4]+\".kea\"\n",
    "    print (gdaltranscmd)\n",
    "    os.system(gdaltranscmd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a new header file referring to .kea files rather than .TIF\n",
    "for m in MTLFileList:\n",
    "    inputtext = file(m).readlines()\n",
    "    outputtext = replaceGTIFF_kea(inputtext)\n",
    "    outputfilebase = m[:-4]\n",
    "    outputfile = outputfilebase + \"_kea.txt\"\n",
    "    out = file(outputfile,\"w\")\n",
    "    out.write(outputtext)\n",
    "    out.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Charlie.Howarth\\Miniconda3\\Scripts\\output\\20210713T144456-APITEST130721\\*stdsref.kea\n"
     ]
    }
   ],
   "source": [
    "#Setting up the paths for finding the created .kea files\n",
    "dirpath= my_path\n",
    "search_critiria_kea = \"*stdsref.kea\"\n",
    "q = os.path.join(dirpath,search_critiria_kea)\n",
    "S2 = glob.glob(q)\n",
    "print(q)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['C:\\\\Users\\\\Charlie.Howarth\\\\Miniconda3\\\\Scripts\\\\output\\\\20210713T144456-APITEST130721\\\\S2A_20210404_lat54lon066_T30UXE_ORB037_utm30n_osgb_vmsk_sharp_rad_srefdem_stdsref.kea',\n",
       " 'C:\\\\Users\\\\Charlie.Howarth\\\\Miniconda3\\\\Scripts\\\\output\\\\20210713T144456-APITEST130721\\\\S2A_20210531_lat54lon066_T30UXE_ORB137_utm30n_osgb_vmsk_sharp_rad_srefdem_stdsref.kea']"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "S2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 2 images with overlap to create the composite.\n",
      "In Image (1):\tC:\\Users\\Charlie.Howarth\\Miniconda3\\Scripts\\output\\20210713T144456-APITEST130721\\S2A_20210404_lat54lon066_T30UXE_ORB037_utm30n_osgb_vmsk_sharp_rad_srefdem_stdsref.kea\n",
      "In Image (2):\tC:\\Users\\Charlie.Howarth\\Miniconda3\\Scripts\\output\\20210713T144456-APITEST130721\\S2A_20210531_lat54lon066_T30UXE_ORB137_utm30n_osgb_vmsk_sharp_rad_srefdem_stdsref.kea\n",
      "In Image (1):\tC:\\Users\\Charlie.Howarth\\Miniconda3\\Scripts\\output\\20210713T144456-APITEST130721\\S2A_20210404_lat54lon066_T30UXE_ORB037_utm30n_osgb_vmsk_sharp_rad_srefdem_stdsref.kea\n",
      "In Image (2):\tC:\\Users\\Charlie.Howarth\\Miniconda3\\Scripts\\output\\20210713T144456-APITEST130721\\S2A_20210531_lat54lon066_T30UXE_ORB137_utm30n_osgb_vmsk_sharp_rad_srefdem_stdsref.kea\n"
     ]
    }
   ],
   "source": [
    "# Mosaicing all the .kea files \n",
    "#inImages = S2\n",
    "#refImg = FG_Sentinel_ROI.kea\n",
    "#outCompImg = CompSentinel.kea\n",
    "#outMskImg = Mask.kea\n",
    "\n",
    "#rsgislib.imageutils.imagecomp.createMaxNDVINDWIComposite(refImg, inImages, 3, 8, 9, outRefImg, outCompImg, outMskImg, tmpPath='./tmp', gdalformat='KEA', dataType=None, calcStats=True, reprojmethod='cubic', use_mode=True)\n",
    "\n",
    "# Mosaicing all the .kea files \n",
    "inImages = q\n",
    "# refImg is any auxrillary file that will have the same extent and projection as the images in S2.\n",
    "# This could be a raster that you create if your covering more than 1 granule\n",
    "# alternatively refImg can be downloaded from The CEDA Archive as the granule will have a valid auxrillary file. \n",
    "refImg = r\"C:\\Users\\Charlie.Howarth\\Miniconda3\\Scripts\\output\\20210713T144456-APITEST130721\\S2A_20210531_lat54lon066_T30UXE_ORB137_utm30n_osgb_valid.kea\"\n",
    "outCompImg = 'CompSentinel.kea'\n",
    "outMskImg = 'Mask.kea'\n",
    "outRefImg = 'CompRefImage.kea'\n",
    "\n",
    "rsgislib.imageutils.imagecomp.createMaxNDVINDWIComposite(refImg, inImages, 3, 8, 9, outRefImg, outCompImg, outMskImg, tmpPath='./tmp', gdalformat='KEA', dataType=None, calcStats=True, reprojmethod='cubic')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Converting the created .kea mosaic back to .tif \n",
    "def replaceGTIFF_kea(inputtext):\n",
    "    outputtext = \"\"    \n",
    "    for w in inputtext:\n",
    "        w = w.replace(\"KEA\",\"GEOTIFF\")\n",
    "        w = w.replace(\".kea\",\".TIF\")\n",
    "        # this line should be unnecessary since Landsat MTL files use capital letters       \n",
    "        # but just in case you have one that doesn't\n",
    "        w = w.replace(\".kea\",\".tif\")\n",
    "        outputtext += w\n",
    "    return outputtex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find all *.kea files and *MTL.txt files in the current directory\n",
    "directory = os.chdir(r\"C:\\Users\\Charlie.Howarth\\Miniconda3\\Scripts\\output\\20210713T144456-APITEST130721\")\n",
    "dirFileList = os.listdir(directory)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print dirFileList\n",
    "tifFileList = [f for f in dirFileList if ((f[-4:]=='.KEA')or(f[-4:]=='.kea'))]\n",
    "MTLFileList = [f for f in dirFileList if (f[-7:]=='MTL.txt')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "#output format (GDAL code)\n",
    "outFormat = 'GTiff'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gdal_translate -of GTiff CompRefImage.kea CompRefImage.tif\n",
      "gdal_translate -of GTiff CompSentinel.kea CompSentinel.tif\n",
      "gdal_translate -of GTiff Mask.kea Mask.tif\n",
      "gdal_translate -of GTiff S2A_20210404_lat54lon066_T30UXE_ORB037_utm30n_osgb_vmsk_sharp_rad_srefdem_stdsref.kea S2A_20210404_lat54lon066_T30UXE_ORB037_utm30n_osgb_vmsk_sharp_rad_srefdem_stdsref.tif\n",
      "gdal_translate -of GTiff S2A_20210531_lat54lon066_T30UXE_ORB137_utm30n_osgb_valid.kea S2A_20210531_lat54lon066_T30UXE_ORB137_utm30n_osgb_valid.tif\n",
      "gdal_translate -of GTiff S2A_20210531_lat54lon066_T30UXE_ORB137_utm30n_osgb_vmsk_sharp_rad_srefdem_stdsref.kea S2A_20210531_lat54lon066_T30UXE_ORB137_utm30n_osgb_vmsk_sharp_rad_srefdem_stdsref.tif\n"
     ]
    }
   ],
   "source": [
    "#run gdal_translate on the .KEA Mosaic  to convert to .tiff\n",
    "for t in tifFileList:\n",
    "    gdaltranscmd = \"gdal_translate -of \"+outFormat+\" \"+t+\" \"+t[:-4]+\".tif\"\n",
    "    print (gdaltranscmd)\n",
    "    os.system(gdaltranscmd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}