examples/exporting_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exporting data with numpy and h5py\n",
    "\n",
    "This notebook shows different ways to export the data for eoflow using numpy or h5py."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import h5py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create temp dir\n",
    "os.makedirs('temp')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Method 1: saving arrays using numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create some numpy arrays to represent our features and labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = np.random.random(size=(1024, 32, 32, 13))\n",
    "labels = np.random.randint(10, size=(1024,))\n",
    "\n",
    "features.shape, labels.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For numpy use the `np.savez` function to save multiple arrays into a single `.npz` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.savez('temp/data.npz', features=features, labels=labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numpy reads and writes the whole file at the same time. Therefore the file size should be small to reduce the overhead.\n",
    "\n",
    "If the dataset size is large (can't fit into memory) it is better to split the dataset into multiple .npz files, or use the hdf5 format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Method 2: saving arrays using h5py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's save the same data using the h5py library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with h5py.File('temp/data.hdf5', 'w') as file:\n",
    "    file.create_dataset('features', data=features)\n",
    "    file.create_dataset('labels', data=labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The h5py allows us to create seperate datasets (and groups of datasets) and save the data to it. The format also allows for sequential reading. This means that only part of the data that is needed can be loaded. Therefore the spliting of the dataset into smaller pieces is not needed anymore.\n",
    "\n",
    "However, if the dataset we want to export is too big to fit into memory we cannot use this method to export the data. That's where the **Method 3** comes in."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Method 3: saving arrays iteratively using h5py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The h5py allows us to write the data in parts (e.g. row by row). The datasets we create can be indexed and written to similarly to numpy arrays. Let's export a dataset produced from a generator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def _generate_data(num_examples):\n",
    "    \"\"\" Generates specified number of examples (example by example).\"\"\"\n",
    "    \n",
    "    for i in range(num_examples):\n",
    "        features = np.random.random(size=(32, 32, 13))\n",
    "        labels = np.random.randint(10, size=())\n",
    "        \n",
    "        yield features, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with h5py.File('temp/data_gen.hdf5', 'w') as file:\n",
    "    num_examples = 1024\n",
    "    \n",
    "    # Define datasets (total shape)\n",
    "    features_ds = file.create_dataset('features', (num_examples, 32, 32, 13), dtype=np.float32)\n",
    "    labels_ds = file.create_dataset('labels', (num_examples,), dtype=np.int32)\n",
    "    \n",
    "    # Store the generated data into the datasets\n",
    "    for i, (features, labels) in enumerate(_generate_data(num_examples)):\n",
    "        features_ds[i] = features\n",
    "        labels_ds[i] = labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE**: the `data_gen.hdf5` size is smaller, because we specified the dtype of the features to be float32, while the original dtype of the array is float64."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tf2",
   "language": "python",
   "name": "tf2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}