Deep_Learning_Intro.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"NEWVERSION_Deep_Learning_Intro.ipynb","provenance":[],"collapsed_sections":["16DWjms_doQ6","1FWXTjf4cBBv","g03N8YTOmlb4","tD2sucV6dGl1","k5w5_4z8czaR","K_zRgGyseTEE","la_XtomXekfM"]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"16DWjms_doQ6","colab_type":"text"},"source":["## GPU General Information"]},{"cell_type":"code","metadata":{"id":"Ge8W3GuTEZD0","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":697},"executionInfo":{"status":"ok","timestamp":1597306648017,"user_tz":-180,"elapsed":2071,"user":{"displayName":"Spyros K","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi-WucjzMGe1cR2X3d0JEuHKhJ0GeeFGTohupjjaw=s64","userId":"10388578916492038588"}},"outputId":"f0223a53-a80e-4b46-8f00-49f6613e2c02"},"source":["%tensorflow_version 1.x\n","import tensorflow as tf\n","from tensorflow.python.client import device_lib\n","\n","print(f\"--- {tf.test.is_gpu_available()}\\n\")\n","print(f\"--- {tf.test.gpu_device_name()}\\n\")\n","print(f\"--- {tf.test.is_built_with_cuda()}\\n\")\n","print(f\"---\\n{device_lib.list_local_devices()}\\n\")"],"execution_count":1,"outputs":[{"output_type":"stream","text":["TensorFlow 1.x selected.\n","--- True\n","\n","--- /device:GPU:0\n","\n","--- True\n","\n","---\n","[name: \"/device:CPU:0\"\n","device_type: \"CPU\"\n","memory_limit: 268435456\n","locality {\n","}\n","incarnation: 9553709449716076208\n",", name: \"/device:XLA_CPU:0\"\n","device_type: \"XLA_CPU\"\n","memory_limit: 17179869184\n","locality {\n","}\n","incarnation: 541715707723116681\n","physical_device_desc: \"device: XLA_CPU device\"\n",", name: \"/device:XLA_GPU:0\"\n","device_type: \"XLA_GPU\"\n","memory_limit: 17179869184\n","locality {\n","}\n","incarnation: 14126520263883718962\n","physical_device_desc: \"device: XLA_GPU device\"\n",", name: \"/device:GPU:0\"\n","device_type: \"GPU\"\n","memory_limit: 14912199066\n","locality {\n","  bus_id: 1\n","  links {\n","  }\n","}\n","incarnation: 5956749867016241656\n","physical_device_desc: \"device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5\"\n","]\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"z7bZQb-YjxmA","colab_type":"text"},"source":["Previous execution some weeks ago showed that a **Tesla K80** was used instead.\n","\n","```\n","[name: \"/device:CPU:0\"\n","device_type: \"CPU\"\n","memory_limit: 268435456\n","locality {\n","}\n","incarnation: 10550069380342432296\n",", name: \"/device:XLA_CPU:0\"\n","device_type: \"XLA_CPU\"\n","memory_limit: 17179869184\n","locality {\n","}\n","incarnation: 16510589324407396482\n","physical_device_desc: \"device: XLA_CPU device\"\n",", name: \"/device:XLA_GPU:0\"\n","device_type: \"XLA_GPU\"\n","memory_limit: 17179869184\n","locality {\n","}\n","incarnation: 12770818163539961133\n","physical_device_desc: \"device: XLA_GPU device\"\n",", name: \"/device:GPU:0\"\n","device_type: \"GPU\"\n","memory_limit: 11276884378\n","locality {\n","  bus_id: 1\n","  links {\n","  }\n","}\n","incarnation: 13215302358784350175\n","physical_device_desc: \"device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7\"\n","]\n","```\n","\n","Followed by a **Tesla T4**.\n","\n"," ```\n","[name: \"/device:CPU:0\"\n","device_type: \"CPU\"\n","memory_limit: 268435456\n","locality {\n","}\n","incarnation: 13999904789832453405\n",", name: \"/device:XLA_CPU:0\"\n","device_type: \"XLA_CPU\"\n","memory_limit: 17179869184\n","locality {\n","}\n","incarnation: 2474607553434206280\n","physical_device_desc: \"device: XLA_CPU device\"\n",", name: \"/device:XLA_GPU:0\"\n","device_type: \"XLA_GPU\"\n","memory_limit: 17179869184\n","locality {\n","}\n","incarnation: 5105918168026004920\n","physical_device_desc: \"device: XLA_GPU device\"\n",", name: \"/device:GPU:0\"\n","device_type: \"GPU\"\n","memory_limit: 14800630580\n","locality {\n","  bus_id: 1\n","  links {\n","  }\n","}\n","incarnation: 8208043236747855715\n","physical_device_desc: \"device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5\"\n","] \n","```"]},{"cell_type":"markdown","metadata":{"id":"1FWXTjf4cBBv","colab_type":"text"},"source":["## GPU Memory Availability Testing\n","\n","Just to make sure that Google isn't purposely limiting access to the GPU memory."]},{"cell_type":"code","metadata":{"id":"EUwDbuBDVKNO","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":102},"executionInfo":{"status":"ok","timestamp":1597306656159,"user_tz":-180,"elapsed":10196,"user":{"displayName":"Spyros K","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi-WucjzMGe1cR2X3d0JEuHKhJ0GeeFGTohupjjaw=s64","userId":"10388578916492038588"}},"outputId":"981e1afd-cc9e-492a-a8d0-f196026b47f0"},"source":["# Source: https://stackoverflow.com/questions/48750199/google-colaboratory-misleading-information-about-its-gpu-only-5-ram-available\n","!pip install gputil\n","!pip install psutil\n","!pip install humanize\n","\n","import psutil\n","import humanize\n","import os\n","import GPUtil as GPU\n","\n","GPUs = GPU.getGPUs()\n","# Only one GPU on Colab and isn’t guaranteed\n","gpu = GPUs[0]\n","def printX():\n","    process = psutil.Process(os.getpid())\n","    print(f\"Gen RAM Free: {humanize.naturalsize(psutil.virtual_memory().available)} | Proc size: {humanize.naturalsize(process.memory_info().rss)}\")\n","    print(f\"GPU RAM Free: {gpu.memoryFree:.0f} MB | Used: {gpu.memoryUsed:.0f} MB | Util {gpu.memoryUtil*100:.0f}% | Total {gpu.memoryTotal:.0f} MB\")\n","printX()"],"execution_count":2,"outputs":[{"output_type":"stream","text":["Requirement already satisfied: gputil in /usr/local/lib/python3.6/dist-packages (1.4.0)\n","Requirement already satisfied: psutil in /usr/local/lib/python3.6/dist-packages (5.4.8)\n","Requirement already satisfied: humanize in /usr/local/lib/python3.6/dist-packages (0.5.1)\n","Gen RAM Free: 12.6 GB | Proc size: 468.3 MB\n","GPU RAM Free: 14968 MB | Used: 111 MB | Util 1% | Total 15079 MB\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"g03N8YTOmlb4","colab_type":"text"},"source":["## CPU General Information\n","\n","The CPU appears to have two cores available."]},{"cell_type":"code","metadata":{"id":"M1cilSMMmjkA","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":445},"executionInfo":{"status":"ok","timestamp":1597306657629,"user_tz":-180,"elapsed":11653,"user":{"displayName":"Spyros K","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi-WucjzMGe1cR2X3d0JEuHKhJ0GeeFGTohupjjaw=s64","userId":"10388578916492038588"}},"outputId":"da693e83-cd3d-4ab0-dc25-2e1ff5b424f8"},"source":["!lscpu"],"execution_count":3,"outputs":[{"output_type":"stream","text":["Architecture:        x86_64\n","CPU op-mode(s):      32-bit, 64-bit\n","Byte Order:          Little Endian\n","CPU(s):              2\n","On-line CPU(s) list: 0,1\n","Thread(s) per core:  2\n","Core(s) per socket:  1\n","Socket(s):           1\n","NUMA node(s):        1\n","Vendor ID:           GenuineIntel\n","CPU family:          6\n","Model:               79\n","Model name:          Intel(R) Xeon(R) CPU @ 2.20GHz\n","Stepping:            0\n","CPU MHz:             2200.000\n","BogoMIPS:            4400.00\n","Hypervisor vendor:   KVM\n","Virtualization type: full\n","L1d cache:           32K\n","L1i cache:           32K\n","L2 cache:            256K\n","L3 cache:            56320K\n","NUMA node0 CPU(s):   0,1\n","Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat md_clear arch_capabilities\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"tD2sucV6dGl1","colab_type":"text"},"source":["## A basic Deep Learning Example #1\n","\n","A text classification task on the IMDb Dataset."]},{"cell_type":"code","metadata":{"id":"Ok_qXRgOQ2rH","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":666},"executionInfo":{"status":"ok","timestamp":1597306667560,"user_tz":-180,"elapsed":21572,"user":{"displayName":"Spyros K","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi-WucjzMGe1cR2X3d0JEuHKhJ0GeeFGTohupjjaw=s64","userId":"10388578916492038588"}},"outputId":"f5b01dd6-8cf8-40b5-bc56-09fcb947f629"},"source":["from __future__ import absolute_import, division, print_function\n","\n","import tensorflow as tf\n","from tensorflow import keras\n","\n","import numpy as np\n","\n","imdb = keras.datasets.imdb\n","(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)\n","\n","print(\"Training entries: {}, labels: {}\".format(len(train_data), len(train_labels)))\n","\n","word_index = imdb.get_word_index()\n","word_index[\"<PAD>\"] = 0\n","word_index[\"<START>\"] = 1\n","word_index[\"<UNK>\"] = 2  # unknown\n","word_index[\"<UNUSED>\"] = 3\n","\n","train_data = keras.preprocessing.sequence.pad_sequences(train_data,\n","                                                        value=word_index[\"<PAD>\"],\n","                                                        padding='post',\n","                                                        maxlen=256)\n","\n","test_data = keras.preprocessing.sequence.pad_sequences(test_data,\n","                                                       value=word_index[\"<PAD>\"],\n","                                                       padding='post',\n","                                                       maxlen=256)\n","\n","#print(train_data[0], type(train_data[0]))\n","\n","# input shape is the vocabulary count used for the movie reviews (10,000 words)\n","vocab_size = 10000\n","\n","model = keras.Sequential()\n","model.add(keras.layers.Embedding(vocab_size, 16))\n","#model.add(keras.layers.Bidirectional(tf.keras.layers.LSTM(16)))\n","model.add(keras.layers.GlobalAveragePooling1D())\n","model.add(keras.layers.Dense(16, activation=tf.nn.relu))\n","model.add(keras.layers.Dense(1, activation=tf.nn.sigmoid))\n","\n","model.summary()\n","\n","model.compile(optimizer='adam',\n","              loss='binary_crossentropy',\n","              metrics=['acc'])\n","\n","x_val = train_data[:10000]\n","partial_x_train = train_data[10000:]\n","\n","y_val = train_labels[:10000]\n","partial_y_train = train_labels[10000:]\n","\n","history = model.fit(partial_x_train,\n","                    partial_y_train,\n","                    epochs=4,\n","                    batch_size=128,\n","                    validation_data=(x_val, y_val),\n","                    verbose=1)\n","\n","results = model.evaluate(test_data, test_labels)\n","\n","print(results)"],"execution_count":4,"outputs":[{"output_type":"stream","text":["Training entries: 25000, labels: 25000\n","WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/keras/initializers.py:119: calling RandomUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Call initializer instance with the dtype argument instead of passing it to the constructor\n","WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n","Instructions for updating:\n","If using Keras pass *_constraint arguments to layers.\n","Model: \"sequential\"\n","_________________________________________________________________\n","Layer (type)                 Output Shape              Param #   \n","=================================================================\n","embedding (Embedding)        (None, None, 16)          160000    \n","_________________________________________________________________\n","global_average_pooling1d (Gl (None, 16)                0         \n","_________________________________________________________________\n","dense (Dense)                (None, 16)                272       \n","_________________________________________________________________\n","dense_1 (Dense)              (None, 1)                 17        \n","=================================================================\n","Total params: 160,289\n","Trainable params: 160,289\n","Non-trainable params: 0\n","_________________________________________________________________\n","WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/nn_impl.py:183: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use tf.where in 2.0, which has the same broadcast rule as np.where\n","Train on 15000 samples, validate on 10000 samples\n","Epoch 1/4\n","15000/15000 [==============================] - 1s 68us/sample - loss: 0.6870 - acc: 0.6177 - val_loss: 0.6734 - val_acc: 0.7604\n","Epoch 2/4\n","15000/15000 [==============================] - 1s 38us/sample - loss: 0.6269 - acc: 0.7734 - val_loss: 0.5707 - val_acc: 0.7994\n","Epoch 3/4\n","15000/15000 [==============================] - 1s 39us/sample - loss: 0.4893 - acc: 0.8323 - val_loss: 0.4393 - val_acc: 0.8419\n","Epoch 4/4\n","15000/15000 [==============================] - 1s 39us/sample - loss: 0.3716 - acc: 0.8737 - val_loss: 0.3626 - val_acc: 0.8621\n","25000/25000 [==============================] - 1s 39us/sample - loss: 0.3698 - acc: 0.8578\n","[0.36979981117248534, 0.8578]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"k5w5_4z8czaR","colab_type":"text"},"source":["## A basic Deep Learning Example #2\n","\n","A text classification task on the IMDb Dataset."]},{"cell_type":"code","metadata":{"id":"AjP5_1GHGWjp","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":408},"executionInfo":{"status":"ok","timestamp":1597306734018,"user_tz":-180,"elapsed":88020,"user":{"displayName":"Spyros K","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi-WucjzMGe1cR2X3d0JEuHKhJ0GeeFGTohupjjaw=s64","userId":"10388578916492038588"}},"outputId":"5680adbe-40f7-4ee5-b96c-dae6b47ca2da"},"source":["'''\n","#Trains an LSTM model on the IMDB sentiment classification task.\n","The dataset is actually too small for LSTM to be of any advantage\n","compared to simpler, much faster methods such as TF-IDF + LogReg.\n","**Notes**\n","- RNNs are tricky. Choice of batch size is important,\n","choice of loss and optimizer is critical, etc.\n","Some configurations won't converge.\n","- LSTM loss decrease patterns during training can be quite different\n","from what you see with CNNs/MLPs/etc.\n","'''\n","from __future__ import print_function\n","\n","from keras.preprocessing import sequence\n","from keras.models import Sequential\n","from keras.layers import Dense, Embedding\n","from keras.layers import LSTM\n","from keras.datasets import imdb\n","\n","max_features = 20000\n","# cut texts after this number of words (among top max_features most common words)\n","maxlen = 80\n","batch_size = 128\n","\n","print('Loading data...')\n","(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)\n","print(len(x_train), 'train sequences')\n","print(len(x_test), 'test sequences')\n","\n","print('Pad sequences (samples x time)')\n","x_train = sequence.pad_sequences(x_train, maxlen=maxlen)\n","x_test = sequence.pad_sequences(x_test, maxlen=maxlen)\n","print('x_train shape:', x_train.shape)\n","print('x_test shape:', x_test.shape)\n","\n","print('Build model...')\n","model = Sequential()\n","model.add(Embedding(max_features, 128))\n","model.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))\n","model.add(Dense(1, activation='sigmoid'))\n","\n","# try using different optimizers and different optimizer configs\n","model.compile(loss='binary_crossentropy',\n","              optimizer='adam',\n","              metrics=['accuracy'])\n","\n","print('Train...')\n","model.fit(x_train, y_train,\n","          batch_size=batch_size,\n","          epochs=4,\n","          validation_data=(x_test, y_test))\n","score, acc = model.evaluate(x_test, y_test,\n","                            batch_size=batch_size)\n","print('Test score:', score)\n","print('Test accuracy:', acc)"],"execution_count":5,"outputs":[{"output_type":"stream","text":["Loading data...\n"],"name":"stdout"},{"output_type":"stream","text":["Using TensorFlow backend.\n"],"name":"stderr"},{"output_type":"stream","text":["25000 train sequences\n","25000 test sequences\n","Pad sequences (samples x time)\n","x_train shape: (25000, 80)\n","x_test shape: (25000, 80)\n","Build model...\n","Train...\n","WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n","\n","Train on 25000 samples, validate on 25000 samples\n","Epoch 1/4\n","25000/25000 [==============================] - 14s 576us/step - loss: 0.4803 - accuracy: 0.7710 - val_loss: 0.4145 - val_accuracy: 0.8135\n","Epoch 2/4\n","25000/25000 [==============================] - 14s 577us/step - loss: 0.3027 - accuracy: 0.8771 - val_loss: 0.3726 - val_accuracy: 0.8326\n","Epoch 3/4\n","25000/25000 [==============================] - 15s 590us/step - loss: 0.2329 - accuracy: 0.9116 - val_loss: 0.4107 - val_accuracy: 0.8274\n","Epoch 4/4\n","25000/25000 [==============================] - 15s 589us/step - loss: 0.1827 - accuracy: 0.9320 - val_loss: 0.4443 - val_accuracy: 0.8256\n","25000/25000 [==============================] - 3s 103us/step\n","Test score: 0.4442848878669739\n","Test accuracy: 0.8256400227546692\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"K_zRgGyseTEE","colab_type":"text"},"source":["## A basic Deep Learning Example #3\n","\n","An image classification task on the MNIST dataset."]},{"cell_type":"code","metadata":{"id":"IwDWOGuaeHi1","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":493},"executionInfo":{"status":"ok","timestamp":1597306868955,"user_tz":-180,"elapsed":222950,"user":{"displayName":"Spyros K","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi-WucjzMGe1cR2X3d0JEuHKhJ0GeeFGTohupjjaw=s64","userId":"10388578916492038588"}},"outputId":"19376e41-1767-486c-d250-4d52fe0a1919"},"source":["from __future__ import print_function\n","import keras\n","from keras.datasets import mnist\n","from keras.models import Sequential\n","from keras.layers import Dense, Dropout, Flatten\n","from keras.layers import Conv2D, MaxPooling2D\n","from keras import backend as K\n","from keras.callbacks import TensorBoard\n","\n","batch_size = 128\n","num_classes = 10\n","epochs = 4\n","\n","# input image dimensions\n","img_rows, img_cols = 28, 28\n","\n","# the data, shuffled and split between train and test sets\n","(x_train, y_train), (x_test, y_test) = mnist.load_data()\n","\n","if K.image_data_format() == 'channels_first':\n","    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n","    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n","    input_shape = (1, img_rows, img_cols)\n","else:\n","    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n","    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n","    input_shape = (img_rows, img_cols, 1)\n","\n","x_train = x_train.astype('float32')\n","x_test = x_test.astype('float32')\n","x_train /= 255\n","x_test /= 255\n","print('x_train shape:', x_train.shape)\n","print(x_train.shape[0], 'train samples')\n","print(x_test.shape[0], 'test samples')\n","\n","# convert class vectors to binary class matrices\n","y_train = keras.utils.to_categorical(y_train, num_classes)\n","y_test = keras.utils.to_categorical(y_test, num_classes)\n","\n","model = Sequential()\n","model.add(Conv2D(32, kernel_size=(3, 3),\n","                 activation='relu',\n","                 input_shape=input_shape))\n","model.add(Conv2D(64, (3, 3), activation='relu'))\n","model.add(MaxPooling2D(pool_size=(2, 2)))\n","model.add(Dropout(0.25))\n","model.add(Flatten())\n","model.add(Dense(128, activation='relu'))\n","model.add(Dropout(0.5))\n","model.add(Dense(num_classes, activation='softmax'))\n","\n","model.compile(loss=keras.losses.categorical_crossentropy,\n","              optimizer=keras.optimizers.Adadelta(),\n","              metrics=['accuracy'])\n","\n","\n","tbCallBack = TensorBoard(histogram_freq=1,\n","                         write_graph=True,\n","                         write_grads=True,\n","                         batch_size=batch_size,\n","                         write_images=True)\n","\n","model.fit(x_train, y_train,\n","          batch_size=batch_size,\n","          epochs=epochs,\n","          verbose=1,\n","          validation_data=(x_test, y_test),\n","          callbacks=[tbCallBack])\n","score = model.evaluate(x_test, y_test, verbose=0)\n","print('Test loss:', score[0])\n","print('Test accuracy:', score[1])"],"execution_count":6,"outputs":[{"output_type":"stream","text":["Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz\n","11493376/11490434 [==============================] - 1s 0us/step\n","x_train shape: (60000, 28, 28, 1)\n","60000 train samples\n","10000 test samples\n","WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/keras/backend/tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.\n","\n","Train on 60000 samples, validate on 10000 samples\n","WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/keras/callbacks/tensorboard_v1.py:146: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.\n","\n","WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/keras/callbacks/tensorboard_v1.py:190: The name tf.summary.image is deprecated. Please use tf.compat.v1.summary.image instead.\n","\n","WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/keras/callbacks/tensorboard_v1.py:200: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.\n","\n","WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/keras/callbacks/tensorboard_v1.py:203: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.\n","\n","Epoch 1/4\n","60000/60000 [==============================] - 10s 169us/step - loss: 0.2677 - accuracy: 0.9170 - val_loss: 0.0574 - val_accuracy: 0.9819\n","WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/keras/callbacks/tensorboard_v1.py:343: The name tf.Summary is deprecated. Please use tf.compat.v1.Summary instead.\n","\n","Epoch 2/4\n","60000/60000 [==============================] - 5s 80us/step - loss: 0.0877 - accuracy: 0.9733 - val_loss: 0.0424 - val_accuracy: 0.9856\n","Epoch 3/4\n","60000/60000 [==============================] - 5s 80us/step - loss: 0.0657 - accuracy: 0.9804 - val_loss: 0.0313 - val_accuracy: 0.9892\n","Epoch 4/4\n","60000/60000 [==============================] - 5s 79us/step - loss: 0.0542 - accuracy: 0.9832 - val_loss: 0.0310 - val_accuracy: 0.9890\n","Test loss: 0.03100653666415019\n","Test accuracy: 0.9890000224113464\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"la_XtomXekfM","colab_type":"text"},"source":["## A basic Deep Learning Example #4\n","\n","A text classification task on the IMDb dataset, with an attached benchmark score for the K520 GPU for batch_size equal to 30.  \n","Refer to the 'GPU Specification List'."]},{"cell_type":"code","metadata":{"id":"HuCD4WgTeieu","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":357},"executionInfo":{"status":"ok","timestamp":1597307002230,"user_tz":-180,"elapsed":356220,"user":{"displayName":"Spyros K","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Gi-WucjzMGe1cR2X3d0JEuHKhJ0GeeFGTohupjjaw=s64","userId":"10388578916492038588"}},"outputId":"2a2a3daa-f7a0-4c95-80e1-a4de604f8289"},"source":["'''\n","#Train a recurrent convolutional network on the IMDB sentiment classification task.\n","Gets to 0.8498 test accuracy after 2 epochs. 41 s/epoch on K520 GPU.\n","'''\n","from __future__ import print_function\n","\n","from keras.preprocessing import sequence\n","from keras.models import Sequential\n","from keras.layers import Dense, Dropout, Activation\n","from keras.layers import Embedding\n","from keras.layers import LSTM\n","from keras.layers import Conv1D, MaxPooling1D\n","from keras.datasets import imdb\n","\n","# Embedding\n","max_features = 20000\n","maxlen = 100\n","embedding_size = 128\n","\n","# Convolution\n","kernel_size = 5\n","filters = 64\n","pool_size = 4\n","\n","# LSTM\n","lstm_output_size = 70\n","\n","# Training\n","batch_size = 30\n","epochs = 4\n","\n","'''\n","Note:\n","batch_size is highly sensitive.\n","Only 2 epochs are needed as the dataset is very small.\n","'''\n","\n","print('Loading data...')\n","(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)\n","print(len(x_train), 'train sequences')\n","print(len(x_test), 'test sequences')\n","\n","print('Pad sequences (samples x time)')\n","x_train = sequence.pad_sequences(x_train, maxlen=maxlen)\n","x_test = sequence.pad_sequences(x_test, maxlen=maxlen)\n","print('x_train shape:', x_train.shape)\n","print('x_test shape:', x_test.shape)\n","\n","print('Build model...')\n","\n","model = Sequential()\n","model.add(Embedding(max_features, embedding_size, input_length=maxlen))\n","model.add(Dropout(0.25))\n","model.add(Conv1D(filters,\n","                 kernel_size,\n","                 padding='valid',\n","                 activation='relu',\n","                 strides=1))\n","model.add(MaxPooling1D(pool_size=pool_size))\n","model.add(LSTM(lstm_output_size))\n","model.add(Dense(1))\n","model.add(Activation('sigmoid'))\n","\n","model.compile(loss='binary_crossentropy',\n","              optimizer='adam',\n","              metrics=['accuracy'])\n","\n","print('Train...')\n","model.fit(x_train, y_train,\n","          batch_size=batch_size,\n","          epochs=epochs,\n","          validation_data=(x_test, y_test))\n","score, acc = model.evaluate(x_test, y_test, batch_size=batch_size)\n","print('Test score:', score)\n","print('Test accuracy:', acc)"],"execution_count":7,"outputs":[{"output_type":"stream","text":["Loading data...\n","25000 train sequences\n","25000 test sequences\n","Pad sequences (samples x time)\n","x_train shape: (25000, 100)\n","x_test shape: (25000, 100)\n","Build model...\n","Train...\n","Train on 25000 samples, validate on 25000 samples\n","Epoch 1/4\n","25000/25000 [==============================] - 31s 1ms/step - loss: 0.3876 - accuracy: 0.8157 - val_loss: 0.3208 - val_accuracy: 0.8631\n","Epoch 2/4\n","25000/25000 [==============================] - 30s 1ms/step - loss: 0.1942 - accuracy: 0.9250 - val_loss: 0.3729 - val_accuracy: 0.8533\n","Epoch 3/4\n","25000/25000 [==============================] - 29s 1ms/step - loss: 0.0944 - accuracy: 0.9684 - val_loss: 0.4419 - val_accuracy: 0.8468\n","Epoch 4/4\n","25000/25000 [==============================] - 29s 1ms/step - loss: 0.0424 - accuracy: 0.9864 - val_loss: 0.5679 - val_accuracy: 0.8350\n","25000/25000 [==============================] - 9s 346us/step\n","Test score: 0.5678787619415671\n","Test accuracy: 0.8350399732589722\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"ywgUBeSbk4l5","colab_type":"text"},"source":["## Other Information - GPU Specifications List\n","\n","| Model | Used in Tensorflow | Generation | Memory | Bus | Shading Units | GPU Clock | Memory Clock | Theoretical FP32 | Note |\n","| - | - | - | - | - | - | - | - | - | - |\n","| Tesla T4 | Yes | Tesla | 16 GB | 256bit | 2560 | 585MHz (1590 boosted) | 10000MHz effective | 8.141 TFLOPS | |\n","| Tesla  K80 | Yes | Tesla | 12 GB x2 | 384bit x2| 2496 x2 | 562MHz (824 boosted) | 5012MHz effective | 4.113 TFLOPS x2 | The 'x2' is irrelevant, only 1 is used |\n","| Grid K520 | No | Grid K5 | 4 GB x2 | 256bit x2 | 1536 x2 | 745MHz | 5000MHz effective | 2.289 TFLOPS x2 | The 'x2' is irrelevant, only 1 is used |"]}]}