rekognition-apis/4-custom-celebrity-recognition.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Custom Celebrity Recognition Using Amazon Rekognition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "This notebook provides a walkthrough of recognizing custom celebrities using Amazon Rekognition. You will first index faces of custom celebrities and then use SearchFaces API (https://docs.aws.amazon.com/rekognition/latest/APIReference/API_SearchFacesByImage.html and https://docs.aws.amazon.com/rekognition/latest/APIReference/API_StartFaceSearch.html) with sample image and video to detect custom celebrities.\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Initialize Stuff\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialise Notebook\n",
    "import boto3\n",
    "from IPython.display import HTML, display, Image as IImage\n",
    "from PIL import Image, ImageDraw, ImageFont\n",
    "import time\n",
    "import os\n",
    "from io import BytesIO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get current region to choose correct bucket\n",
    "mySession = boto3.session.Session()\n",
    "awsRegion = mySession.region_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize clients\n",
    "rekognition = boto3.client('rekognition')\n",
    "dynamodb = boto3.client('dynamodb')\n",
    "s3 = boto3.client('s3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# S3 bucket that contains sample images and videos\n",
    "\n",
    "# We are providing sample images and videos in this bucket so\n",
    "# you do not have to manually download/upload test images and videos.\n",
    "bucketName = \"aws-rek-immersionday-\" + awsRegion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DynamoDB Table and Rekognition Collection names. We will be creating these in this module.\n",
    "ddbTableName = \"my-celebrities\"\n",
    "collectionId = \"my-celebrities\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create temporary directory\n",
    "# This directory is not needed to call Rekognition APIs.\n",
    "# We will only use this directory to download images from S3 bucket and draw bounding boxes\n",
    "\n",
    "!mkdir m2tmp\n",
    "tempFolder = 'm2tmp/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DynamoDB table to store custom celebrity metadata\n",
    "***\n",
    "In this step we will create a DynamoDB table to store custom celebrity metadata including id, name and url. You can store additional attributes for each celebrity if needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### List existing DynamoDB tables in your account"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List existing DynamoDB Tables\n",
    "# Before creating DynamoDB table, let us first look at the list of existing DynamoDB tables in our account.\n",
    "\n",
    "listTablesResponse = dynamodb.list_tables()\n",
    "display(listTablesResponse[\"TableNames\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create new DynamoDB Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create new DynamoDB Table\n",
    "        \n",
    "createTableResponse = dynamodb.create_table(\n",
    "    TableName=ddbTableName,\n",
    "    KeySchema=[\n",
    "        {\n",
    "            'AttributeName': 'id',\n",
    "            'KeyType': 'HASH'  #Partition key\n",
    "        }\n",
    "    ],\n",
    "    AttributeDefinitions=[\n",
    "        {\n",
    "            'AttributeName': 'id',\n",
    "            'AttributeType': 'S'\n",
    "        },\n",
    "\n",
    "    ],\n",
    "    BillingMode='PAY_PER_REQUEST'\n",
    ")\n",
    "\n",
    "display(createTableResponse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### List DynamoDB Tables in your account to see newly created table "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List DynamoDB Tables\n",
    "# Let us look at list of our DynamoDB tables again to make sure that table we just created appears in the list.\n",
    "\n",
    "listTablesResponse = dynamodb.list_tables()\n",
    "display(listTablesResponse[\"TableNames\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Rekogniton Collection to store faces\n",
    "***\n",
    "In this step we will create a Rekognition Collection.\n",
    "\n",
    "Amazon Rekognition can store information about detected faces in server-side containers known as [collections](https://docs.aws.amazon.com/rekognition/latest/dg/collections.html). You can use the facial information that's stored in a collection to search for known faces in images, stored videos, and streaming videos. In this section you will learn how you can create and manage Rekognition Collections."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### List Rekognition Collections"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List Rekognition Collections\n",
    "# Let us first see if we have already created any Rekognition collections in our account.\n",
    "# If there is not an existing Rekognition in your account, you will see empty list\n",
    "# otherwise you will a list with names of rekognition collections and face model version.\n",
    "\n",
    "listCollectionsResponse = rekognition.list_collections()\n",
    "\n",
    "display(listCollectionsResponse[\"CollectionIds\"])\n",
    "display(listCollectionsResponse[\"FaceModelVersions\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create new Rekognition collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#cids = listCollectionsResponse[\"CollectionIds\"]\n",
    "#for cid in cids:\n",
    "#    rekognition.delete_collection(CollectionId=cid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create Rekognition Collection\n",
    "# Let us now create a new Rekognition collection that we will use to store faces of custom celebrities.\n",
    "\n",
    "createCollectionResponse = rekognition.create_collection(\n",
    "    CollectionId=collectionId\n",
    ")\n",
    "display(createCollectionResponse)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### List Rekognition collections to see newly created Rekognition collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List Rekognition Collections\n",
    "# Let us make sure that Recognition we just created now appears in the list of collections in our AWS account.\n",
    "listCollectionsResponse = rekognition.list_collections()\n",
    "\n",
    "display(listCollectionsResponse[\"CollectionIds\"])\n",
    "display(listCollectionsResponse[\"FaceModelVersions\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### View additional information about the collection we just created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Describe Rekognition Collection\n",
    "# You can use DescribeCollection to get information, \n",
    "# such as the number of faces indexed into a collection \n",
    "# and the version of the model used by the collection for face detection etc.\n",
    "# https://docs.aws.amazon.com/rekognition/latest/APIReference/API_DescribeCollection.html\n",
    "\n",
    "# Since we have not indexed any faces yet, you should see FaceCount: 0\n",
    "\n",
    "describeCollectionResponse = rekognition.describe_collection(\n",
    "    CollectionId=collectionId\n",
    ")\n",
    "display(describeCollectionResponse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Index Custom Celebrity Faces\n",
    "***\n",
    "\n",
    "In this step, you will index faces of custom celebrities in Rekognition collection and store their additional information in the DynamoDB table created in earlier steps.\n",
    "\n",
    "We will index multiple images for each celebrity. By indexing multiple faces we increase the likelihood of detecting celebrities when their face is at different angles, etc. We will use [IndexFaces](https://docs.aws.amazon.com/rekognition/latest/APIReference/API_IndexFaces.html) to detect faces in the input image and [add them](https://docs.aws.amazon.com/rekognition/latest/dg/add-faces-to-collection-procedure.html) to the specified collection.\n",
    "\n",
    "You can read more about some of the best practices around [indexing faces here in the blog](https://aws.amazon.com/blogs/machine-learning/save-time-and-money-by-filtering-faces-during-indexing-with-amazon-rekognition/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define methods to add face to Rekognition collection and add related attributes to DynamoDB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will define a method to index a face along with the celebrity id\n",
    "# https://docs.aws.amazon.com/rekognition/latest/APIReference/API_IndexFaces.html\n",
    "\n",
    "def indexFace (bucketName, imageName, celebrityId):\n",
    "\n",
    "    indexFaceResponse = rekognition.index_faces(\n",
    "        CollectionId=collectionId,\n",
    "        Image={\n",
    "            'S3Object': {\n",
    "                'Bucket': bucketName,\n",
    "                'Name': imageName,\n",
    "            }\n",
    "        },\n",
    "        ExternalImageId=celebrityId,\n",
    "        DetectionAttributes=[\n",
    "            'DEFAULT' #'DEFAULT'|'ALL',\n",
    "        ],\n",
    "        MaxFaces=1,\n",
    "        QualityFilter='AUTO' #NONE | AUTO | LOW | MEDIUM | HIGH\n",
    "    )\n",
    "    \n",
    "    display(indexFaceResponse)\n",
    "\n",
    "# We will define a method to write metadata (id, name, url) of celebrity to DynamoDB\n",
    "def addCelebrityToDynamoDB(celebrityId, celebrityName, celebrityUrl):\n",
    "    ddbPutItemResponse = dynamodb.put_item(\n",
    "        Item={\n",
    "            'id': {'S': celebrityId},\n",
    "            'name': {'S': celebrityName},\n",
    "            'url': { 'S': celebrityUrl},\n",
    "        },\n",
    "        TableName=ddbTableName,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Index first celebrity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Index Celebrity 1\n",
    "celebrityId = \"1\"\n",
    "celebrityName = \"Chris Munns\"\n",
    "celebrityUrl = \"http://www.amazon.com\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "addCelebrityToDynamoDB(celebrityId, celebrityName, celebrityUrl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Index face 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(IImage(url=s3.generate_presigned_url('get_object', Params={'Bucket': bucketName, 'Key': \"media/custom-celebrity-recognition/chris01.png\"})))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# After you run this cell the biggest face from the image will be indexed.\n",
    "# You will get a JSON response with a variety of information, but notice FaceId, ImageId and ExternalImageId\n",
    "# Later when we will search celebrities, we will use this ExteralImageId to extract metadata from DynamoDB.\n",
    "\n",
    "indexFace(bucketName, \"media/custom-celebrity-recognition/chris01.png\", celebrityId)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Index face 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(IImage(url=s3.generate_presigned_url('get_object', Params={'Bucket': bucketName, 'Key': \"media/custom-celebrity-recognition/chris02.png\"})))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "indexFace(bucketName, \"media/custom-celebrity-recognition/chris02.png\", celebrityId)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Index face 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(IImage(url=s3.generate_presigned_url('get_object', Params={'Bucket': bucketName, 'Key': \"media/custom-celebrity-recognition/chris03.png\"})))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "indexFace(bucketName, \"media/custom-celebrity-recognition/chris03.png\", celebrityId)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Describe Rekognition Collection\n",
    "# With three faces indexed for celebrity 1, you shoud now see FaceCount: 3\n",
    "\n",
    "describeCollectionResponse = rekognition.describe_collection(\n",
    "    CollectionId=collectionId\n",
    ")\n",
    "display(\"FaceCount: {0}\".format(describeCollectionResponse[\"FaceCount\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Index second celebrity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Index Celebrity 2\n",
    "celebrityId = \"2\"\n",
    "celebrityName = \"Kashif Imran\"\n",
    "celebrityUrl = \"http://aws.amazon.com\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "addCelebrityToDynamoDB(celebrityId, celebrityName, celebrityUrl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Index face 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(IImage(url=s3.generate_presigned_url('get_object', Params={'Bucket': bucketName, 'Key': \"media/custom-celebrity-recognition/kashif01.jpg\"})))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "indexFace(bucketName, \"media/custom-celebrity-recognition/kashif01.jpg\", celebrityId)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Index face 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(IImage(url=s3.generate_presigned_url('get_object', Params={'Bucket': bucketName, 'Key': \"media/custom-celebrity-recognition/kashif02.jpg\"})))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "indexFace(bucketName, \"media/custom-celebrity-recognition/kashif02.jpg\", celebrityId)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Index face 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(IImage(url=s3.generate_presigned_url('get_object', Params={'Bucket': bucketName, 'Key': \"media/custom-celebrity-recognition/kashif03.jpg\"})))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "indexFace(bucketName, \"media/custom-celebrity-recognition/kashif03.jpg\", celebrityId)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Describe Rekognition Collection\n",
    "# You should now have FaceCount: 6 since we have indexed 3 faces for each of the 2 celebrities we indexed.\n",
    "describeCollectionResponse = rekognition.describe_collection(\n",
    "    CollectionId=collectionId\n",
    ")\n",
    "display(\"FaceCount: {0}\".format(describeCollectionResponse[\"FaceCount\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recognize custom celebrities in image\n",
    "***\n",
    "Now let us try the image with custom celebrities and see if we can recognize people in that image.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "imageName = \"media/custom-celebrity-recognition/serverless-bytes.png\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(IImage(url=s3.generate_presigned_url('get_object', Params={'Bucket': bucketName, 'Key': imageName})))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Call Rekognition to recognize custom celebrity in image by using face search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "searchFacesResponse = rekognition.search_faces_by_image(\n",
    "    CollectionId=collectionId,\n",
    "    Image={\n",
    "        'S3Object': {\n",
    "            'Bucket': bucketName,\n",
    "            'Name': imageName,\n",
    "        }\n",
    "    },\n",
    "    MaxFaces=2,\n",
    "    FaceMatchThreshold=95\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Review raw JSON response of search face by image API call"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You will see Rekognition response with SearchedFaceBoundingBox (which contains information about the biggest face\n",
    "# in the image). Rekognition also returns FaceMatches, a list of matched faces. Each matched face has additional\n",
    "# information including FaceId, ImageId and ExternalImageId. We will use ExternalImageId to extract information\n",
    "# from DynamoDB about this celebrity.\n",
    "\n",
    "display(searchFacesResponse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display image with bounding box around recognized custom celebrity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define functions to show image and bounded boxes around recognized celebrities\n",
    "  \n",
    "def displayWithBoundingBoxes (sourceImage, boxes):\n",
    "    # blue, green, red, grey\n",
    "    colors = ((220,220,220),(230,230,230),(76,182,252),(52,194,123))\n",
    "    \n",
    "    # Download image locally\n",
    "    imageLocation = tempFolder+os.path.basename(sourceImage)\n",
    "\n",
    "    s3.download_file(bucketName, sourceImage, imageLocation)\n",
    "\n",
    "    # Draws BB on Image\n",
    "    bbImage = Image.open(imageLocation)\n",
    "    draw = ImageDraw.Draw(bbImage)\n",
    "    width, height = bbImage.size\n",
    "    col = 0\n",
    "    maxcol = len(colors)\n",
    "    line= 3\n",
    "    for box in boxes:\n",
    "        x1 = int(box[1]['Left'] * width)\n",
    "        y1 = int(box[1]['Top'] * height)\n",
    "        x2 = int(box[1]['Left'] * width + box[1]['Width'] * width)\n",
    "        y2 = int(box[1]['Top'] * height + box[1]['Height']  * height)\n",
    "        \n",
    "        draw.text((x1,y1),box[0],colors[col])\n",
    "        for l in range(line):\n",
    "            draw.rectangle((x1-l,y1-l,x2+l,y2+l),outline=colors[col])\n",
    "        col = (col+1)%maxcol\n",
    "    \n",
    "    imageFormat = \"PNG\"\n",
    "    ext = sourceImage.lower()\n",
    "    if(ext.endswith('jpg') or ext.endswith('jpeg')):\n",
    "       imageFormat = 'JPEG'\n",
    "\n",
    "    bbImage.save(imageLocation,format=imageFormat)\n",
    "\n",
    "    display(bbImage)\n",
    "    \n",
    "def getDynamoDBItem(itemId):\n",
    "    ddbGetItemResponse = dynamodb.get_item(\n",
    "        Key={'id': {'S': itemId} },\n",
    "        TableName=ddbTableName\n",
    "    )\n",
    "    \n",
    "    itemToReturn = ('', '', '')\n",
    "    \n",
    "    if('Item' in ddbGetItemResponse):\n",
    "        itemToReturn = (ddbGetItemResponse['Item']['id']['S'], \n",
    "                ddbGetItemResponse['Item']['name']['S'],\n",
    "                ddbGetItemResponse['Item']['url']['S'])\n",
    "    \n",
    "    return itemToReturn\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# After your run this cell you should see one of the faces recognized using Amazon Rekognition.\n",
    "# You only see one face recognized in this example because\n",
    "# SearchFacesByImage, for a given input image, first detects the largest face in the image,\n",
    "# and then searches the specified collection for matching faces.\n",
    "\n",
    "# In next section we will use DetectFaces API call to first detect faces in the image and then\n",
    "# use SearchFacesByImage for each detected face to get it recognized.\n",
    "\n",
    "def displaySearchedFace(sfr):  \n",
    "\n",
    "    boxes = []\n",
    "    \n",
    "    if(len(sfr['FaceMatches']) > 0):\n",
    "        bb = sfbb = sfr['SearchedFaceBoundingBox']\n",
    "        eid = sfr['FaceMatches'][0]['Face']['ExternalImageId']\n",
    "        conf = sfr['FaceMatches'][0]['Similarity']\n",
    "\n",
    "        celeb = getDynamoDBItem(eid)\n",
    "\n",
    "        boxes.append((\"{0}-{1}-{2}%\".format(celeb[0], celeb[1], round(conf,2)), bb))\n",
    "\n",
    "        displayWithBoundingBoxes(imageName, boxes)\n",
    "\n",
    "displaySearchedFace(searchFacesResponse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recognize all custom celebrities in image\n",
    "***\n",
    "Now let us try an image with custom celebrities and see if we can recognize all people in that image. To recognize all faces in the image, we will first call detect faces and then for each face using face search API to recognize each face in the image. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "imageName = \"media/custom-celebrity-recognition/serverless-bytes.png\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define helper functions to detect faces, crop faces in the main image, and then recognize each face"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def detectFaces():\n",
    "    detectFacesResponse = rekognition.detect_faces(\n",
    "        Image={\n",
    "            'S3Object': {\n",
    "                'Bucket': bucketName,\n",
    "                'Name': imageName\n",
    "                }\n",
    "            },\n",
    "        Attributes=['DEFAULT'])\n",
    "    return detectFacesResponse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getFaceCrop(imageBinary, box, image_width, image_height):\n",
    "    \n",
    "    x1 = int(box['Left'] * image_width)-25\n",
    "    y1 = int(box['Top'] * image_height)-25\n",
    "    x2 = int(box['Left'] * image_width + box['Width'] * image_width)+25\n",
    "    y2 = int(box['Top'] * image_height + box['Height']  * image_height)+25\n",
    "    if x1 < 0 : x1=0\n",
    "    if y1 < 0 : y1=0\n",
    "    if x2 < 0 : x2=image_width\n",
    "    if y2 < 0 : y2=image_height \n",
    "           \n",
    "    coordinates = (x1,y1,x2,y2)\n",
    "\n",
    "    image_crop = imageBinary.crop(coordinates)\n",
    "    stream2 = BytesIO() \n",
    "    \n",
    "    iformat = \"JPEG\"\n",
    "    if(imageName.lower().endswith(\"png\")):\n",
    "        iformat = \"PNG\"\n",
    "    \n",
    "    image_crop.save(stream2,format=iformat)\n",
    "    image_region_binary = stream2.getvalue()    \n",
    "    stream2.close()\n",
    "    \n",
    "    return image_region_binary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def recognizeFace(faceCrop):\n",
    "    searchFacesResponse = rekognition.search_faces_by_image(\n",
    "        CollectionId=collectionId,\n",
    "        Image={\n",
    "            'Bytes': faceCrop\n",
    "            },\n",
    "        MaxFaces=2,\n",
    "        FaceMatchThreshold=95\n",
    "    )\n",
    "\n",
    "    if(len(searchFacesResponse['FaceMatches']) > 0):\n",
    "        eid = searchFacesResponse['FaceMatches'][0]['Face']['ExternalImageId']\n",
    "        conf = searchFacesResponse['FaceMatches'][0]['Similarity']\n",
    "        celeb = getDynamoDBItem(eid)\n",
    "\n",
    "        return \"{0}-{1}-{2}%\".format(celeb[0], celeb[1], round(conf,2))\n",
    "    else:\n",
    "        return \"\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def recognizeAllCustomCelebrities():\n",
    "    detectedFaces = detectFaces()\n",
    "    \n",
    "    # Download image locally\n",
    "    imageLocation = tempFolder+os.path.basename(imageName)\n",
    "    s3.download_file(bucketName, imageName, imageLocation)\n",
    "\n",
    "    imageBinary = Image.open(imageLocation)\n",
    "    width, height = imageBinary.size \n",
    "    \n",
    "    boxes = []\n",
    "    for detectedFace in detectedFaces['FaceDetails']:\n",
    "        faceCrop = getFaceCrop(imageBinary, detectedFace['BoundingBox'], width, height)\n",
    "        recognizedFace = recognizeFace(faceCrop)\n",
    "        if(recognizedFace):\n",
    "            boxes.append((recognizedFace, detectedFace['BoundingBox']))\n",
    "        else:\n",
    "            boxes.append((\"Unrecognized Face\", detectedFace['BoundingBox']))\n",
    "    displayWithBoundingBoxes(imageName, boxes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "recognizeAllCustomCelebrities()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recognize custom celebrities in video\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "videoName = \"media/custom-celebrity-recognition/serverless-bytes.mov\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Start face search job to find faces in the video that match faces in our Rekognition collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "startFaceSearchResponse = rekognition.start_face_search(\n",
    "    Video={\n",
    "        'S3Object': {\n",
    "            'Bucket': bucketName,\n",
    "            'Name': videoName\n",
    "        }\n",
    "    },\n",
    "    FaceMatchThreshold=90,\n",
    "    CollectionId=collectionId,\n",
    ")\n",
    "\n",
    "\n",
    "faceSearchJobId = startFaceSearchResponse['JobId']\n",
    "display(\"Job ID: {0}\".format(faceSearchJobId))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Wait until the face search job is complete"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "getFaceSearch = rekognition.get_face_search(\n",
    "    JobId=faceSearchJobId,\n",
    "    SortBy='TIMESTAMP'\n",
    ")\n",
    "\n",
    "while(getFaceSearch['JobStatus'] == 'IN_PROGRESS'):\n",
    "    time.sleep(5)\n",
    "    print('.', end='')\n",
    " \n",
    "    getFaceSearch = rekognition.get_face_search(\n",
    "    JobId=faceSearchJobId,\n",
    "    SortBy='TIMESTAMP'\n",
    ")\n",
    "    \n",
    "display(getFaceSearch['JobStatus'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Review raw JSON response from Rekognition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(getFaceSearch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Show recognized custom celebrities in the video"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "theCelebs = {}\n",
    "\n",
    "# Display timestamps and celebrites detected at that time\n",
    "strDetail = \"Celebrites detected in vidoe<br>=======================================<br>\"\n",
    "strOverall = \"Celebrities in the overall video:<br>=======================================<br>\"\n",
    "\n",
    "# Faces detected in each frame\n",
    "for person in getFaceSearch['Persons']:\n",
    "    if('FaceMatches' in person and len(person[\"FaceMatches\"])> 0):\n",
    "        ts = person[\"Timestamp\"]\n",
    "        theFaceMatches = {}\n",
    "        for fm in person[\"FaceMatches\"]:\n",
    "            conf = fm[\"Similarity\"]\n",
    "            eid =  fm[\"Face\"][\"ExternalImageId\"]\n",
    "            if(eid not in theFaceMatches):\n",
    "                theFaceMatches[eid] = (eid, ts, round(conf,2))\n",
    "            if(eid not in theCelebs):\n",
    "                theCelebs[eid] = (getDynamoDBItem(eid))\n",
    "        for theFaceMatch in theFaceMatches:\n",
    "            celeb = theCelebs[theFaceMatch]\n",
    "            fminfo = theFaceMatches[theFaceMatch]\n",
    "            strDetail = strDetail + \"At {0} ms<br> {2} (ID:{1}) Conf: {4}%<br>\".format(fminfo[1],\n",
    "                       celeb[0], celeb[1], celeb[2], fminfo[2])\n",
    "\n",
    "# Unique faces detected in video\n",
    "for theCeleb in theCelebs:\n",
    "    tc = theCelebs[theCeleb]\n",
    "    strOverall = strOverall + \"{1} (ID: {0})<br>\".format(tc[0], tc[1], tc[2])\n",
    "\n",
    "# Display results\n",
    "display(HTML(strOverall))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display video in player"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display video in player\n",
    "\n",
    "s3VideoUrl = s3.generate_presigned_url('get_object', Params={'Bucket': bucketName, 'Key': videoName})\n",
    "\n",
    "videoTag = \"<video controls='controls' autoplay width='640' height='360' name='Video' src='{0}'></video>\".format(s3VideoUrl)\n",
    "\n",
    "videoui = \"<table><tr><td style='vertical-align: top'>{}</td><td>{}</td></tr></table>\".format(videoTag, strDetail)\n",
    "\n",
    "display(HTML(videoui))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Index additional faces of known celebrities to improve recognition of these celebrities\n",
    "You can further improve the performance of your solution by indexing faces of celebrities that the Rekognition celebrity API can already recognize for most of your media, but might not perfom as well in certain situations. Below we are indexing a few images of Jeremy Clarkson and Richard Hammond even though they are recognized well by Rekognition's celebrity API. We are using same ID for them that Rekognition Celebrity API returns, so we can detect when both Celebrity API and FaceAPI recognize same celebrity in a frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Index Celebrity 3\n",
    "celebrityId = \"2mW0ey5n\"\n",
    "celebrityName = \"Jeremy Clarkson\"\n",
    "celebrityUrl = \"https://www.imdb.com/name/nm0165087/\"\n",
    "\n",
    "addCelebrityToDynamoDB(celebrityId, celebrityName, celebrityUrl)\n",
    "\n",
    "indexFace(bucketName, \"media/celebrity-recognition/jc04.png\", celebrityId)\n",
    "indexFace(bucketName, \"media/celebrity-recognition/jc05.png\", celebrityId)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Index Celebrity 4\n",
    "celebrityId = \"4TK3NJ\"\n",
    "celebrityName = \"Richard Hammond\"\n",
    "celebrityUrl = \"https://www.imdb.com/name/nm1414369/\"\n",
    "\n",
    "addCelebrityToDynamoDB(celebrityId, celebrityName, celebrityUrl)\n",
    "\n",
    "indexFace(bucketName, \"media/celebrity-recognition/rh01.png\", celebrityId)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}