notebooks/ct-domain-demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Detecting Phishing Domains with UniSim\n",
    "\n",
    "This colab showcases how to use UniSim near-duplicate text embeddings to find look-a-like phishing domains.\n",
    "\n",
    "This technique can be used to monitor Certificate Transparency logs to find look-a-like phishing domains in near-realtime. If you are interested in an end-to-end implementation of this idea, you can find a complete server side / client side implementation at [CThunter GitHub repository](https://github.com/ebursztein/cthunter).\n",
    "\n",
    "For this colab, we are going to use a static list of domains and fake domains to demonstrate the underlying technical concepts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging, os\n",
    "logging.getLogger('tensorflow').disabled = True\n",
    "os.environ[\"TF_CPP_MIN_LOG_LEVEL\"] = \"3\" # ignore TensorFlow debugging info\n",
    "\n",
    "from time import time\n",
    "import json\n",
    "from unisim import TextSim\n",
    "from tabulate import tabulate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Data\n",
    "\n",
    "First, we are loading made-up data that simulates our use-case (finding if the domains present in the logs look like the domains we are monitoring).\n",
    "\n",
    "The `domains.json` file contains two lists:\n",
    "\n",
    "- **domains**: this is a list of domains that are monitored\n",
    "- **logs**: those are the made-up domains seen in the logs\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Some Domains: ['google', 'facebook', 'twitter', 'linkedin', 'instagram']\n",
      "Some logs: ['gooolgle', 'g00gle', 'g𝙤ogl', 'googglee', 'google']\n"
     ]
    }
   ],
   "source": [
    "data = json.load(open('./data/domains.json'))\n",
    "domains = data['domains']\n",
    "logs = data['logs']\n",
    "print('Some Domains:', domains[:5])\n",
    "print(\"Some logs:\", logs[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How Unisim Embeddings Work\n",
    "\n",
    "UniSim text embeddings leverage the RETSim model (a tiny transformer model trained with metric learning) to embed texts into vector representations that can be compared using cosine similarity.\n",
    "\n",
    "Let's demonstrate how this works in practice so we get a sense of the value returned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/keras/initializers/initializers.py:120: UserWarning: The initializer RandomNormal is unseeded and being called multiple times, which will return identical values each time (even if the initializer is unseeded). Please update your code to provide a seed to the initializer, or avoid using the same initalizer instance more than once.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Similarity between Google and g𝙤ogle is 0.8847256302833557\n",
      "Similarity between Google and googlee is 0.894428014755249\n",
      "Similarity between Google and loogle is 0.7688487768173218\n",
      "Similarity between Google and thisisnotamatch is 0.4283703863620758\n"
     ]
    }
   ],
   "source": [
    "ts = TextSim()  # init UniSim text similarity\n",
    "\n",
    "test_domain = \"Google\"\n",
    "test_logs = [\"g𝙤ogle\", 'googlee', 'loogle', \"thisisnotamatch\"]\n",
    "for log in test_logs:\n",
    "    s = ts.similarity(test_domain, log)\n",
    "    print(f\"Similarity between {test_domain} and {log} is {s}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As visible in the example above, the father the domains, the lower is the similarity so we need a threshold. 0.85 is usually a good starting point until you calibrate the value based on your own data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing and Searching\n",
    "\n",
    "A key issue with traditional text similarity algorithms based on edit distance is that they have are $O(N^2)$ in complexity, which makes them prohibitively expensive to compute at scale.\n",
    "\n",
    "However, like other text embeddings, UniSim transforms strings into vector representations and we can take advantage of this in multiple ways to speed up computation:\n",
    "\n",
    "1. We can pre-compute the domain embeddings and only compute the log domains' embeddings as they arrive, saving computation time.\n",
    "2. We can leverage GPU to do batch computation to compute domain embeddings.\n",
    "3. If the list of domains become very large, we can use an Approximate Nearest Neighboor (ANN) algorithm to do search in sub-linear time.\n",
    "\n",
    "Here our list of domains is quite small, so we are only going to use batching and use an exact SIMD accelerated search. To do so, we are going to perform three steps:\n",
    "\n",
    "1. Index our domains\n",
    "2. Create a batch of logs\n",
    "3. Look through the results to find matches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "match_threshold = 0.85  # 0.8-0.9 is typically a good similarity threshold for matching texts\n",
    "ts.reset_index()  # reset index in case the cell is ran multiple times\n",
    "idxs = ts.add(domains)  # index the domains we want to monitor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Search took 0.12s\n",
      "Log        Match       Similarity\n",
      "---------  --------  ------------\n",
      "gooolgle   google        0.88643\n",
      "g𝙤ogl      google        0.875504\n",
      "googglee   google        0.936634\n",
      "google     google        1\n",
      "goolgle    google        0.887561\n",
      "thegoogle  google        0.901916\n",
      "go0gl3     google        0.878335\n",
      "bookface   facebook      0.952432\n",
      "faceb0ok   facebook      0.965319\n",
      "twiter     twitter       0.954167\n",
      "twittter   twitter       0.989152\n",
      "twiiter    twitter       0.935445\n",
      "Eebay      ebay          0.917119\n",
      "ebayy      ebay          0.951802\n",
      "eb4y       ebay          0.899173\n",
      "amazoon    amazon        0.979939\n",
      "ammazonn   amazon        0.898798\n"
     ]
    }
   ],
   "source": [
    "start = time()\n",
    "results = ts.search(logs, k=1, similarity_threshold=match_threshold)  # match the logs against the dataset\n",
    "print(f\"Search took {time() - start:.2f}s\")\n",
    "\n",
    "# get matching matching results\n",
    "matches = []\n",
    "not_matches = []   # just to show what it looks like\n",
    "for result in results.results:\n",
    "    log_match = result.query_data\n",
    "    for match in result.matches:\n",
    "        if match.is_match:\n",
    "            matches.append((log_match, match.data, match.similarity))\n",
    "        else:\n",
    "            not_matches.append((log_match, match.data, match.similarity))\n",
    "print(tabulate(matches, headers=['Log', 'Match', 'Similarity']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Log           Match        Similarity\n",
      "------------  ---------  ------------\n",
      "g00gle        google         0.787034\n",
      "goolge        google         0.741084\n",
      "goog-le.net   google         0.84237\n",
      "G00gleLogin   google         0.701801\n",
      "faecbook      facebook       0.849864\n",
      "faecbook      facebook       0.849864\n",
      "f@ceb00k.bl   facebook       0.689276\n",
      "faecbo-ok     facebook       0.798683\n",
      "myfaecbook    facebook       0.739695\n",
      "ficebo0k.com  facebook       0.781612\n",
      "amazin        amazon         0.721841\n",
      "theamazon     amazon         0.839458\n",
      "myamazin      amazon         0.609729\n",
      "tw1tt3r       twitter        0.74809\n",
      "3b4y          ebay           0.791277\n",
      "paypall       apple          0.607747\n",
      "palpay        apple          0.529317\n",
      "p4yoal        youtube        0.605841\n",
      "p4ypal        apple          0.620593\n",
      "p4ypa1        pinterest      0.572656\n",
      "p4yp4l        pinterest      0.574713\n",
      "Amazz0n       amazon         0.730577\n",
      "amaz00n       amazon         0.791594\n",
      "amz𝙤n         amazon         0.822388\n"
     ]
    }
   ],
   "source": [
    "# let's display some non-matches to show what it looks like\n",
    "print(tabulate(not_matches, headers=['Log', 'Match', 'Similarity']))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.6 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "767d51c1340bd893661ea55ea3124f6de3c7a262a8b4abca0554b478b1e2ff90"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}