examples/ballots.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "21d29912-986f-41da-8c3a-b903c13414cb",
   "metadata": {},
   "source": [
    "# Auditing electronic voting\n",
    "\n",
    "The goal is to construct a confidence interval for the proportion of people voting for each candidate using a small number of hand-labeled ballots and a large number of ballots read with an optical scanner. On Election Day in the United States, most voters use electronic or optical-scan ballots, neither of which is perfectly accurate. The data is taken from a special election in San Francisco for the Assembly District 17 seat on April 19, 2022. The candidates were David Campos and Matt Haney. This is an example of a risk-limiting audit: a statistically valid way to check the results of an election by inspecting subsets of ballots (see, e.g., (1)).\n",
    "\n",
    "1. M. Lindeman and P. B. Stark, A gentle introduction to risk-limiting audits. IEEE Security & Privacy 10(5), 42–49 (2012)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d70aaddf-81bb-4364-ada6-a0af5d5bbcce",
   "metadata": {},
   "source": [
    "### Import necessary packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4f57ecd3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os, sys\n",
    "\n",
    "sys.path.append(os.path.abspath(os.path.join(os.getcwd(), os.pardir)))\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from ppi_py.datasets import load_dataset\n",
    "from ppi_py import ppi_mean_ci, classical_mean_ci\n",
    "from tqdm import tqdm\n",
    "from scipy.optimize import brentq\n",
    "from utils import *"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cf90ae6",
   "metadata": {},
   "source": [
    "### Import the ballots data set\n",
    "\n",
    "Load the data. The data set contains a small number of hand-labeled votes (```Y```), corresponding predicted votes based on computer vision (```Yhat```), and predicted votes for the remaining ballots (```Yhat_unlabeled```)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a6da3138",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset ballots not found at location ./data/; downloading now...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading...\n",
      "From: https://drive.google.com/uc?id=1DJvTWvPM6zQD0V4yGH1O7DL3kfnTE06u\n",
      "To: /Users/angelopoulos/Code/working/ppi_py/examples/data/ballots.npz\n",
      "100%|██████████| 642k/642k [00:00<00:00, 1.36MB/s]\n"
     ]
    }
   ],
   "source": [
    "dataset_folder = \"./data/\"\n",
    "data = load_dataset(dataset_folder, \"ballots\")\n",
    "Y = data[\"Y\"]\n",
    "Yhat = data[\"Yhat\"]\n",
    "Yhat_unlabeled = data[\"Yhat_unlabeled\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8969f9db",
   "metadata": {},
   "source": [
    "### Problem setup\n",
    "\n",
    "Specify the error level (```alpha```), range of values for the labeled data set size (```ns```), and number of trials (```num_trials```).\n",
    "\n",
    "Specify the ground-truth value of the estimand (reported election outcome)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5b3c8f29",
   "metadata": {},
   "outputs": [],
   "source": [
    "alpha = 0.05\n",
    "n_max = Y.shape[0]  # Total number of labeled ballots\n",
    "ns = np.linspace(100, n_max, 20).astype(\n",
    "    int\n",
    ")  # Test for different numbers of labeled ballots\n",
    "num_trials = 500\n",
    "\n",
    "# Reported result (official outcome)\n",
    "reported_estimate = 0.6237"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83ce18be",
   "metadata": {},
   "source": [
    "### Construct intervals\n",
    "\n",
    "Form confidence intervals for all methods and problem parameters. A dataframe with the following columns is formed:\n",
    "1. ```method``` (one of ```PPI```, ```Classical```, and ```Imputation```)\n",
    "2. ```n``` (labeled data set size, takes values in ```ns```)\n",
    "3. ```lower``` (lower endpoint of the confidence interval)\n",
    "4. ```upper``` (upper endpoint of the confidence interval)\n",
    "5. ```trial``` (index of trial, goes from ```0``` to ```num_trials-1```)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "812f8fd5",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [03:20<00:00, 10.04s/it]\n"
     ]
    }
   ],
   "source": [
    "# Run prediction-powered inference and classical inference for many values of n\n",
    "results = []\n",
    "for i in tqdm(range(ns.shape[0])):\n",
    "    for j in range(num_trials):\n",
    "        # Prediction-Powered Inference\n",
    "        n = ns[i]\n",
    "        rand_idx = np.random.permutation(n_max)\n",
    "        _Yhat = Yhat[rand_idx[:n]]\n",
    "        _Y = Y[rand_idx[:n]]\n",
    "\n",
    "        ppi_ci = ppi_mean_ci(_Y, _Yhat, Yhat_unlabeled, alpha=alpha)\n",
    "\n",
    "        # Classical interval\n",
    "        classical_ci = classical_mean_ci(_Y, alpha=alpha)\n",
    "\n",
    "        # Append results\n",
    "        results += [\n",
    "            pd.DataFrame(\n",
    "                [\n",
    "                    {\n",
    "                        \"method\": \"PPI\",\n",
    "                        \"n\": n,\n",
    "                        \"lower\": ppi_ci[0],\n",
    "                        \"upper\": ppi_ci[1],\n",
    "                        \"trial\": j,\n",
    "                    }\n",
    "                ]\n",
    "            )\n",
    "        ]\n",
    "        results += [\n",
    "            pd.DataFrame(\n",
    "                [\n",
    "                    {\n",
    "                        \"method\": \"Classical\",\n",
    "                        \"n\": n,\n",
    "                        \"lower\": classical_ci[0],\n",
    "                        \"upper\": classical_ci[1],\n",
    "                        \"trial\": j,\n",
    "                    }\n",
    "                ]\n",
    "            )\n",
    "        ]\n",
    "\n",
    "# Imputed CI\n",
    "imputed_ci = classical_mean_ci(Yhat_unlabeled, alpha=alpha)\n",
    "results += [\n",
    "    pd.DataFrame(\n",
    "        [\n",
    "            {\n",
    "                \"method\": \"Imputation\",\n",
    "                \"n\": np.nan,\n",
    "                \"lower\": imputed_ci[0],\n",
    "                \"upper\": imputed_ci[1],\n",
    "                \"trial\": 0,\n",
    "            }\n",
    "        ]\n",
    "    )\n",
    "]\n",
    "\n",
    "df = pd.concat(results, axis=0, ignore_index=True)\n",
    "df[\"width\"] = df[\"upper\"] - df[\"lower\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d15ba288",
   "metadata": {},
   "source": [
    "### Plot results\n",
    "\n",
    "Plot:\n",
    "1. Five randomly chosen intervals from the dataframe for PPI and the classical method, and the imputed interval;\n",
    "2. The average interval width for PPI and the classical method, together with a scatterplot of the widths from the five random draws."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6077b2c4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnsAAADMCAYAAAALZGVzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAwQElEQVR4nO3deZxcd3nn+89zaunqlroltVotS7JWW7EEWMYgGxOEbTDGZjFLICRx7uUCIR4mybxemWTmkiEhmYRh7uQO3CQMIcQMhJCE3OGaMQ4QbGe8EEvGMTI23iVrtWS11Ita3a1eajnnuX9Udala6lW9nK6q79uvcp8663Oq+3Q/+q3m7oiIiIhIbQriDkBERERE5o+SPREREZEapmRPREREpIYp2RMRERGpYUr2RERERGqYkj0RERGRGpacYrvGZZHFyuIOIAZ6HmUxqrtn8dZbb/V777037jBExjPu8zhVsicisujln/lBeTl15TtijETqQXd3d9whiMyIkj0RqXrhgUfLy0r2RETGUps9ERERkRqmkj0RqXrJ19wadwgiDPT3093ZSXZkhIZMhrb2dppbWuIOS0TJnohUv+TWN8UdgtS5gf5+jh89igPpVIp8Ps/xo0e5dONGJXwSO1XjioiIzFJ3ZyfZbJbOjg46XnkFMyORTNLd2Rl3aCJK9kRERGYrOzJCIZ8HIAxDBs+eJZlMkh0ZiTkyESV7IiIis9aQyZBMpcrvhwYHKRQKNGQyMUYlUqQ2eyJS9fJP3lNeTl393hgjkXrV1t5OX29v+X0um2Xo7Fkuu+KKGKMSKVKyJyJVLzyyt7ysZE/iErljZrgXJ7sZURWuLBKqxhUREZmlE8eOkc/lSKXT5XW5bJYTx47FGJVIkUr2RKTqJV/7nrhDkDrX39dHkEiQSCTIZbMARFFE35kz8QYmgpI9EakByc3XxB2CCAYkEgmCICCKIqDYM1ckbqrGFRERmaXmZcsIw5AoDMf0yg0C/ZmV+OmnUEREZJbWrV9PY2MjbjYmwcvncuUOGyJxUbInIiIyS80tLWzeupWVbW0sW76cRCIBQKFQULs9iZ3a7IlI1cv9+Fvl5fQ1H4oxEqlnzS0t5Xlw9z3/PB3HjwPQ09nJ8hUr4gxN6pxK9kSk6kXHnym/RBaDtlWrysvdXV2qypVYKdkTERGZY8tbWwlKVbnDQ0MMDQ7GHJHUM1XjikgsHj/9L3N3suvfcW55gvNe2/qGubueyBQSiQStK1fS3dkJFEv3lixdGnNUUq+U7IlILFrTraxqaF+w642EI2QSmpReFk5be3s52evp7GTj5s0xRyT1StW4IiIi82BlW1t5ub+vrzyzhshCU7InIiIyD1LpNMsqeuF2d3XFGI3UMyV7IiIi86SyV26Pkj2JiZI9EYmF6deP1IG29nPtUnt7eggLhRijkXqlDhoiEovLll42Z+caufvT5eXM+z8zZ+cVma3GpiaalixhaHCQKIo4ffo0q9oXrmOSCKhkT0REZF5Vlu4dOXCAF599lsMHDjDQ3x9jVFJPVLInIlUvdd0vxx2CyITaVq3i5cOHgeIAy61tbeTzeY4fPcqlGzeWp1gTmS9K9kQkFj/sfnjuTpaqWO4+Oe4uN7TdOHfXE5mB5mXLSCQShGFIFEV0nTpFQ0MD6YYGujs7lezJvFOyJyKxWJVuZ01mzYJdb7gwTGOyccGuJzLKzEimUoRhCEAURYRRxODZs0RRFHN0Ug/UZk9ERGSeJUrz5AIU8nkSQQBmFPL5GKOSeqFkT0REZJ41ZM5N1ReNluqFIYmkKthk/inZExERmWeZTIYgOPcnd7T9XkNDQ4xRSb1QsicisQjM4g5BaoyZBWb2ZTP7kZk9bGaXj7NPk5ntMbNtFeueLO3/sJn91XzE5hSnTyu/dy+vF5lvKj8WkVhsa94+Z+fSoMpS8j4g4+5vNLPrgM8D7x3daGY7gS8Dl1asywC4+43zGpk7y1tbOd3VRb6ynZ4r3ZP5p2RPRKpeatfH4g5BFoddwL0A7v5YKbmr1AC8H/ibinVXAU1mdj/Fv4mfcvfH5jqwhkyGfD7PqtWrOXH8OAC5bFYle7IglOyJSNVLrNocdwiyOLQAfRXvQzNLunsBwN33QHEolApDwOeA/w5sBX5gZleMHjPKzO4A7gDYsGHDjANra2/n+NGjJJJJlixdyuDZswAMDw7O+FwiM6U2eyIiUiv6geaK98H5Sds49gN/60X7gR7gggEg3f1Od9/p7jtXrVo148CaW1q4dONGUqkUjU1N5wLu66PvzJkZn09kJlSyJyKx+N7J7y7o9d59yW0Lej2JxR7gNuBbpTZ7z0zjmI8BVwK/ZmZrKZYOdsxHcM0tLeXZMl545hlOdRQvc+TgQa56/evn45IigJI9EYnJmsxaNjTOvDrsYg0WBlmSXLJg15NY3A3cbGaPAgZ81MxuB5a6+50THPNV4Otmtpti59iPTaM0cNY2btlSTvZ6e3roO3OGZcuXz/dlpU4p2RMRkZrg7hHwifNWvzjOfjdWLOeA2+c3sgs1LVnC6jVrVLonC0Jt9kRERGKw8bLLoNRZpLenh77e3pgjklqlZE9ERCQGTU1NrF5zri/IkUOHYoxGapmSPRGJRaBfPyJs3LJFpXsy79RmT0RicfXyq+MOQSR2o6V7p06cAOCFZ59l+YoVNGQytLW3l3vvisyG/mktIiISo41btpSXR4aHcXfy+TzHjx5loL8/xsikVijZExERiVFTUxNLli4tv+/q7ORsfz9RFNHd2RljZFIrVI0rIrH44sH/tqDX+43L/s2CXk9kJlLpdHk5CkMGz54lm80SRVGMUUmtULInIrG4fOnlvKr51Qt2vTO5MyxPL1+w64nMRCGfx8xwdwDCMMTdyY6MxByZ1AJV44qIiMSsUChgwdg/yVEUkc/nY4pIaomSPRERkZglEgka0mlSqdSY9WFh3mdukzqgZE9ERCRmzcuWgRmZTIZMJlNeXygU6Dp1KsbIpBYo2RORWCRIxB2CyKKxbv16GhsbcTOCRIJE8lyT+hefe47hoaEYo5NqZ6ONQScw6UaRGFncAcRAz6MsRnX3LO7cudP37t075+cd6O+nu7OT7MgIqVSKUydPkstmgWLJX2tbGyePH6dQKJBMJlmzfj2bKsboE2GC51G9cUVERBaB5paWMTNmrLrkEp58/HHcnYG+Pgb6+kgkkwSJBIUw5OXSXLpK+GQqqsYVERFZhFqWLWPz1q1j1gVmBGYkEwksCOg4diym6KSaqGRPRGLxqed+Z+5OVsieW042jLvLf371f5m764kskPUbN3Lm9GlOd3cDkM/nSafTmBlBEFBQb12ZBiV7IhKL17RcyTUrrl2w6/Vke1jZsHLBricyF8yMba95DY8+/HB5XT6fJ5VKEUURyaT+jMvUVI0rIiKyiKXTaVavXVt+7+7k8nmiMGTN+vUxRibVQsmeiIjIIrf9Na9h+cqKkml3Mo2NbNi0KbaYpHoo2RMREakCV73udWys6Hk7PDTEc089RRRFMUYl1UDJnojEImlqayQyE2bGpssuY8PmzeV1Pd3dPPfTnyrhk0lpUGWpVnU3kCt6HmVxqrtncb4GVZ4ud+fwgQO8fPhwed3KVat49VVXEQQqw6lzGlRZRESk2pkZmy+/HKCc8PV0dfGTxx8nMCOfy5FuaGDt+vWsXrMmzlBlkVCyJyJVb+S7nykvZ277dIyRiCyM0YTP3Tl25AgAZ/v7y+Pv5QsFDu7bB6CET9RmT0RqQCF37iVSJ8yMLVu3sr6iR667E7ljQBiGvFxKBKW+KdkTERGpUqMJn9m5ploeRcWZNczIDg/HGJ0sFqrGFZGq1/Du34s7BJkHZvarwG8CjRQbnru7b5n0oDpkZiQSCcIowku9ct2dsFAgkUgAMNDfT3dnJ9mRERoyGdra22luaYkzbFlASvZEpOpZavz5cKXqfQJ4J3Ay7kAWu2UrVtB7+jSWSBCFYXl9GIa88Mwz5HI5CrkcYRSRGBhgoL+fzZdfroSvTqgaV0REFqtudz/q7tnRV9wBLVabLruMpUuXkkwkCEqleaNOdXTQ19tL6E4ylcLNGB4a4sSxYzFFKwtNJXsiIrKomNl/Li2mzew+4CeUxpl090/FFtgi1tzSwtbt28tVtclUioG+PvrOnAEgiiKGBwdpWrKEZDKJJxL09/XFG7QsGCV7IiKy2Ow77+uoSQcWN7MA+BJwFZAFPu7uB87bpwn4J+BX3P3F6RxTLZpbWsZUy7o7Rw8f5siBA+X3g2fPkkgkSKVSJJJKAeqFvtMiUvVG7j43tl7m/Z+ZZE+pBu7+1wBm9kV3/43R9Wb2DeAbkxz6PiDj7m80s+uAzwPvrTh+J/Bl4NLpHlPNzIxNW7bQ3dXF2YpSvDAMCcOQlhUrYoxOFpLa7ImIyKJiZr9uZh3Ar5rZidKrA1g3xaG7gHsB3P0xYOd52xuA9wMvzuCYqte8dCmpdHrM8CwA/b29HNy/n7CiQ4fUJiV7IiKyqLj7n7v7GuAP3X1t6bXG3W+a4tAWoLIhWmhm5Rosd9/j7uf3Spj0mFFmdoeZ7TWzvV1dXTO8o3i5Oyvb2mhuaSGVTo/ZduzIEX786KP09vTEFJ0sBFXjikjVU9VtbTGzD5cWj1csA+Duk1Xj9gPNFe8Ddy9McblpHePudwJ3AuzcuXPStoOLTUMmQz6fZ8XKlawA8rkc3V1d5LLFzs0jw8P89IknCIJi+U8qlWLN+vVs2qIhDWuFSvZERGSx2V56fQT4BWA98HOl5cnsoTguH6X2d89M41oXc0xVaWtvJywUyOfzuDuYsXzFCjZs3jymk0YURURRRC6f5+jBgxw5dCjGqGUuqWRPREQWFXf/DwBmdq+7v2t0vZndP8WhdwM3m9mjFGfc+KiZ3Q4sLZXMTeuYWd/AItPc0sKlGzeOmUFjzbp1NLe0sG7DBv7lkUeISjNvAOVZOI4fOcLGzZsvaOsn1UfJnoiILFbtZrbc3c+Y2Upg5WQ7u3tEcdaNSi+Os9+NUxxTc84flmVUQ0NDcbq1ZJKwMLb2ulAo8MRjj7H58stpbWtT0lfFlOyJ1IDuzk7y+XzcYUgNSqVStLW3x3X5zwJ7zayfYkeKj8UVSC1LJpMUwpB0Ol0elmXU2YEBnnnySRoyGaIwJIoiksmk2vRVGSV7IjUgkUzSvGxZ3GFIjSoUCiRjGIDX3b8NfNvM2oEed9cYIfNgzfr1vHzoECEQJBI4EIUhZlZs4wdkR0aA4th9+UKBowcPAijhqxJK9kREZFEZHUzZzJ6gOKvF6Hrc/WdjDK0mjSZsHceOUSgUSCWTrNm8mTXr1vHy4cO88vLL5X3dHS+V/B0/coQNmzaVe/HK4qVkT0REFpvRsXQ2A/cBTwA/AM7GFlGN27Rly7ildFu3baPj+HGccx03RhUKBR575BHWrV9PIQzpPHGiXAqsat7FRcmeiIgsKu5+qvS11cy2A++hOMZdJ8UZMGQBpVIpCmFIkEgQltrtjcplsxwuzb072tGjEIa8XBq2RQnf4qCyV5EakEgk4g5BZM6Z2VXAu4C3lla9EGM4dWvN+vV4FBG5k0gmCRKJYmJ33u8dd6eQzxOFIQ6cqKj+lXipZE+kBrSoc4bUpn8GDgG/6+7/GHcw9WrcNn3r17Nh0yY6T57kxWefHbO/u4M7uVyOfc8/z5q1a8GMnq6u8jh/be3t4w4FI/PDRnvaTKCqpoSRulKPAz7peZTFaN6exdIctbuAW4BrgU53/6X5ut507dy50/fu3Rt3GIvGow8/TKE0Rl90Xru+UUEQgBlGscdvYybD1le9Sgnf3Bv3eVQ1rohUPR/uL7+kpiwH1gEbgSbgaKzRyLjWbtgAgAUByVQKG6d3bhRFRKUx/Ar5PAMDAxx86aXy9oH+fg4fOMCLzz7L4QMHGOjXszyXVI0rIlUve+9/LS9n3v+ZSfaUKnMfxenMPuvuz8UdjIzv/GredCrFJZdeyorWVk6+8gonT5wYs/9ojeKZnh6efuIJmpct43RXF9lsljCKSAQBvT09bN2+XSV/c0TVuFKtVI1b4VRHB7lcbiFjWVzyI+eWU5n44qhB6XSa1WvWTLZL3T2LqsadmUceeKA8dMtkOYeZlat73Z0Vra1cefXVCxdobRj3eVTJnkgNSKZSLFuxIu4wpEbl83lSqVTcYUiVamxqYnhoiETpZygMQ6LwwslQ3P3cVG1m9Pb0MNDfz9LmZjpPnuTEsWPkslnSDQ2sXb9+qn+ESAUleyIiIjJv1m/axMH9+88N3xIEJJNJNmzahLtzcP/+Cw9yJ3LnicceI5FMEkURZlaeru2lffsAlPBNk5I9ERERmTejCdlEJXMnOzoYHhzEzIjGqeoNSz19R9eaFWsqDx84wKrVqwmCgFMdHSr5m4SSPREREZlXq9esmTD52rBpEwf37SuW+pkVkzp3ljQ3MzQ4SCGfH7P/aDI4MjzM7oceorGxkeHh4XISqJK/CynZE6kBmkFDRKrVZCV/7s7uhx4qt/E7v9QvCkMGz46dMnm8kr+B/n66OzvrdlBn9caValV3PQDR8zihaKC7vBw0t8UYSV2qu2dRvXEX1rNPPcXpnh6CIMDMyvPzBqVq38kkEgmWLF1aHNalUCCKIpLJJJmmJrZu21aLCZ9644pIbcr9rz8rL2ucPZHasnHLFnK5HCNDQ4RhSDKRINPczNZt20imUux99NFx2/pBsedvf1/fmHW5fJ5cXx/7nnuO7Tt20NTUxNHDh8vjBCZL08GNjh9YC5TsiYiIyKLV3NLC1m3bJqyGXbFyZbHkz4zAjDCKJi/5KyWFZwcG+PGePVgQ4KXevkEiQb5Q4OjBg8C5AaOrvRpYyZ5IDTh+9CjZbDbuMOJzxYfOLY83jINctIaGBi7duDHuMKTONbe0TJhcTVbyl0qleHzPnnKnj/FK/7yUELp7uecvwNGDBwlLJX2dJ0+Sy2bLQ8dU2wwfSvZEakAqnaZ11aq4w5AalcvlSKfTcYchMq6pSv4am5oYGR4mkUzipYQvDEOCICAIAvLn9fYd5e4cO3JkzLrR4WHy/f289OKL7Lj6apKpFEcOHVrU1cBK9kRERKSqTVbyt2HzZl4qDe0CgBmJVIqtV1xB+yWX8KMf/pBCoTDhOH+V3L1cDdx/5gy7H3qIRDJJWDo+CIJxq4HjTgaV7ImIiEjNmmpQ57UbNvDyoUNgRjKVIgxDPIpoW72axsZGjh05MmkCWB70uXK6N+DIgQMM9PWRz+cZ6OvDSiWJhTAsXg8WLOFTsiciIiI1bbJBnUcTrtGSt9R5JW893d0MDw6SSCTK1cCj07f5BO0AR/V0dZWXvXJOYDNePnSIZCJB45IlnO7poaujo9jmcB5K/jTOnlSruhvbi0mex56uLhqXLFnIWKSOJJPJydrs1d2zqHH26supjo7yDB+4Q6nX72VXXMGq1auL1cBhSFCR/E2RW02sNCA07qxctYpLN2ygsamJjhMnOHn8+HSqgTXOnkitWlnnnTNG7v50eVnj7InIXJqqGnjdxo3lauBEIkEURURhyJpLL2XZ8uXsf/55wijCuHAGkAtUbO/p6hpTMgiU2wTOtBpYyZ6IiIjIJGZSDZxMJlmzaVN5/fDwMC8fOoSVZgCJogiPIpa3tpLJZOh45ZVpxxFFUXF6zCCg49gxJXsiUj9s+dq4QxCROrZpy5YJE6+p2gSe7u6mEIYkguDc0DClksClLS30nzkz5nxBEIAZhYoxAaeiZE9Eql7DW/513CGIiExosmRwzfr1vHzoECHFRM7dMWBD6Zgf/fCH5AsFEqWSQTOjUOrIMV1K9kRqwL7nnmNoaCjuMKQGNTU1ccWrXx13GCI1a6pq4NFkMHIvTgdXGhpmzaZN076Gkj2RGpDOZFizfn3cYUiNGhkZIZPJxB2GSM2aSTXw+cngdCjZExGRmmBmAfAl4CogC3zc3Q9UbL8N+H2gAHzN3b9SWv8k0Ffa7bC7f3RBAxeZwmTJ4HQo2RMRkVrxPiDj7m80s+uAzwPvBTCzFPAnwDXAILDHzL4LnAFw9xtjiFdkQQRxByAiIjJHdgH3Arj7Y8DOim3bgQPu3uvuOWA38GaKpYBNZna/mT1YShJFaoqSPZEaMMnsBiL1pIVz1bEAoZklJ9g2ACwDhoDPAbcAnwD+ruKYMjO7w8z2mtnervMGuhVZ7FSNK1ID1tV554zsI18tLze8+VdijERi1g80V7wP3L0wwbZmilW4+ymW+Dmw38x6gDXAscoTu/udwJ1QnC5tXqIXmSdK9kSk6inBk5I9wG3At0rVsc9UbHsB2GpmrcBZ4HqKJXofA64Efs3M1lIsAexY0KhF5pmSPRERqRV3Azeb2aMUJ4T/qJndDix19zvN7LeA+yg2Yfqau79iZl8Fvm5muwEHPlZRGihSE2yKSXlVVC2LlcUdQAwmfB6f/PGPOTswsJCxSJ1Y2tzM1ddcM9kudfcs7ty50/fu3Rt3GCLjGfd5VMmeSA3INDay6bLL4g5DatTw0BCNTU1xhyEiF0nJnoiIyAI4MfwKT/c/TW++lxWpFexo2cHaxnVxhyV1QEOviIiIzLMTw6/wcPdDDIVDLE8uZygc4uHuhzgx/ErcoUkdULInIiIyz57uf5pMopGmRBNmRlOiiUyikaf7n447NKkDqsYVqQGapF4kfpNV0/bme1meXD5m/8agkd58bwyRSr1RsidSAzZffnncIYjUtdFq2kyicUw17Y1tb2Ft4zpWpFYwFA7RlDjX0WU4GmZFakWMUUu9ULInIiIyS5XVtED569P9T7O2cR07Wnbw7VfuoifXQ97zpCzFyvRKPrDug3GGLXVCbfZERERmqTffS2PQOGZdZTVt50gnp3OniYgICIiIOJ07TedIZxzhSp1RyZ6IiMgsTVVNu/v0bpqSS2hMnEsIh8Nhdp/ezWtXXA1oaBaZPyrZE5GqFx77afklEocdLTsYCYcZCodwd4bCIUbCYXa07ACgv9BHJmgYc0wmaKC/0AdoaBaZXyrZE5Gql997V3k5sf6qGCORerW2cR3Xr7yB+zrv42TUwaamzVy34rpyyVxLchnD4fCYkr2RKEtLchkwdZs/kdlQyZ6IiMgceH7geZ4beJYDgwd44sxeDgweIBtlAdjVuotsNMJwOIx7xHA4TDYaYVfrLmDqNn8is6GSPRGpesGlV8YdggiBnSs/6c338p2Ou7m/8z52rXwzu1a+mXdzG7tP76a/0EdLchk3r7q53F5vRWoFnSOd9Bf6GYmGyQSNtCRbaM+0l8+pNn1ysczdJ9s+6UaRGFncAcRAz6MsRnX3LO7cudP37t17wfrIIx7seoB/7v4hZ8OzY7alLMW1K97AjatuZGW67YJjn+p9ku+d+i4NQYZM0MBIlCUbjfDu1bfx2hVXjxnHrzFoZDgaZiQcLo/jJ1Iy7vOokj0REZE5EFjA29pv5vq2G/hx7+M83P0QPbkeAPKeZ8/p3Tx6eg+bmzZzaeN6Lm1cz/rG9axqWMXJ3Ek2Nm6ir9DHSFRs23dJwyWczJ0E1KZPZkfJnoiIyBxKB2netHIX17W+kWf6nubBrgc4PnIcAMc5NHSIQ0OHyvs3BA2kgjStqRW0JJfRmm4lZUkSlqQ7142705vvJeEJDgwfKFfztqfb6Y2m16ZPVcD1TdW4Uq3qruoIPY+yONXdszhRNe5E3J2XBl/iwa4H2H9234yvFxAQWIC7k7AE6SBN0pI4ztYlP8PtG3550uPnogpYyWLVGPd5VLIn1aru/sCg53FChcM/Li8nN18TYyR1qe6exZkme5X68n0cGz7G8dLr2PBxBgr9s4pneWo56zLrSJDgVPYUI9EILcllvKn1TVzTei33dd57wYDPo+9vXf0OYPJkbiHaCyqZnDNK9qSm1N0fGPQ8Tmjk7k+XlzPv/0yMkdSlunsWJ0r2LjZh6cv3cU/Hd+gY7mAgHKAQ5fHS4x56SER00bEaRsKSNARpUkGadJCiIWggbQ2EhLy9/e1kwyw/7XuKJcmlNCWaLkjm7j31g1kli1NR55M5pQ4aIjIHnvq/Z3+OpkvgZz48+/OILBKVCUvlDBjTSViWpZaRClLsWLYDs3N/q92dM4Uz/PzaD/H9k9/j1MhJ+gr9DIWDRB5R8ALZKDtpMug4Bc9TCPMQDl6w/c4jf1leDghIBw00BGkSluTkyEm2t7yKfQP7WJ5aTjbMkg7SpIM0GcuUxwA8MfwK337lLnpyPeQ9T8pSvDSwnw+s++C0kjV1Ppl/SvZEZGaWboBlW2d/nsETsGTt7M8DJDbtnJPziFys2SYsk82tmwySbGzcyDP9T9MQZGhNtZaHZvnAmg+ypmktf3HozwksIB/ly6WBoYflEsLpiIgYiYYZiYYBOJ2n3LHk5eGjY/Y1jFSQ4tjQywwUBhgOhwksIEGCER/h5eGX+ftj3+SGVTcW9xs8xgtnX2AoHKQ52cK1y6/l6hWvoyFo4HTudHkO4VHnDyg9Vcnhg6ceYPfpRxiOhmkMGtnV+mbeuvqmaR//VO+TY8ZA3NW6qzwGYi1QsiciVS919XvjDkHqXG++l+XJ5WPWzWQGjB0tO3i4+6HycaNVmdetuA5gwqFZOvOdvC7zetobVhenY0ufm4VjOBymIWjg/9j4EY4MHuK5gefozfeStBQtyRYiIvryfRwfPk42GplRYug4uSjHyezJ8rrIIwoUyu9fyb7CN4//3QXH9hX6OH7yGP/z5LeBYuJoBJhRWgpIBSkyiQx/dfRrhF6gK9tFKkjTEDTQOXKK5/ufY1vzNtoaVrG/fx9P9z9NwhIkLclIOMJ9nfcyHA5xQ/tb6Ml28w8d99Cd6yEf5UgFaQ6cfYmfW/sB1jau46neJ7mn4zuEhLg7I+EI93R8B6Cc8E2VLM42GZ1q+2wp2RMREZmlyUrmpmNt4zpubHvLmIShcm7d3nwvbQ1trMqsKh8zOiQLFKdj+96p7wKMGZT55lU3syy1DHc4nestl1xtX7q9nMh86/j/YN/AixSiAiEhUEy6VjW0s7V5K0cGj3B48BAREV76by4VzxiOaZWcC3MMhoPlcQrHM1rqOKrgBbKeLb9/oOcBHuh5YMw+hpENswwM9fOFg19gZUMr3dluCl4gIChWo0fFUs67Ou7iyPARstEI+wb2MRwOExGRIMFPep/g2tZrWZluoy9/hsdOP8ZAYYDQQ5JBkmfOPM1N7W/jkswa9p5+nD2n9xR7UluabJjlvs57iTziptVv46HOB7m/6z6SliRjGXJRjvu77gMoJ3yzTQbVQUOqVd01CmexPI8H/t+5qcZtWjNn1bgSq7p7FsfroDHfnQym00lioqrIqWbn+O9HvkLnyCmyUY7IQwJL0BCkac+s5uObfpUvHvxvxVLDxLlSw6HCEOkgzYcu/QW+fvRr9OX7SFgCrFjCF3pIQyLD9ubtPHnmJxhG8UfFx/yXsAShh7P+fGqBnfcoGUZbQxvDhWHOhmdLnW0SJEkWO9esumW8hE8dNERkDlgi7ghEFp2pSuZma6pqXihWOY7Xzmz36d00BJlysjb6dffp3bx2xdWMhCOkgjQtqWXlY4bDYUbCEQD6C300J5rHnLMxkWEgHGBt41reufpdY6pBk0GSBAneu+Z9vHbF1ewbeBF3SAWp8vH5KI8Z/NGr/hP/df8fYwQkLVksPfSInOcIPeS2Ne/hno57OJsfIGEJzGxMMvmq5u38tO+nhB6WxyIEyp1WGhONDIVDs/78F8L5JaaO05ntHPM+8ohEkCRJkt2nH5l26Z6SPRGZmct+Pu4ILlB4aU95Obn1TTFGIjI/ZpNMjpesZYIG+gt9QCkhKgyWetImyXsB96icFLYkl11QsjcSZWlJFpPD0QRzog4OW5q2sO/sPswhQYKQkILnuWLJFQCYGQkCksG5lCSIAkJCrlr2Wr7f8T3SpR7CAQGRRYReoClo5Pb1/xuXpNeUq0FTQYq85yl4oVzy9fn9n6Mn103SUhhGRETB8yxLLud/3/hh7jlxN0eHjpIIEsXzE1GICqxuWM3rV+zkH099n8CLVbyjCVnkEW7OlS1X8nTf08V2h6We1O7gpSrv1Q2X0Jk9NSaR84r/X4wECZKWYLjUkWY6lOyJSNUrPHtveVnJnsRhNkOvTNfaxnWTnmuiTgJTJWvrGteRtnS580cmaKQtc6594GTtAUdNVKoI8PbVtzBQGKAn10POc6QsxdrMWt6++hYANjRu5NDgQcBIWrI8pMyWJZcBkE6kCUiQ8ywFD0lagsagmWSiWMswWrpV2abtra03ldfftOomvtNxdzlBMy9OUXfr6ltZ17iO9619/5ihY9KWZk3jmvLQMU+ceYKebDfpIE1gAZFH5KIcKxva+PCGj/D/vPQ5urPdpTiLyWIuzNHW0MZvbf13PHjqgXPJqI1NRm9sfwsPnnqAf+q6n6QlSVqSvBd7VO9qfTPXrnwDXzr0RfJRnmSQKs+mkouyNAaNF37YE1CyJyIiMktxjxU3WbI5VbK2o2UHD+ceYm3j2jFVxDtadgDjl9zdvOrmaQ9NsrZxHR9Y98EJe6vuWrmL/nwfA+FZstEIiSBJW7qNXSt3AbCxcRMHBl+iOdlSLnkcCYfZ2HhZ+RpvXX3ThFWaU5U8ThXfTW03laupwyjEzGgIGriprXi9t7bdxD0nv0MYhYSEGEZDooG3jm6fIhl92yU3E1hwbntibAeMG1beyP1d9xF5SKKU6BW8wFtb1UFDal/dNQpHz+OE8s/8oLycuvIdMUZSl+ruWRyvg8bfH/8my5PLxx0U+ZcuvX3eY5qqA8dU48jFPV3ZVNO1/eOp7zOQP0vB8yQtRXNqKe9c/a4Fi3Gqz2++x+mbQW9cTZcmNaXu/sCg51EWp7p7FsdL9qbTW3Y+xZ1szre4k9Eqot64IiJSu8wsAL4EXAVkgY+7+4GK7bcBvw8UgK+5+1emOma6ptNbdj7Ndpy/xW6q9ooyuSDuAERERObI+4CMu78R+B3g86MbzCwF/AnwduAG4A4zu2SyY2ZitLdsU6KJM4UzNCWa5rRzxlR2tOxgJBxmKBzC3RkKh8a0u5P6ppI9ERGpFbuAewHc/TEzq5w0eTtwwN17AcxsN/Bm4I2THDMjcZY+zfc4f1LdlOyJSNXLv/BgeTm1/a0xRiIxawH6Kt6HZpZ098I42waAZVMcU2ZmdwB3AGzYsGE+Yp81VXXKRJTsiUjVC198qLysZK+u9QOVowcHFUnb+duagTNTHFPm7ncCd0Kxg8Ycxiwy79RmT0REasUe4J0AZnYd8EzFtheArWbWamZp4HrgR1McI1ITphp6RUQWCTO7F2ib5u5tQPc8hrOQauleoLbupw140d1vjTsQGNMbdwfFISg+CrwOWOrud1b0xg0o9sb98/GOcfcXp7hOF3B0/u5kStXwM6QY58ZMY+we73lUsidSg8xsr7tfdEPzxaSW7gVq635q6V6qSTV87opxbsxVjKrGFREREalhSvZEREREapiSPZHadGfcAcyhWroXqK37qaV7qSbV8LkrxrkxJzGqzZ6IiIhIDVPJnoiIiEgNU7InUmXMLDCzL5vZj8zsYTO7/Lzt15jZI2a228zuMrPMVMfE5SLvJWVmf1Na/7iZvSeu+CtdzL1UbGs3s2Nmtm3hI7/Qxd6Lmf2H0jFPmNmvxBN99RvvZ9zMXmdmr5S+Hw+b2S+U9v1VM9trZo+Z2bsXOM4nK+L5KzO7vPQz8YiZ/UVpWJvYYjSzj1TE95iZjSymz9HM3mBmD5eWp/3ZmVmjmX27tO8/mtmqKS/m7nrppVcVvYCfA75eWr4OuKdimwFPAZeX3n8cuGKyY6rwXj4K/Glp3Urg5bjv42LvpbScAu4G9gPb4r6PWXxfbgS+S7EQYSnwH+O+j2p9jfczXvqcf/u8/S6hOAh0A8Wp354BGhYoxgzw5Hnr/gG4sbT8ZeD9ccZ4Xmx/TnG6u0XxOQL/Z+k6j830swN+a/T5An4R+LOprqeSPZHqM2ayd6ByDKafAXqA3zSzHwKt7r5vimPidDH38v8Bn67Y74KprWJyMfcC8DmKv9xPLGCsU7mYe7mF4h+juykmfd9b0Ihry3g/468H3mVm/2xmXzWzZuBaYI+7Z929DzhAcXDohXAV0GRm95vZg1acfeT1wA9L238AvC3mGAEws53Aq7045d1i+RwPUvxH1aiZfHbl57Ni30kp2ROpPuNO3F5abgN+luKMAG8DbjKzm6Y4Jk4zvhd3P+vuA6Vf0ncBv7egEU9sxvdiZh8Butz9vgWNdGoX8zPWRjEp/HngE8DfmZktXMi1Y4Kf8ceBf+/u1wOHgD/gwu/TAMUSoIUwRPEfKrdQ+n5T7PQ52utzNJY4Yxz1KeAPS8uL4nN0928D+YpVM/nsKtdPK1YleyLVZ7KJ23uAA+7+vLvnKf7r7/VTHBOni7kXzGw98BDwN+7+zYUMeBIXcy8fA24utdt5LfANM7tk4UKe0MXcSw9wn7vnSiV9I8DUbYlkXOP8jN/t7k+UNt8NXM2F36dm4MwChbgf+Fsv2k/x+796nFjijBEzW06xecRDpVWL7XMcFY1z/Yniqlw/rViV7IlUn8kmbj8ELK1oUP9m4LkpjonTjO/FzFYD9wOfdPevLWSwU5jxvbj79e5+g7vfSLEd3Ifd/eTChTyhi/kZ2w3cakVrgSUUEwCZoQl+xu8zs2tLyzcBT1AspXqzFTsuLQO2A88uUJgfAz5finctxdKm+83sxtL2dwCPxBwjwPXA/6p4v9g+x1FPzuCzKz+fFftOajFU44jIzNxNsTToUUoTt5vZ7Zyb7P1XgG+WqtAedffvl3p2jTkmtujHuph7+TNgBfBpMxtt1/QOdx+O5Q7OmfG9xBnsFC7qXszseop/oALg1909jCn+avcpLvwZ/y3gT80sB5wE7nD3fjP7AsU/9gHwu+4+skAxfhX4upntBpxi8tcNfMXM0sALwF3uHsYYIxQ7Dx2qeP+vgS8uos9x1G8zzc/OzP4C+OvSZ58Dbp/q5BpUWURERKSGqRpXREREpIYp2RMRERGpYUr2RERERGqYkj0RERGRGqZkT0RERKSGKdkTERERqWFK9kRERERqmJI9ERERkRqmZE9ERESkhinZExEREalhSvZEREREapiSPRGZlJltMrN+M3u44vX7szznb5S+3mpmd8zyXJ81s71mduMMj/uImbmZvaFiXcrMus3sP05yXKuZ3V5a3mBmt42zzxEzy1S832ZmD88kPhGRuZKMOwARqQrPu/uNc3i+3wO+6O73zsG5fgG42t0HLuLYF4FfAv6l9P5WoG+KY3YA7wG+CbwV2AZ89yKuLSKyIJTsichFKZWk/TGQA+4EhoFfB6y0yweB08AXgGuBNPAHwGuAVjP7EvA4sM3df8fMfhv4RaAA/LO7f7JUwrYZaAc2Av/W3e+riOH3gUuB75vZLcB/AnaVNn/T3f/MzL4OrCy93uXuvRW38QPgFjML3D2imPj9fcX5/y9gJ9AMvODuHwV+F7jKzP4V8G+BJjN71N3/YZqf2wfH+ZxeA3yy9FluBv6Hu3/WzNaXPtsMMALcAbwD2Oru/97MEsBTwE53z07n+iKLgZl9BHgn0ARcBvyxu389zphqmZI9EZmOV51XDfnLpa8Zd38DgJl9imIyNWRmfwncAgwBbe5+rZldAvyGu/+emf0bd/+10i98zOxK4EPAz1JM9r5tZu8uXSPr7u8ws5uB3wbKyZ67/5GZfQx4O/A2ionSdRR/t+02swdLuz7o7n8yzn3lgB8BN5jZXqAFOA5cYmYtQK+732xmAfCcma0DPgt8wt3/0syyFJPV8RK9+80sKi03lT4LgJ8Z53N6hWIyuwNoAE6UrvM54Avu/gMzuwn4L8AngJ+Y2e9QLIl8SImeVKll7n6LmW2lWDr+9ZjjqVlK9kRkOi6oxi39gt5XsaoT+GszO0uxavNHwBWlr7j7SYrVt+PZBjzm7vnSuR8BXl3a9mTp6zGKJVwT2Q484u4O5M3sMeBVpW37Jj6Mb1Is0dsA/E+KJZBQLKlsN7O/B84CS4HUJOc539vdfaR0P9uAL5fWj/c5ATzj7gWgYGbDpXVXAp8ys09SLAnMufuAmf2QYpL4UeCPZhCTyGLyVOnrVM+2zJI6aIjIbEQAZrYM+EOK1bAfp5goGfACcM3oPmY2Wipn553nReANZpY0MwOuB/aXtvk0Y3mBUhWumaUolhK+VBnnBB6mWBr488BdFevfAax3918CPgU0luKOOPe7s3J5SpN8TjD+fb4IfLKUaP+rivi+Ujq+3d2fnu71RRaZ6T7bMktK9kRkLvQDe4CfAI9QTGLWAv8A9JrZborVr39a2v95M/vb0YPd/RngW6VzPA4cAb4zkwDc/XvAYTP7EfAYcJe7/2Qax0XAPwFD7t5fselxYEuphPAu4FDpng4CV5rZbwLPAO81s1+cZpgTfU4T+XfAH5RK8r4BPF2K+V+Ay4G/m+Z1RaSOWbHGQ0REqkWpDeEe4JbzElQRkQuoZE9EpIqY2WaKJYPfUKInItOhkj0RERGRGqaSPREREZEapmRPREREpIYp2RMRERGpYUr2RERERGqYkj0RERGRGqZkT0RERKSG/f/BVpQGc6XrMgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 648x216 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "make_plots(\n",
    "    df,\n",
    "    \"./plots/ballots.pdf\",\n",
    "    intervals_xlabel=\"Fraction for Matt Haney\",\n",
    "    true_theta=reported_estimate,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17d27ff3-a864-42f8-aa93-d010e22227d1",
   "metadata": {},
   "source": [
    "### Power experiment\n",
    "\n",
    "For PPI and the classical approach, find the smallest value of ```n``` such that the method has power 80% against the null $H_0: \\text{fraction for Matt Haney} \\leq 50\\%$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "591efd2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find n such that we reject H0: fraction for Matt Haney < 0.5 with probability 80% using a test at level alpha\n",
    "num_experiments = 100\n",
    "list_rand_idx = [np.random.permutation(n_max) for i in range(num_experiments)]\n",
    "\n",
    "alpha_pval = 0.05\n",
    "\n",
    "\n",
    "def _to_invert_ppi(n):\n",
    "    n = int(n)\n",
    "    nulls_rejected = 0\n",
    "    # Data setup\n",
    "    for i in range(num_experiments):\n",
    "        rand_idx = list_rand_idx[i]\n",
    "        _Yhat = Yhat[rand_idx[:n]]\n",
    "        _Y = Y[rand_idx[:n]]\n",
    "\n",
    "        ppi_ci = ppi_mean_ci(_Y, _Yhat, Yhat_unlabeled, alpha=alpha_pval)\n",
    "        if ppi_ci[0] > 0.5:\n",
    "            nulls_rejected += 1\n",
    "    return nulls_rejected / num_experiments - 0.8\n",
    "\n",
    "\n",
    "def _to_invert_classical(n):\n",
    "    n = int(n)\n",
    "    nulls_rejected = 0\n",
    "    # Data setup\n",
    "    for i in range(num_experiments):\n",
    "        rand_idx = list_rand_idx[i]\n",
    "        _Y = Y[rand_idx[:n]]\n",
    "\n",
    "        classical_ci = classical_mean_ci(_Y, alpha=alpha_pval)\n",
    "\n",
    "        if classical_ci[0] > 0.5:\n",
    "            nulls_rejected += 1\n",
    "    return nulls_rejected / num_experiments - 0.8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3403fb69",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The PPI test requires n=1 labeled data points to reject the null.\n",
      "The classical test requires n=112 labeled data points to reject the null.\n"
     ]
    }
   ],
   "source": [
    "# n_ppi = int(brentq(_to_invert_ppi, 1, 1000, xtol=1)) # This rejects for any value n. Try it!\n",
    "n_classical = int(brentq(_to_invert_classical, 1, 1000, xtol=1))\n",
    "print(f\"The PPI test requires n={1} labeled data points to reject the null.\")\n",
    "print(\n",
    "    f\"The classical test requires n={n_classical} labeled data points to reject the null.\"\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}