hp_xg_boost old.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8FLud1n-3pVm"
   },
   "source": [
    "<font size=\"+3\" ><b> <center>Learning XGBoost using Kaggle House Price competition</center></b></font>\n",
    ">\n",
    "_**Objectives**_\n",
    ">\n",
    "> This is a work in progress, not a polished, complete notebook\n",
    ">\n",
    "- Explore Kaggle House Price dataset\n",
    "- Learn XGBoost\n",
    "- Learn XGBoost inbuilt k-Fold crossvalidation\n",
    "- Learn GridsearchCV\n",
    "- Learn RandomSerchCV\n",
    "- Learn hyperopt\n",
    "- Compare model accuracy scores with Kaggle competition submission scores\n",
    "\n",
    "This is a _long_ Notebook, so care has been taken to use a hierarchy for sections.  Suggestions for how to make this Notebook more user friendly will be much appreciate.\n",
    ">\n",
    "_With acknowledgement to [Machine Learning A-Z™: Hands-On Python & R In Data Science](https://www.udemy.com/course/machinelearning/)_\n",
    ">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sO8VPU6n3vES"
   },
   "source": [
    "# Importing the libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "clDSsF7P33NU"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import qgrid  #   To explore Pandas DataFrames like a \"spreasheet\"\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "import math\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "\n",
    "#  For measuring model performance\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.metrics import r2_score\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "#  Make sure *all* print() lines are printed, not just the last one\n",
    "InteractiveShell.ast_node_interactivity = \"all\"\n",
    "\n",
    "# Make sure matplotlib charts and graphs are displayed in the cell outputs\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zGpwK5XD386E"
   },
   "source": [
    "# Import and explore the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Zcksk88u4Ae8"
   },
   "outputs": [],
   "source": [
    "#   This template assumes the data file contains the dependent (y) variable\n",
    "#   in the *last column*\n",
    "\n",
    "#   Load training data\n",
    "training_data = pd.read_csv('hp_train.csv')  #  Kaggle house price training data\n",
    "X = training_data.iloc[:, :\n",
    "                       -1]  #  To get Numpy ndarray add .value.  This gives Pandas DataFrame\n",
    "y = training_data.iloc[:,\n",
    "                       -1]  #  To get Numpy ndarray add .value.  This gives Pandas DataFrame\n",
    "\n",
    "# NB: convert y into type float\n",
    "y = y.astype(float)  #  This makes y a Pandas Series\n",
    "\n",
    "#   Load test data for competition\n",
    "competition_data = pd.read_csv('hp_test.csv')  #  Kaggle house price test data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">*Note*\n",
    ">\n",
    ">Categorical features not supported\n",
    ">\n",
    ">Note that XGBoost does not provide specialization for categorical features; if your data contains categorical features, load it as a NumPy array first and then perform corresponding preprocessing steps like one-hot encoding."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Processing the data\n",
    ">\n",
    "> With acknowledgement to [Kaggle Notebook by Dominik Gawlik](https://www.kaggle.com/dgawlik/house-prices-eda) for a lot of the techniques used"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dealing with missing data\n",
    "\n",
    "XGBoost can automatically handle missing data.  XGBoost was designed to work with sparse data.  However, using k-Nearest Neighbour can, under certain circumstances, beat the built-in function.  _(Acknowledgement to [Massimo Belloni](https://towardsdatascience.com/xgboost-is-not-black-magic-56ca013144b4))_.  This is generally where there is a high number of missing values.  In this Notebook the kNN method is followed for learning purposes.\n",
    "\n",
    "To outperform the XGBoost built-in default strategy we need two things:\n",
    "\n",
    "* A distance metric that takes into account missing values [(see this post by AirBnb)](https://medium.com/airbnb-engineering/overcoming-missing-values-in-a-random-forest-classifier-7b1fc1fc03ba)\n",
    "* To normalise the dataset to have meaningful distances, obtained by summing up differences among features with different domains (this is not strictly required by XGBoost but it’s needed for kNN imputation)  _**Note:**_ This normalisation should not hurt XGBoost performance - see answer by [Sycorax](https://stats.stackexchange.com/users/22311/sycorax) on [StackExchange](https://stats.stackexchange.com/questions/353462/what-are-the-implications-of-scaling-the-features-to-xgboost)\n",
    "\n",
    "For resources on using kNN to impute missing values, see posts by:\n",
    "\n",
    "* [Yohan Obadia](https://towardsdatascience.com/the-use-of-knn-for-missing-values-cf33d935c637)\n",
    "* [Kaushik](https://www.analyticsvidhya.com/blog/2020/07/knnimputer-a-robust-way-to-impute-missing-values-using-scikit-learn/)\n",
    "* [Jason Brownlee](https://machinelearningmastery.com/knn-imputation-for-missing-values-in-machine-learning/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore missing data\n",
    "\n",
    "Using the **missingno** library as explained by [Soner Yildirim](https://towardsdatascience.com/visualize-missing-values-with-missingno-ad4d938b00a1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the missingno library\n",
    "import missingno as msno"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make DataFrame with only missing data\n",
    "# Note the full training data set is used, which includes y - the dependent variable\n",
    "\n",
    "# isna returns missing values\n",
    "missingdata = (training_data.isna().sum() > 0\n",
    "              )  # this is a series with dtype: bool\n",
    "missingdata = missingdata.to_numpy(\n",
    ")  # converts series into numpy array, so now have a boolean array\n",
    "only_missing_data = training_data[training_data.columns[\n",
    "    missingdata]]  # data.columns returns an index, convert this to a DataFrame\n",
    "only_missing_data  # this is now a DataFrame containing only the columns and rows where there is missing data in the column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use qgrid to show lika a spreadsheet - https://github.com/quantopian/qgrid\n",
    "import qgrid\n",
    "qgrid_widget = qgrid.show_grid(only_missing_data, show_toolbar=True)\n",
    "qgrid_widget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Want to sort the dataframe columns based on number of missing values in each column\n",
    "\n",
    "number_missing_data = only_missing_data.isna().sum().to_numpy(\n",
    ")  # array with number of missing values in each column\n",
    "column_names = only_missing_data.columns.to_numpy(\n",
    ")  # this gives an array containing column names\n",
    "missing = np.vstack(\n",
    "    (column_names, number_missing_data\n",
    "    ))  # make array by combining number of missing data and column name\n",
    "missing = pd.DataFrame(\n",
    "    missing\n",
    ")  # convert to pandas dataframe with row 0 containing column names, row 1 sum of missing data\n",
    "missing = missing.T  # transpose - first column now column names, second column sum of missing values\n",
    "missing = missing.sort_values(\n",
    "    by=1, axis=0, ascending=False\n",
    ")  # column names = 0,1, so sort on second column, largest to smallest\n",
    "missing = missing.T  # missing now has sorted column names in first row\n",
    "missing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    ">Conclusion: the majority of missing data is in PoolQC, MiscFeature, Alley, Fence and FireplaceQu\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now sort DataFrame \"only_missing_data\" columns based on column order in \"missing\" DataFrame\n",
    "column_order = missing.iloc[0, :]  # this is a series\n",
    "column_order = column_order.to_numpy()  # convert series to array\n",
    "#column_order\n",
    "only_missing_data = only_missing_data.reindex(column_order, axis=1)\n",
    "only_missing_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Visualize missing data with missingno"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Matrix plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# White lines represent missing data\n",
    "msno.matrix(only_missing_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Heatmap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Positive correlation is proportional to the level of darkness in blue as indicated by the bar on the right side.\n",
    "\n",
    "msno.heatmap(only_missing_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dendogram\n",
    "\n",
    "To interpret this graph, read it from a top-down perspective. Cluster leaves which linked together at a distance of zero fully predict one another's presence—one variable might always be empty when another is filled, or they might always both be filled or both empty, and so on. \n",
    "\n",
    "Cluster leaves which split close to zero, but not at it, predict one another very well, but still imperfectly. If your own interpretation of the dataset is that these columns actually are or ought to be match each other in nullity, then the height of the cluster leaf tells you, in absolute terms, how often the records are \"mismatched\", how many values you would have to fill in or drop, if you are so inclined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(only_missing_data)\n",
    "msno.dendrogram(only_missing_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bar plot of missing values\n",
    "\n",
    "It shows bars that are proportional to the number of non-missing values \n",
    "as well as providing the actual number of missingvalues. \n",
    "We get an idea of how much of each column is missing.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# msno.bar(only_missing_data)\n",
    "missing = training_data.isnull().sum()\n",
    "missing = missing[missing > 0]\n",
    "missing.sort_values(inplace=True)\n",
    "missing.plot.bar()  #   Height of bar shown number of rows with missing data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">_**Conclusions**_\n",
    "> It is prudent to drop the PoolQC, MiscFeature, Alley and Fence features - there is simply too much data missing.  For the FireplaceQu feature, around 50% data is missing, but this seems to be the \"rule of thumb\" threshhold where it should be explored how much information is in the feature, and whether it is worth imputing values.  There is no objective value (that I could find), and what percentage missing data suggests dropping the whole feature depends on the situation.\n",
    "\n",
    "Some resources with information on a rule of thumb to follow:\n",
    "\n",
    "* [AnalyticsVidhya](https://discuss.analyticsvidhya.com/t/what-should-be-the-allowed-percentage-of-missing-values/2456): maybe 50%\n",
    "* [Dan Berdikulov](https://medium.com/@danberdov/dealing-with-missing-data-8b71cd819501#:~:text=As%20a%20rule%20of%20thumb,the%20variable%20should%20be%20considered.): 60% - 70%\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Drop columns\n",
    ">_Make sure to drop in both training and competition sets_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd = competition_data.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(type(X))\n",
    "print(type(competition_data))\n",
    "print(type(cd))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = X.drop(['Id', 'PoolQC', 'MiscFeature', 'Alley', 'Fence', 'FireplaceQu'],\n",
    "           axis=1).copy()  #   Our training data\n",
    "cd = cd.drop(['Id', 'PoolQC', 'MiscFeature', 'Alley', 'Fence', 'FireplaceQu'],\n",
    "             axis=1).copy()  #   Our competition data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.info()\n",
    "cd.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dealing with categorical data\n",
    ">\n",
    ">Internally, XGBoost models represent all problems as a regression predictive modeling problem that only takes numerical values as input. If your data is in a different form, it must be prepared into the expected format. See [Data Preparation for Gradient Boosting with XGBoost in Python](https://machinelearningmastery.com/data-preparation-gradient-boosting-xgboost-python/)\n",
    ">\n",
    ">_**Remember to save the label encoder as a separate object so that we can transform both the training and later the test and validation datasets using the same encoding scheme.**_\n",
    ">\n",
    "> >xgboost only deals with numeric columns.\n",
    "\n",
    "if you have a feature [a,b,b,c] which describes a categorical variable (i.e. no numeric relationship)\n",
    "\n",
    "Using LabelEncoder you will simply have this:\n",
    "\n",
    "array([0, 1, 1, 2])\n",
    "Xgboost will wrongly interpret this feature as having a numeric relationship! This just maps each string ('a','b','c') to an integer, nothing more.\n",
    "\n",
    "Proper way\n",
    "\n",
    "Using OneHotEncoder you will eventually get to this:\n",
    "\n",
    "array([[ 1.,  0.,  0.],\n",
    "       [ 0.,  1.,  0.],\n",
    "       [ 0.,  1.,  0.],\n",
    "       [ 0.,  0.,  1.]])\n",
    "This is the proper representation of a categorical variable for xgboost or any other machine learning tool.\n",
    "\n",
    "Pandas get_dummies is a nice tool for creating dummy variables (which is easier to use, in my opinion)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One suggestion of code:\n",
    ">ONE_HOT_COLS = [\"categorical_col1\", \"categorical_col2\", \"categorical_col3\"]\n",
    "print(\"Starting DF shape: %d, %d\" % df.shape)\n",
    "\n",
    "\n",
    "for col in ONE_HOT_COLS:\n",
    "    s = df[col].unique()\n",
    "\n",
    "    # Create a One Hot Dataframe with 1 row for each unique value\n",
    "    one_hot_df = pd.get_dummies(s, prefix='%s_' % col)\n",
    "    one_hot_df[col] = s\n",
    "\n",
    "    print(\"Adding One Hot values for %s (the column has %d unique values)\" % (col, len(s)))\n",
    "    pre_len = len(df)\n",
    "\n",
    "    # Merge the one hot columns\n",
    "    df = df.merge(one_hot_df, on=[col], how=\"left\")\n",
    "    assert len(df) == pre_len\n",
    "    print(df.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OneHotEncode Categorical Data\n",
    ">\n",
    "[Article on encoding categorical data](https://www.datacamp.com/community/tutorials/categorical-data) - used as inspiration for this code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_features_indices = np.where(\n",
    "    X.dtypes != np.float)[0]  # List of categorical features indices, int64\n",
    "#  Off CatBoost training website"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_features_indices.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = X.fillna(-999).copy()\n",
    "cd = cd.fillna(-999).copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_play = X.iloc[:5, 0:2].copy()\n",
    "X_play"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.get_dummies(X_play['MSSubClass'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.get_dummies(X_play['MSZoning'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.get_dummies(data=X_play, columns=['MSSubClass', 'MSZoning'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list = ['MSSubClass', 'MSZoning']\n",
    "for element in list:\n",
    "    print(type(element))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.get_dummies(data=X_play, columns=list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "# From https://stackoverflow.com/questions/39923927/pandas-sklearn-one-hot-encoding-dataframe-or-numpy\n",
    "\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "columns_to_encode = categorical_features_indices\n",
    "#ct = ColumnTransformer(transformers=[('encoder', OneHotEncoder(), X.columns[columns_to_encode].tolist())], remainder='passthrough')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "metadata": {},
     "execution_count": 67
    }
   ],
   "source": [
    "# print(X.info(), \"\\n\\n\")\n",
    "# print(type(columns_to_encode))\n",
    "# columns_to_encode\n",
    "# print(\"\\n\\n\", X.columns[columns_to_encode].tolist())\n",
    "list_to_encode = X.columns[columns_to_encode].tolist()\n",
    "type(list_to_encode)\n",
    "#list_to_encode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\nRangeIndex: 1460 entries, 0 to 1459\nColumns: 8574 entries, LotFrontage to SaleCondition_Partial\ndtypes: float64(3), uint8(8571)\nmemory usage: 12.0 MB\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "NoneType"
      ]
     },
     "metadata": {},
     "execution_count": 68
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\nRangeIndex: 1460 entries, 0 to 1459\nColumns: 8574 entries, LotFrontage to SaleCondition_Partial\ndtypes: float64(3), uint8(8571)\nmemory usage: 12.0 MB\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "NoneType"
      ]
     },
     "metadata": {},
     "execution_count": 68
    }
   ],
   "source": [
    "X_enc = pd.get_dummies(data=X, columns=list_to_encode).copy()\n",
    "cd_enc = pd.get_dummies(data=X, columns=list_to_encode).copy()\n",
    "type(X_enc.info())\n",
    "type(cd_enc.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XNn2RnST6_Q-"
   },
   "source": [
    "# Splitting the dataset into the Training set and Test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ajhBL-er7Gry"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X_enc,\n",
    "                                                    y,\n",
    "                                                    test_size=0.2,\n",
    "                                                    random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\">  \n",
    "\n",
    "_**Now everything is done up to X_train, X_test, y_train, y_test**_\n",
    "\n",
    "We are now in familiar territory  \n",
    ">\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1Y89ctGZ7Mcx"
   },
   "source": [
    "# Training XGBoost on the Training set - baseline model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ude1J0E47SKN"
   },
   "outputs": [],
   "source": [
    "import xgboost as xgb\n",
    "\n",
    "#########################################################################\n",
    "#\n",
    "# With gblinear the base model scores RMSE 57,877, R^2 0.51\n",
    "# With gbtree the base model scores RMSE 39,527, R^2 0.77\n",
    "#\n",
    "#########################################################################\n",
    "\n",
    "xg_reg = xgb.XGBRegressor(booster = 'gbtree', objective ='reg:squarederror', learning_rate = 0.3,\n",
    "                n_estimators = 1000, eval_metric = 'rmse')\n",
    "\n",
    "xg_reg.fit(X_train,y_train)\n",
    "\n",
    "y_pred = xg_reg.predict(X_test)\n",
    "rmse = math.sqrt(mean_squared_error(y_test, y_pred))\n",
    "r2 = r2_score(y_test, y_pred)\n",
    "print(\"RMSE: %f\\n\\n\" % (rmse))\n",
    "print(\"R^2: %f\\n\\n\" % (r2))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "outputPrepend"
    ]
   },
   "outputs": [],
   "source": [
    "# Convert dataset to special XGBoost optimised data structure\n",
    "dtrain = xgb.DMatrix(X_train, label=y_train, missing=-999.0)\n",
    "dtest = xgb.DMatrix(X_test, label=y_test)\n",
    "dcomp = xgb.DMatrix(cd_enc)\n",
    "\n",
    "#   Specify parameters\n",
    "\n",
    "evallist = [(dtest, 'eval'), (dtrain, 'train')]\n",
    "\n",
    "#evals_result = {}\n",
    "\n",
    "params = {\n",
    "    'booster': 'gbtree', # gbtree beats gblinear - don't feel forced to fit linear to this dataset\n",
    "    'objective': 'reg:squarederror',\n",
    "    'learning_rate': 0.3,\n",
    "#     'reg_alpha': 0,  #  L1 regularisation\n",
    "#     'reg_lambda': 0,  # L2 regularisation\n",
    "    'eval_metric': 'rmse',  # Used by Kaggle house price competition - this parameter is used for VALIDATION data\n",
    "#    'evals': evallist,\n",
    "    'verbose_eval': 1,  # If integer, value is printed every x boosting stages\n",
    "    'early_stopping_rounds': 10,\n",
    "    'verbosity': 1,  # 0 is silent, 3 is debug\n",
    "}\n",
    "\n",
    "#  Train the model\n",
    "num_round = 1000\n",
    "bst = xgb.train(params, dtrain, num_round, evallist)\n",
    "\n",
    "# Evaluate how good the model is\n",
    "y_pred = bst.predict(dtest)\n",
    "rmse = math.sqrt(mean_squared_error(y_test, y_pred))\n",
    "r2 = r2_score(y_test, y_pred)\n",
    "print(\"RMSE: %f\\n\\n\" % (rmse))\n",
    "print(\"R^2: %f\\n\\n\" % (r2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "dict"
      ]
     },
     "metadata": {},
     "execution_count": 80
    }
   ],
   "source": [
    "type(params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# At this stage, the basic model works with the special DMatrix format and parametrisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EnbCjHgQ8XPn"
   },
   "source": [
    "# Applying k-Fold Cross Validation to measure performance\n",
    ">\n",
    "> The result is a more reliable estimate of the performance of the algorithm on new data given your test data. It is more accurate because the algorithm is trained and evaluated multiple times on different data.\n",
    ">\n",
    "> We can use k-fold cross validation support provided in scikit-learn. First we must create the KFold object specifying the number of folds and the size of the dataset. We can then use this scheme with the specific dataset. The cross_val_score() function from scikit-learn allows us to evaluate a model using the cross validation scheme and returns a list of the scores for each model trained on each fold.\n",
    ">\n",
    ">\n",
    "_**What about  XGBoost’s built-in Cross Validation ???**_\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yYbfiITD8ZAz"
   },
   "outputs": [],
   "source": [
    "# from sklearn.model_selection import cross_val_score\n",
    "# accuracies = cross_val_score(estimator = xg_reg, X = X_train, y = y_train, cv = 10)\n",
    "# print(\"Accuracy: {:.2f} %\".format(accuracies.mean()*100))\n",
    "# print(\"Standard Deviation: {:.2f} %\".format(accuracies.std()*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import matplotlib.pyplot as plt\n",
    "\n",
    "# xgb.plot_tree(xg_reg,num_trees=0)\n",
    "# plt.rcParams['figure.figsize'] = [50, 10]\n",
    "# plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://xgboost.readthedocs.io/en/latest/get_started.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ude1J0E47SKN"
   },
   "outputs": [],
   "source": [
    "\n",
    "#   Specify parameters\n",
    "\n",
    "evallist = [(dtest, 'eval'), (dtrain, 'train')]\n",
    "\n",
    "evals_result = {}\n",
    "\n",
    "params = {\n",
    "    'booster': 'gbtree', # gbtree beats gblinear - don't feel forced to fit linear to this dataset\n",
    "    'objective': 'reg:squarederror',\n",
    "    'learning_rate': 0.3,\n",
    "#     'reg_alpha': 0,  #  L1 regularisation\n",
    "#     'reg_lambda': 0,  # L2 regularisation\n",
    "    'eval_metric': 'rmse',  # Used by Kaggle house price competition - this parameter is used for VALIDATION data\n",
    "#    'evals': evallist,\n",
    "    'verbose_eval': 1,  # If integer, value is printed every x boosting stages\n",
    "    'early_stopping_rounds': 10,\n",
    "    'verbosity': 1,  # 0 is silent, 3 is debug\n",
    "}\n",
    "\n",
    "#  Train the model\n",
    "num_round = 1000\n",
    "bst = xgb.train(params, dtrain, num_round, evallist)\n",
    "\n",
    "# Evaluate how good the model is\n",
    "y_pred = bst.predict(dtest)\n",
    "rmse = math.sqrt(mean_squared_error(y_test, y_pred))\n",
    "r2 = r2_score(y_test, y_pred)\n",
    "print(\"RMSE: %f\\n\\n\" % (rmse))\n",
    "print(\"R^2: %f\\n\\n\" % (r2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # https://xgboost.readthedocs.io/en/latest/python/python_api.html\n",
    "# # https://stackoverflow.com/questions/47152610/what-is-the-difference-between-xgb-train-and-xgb-xgbregressor-or-xgb-xgbclassif\n",
    "# model = xgb.train(\n",
    "#     params,\n",
    "#     dtrain,\n",
    "#     num_boost_round=num_rounds,\n",
    "#     evals_result=evals_result\n",
    "# )\n",
    "# print(evals_result)\n",
    "# y_pred = model.predict(dtest)\n",
    "# rmse = np.sqrt(mean_squared_error(y_test, y_pred))\n",
    "# print(\"RMSE: %f\" % (rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#from sklearn.model_selection import cross_val_score\n",
    "\n",
    "num_round = 10\n",
    "#model = xgb.train(params, dtrain, num_round, evallist)\n",
    "\n",
    "cv_results = xgb.cv(\n",
    "    params,\n",
    "    dtrain,\n",
    "    num_boost_round=num_round,\n",
    "    seed=42,\n",
    "    nfold=5,\n",
    "    metrics={'rmse'},\n",
    "    early_stopping_rounds=10\n",
    ")\n",
    "cv_results\n",
    "\n",
    "# result = xgb.cv(params=params, dtrain=dtrain, num_boost_round=num_round, early_stopping_rounds=10)\n",
    "# print(result)\n",
    "# accuracies = cross_val_score(estimator=model, X=X_enc, y=y_train, cv=10)\n",
    "# print(\"Accuracy: {:.2f} %\".format(accuracies.mean() * 100))\n",
    "# print(\"Standard Deviation: {:.2f} %\".format(accuracies.std()*100))"
   ]
  },
  {
   "source": [
    "# Now start tuning\n",
    "From https://blog.cambridgespark.com/hyperparameter-tuning-in-xgboost-4ff9100a3b2f"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parameters max_depth and min_child_weight\n",
    "\n",
    "gridsearch_params = [\n",
    "    (max_depth, min_child_weight)\n",
    "    for max_depth in range(9,12)\n",
    "    for min_child_weight in range(5,8)\n",
    "]\n",
    "\n",
    "# Define initial best params and RMSE\n",
    "min_rmse = float(\"Inf\")\n",
    "best_params = None\n",
    "for max_depth, min_child_weight in gridsearch_params:\n",
    "    print(\"CV with max_depth={}, min_child_weight={}\".format(\n",
    "                             max_depth,\n",
    "                             min_child_weight))\n",
    "    # Update our parameters\n",
    "    params['max_depth'] = max_depth\n",
    "    params['min_child_weight'] = min_child_weight\n",
    "    # Run CV\n",
    "    cv_results = xgb.cv(\n",
    "        params,\n",
    "        dtrain,\n",
    "        num_boost_round=num_round,\n",
    "        seed=42,\n",
    "        nfold=5,\n",
    "        metrics={'rmse'},\n",
    "        early_stopping_rounds=10\n",
    "    )\n",
    "    # Update best RMSE\n",
    "    mean_rmse = cv_results['test-rmse-mean'].min()\n",
    "    boost_rounds = cv_results['test-rmse-mean'].argmin()\n",
    "    print(\"\\tRMSE {} for {} rounds\".format(mean_rmse, boost_rounds))\n",
    "    if mean_rmse < min_rmse:\n",
    "        min_rmse = mean_rmse\n",
    "        best_params = (max_depth,min_child_weight)\n",
    "print(\"Best params: {}, {}, RMSE: {}\".format(best_params[0], best_params[1], min_rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "outputPrepend"
    ]
   },
   "outputs": [],
   "source": [
    "# From previous round optimization\n",
    "params['max_depth'] = 10\n",
    "params['min_child_weight'] = 6\n",
    "\n",
    "#  Parameters subsample and colsample_bytree\n",
    "gridsearch_params = [\n",
    "    (subsample, colsample)\n",
    "    for subsample in [i/10. for i in range(7,11)]\n",
    "    for colsample in [i/10. for i in range(7,11)]\n",
    "]\n",
    "\n",
    "min_rmse = float(\"Inf\")\n",
    "best_params = None\n",
    "\n",
    "# We start by the largest values and go down to the smallest\n",
    "for subsample, colsample in reversed(gridsearch_params):\n",
    "    print(\"CV with subsample={}, colsample={}\".format(\n",
    "                             subsample,\n",
    "                             colsample))\n",
    "    # We update our parameters\n",
    "    params['subsample'] = subsample\n",
    "    params['colsample_bytree'] = colsample\n",
    "    # Run CV\n",
    "    cv_results = xgb.cv(\n",
    "        params,\n",
    "        dtrain,\n",
    "        num_boost_round=num_round,\n",
    "        seed=42,\n",
    "        nfold=5,\n",
    "        metrics={'rmse'},\n",
    "        early_stopping_rounds=10\n",
    "    )\n",
    "    # Update best score\n",
    "    mean_rmse = cv_results['test-rmse-mean'].min()\n",
    "    boost_rounds = cv_results['test-rmse-mean'].argmin()\n",
    "    print(\"\\tRMSE {} for {} rounds\".format(mean_rmse, boost_rounds))\n",
    "    if mean_rmse < min_rmse:\n",
    "        min_rmse = mean_rmse\n",
    "        best_params = (subsample,colsample)\n",
    "print(\"Best params: {}, {}, RMSE: {}\".format(best_params[0], best_params[1], min_rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Update params from above\n",
    "params['subsample'] = .7\n",
    "params['colsample_bytree'] = 1.0\n",
    "\n",
    "#  Now run model with these optimal values\n",
    "num_round = 1000\n",
    "model = xgb.train(\n",
    "    params,\n",
    "    dtrain,\n",
    "    num_boost_round=num_round,\n",
    "    evals=[(dtest, \"Test\")],\n",
    "    early_stopping_rounds=10\n",
    ")\n",
    "\n",
    "print(\"Best RMSE: {:.2f} in {} rounds\".format(model.best_score, model.best_iteration+1))\n"
   ]
  },
  {
   "source": [
    "# Hyperopt\n",
    "From here: https://towardsdatascience.com/an-example-of-hyperparameter-optimization-on-xgboost-lightgbm-and-catboost-using-hyperopt-12bc41a271e\n",
    "\n",
    "\n",
    "fmin() is the main function in hyperopt for optimization. \n",
    "It accepts four basic arguments and output the optimized parameter set:\n",
    "* Objective Function — fn\n",
    "* Search Space — space\n",
    "* Search Algorithm — algo\n",
    "* (Maximum) no. of evaluations — max_evals\n",
    "\n",
    "We may also pass a Trials object to the trials argument which keeps track of the whole process. In order to run with trails the output of the objective function has to be a dictionary including at least the keys 'loss' and 'status' which contain the result and the optimization status respectively. The interim values could be extracted by the following:\n",
    "* trials.trials - a list of dictionaries contains all relevant information\n",
    "* trials.results - a list of dictionaries collecting the function outputs\n",
    "* trials.losses() - a list of losses (float for each 'ok' trial)\n",
    "* trials.statuses() - a list of status strings\n",
    "* trials.vals - a dictionary of sampled parameters\n",
    "\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hyperopt import fmin, tpe, hp, STATUS_OK, Trials, space_eval\n",
    "\n",
    "###############################################################\n",
    "#\n",
    "#   Need to compare the output with the \"manual\" result above - 36,106.71\n",
    "#\n",
    "###############################################################\n",
    "\n",
    "# Choose hyperparameter domain to search over\n",
    "space = {\n",
    "    'booster': 'gbtree',\n",
    "    'learning_rate': 0.3,\n",
    "    'max_depth': hp.choice('max_depth', np.arange(5, 30, 5, dtype=int)),\n",
    "    'n_estimators': hp.choice('n_estimators', np.arange(100, 1000, 200, dtype=int)),\n",
    "    'colsample_bytree': hp.quniform('colsample_bytree', 0.1, 1.5, 0.4),\n",
    "    'min_child_weight': hp.choice('min_child_weight', np.arange(1, 100, 50, dtype=int)),\n",
    "    'subsample': hp.quniform('subsample', 0.5, 0.9, 0.4),\n",
    "    'eta': hp.quniform('eta', 0.05, 0.7, 0.2),\n",
    "    'objective': 'reg:squarederror',\n",
    "    'eval_metric': 'rmse',\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# from hyperopt import hp, fmin, tpe, Trials, STATUS_OK\n",
    "# import matplotlib.pyplot as plt\n",
    "\n",
    "# # Define objective function\n",
    "# def f(x):\n",
    "#     return {'loss': x ** 2 - x, 'status': STATUS_OK}\n",
    "\n",
    "# # Run hyperopt optimization\n",
    "# trials = Trials()\n",
    "# result = fmin(\n",
    "#     fn=f,                           # objective function\n",
    "#     space=hp.uniform('x', -1, 1),   # parameter space\n",
    "#     algo=tpe.suggest,               # surrogate algorithm\n",
    "#     max_evals=500,                  # no. of evaluations\n",
    "#     trials=trials                   # trials object that keeps track of the sample results (optional)\n",
    "# )\n",
    "\n",
    "# # Print the optimized parameters\n",
    "# print(result)   # {'x': 0.5000833960783931}\n",
    "\n",
    "# # Extract and plot the trials \n",
    "# x = trials.vals['x']\n",
    "# y = [x['loss'] for x in trials.results]\n",
    "# plt.scatter(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hyperopt import hp\n",
    "import numpy as np\n",
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "\n",
    "# XGB parameters\n",
    "xgb_reg_params = {\n",
    "    'learning_rate':    hp.choice('learning_rate',    np.arange(0.05, 0.31, 0.05)),\n",
    "    'max_depth':        hp.choice('max_depth',        np.arange(5, 16, 1, dtype=int)),\n",
    "    'min_child_weight': hp.choice('min_child_weight', np.arange(1, 8, 1, dtype=int)),\n",
    "    'colsample_bytree': hp.choice('colsample_bytree', np.arange(0.3, 0.8, 0.1)),\n",
    "    'subsample':        hp.uniform('subsample', 0.8, 1),\n",
    "    'n_estimators':     100,\n",
    "}\n",
    "xgb_fit_params = {\n",
    "    'eval_metric': 'rmse',\n",
    "    'early_stopping_rounds': 10,\n",
    "    'verbose': False\n",
    "}\n",
    "xgb_para = dict()\n",
    "xgb_para['reg_params'] = xgb_reg_params\n",
    "xgb_para['fit_params'] = xgb_fit_params\n",
    "xgb_para['loss_func' ] = lambda y, pred: np.sqrt(mean_squared_error(y, pred))\n",
    "\n",
    "\n",
    "# LightGBM parameters\n",
    "lgb_reg_params = {\n",
    "    'learning_rate':    hp.choice('learning_rate',    np.arange(0.05, 0.31, 0.05)),\n",
    "    'max_depth':        hp.choice('max_depth',        np.arange(5, 16, 1, dtype=int)),\n",
    "    'min_child_weight': hp.choice('min_child_weight', np.arange(1, 8, 1, dtype=int)),\n",
    "    'colsample_bytree': hp.choice('colsample_bytree', np.arange(0.3, 0.8, 0.1)),\n",
    "    'subsample':        hp.uniform('subsample', 0.8, 1),\n",
    "    'n_estimators':     100,\n",
    "}\n",
    "lgb_fit_params = {\n",
    "    'eval_metric': 'l2',\n",
    "    'early_stopping_rounds': 10,\n",
    "    'verbose': False\n",
    "}\n",
    "lgb_para = dict()\n",
    "lgb_para['reg_params'] = lgb_reg_params\n",
    "lgb_para['fit_params'] = lgb_fit_params\n",
    "lgb_para['loss_func' ] = lambda y, pred: np.sqrt(mean_squared_error(y, pred))\n",
    "\n",
    "\n",
    "# CatBoost parameters\n",
    "ctb_reg_params = {\n",
    "    'learning_rate':     hp.choice('learning_rate',     np.arange(0.05, 0.31, 0.05)),\n",
    "    'max_depth':         hp.choice('max_depth',         np.arange(5, 16, 1, dtype=int)),\n",
    "    'colsample_bylevel': hp.choice('colsample_bylevel', np.arange(0.3, 0.8, 0.1)),\n",
    "    'n_estimators':      100,\n",
    "    'eval_metric':       'RMSE',\n",
    "}\n",
    "ctb_fit_params = {\n",
    "    'early_stopping_rounds': 10,\n",
    "    'verbose': False\n",
    "}\n",
    "ctb_para = dict()\n",
    "ctb_para['reg_params'] = ctb_reg_params\n",
    "ctb_para['fit_params'] = ctb_fit_params\n",
    "ctb_para['loss_func' ] = lambda y, pred: np.sqrt(mean_squared_error(y, pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://towardsdatascience.com/an-example-of-hyperparameter-optimization-on-xgboost-lightgbm-and-catboost-using-hyperopt-12bc41a271e\n",
    "\n",
    "import lightgbm as lgb\n",
    "import xgboost as xgb\n",
    "import catboost as ctb\n",
    "from hyperopt import fmin, tpe, STATUS_OK, STATUS_FAIL, Trials\n",
    "\n",
    "\n",
    "class HPOpt(object):\n",
    "\n",
    "    def __init__(self, X_train, X_test, y_train, y_test):\n",
    "        self.X_train = X_train\n",
    "        self.X_test  = X_test\n",
    "        self.y_train = y_train\n",
    "        self.y_test  = y_test\n",
    "\n",
    "    def process(self, fn_name, space, trials, algo, max_evals):\n",
    "        fn = getattr(self, fn_name)\n",
    "        try:\n",
    "            result = fmin(fn=fn, space=space, algo=algo, max_evals=max_evals, trials=trials)\n",
    "        except Exception as e:\n",
    "            return {'status': STATUS_FAIL,\n",
    "                    'exception': str(e)}\n",
    "        return result, trials\n",
    "\n",
    "    def xgb_reg(self, para):\n",
    "        reg = xgb.XGBRegressor(**para['reg_params'])\n",
    "        return self.train_reg(reg, para)\n",
    "\n",
    "    def lgb_reg(self, para):\n",
    "        reg = lgb.LGBMRegressor(**para['reg_params'])\n",
    "        return self.train_reg(reg, para)\n",
    "\n",
    "    def ctb_reg(self, para):\n",
    "        reg = ctb.CatBoostRegressor(**para['reg_params'])\n",
    "        return self.train_reg(reg, para)\n",
    "\n",
    "    def train_reg(self, reg, para):\n",
    "        reg.fit(self.X_train, self.y_train,\n",
    "                eval_set=[(self.X_train, self.y_train), (self.X_test, self.y_test)],\n",
    "                **para['fit_params'])\n",
    "        pred = reg.predict(self.X_test)\n",
    "        loss = para['loss_func'](self.y_test, pred)\n",
    "        return {'loss': loss, 'status': STATUS_OK}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "100%|██████████| 100/100 [19:20<00:00, 11.60s/trial, best loss: 32335.596084526587]\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "[LightGBM] [Warning] \n",
      "Accuracy may be bad since you didn't set num_leaves and 2^max_depth > num_leaves\n",
      "100%|██████████| 100/100 [01:08<00:00,  1.45trial/s, best loss: 33492.1391549921]\n",
      "100%|██████████| 100/100 [44:04<00:00, 26.44s/trial, best loss: 34184.398720101315]\n"
     ]
    },
    {
     "output_type": "error",
     "ename": "AttributeError",
     "evalue": "'tuple' object has no attribute 'predict'",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-75-f015ab551257>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[0;31m# XGBoost\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m \u001b[0my_pred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mxgb_opt\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_test\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     11\u001b[0m \u001b[0mrmse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqrt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmean_squared_error\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_pred\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m \u001b[0mr2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mr2_score\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_pred\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: 'tuple' object has no attribute 'predict'"
     ]
    }
   ],
   "source": [
    "obj = HPOpt(X_train, X_test, y_train, y_test)\n",
    "\n",
    "xgb_opt = obj.process(fn_name='xgb_reg', space=xgb_para, trials=Trials(), algo=tpe.suggest, max_evals=100)\n",
    "lgb_opt = obj.process(fn_name='lgb_reg', space=lgb_para, trials=Trials(), algo=tpe.suggest, max_evals=100)\n",
    "ctb_opt = obj.process(fn_name='ctb_reg', space=ctb_para, trials=Trials(), algo=tpe.suggest, max_evals=100)\n",
    "\n",
    "#  See how good the models are:\n",
    "\n",
    "# XGBoost\n",
    "y_pred = xgb_opt.predict(y_test)\n",
    "rmse = math.sqrt(mean_squared_error(y_test, y_pred))\n",
    "r2 = r2_score(y_test, y_pred)\n",
    "print(\"XGBoost\\n\")\n",
    "print(\"RMSE: %f\\n\\n\" % (rmse))\n",
    "print(\"R^2: %f\\n\\n\" % (r2))\n",
    "\n",
    "# LightGBM\n",
    "y_pred = lgb_opt.predict(y_test)\n",
    "rmse = math.sqrt(mean_squared_error(y_test, y_pred))\n",
    "r2 = r2_score(y_test, y_pred)\n",
    "print(\"LGB\\n\")\n",
    "print(\"RMSE: %f\\n\\n\" % (rmse))\n",
    "print(\"R^2: %f\\n\\n\" % (r2))\n",
    "\n",
    "# CatBoost\n",
    "y_pred = ctb_opt.predict(y_test)\n",
    "rmse = math.sqrt(mean_squared_error(y_test, y_pred))\n",
    "r2 = r2_score(y_test, y_pred)\n",
    "print(\"CatBoost\\n\")\n",
    "print(\"RMSE: %f\\n\\n\" % (rmse))\n",
    "print(\"R^2: %f\\n\\n\" % (r2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tuple"
      ]
     },
     "metadata": {},
     "execution_count": 84
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "{'colsample_bytree': 3,\n",
       " 'learning_rate': 1,\n",
       " 'max_depth': 4,\n",
       " 'min_child_weight': 3,\n",
       " 'subsample': 0.9376570860012425}"
      ]
     },
     "metadata": {},
     "execution_count": 84
    }
   ],
   "source": [
    "type(xgb_opt)\n",
    "\n",
    "xgb_opt[0]\n",
    "# lgb_opt\n",
    "# ctb_opt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "tags": [
     "outputPrepend"
    ]
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "ain-rmse:62.94508\n",
      "[581]\teval-rmse:43171.11719\ttrain-rmse:62.02114\n",
      "[582]\teval-rmse:43171.52734\ttrain-rmse:61.60399\n",
      "[583]\teval-rmse:43171.05078\ttrain-rmse:61.61120\n",
      "[584]\teval-rmse:43171.35156\ttrain-rmse:61.55329\n",
      "[585]\teval-rmse:43171.42188\ttrain-rmse:61.08472\n",
      "[586]\teval-rmse:43171.30859\ttrain-rmse:60.80474\n",
      "[587]\teval-rmse:43171.66016\ttrain-rmse:60.80100\n",
      "[588]\teval-rmse:43172.24219\ttrain-rmse:59.85804\n",
      "[589]\teval-rmse:43170.60938\ttrain-rmse:59.30514\n",
      "[590]\teval-rmse:43170.81641\ttrain-rmse:58.95732\n",
      "[591]\teval-rmse:43170.30859\ttrain-rmse:58.48189\n",
      "[592]\teval-rmse:43170.25000\ttrain-rmse:58.27849\n",
      "[593]\teval-rmse:43170.47656\ttrain-rmse:58.02854\n",
      "[594]\teval-rmse:43170.23047\ttrain-rmse:57.24585\n",
      "[595]\teval-rmse:43168.30859\ttrain-rmse:56.96907\n",
      "[596]\teval-rmse:43168.37891\ttrain-rmse:56.47749\n",
      "[597]\teval-rmse:43168.40625\ttrain-rmse:56.56363\n",
      "[598]\teval-rmse:43168.53516\ttrain-rmse:55.96230\n",
      "[599]\teval-rmse:43168.77344\ttrain-rmse:55.46189\n",
      "[600]\teval-rmse:43168.36328\ttrain-rmse:54.58975\n",
      "[601]\teval-rmse:43167.66016\ttrain-rmse:53.82209\n",
      "[602]\teval-rmse:43167.97266\ttrain-rmse:53.16275\n",
      "[603]\teval-rmse:43167.72266\ttrain-rmse:52.61136\n",
      "[604]\teval-rmse:43168.55078\ttrain-rmse:51.91663\n",
      "[605]\teval-rmse:43168.76953\ttrain-rmse:51.78213\n",
      "[606]\teval-rmse:43169.05078\ttrain-rmse:51.55018\n",
      "[607]\teval-rmse:43169.55859\ttrain-rmse:51.18023\n",
      "[608]\teval-rmse:43169.16016\ttrain-rmse:50.40289\n",
      "[609]\teval-rmse:43169.36719\ttrain-rmse:50.04166\n",
      "[610]\teval-rmse:43169.50000\ttrain-rmse:49.85007\n",
      "[611]\teval-rmse:43169.10156\ttrain-rmse:49.77644\n",
      "[612]\teval-rmse:43168.88672\ttrain-rmse:49.50281\n",
      "[613]\teval-rmse:43168.29297\ttrain-rmse:48.68285\n",
      "[614]\teval-rmse:43168.38281\ttrain-rmse:48.28859\n",
      "[615]\teval-rmse:43168.36328\ttrain-rmse:47.75573\n",
      "[616]\teval-rmse:43167.95312\ttrain-rmse:47.14860\n",
      "[617]\teval-rmse:43167.55469\ttrain-rmse:46.54856\n",
      "[618]\teval-rmse:43167.98828\ttrain-rmse:46.52555\n",
      "[619]\teval-rmse:43168.21094\ttrain-rmse:46.31016\n",
      "[620]\teval-rmse:43168.38281\ttrain-rmse:45.96356\n",
      "[621]\teval-rmse:43169.12109\ttrain-rmse:45.58131\n",
      "[622]\teval-rmse:43170.41406\ttrain-rmse:45.01575\n",
      "[623]\teval-rmse:43170.79688\ttrain-rmse:44.61128\n",
      "[624]\teval-rmse:43171.27344\ttrain-rmse:44.30582\n",
      "[625]\teval-rmse:43171.21094\ttrain-rmse:44.30842\n",
      "[626]\teval-rmse:43171.49609\ttrain-rmse:44.41461\n",
      "[627]\teval-rmse:43171.32812\ttrain-rmse:44.32741\n",
      "[628]\teval-rmse:43171.53125\ttrain-rmse:43.50872\n",
      "[629]\teval-rmse:43170.98828\ttrain-rmse:42.94233\n",
      "[630]\teval-rmse:43170.94141\ttrain-rmse:42.89074\n",
      "[631]\teval-rmse:43171.00781\ttrain-rmse:42.54197\n",
      "[632]\teval-rmse:43170.71484\ttrain-rmse:42.33101\n",
      "[633]\teval-rmse:43170.54297\ttrain-rmse:42.46713\n",
      "[634]\teval-rmse:43170.47266\ttrain-rmse:42.35274\n",
      "[635]\teval-rmse:43170.62891\ttrain-rmse:41.96874\n",
      "[636]\teval-rmse:43171.07031\ttrain-rmse:41.51138\n",
      "[637]\teval-rmse:43170.89453\ttrain-rmse:41.20113\n",
      "[638]\teval-rmse:43170.89453\ttrain-rmse:40.81920\n",
      "[639]\teval-rmse:43170.90234\ttrain-rmse:40.09754\n",
      "[640]\teval-rmse:43171.04688\ttrain-rmse:39.83342\n",
      "[641]\teval-rmse:43171.21875\ttrain-rmse:39.55609\n",
      "[642]\teval-rmse:43171.23828\ttrain-rmse:39.41230\n",
      "[643]\teval-rmse:43171.23828\ttrain-rmse:39.38483\n",
      "[644]\teval-rmse:43170.78906\ttrain-rmse:39.01130\n",
      "[645]\teval-rmse:43170.94922\ttrain-rmse:38.77307\n",
      "[646]\teval-rmse:43170.83594\ttrain-rmse:38.53837\n",
      "[647]\teval-rmse:43171.17188\ttrain-rmse:38.11756\n",
      "[648]\teval-rmse:43170.95312\ttrain-rmse:37.87278\n",
      "[649]\teval-rmse:43170.83984\ttrain-rmse:37.52539\n",
      "[650]\teval-rmse:43170.99609\ttrain-rmse:37.28092\n",
      "[651]\teval-rmse:43171.18750\ttrain-rmse:36.95834\n",
      "[652]\teval-rmse:43170.86719\ttrain-rmse:36.54853\n",
      "[653]\teval-rmse:43170.76562\ttrain-rmse:36.43578\n",
      "[654]\teval-rmse:43170.39844\ttrain-rmse:36.16367\n",
      "[655]\teval-rmse:43170.39453\ttrain-rmse:36.16130\n",
      "[656]\teval-rmse:43169.82031\ttrain-rmse:35.93647\n",
      "[657]\teval-rmse:43170.08594\ttrain-rmse:35.72480\n",
      "[658]\teval-rmse:43170.07812\ttrain-rmse:35.18507\n",
      "[659]\teval-rmse:43170.10938\ttrain-rmse:35.06399\n",
      "[660]\teval-rmse:43169.66016\ttrain-rmse:34.57388\n",
      "[661]\teval-rmse:43169.59375\ttrain-rmse:34.47660\n",
      "[662]\teval-rmse:43169.26953\ttrain-rmse:34.17954\n",
      "[663]\teval-rmse:43169.15234\ttrain-rmse:33.96467\n",
      "[664]\teval-rmse:43169.52734\ttrain-rmse:33.97475\n",
      "[665]\teval-rmse:43169.48828\ttrain-rmse:33.63245\n",
      "[666]\teval-rmse:43169.41016\ttrain-rmse:33.90086\n",
      "[667]\teval-rmse:43169.41797\ttrain-rmse:33.64673\n",
      "[668]\teval-rmse:43169.16797\ttrain-rmse:33.49665\n",
      "[669]\teval-rmse:43170.21094\ttrain-rmse:33.06065\n",
      "[670]\teval-rmse:43170.21484\ttrain-rmse:32.66839\n",
      "[671]\teval-rmse:43169.94531\ttrain-rmse:31.96062\n",
      "[672]\teval-rmse:43170.36328\ttrain-rmse:31.40008\n",
      "[673]\teval-rmse:43170.34766\ttrain-rmse:31.43908\n",
      "[674]\teval-rmse:43170.61328\ttrain-rmse:31.22001\n",
      "[675]\teval-rmse:43170.42969\ttrain-rmse:31.36559\n",
      "[676]\teval-rmse:43170.54688\ttrain-rmse:30.91423\n",
      "[677]\teval-rmse:43170.17578\ttrain-rmse:30.79821\n",
      "[678]\teval-rmse:43170.26953\ttrain-rmse:30.41444\n",
      "[679]\teval-rmse:43170.28906\ttrain-rmse:30.24529\n",
      "[680]\teval-rmse:43170.17188\ttrain-rmse:29.82127\n",
      "[681]\teval-rmse:43169.99609\ttrain-rmse:29.26848\n",
      "[682]\teval-rmse:43170.17578\ttrain-rmse:29.07854\n",
      "[683]\teval-rmse:43170.28906\ttrain-rmse:28.73293\n",
      "[684]\teval-rmse:43170.25000\ttrain-rmse:28.62022\n",
      "[685]\teval-rmse:43170.08203\ttrain-rmse:28.10434\n",
      "[686]\teval-rmse:43170.09766\ttrain-rmse:27.84747\n",
      "[687]\teval-rmse:43170.44141\ttrain-rmse:27.76680\n",
      "[688]\teval-rmse:43170.61328\ttrain-rmse:27.58642\n",
      "[689]\teval-rmse:43170.74609\ttrain-rmse:27.67175\n",
      "[690]\teval-rmse:43170.73438\ttrain-rmse:27.12897\n",
      "[691]\teval-rmse:43170.71484\ttrain-rmse:27.36613\n",
      "[692]\teval-rmse:43170.91016\ttrain-rmse:27.05990\n",
      "[693]\teval-rmse:43171.12891\ttrain-rmse:26.93498\n",
      "[694]\teval-rmse:43171.16016\ttrain-rmse:26.65629\n",
      "[695]\teval-rmse:43171.12500\ttrain-rmse:26.57791\n",
      "[696]\teval-rmse:43171.12500\ttrain-rmse:26.41799\n",
      "[697]\teval-rmse:43171.37109\ttrain-rmse:26.23245\n",
      "[698]\teval-rmse:43171.42188\ttrain-rmse:25.80263\n",
      "[699]\teval-rmse:43171.24219\ttrain-rmse:25.65300\n",
      "[700]\teval-rmse:43171.21484\ttrain-rmse:25.47848\n",
      "[701]\teval-rmse:43170.46094\ttrain-rmse:25.48947\n",
      "[702]\teval-rmse:43170.49219\ttrain-rmse:25.24302\n",
      "[703]\teval-rmse:43170.26562\ttrain-rmse:25.18459\n",
      "[704]\teval-rmse:43170.23047\ttrain-rmse:25.29214\n",
      "[705]\teval-rmse:43170.14453\ttrain-rmse:24.77268\n",
      "[706]\teval-rmse:43170.34375\ttrain-rmse:24.57793\n",
      "[707]\teval-rmse:43170.32422\ttrain-rmse:24.49795\n",
      "[708]\teval-rmse:43170.35938\ttrain-rmse:24.09733\n",
      "[709]\teval-rmse:43170.67969\ttrain-rmse:23.86957\n",
      "[710]\teval-rmse:43170.69922\ttrain-rmse:23.79324\n",
      "[711]\teval-rmse:43170.49609\ttrain-rmse:23.61076\n",
      "[712]\teval-rmse:43170.49219\ttrain-rmse:23.57617\n",
      "[713]\teval-rmse:43170.49219\ttrain-rmse:23.48275\n",
      "[714]\teval-rmse:43170.24609\ttrain-rmse:23.35334\n",
      "[715]\teval-rmse:43170.32031\ttrain-rmse:23.23237\n",
      "[716]\teval-rmse:43170.33984\ttrain-rmse:23.17793\n",
      "[717]\teval-rmse:43170.43750\ttrain-rmse:23.08243\n",
      "[718]\teval-rmse:43170.29688\ttrain-rmse:22.88996\n",
      "[719]\teval-rmse:43170.36328\ttrain-rmse:22.70406\n",
      "[720]\teval-rmse:43170.48438\ttrain-rmse:22.71311\n",
      "[721]\teval-rmse:43170.40625\ttrain-rmse:22.62148\n",
      "[722]\teval-rmse:43170.28906\ttrain-rmse:22.40130\n",
      "[723]\teval-rmse:43170.29297\ttrain-rmse:22.23523\n",
      "[724]\teval-rmse:43170.46875\ttrain-rmse:22.06180\n",
      "[725]\teval-rmse:43170.28906\ttrain-rmse:21.92718\n",
      "[726]\teval-rmse:43170.31250\ttrain-rmse:21.89904\n",
      "[727]\teval-rmse:43170.22656\ttrain-rmse:21.95694\n",
      "[728]\teval-rmse:43169.98438\ttrain-rmse:21.68871\n",
      "[729]\teval-rmse:43170.02734\ttrain-rmse:21.58174\n",
      "[730]\teval-rmse:43170.12500\ttrain-rmse:21.52345\n",
      "[731]\teval-rmse:43170.02344\ttrain-rmse:21.46167\n",
      "[732]\teval-rmse:43170.10938\ttrain-rmse:21.40822\n",
      "[733]\teval-rmse:43170.06250\ttrain-rmse:21.30835\n",
      "[734]\teval-rmse:43169.94141\ttrain-rmse:21.24852\n",
      "[735]\teval-rmse:43169.60938\ttrain-rmse:20.94031\n",
      "[736]\teval-rmse:43169.64453\ttrain-rmse:20.97974\n",
      "[737]\teval-rmse:43169.47656\ttrain-rmse:20.66317\n",
      "[738]\teval-rmse:43169.58203\ttrain-rmse:20.44507\n",
      "[739]\teval-rmse:43169.52344\ttrain-rmse:20.03824\n",
      "[740]\teval-rmse:43169.48438\ttrain-rmse:19.86028\n",
      "[741]\teval-rmse:43169.48828\ttrain-rmse:19.77112\n",
      "[742]\teval-rmse:43169.54688\ttrain-rmse:19.71384\n",
      "[743]\teval-rmse:43169.56250\ttrain-rmse:19.48387\n",
      "[744]\teval-rmse:43169.33203\ttrain-rmse:19.36022\n",
      "[745]\teval-rmse:43169.41406\ttrain-rmse:19.52241\n",
      "[746]\teval-rmse:43169.25000\ttrain-rmse:19.66032\n",
      "[747]\teval-rmse:43169.35547\ttrain-rmse:19.05861\n",
      "[748]\teval-rmse:43169.45312\ttrain-rmse:18.96552\n",
      "[749]\teval-rmse:43169.42969\ttrain-rmse:18.82957\n",
      "[750]\teval-rmse:43169.39844\ttrain-rmse:18.69157\n",
      "[751]\teval-rmse:43169.31250\ttrain-rmse:18.59872\n",
      "[752]\teval-rmse:43169.17969\ttrain-rmse:18.50775\n",
      "[753]\teval-rmse:43169.18359\ttrain-rmse:18.40082\n",
      "[754]\teval-rmse:43169.21484\ttrain-rmse:18.41440\n",
      "[755]\teval-rmse:43168.97656\ttrain-rmse:18.07165\n",
      "[756]\teval-rmse:43168.87109\ttrain-rmse:17.94081\n",
      "[757]\teval-rmse:43168.32031\ttrain-rmse:17.64549\n",
      "[758]\teval-rmse:43168.14062\ttrain-rmse:17.44771\n",
      "[759]\teval-rmse:43168.25000\ttrain-rmse:17.44585\n",
      "[760]\teval-rmse:43168.27734\ttrain-rmse:17.04942\n",
      "[761]\teval-rmse:43168.26953\ttrain-rmse:16.95393\n",
      "[762]\teval-rmse:43168.35547\ttrain-rmse:16.88857\n",
      "[763]\teval-rmse:43167.87109\ttrain-rmse:16.73029\n",
      "[764]\teval-rmse:43167.61719\ttrain-rmse:16.81967\n",
      "[765]\teval-rmse:43167.50781\ttrain-rmse:16.56236\n",
      "[766]\teval-rmse:43167.43359\ttrain-rmse:16.49739\n",
      "[767]\teval-rmse:43167.41797\ttrain-rmse:16.37069\n",
      "[768]\teval-rmse:43167.27734\ttrain-rmse:16.17793\n",
      "[769]\teval-rmse:43167.48828\ttrain-rmse:16.05360\n",
      "[770]\teval-rmse:43167.46094\ttrain-rmse:16.12666\n",
      "[771]\teval-rmse:43167.46484\ttrain-rmse:15.88576\n",
      "[772]\teval-rmse:43167.45703\ttrain-rmse:15.82861\n",
      "[773]\teval-rmse:43167.48438\ttrain-rmse:15.68877\n",
      "[774]\teval-rmse:43167.46484\ttrain-rmse:15.61076\n",
      "[775]\teval-rmse:43167.49219\ttrain-rmse:15.46608\n",
      "[776]\teval-rmse:43167.52734\ttrain-rmse:15.31345\n",
      "[777]\teval-rmse:43167.53125\ttrain-rmse:15.27004\n",
      "[778]\teval-rmse:43167.59375\ttrain-rmse:15.23520\n",
      "[779]\teval-rmse:43167.73438\ttrain-rmse:15.11023\n",
      "[780]\teval-rmse:43167.69141\ttrain-rmse:15.05125\n",
      "[781]\teval-rmse:43167.76562\ttrain-rmse:15.15245\n",
      "[782]\teval-rmse:43167.35156\ttrain-rmse:15.06623\n",
      "[783]\teval-rmse:43167.55078\ttrain-rmse:14.74801\n",
      "[784]\teval-rmse:43167.42578\ttrain-rmse:14.58033\n",
      "[785]\teval-rmse:43167.30469\ttrain-rmse:14.42752\n",
      "[786]\teval-rmse:43167.42969\ttrain-rmse:14.35066\n",
      "[787]\teval-rmse:43167.34375\ttrain-rmse:14.24954\n",
      "[788]\teval-rmse:43167.41406\ttrain-rmse:14.38876\n",
      "[789]\teval-rmse:43167.39062\ttrain-rmse:14.18428\n",
      "[790]\teval-rmse:43167.62500\ttrain-rmse:14.10116\n",
      "[791]\teval-rmse:43167.49609\ttrain-rmse:14.02835\n",
      "[792]\teval-rmse:43167.35547\ttrain-rmse:13.94417\n",
      "[793]\teval-rmse:43167.65625\ttrain-rmse:13.86845\n",
      "[794]\teval-rmse:43167.69141\ttrain-rmse:13.64457\n",
      "[795]\teval-rmse:43167.72266\ttrain-rmse:13.60058\n",
      "[796]\teval-rmse:43167.69141\ttrain-rmse:13.50432\n",
      "[797]\teval-rmse:43167.56250\ttrain-rmse:13.39755\n",
      "[798]\teval-rmse:43167.60547\ttrain-rmse:13.38548\n",
      "[799]\teval-rmse:43167.64062\ttrain-rmse:13.22674\n",
      "[800]\teval-rmse:43167.60938\ttrain-rmse:13.14379\n",
      "[801]\teval-rmse:43168.00391\ttrain-rmse:12.76346\n",
      "[802]\teval-rmse:43167.96484\ttrain-rmse:12.76538\n",
      "[803]\teval-rmse:43167.93750\ttrain-rmse:12.69145\n",
      "[804]\teval-rmse:43167.96094\ttrain-rmse:12.61386\n",
      "[805]\teval-rmse:43167.96875\ttrain-rmse:12.50520\n",
      "[806]\teval-rmse:43168.03906\ttrain-rmse:12.44104\n",
      "[807]\teval-rmse:43168.06250\ttrain-rmse:12.36491\n",
      "[808]\teval-rmse:43167.98828\ttrain-rmse:12.26086\n",
      "[809]\teval-rmse:43168.05469\ttrain-rmse:12.16225\n",
      "[810]\teval-rmse:43168.09766\ttrain-rmse:12.18098\n",
      "[811]\teval-rmse:43168.13672\ttrain-rmse:12.09691\n",
      "[812]\teval-rmse:43168.14062\ttrain-rmse:11.97915\n",
      "[813]\teval-rmse:43168.14453\ttrain-rmse:11.96702\n",
      "[814]\teval-rmse:43168.05469\ttrain-rmse:11.87059\n",
      "[815]\teval-rmse:43167.98438\ttrain-rmse:11.81306\n",
      "[816]\teval-rmse:43167.98047\ttrain-rmse:11.76725\n",
      "[817]\teval-rmse:43168.08594\ttrain-rmse:11.60673\n",
      "[818]\teval-rmse:43168.11328\ttrain-rmse:11.54949\n",
      "[819]\teval-rmse:43168.08203\ttrain-rmse:11.48682\n",
      "[820]\teval-rmse:43168.14453\ttrain-rmse:11.40595\n",
      "[821]\teval-rmse:43168.36719\ttrain-rmse:11.38313\n",
      "[822]\teval-rmse:43168.35156\ttrain-rmse:11.30227\n",
      "[823]\teval-rmse:43168.41016\ttrain-rmse:11.23620\n",
      "[824]\teval-rmse:43168.26953\ttrain-rmse:11.07556\n",
      "[825]\teval-rmse:43168.16016\ttrain-rmse:11.01190\n",
      "[826]\teval-rmse:43168.10938\ttrain-rmse:10.85600\n",
      "[827]\teval-rmse:43168.20312\ttrain-rmse:10.53295\n",
      "[828]\teval-rmse:43168.49219\ttrain-rmse:10.36449\n",
      "[829]\teval-rmse:43168.48828\ttrain-rmse:10.32476\n",
      "[830]\teval-rmse:43168.27734\ttrain-rmse:10.21677\n",
      "[831]\teval-rmse:43168.46875\ttrain-rmse:10.03628\n",
      "[832]\teval-rmse:43168.41406\ttrain-rmse:10.00892\n",
      "[833]\teval-rmse:43168.43750\ttrain-rmse:9.90640\n",
      "[834]\teval-rmse:43168.51172\ttrain-rmse:9.84248\n",
      "[835]\teval-rmse:43168.31641\ttrain-rmse:9.67048\n",
      "[836]\teval-rmse:43168.29297\ttrain-rmse:9.62823\n",
      "[837]\teval-rmse:43168.25000\ttrain-rmse:9.56873\n",
      "[838]\teval-rmse:43168.19141\ttrain-rmse:9.48512\n",
      "[839]\teval-rmse:43168.15234\ttrain-rmse:9.44393\n",
      "[840]\teval-rmse:43168.23438\ttrain-rmse:9.37463\n",
      "[841]\teval-rmse:43168.23438\ttrain-rmse:9.30293\n",
      "[842]\teval-rmse:43168.24609\ttrain-rmse:9.21876\n",
      "[843]\teval-rmse:43168.21094\ttrain-rmse:9.20318\n",
      "[844]\teval-rmse:43168.23828\ttrain-rmse:9.08355\n",
      "[845]\teval-rmse:43168.13672\ttrain-rmse:8.97413\n",
      "[846]\teval-rmse:43168.13672\ttrain-rmse:8.96673\n",
      "[847]\teval-rmse:43168.11719\ttrain-rmse:8.84860\n",
      "[848]\teval-rmse:43168.12109\ttrain-rmse:8.88457\n",
      "[849]\teval-rmse:43168.24219\ttrain-rmse:8.73156\n",
      "[850]\teval-rmse:43168.19922\ttrain-rmse:8.73708\n",
      "[851]\teval-rmse:43168.21094\ttrain-rmse:8.68205\n",
      "[852]\teval-rmse:43168.26953\ttrain-rmse:8.57322\n",
      "[853]\teval-rmse:43168.19531\ttrain-rmse:8.42699\n",
      "[854]\teval-rmse:43168.16016\ttrain-rmse:8.38711\n",
      "[855]\teval-rmse:43168.12500\ttrain-rmse:8.31090\n",
      "[856]\teval-rmse:43168.08594\ttrain-rmse:8.29490\n",
      "[857]\teval-rmse:43168.09766\ttrain-rmse:8.24510\n",
      "[858]\teval-rmse:43168.10547\ttrain-rmse:8.19676\n",
      "[859]\teval-rmse:43168.21094\ttrain-rmse:8.08815\n",
      "[860]\teval-rmse:43168.21094\ttrain-rmse:8.07002\n",
      "[861]\teval-rmse:43168.22656\ttrain-rmse:8.04308\n",
      "[862]\teval-rmse:43168.20312\ttrain-rmse:8.00978\n",
      "[863]\teval-rmse:43168.21875\ttrain-rmse:7.94502\n",
      "[864]\teval-rmse:43168.29688\ttrain-rmse:7.89599\n",
      "[865]\teval-rmse:43168.12891\ttrain-rmse:7.86888\n",
      "[866]\teval-rmse:43168.13672\ttrain-rmse:7.83058\n",
      "[867]\teval-rmse:43168.15234\ttrain-rmse:7.77085\n",
      "[868]\teval-rmse:43168.08594\ttrain-rmse:7.71923\n",
      "[869]\teval-rmse:43168.04688\ttrain-rmse:7.61822\n",
      "[870]\teval-rmse:43168.24219\ttrain-rmse:7.45434\n",
      "[871]\teval-rmse:43168.28516\ttrain-rmse:7.34719\n",
      "[872]\teval-rmse:43168.28125\ttrain-rmse:7.34220\n",
      "[873]\teval-rmse:43168.21875\ttrain-rmse:7.22826\n",
      "[874]\teval-rmse:43168.23828\ttrain-rmse:7.22299\n",
      "[875]\teval-rmse:43168.13281\ttrain-rmse:7.23890\n",
      "[876]\teval-rmse:43168.13281\ttrain-rmse:7.12551\n",
      "[877]\teval-rmse:43168.03906\ttrain-rmse:7.07003\n",
      "[878]\teval-rmse:43168.01562\ttrain-rmse:6.99186\n",
      "[879]\teval-rmse:43168.02344\ttrain-rmse:6.98092\n",
      "[880]\teval-rmse:43168.07422\ttrain-rmse:6.88665\n",
      "[881]\teval-rmse:43168.08594\ttrain-rmse:6.88064\n",
      "[882]\teval-rmse:43168.08203\ttrain-rmse:6.84670\n",
      "[883]\teval-rmse:43168.05469\ttrain-rmse:6.72561\n",
      "[884]\teval-rmse:43168.12109\ttrain-rmse:6.69851\n",
      "[885]\teval-rmse:43168.08203\ttrain-rmse:6.66190\n",
      "[886]\teval-rmse:43168.16016\ttrain-rmse:6.59542\n",
      "[887]\teval-rmse:43168.20703\ttrain-rmse:6.55974\n",
      "[888]\teval-rmse:43168.16016\ttrain-rmse:6.53577\n",
      "[889]\teval-rmse:43168.17188\ttrain-rmse:6.45692\n",
      "[890]\teval-rmse:43168.17969\ttrain-rmse:6.38329\n",
      "[891]\teval-rmse:43168.17188\ttrain-rmse:6.35227\n",
      "[892]\teval-rmse:43168.08984\ttrain-rmse:6.28709\n",
      "[893]\teval-rmse:43168.12109\ttrain-rmse:6.26963\n",
      "[894]\teval-rmse:43168.03125\ttrain-rmse:6.19425\n",
      "[895]\teval-rmse:43168.11328\ttrain-rmse:6.19464\n",
      "[896]\teval-rmse:43168.12891\ttrain-rmse:6.13168\n",
      "[897]\teval-rmse:43168.06250\ttrain-rmse:6.11040\n",
      "[898]\teval-rmse:43168.08984\ttrain-rmse:6.05179\n",
      "[899]\teval-rmse:43168.05469\ttrain-rmse:6.02634\n",
      "[900]\teval-rmse:43168.08594\ttrain-rmse:5.97550\n",
      "[901]\teval-rmse:43168.07422\ttrain-rmse:5.96319\n",
      "[902]\teval-rmse:43168.13281\ttrain-rmse:5.80624\n",
      "[903]\teval-rmse:43168.00391\ttrain-rmse:5.78178\n",
      "[904]\teval-rmse:43168.03516\ttrain-rmse:5.75696\n",
      "[905]\teval-rmse:43168.09375\ttrain-rmse:5.67659\n",
      "[906]\teval-rmse:43168.09375\ttrain-rmse:5.63466\n",
      "[907]\teval-rmse:43168.14453\ttrain-rmse:5.59689\n",
      "[908]\teval-rmse:43168.14453\ttrain-rmse:5.57713\n",
      "[909]\teval-rmse:43168.18750\ttrain-rmse:5.45033\n",
      "[910]\teval-rmse:43168.21875\ttrain-rmse:5.38448\n",
      "[911]\teval-rmse:43168.22656\ttrain-rmse:5.32799\n",
      "[912]\teval-rmse:43168.17188\ttrain-rmse:5.30822\n",
      "[913]\teval-rmse:43168.18359\ttrain-rmse:5.28196\n",
      "[914]\teval-rmse:43168.11719\ttrain-rmse:5.29358\n",
      "[915]\teval-rmse:43168.14062\ttrain-rmse:5.30317\n",
      "[916]\teval-rmse:43168.10938\ttrain-rmse:5.24868\n",
      "[917]\teval-rmse:43168.15234\ttrain-rmse:5.23621\n",
      "[918]\teval-rmse:43168.17969\ttrain-rmse:5.19265\n",
      "[919]\teval-rmse:43168.16406\ttrain-rmse:5.17483\n",
      "[920]\teval-rmse:43168.10156\ttrain-rmse:5.13557\n",
      "[921]\teval-rmse:43168.08594\ttrain-rmse:5.09793\n",
      "[922]\teval-rmse:43168.06641\ttrain-rmse:5.08254\n",
      "[923]\teval-rmse:43168.05469\ttrain-rmse:5.07632\n",
      "[924]\teval-rmse:43168.05469\ttrain-rmse:5.05383\n",
      "[925]\teval-rmse:43168.05859\ttrain-rmse:5.02089\n",
      "[926]\teval-rmse:43168.06250\ttrain-rmse:5.00401\n",
      "[927]\teval-rmse:43167.96875\ttrain-rmse:4.93564\n",
      "[928]\teval-rmse:43167.92188\ttrain-rmse:4.85091\n",
      "[929]\teval-rmse:43167.98047\ttrain-rmse:4.84912\n",
      "[930]\teval-rmse:43167.97656\ttrain-rmse:4.80778\n",
      "[931]\teval-rmse:43168.01562\ttrain-rmse:4.77412\n",
      "[932]\teval-rmse:43167.93359\ttrain-rmse:4.71201\n",
      "[933]\teval-rmse:43167.97656\ttrain-rmse:4.67072\n",
      "[934]\teval-rmse:43167.95703\ttrain-rmse:4.60611\n",
      "[935]\teval-rmse:43167.93359\ttrain-rmse:4.60605\n",
      "[936]\teval-rmse:43167.92188\ttrain-rmse:4.57092\n",
      "[937]\teval-rmse:43167.94141\ttrain-rmse:4.58684\n",
      "[938]\teval-rmse:43167.96484\ttrain-rmse:4.57128\n",
      "[939]\teval-rmse:43167.97266\ttrain-rmse:4.52905\n",
      "[940]\teval-rmse:43167.99609\ttrain-rmse:4.50744\n",
      "[941]\teval-rmse:43168.01953\ttrain-rmse:4.46812\n",
      "[942]\teval-rmse:43168.02734\ttrain-rmse:4.44154\n",
      "[943]\teval-rmse:43168.05078\ttrain-rmse:4.42877\n",
      "[944]\teval-rmse:43168.04688\ttrain-rmse:4.38893\n",
      "[945]\teval-rmse:43168.04688\ttrain-rmse:4.36366\n",
      "[946]\teval-rmse:43167.93750\ttrain-rmse:4.41235\n",
      "[947]\teval-rmse:43167.93750\ttrain-rmse:4.33413\n",
      "[948]\teval-rmse:43167.93359\ttrain-rmse:4.32406\n",
      "[949]\teval-rmse:43167.92188\ttrain-rmse:4.30440\n",
      "[950]\teval-rmse:43167.97266\ttrain-rmse:4.27850\n",
      "[951]\teval-rmse:43167.94922\ttrain-rmse:4.29250\n",
      "[952]\teval-rmse:43167.93750\ttrain-rmse:4.25365\n",
      "[953]\teval-rmse:43167.96875\ttrain-rmse:4.21410\n",
      "[954]\teval-rmse:43167.96094\ttrain-rmse:4.19240\n",
      "[955]\teval-rmse:43167.97266\ttrain-rmse:4.20927\n",
      "[956]\teval-rmse:43167.97266\ttrain-rmse:4.18028\n",
      "[957]\teval-rmse:43167.98828\ttrain-rmse:4.15750\n",
      "[958]\teval-rmse:43168.00781\ttrain-rmse:4.15007\n",
      "[959]\teval-rmse:43167.98828\ttrain-rmse:4.10846\n",
      "[960]\teval-rmse:43168.00781\ttrain-rmse:4.07668\n",
      "[961]\teval-rmse:43167.94531\ttrain-rmse:4.06302\n",
      "[962]\teval-rmse:43168.03516\ttrain-rmse:4.01573\n",
      "[963]\teval-rmse:43168.04688\ttrain-rmse:3.98818\n",
      "[964]\teval-rmse:43168.06250\ttrain-rmse:3.96453\n",
      "[965]\teval-rmse:43168.05078\ttrain-rmse:3.95641\n",
      "[966]\teval-rmse:43168.05078\ttrain-rmse:3.93826\n",
      "[967]\teval-rmse:43168.05078\ttrain-rmse:3.92513\n",
      "[968]\teval-rmse:43168.03516\ttrain-rmse:3.93104\n",
      "[969]\teval-rmse:43168.02344\ttrain-rmse:3.89075\n",
      "[970]\teval-rmse:43168.02734\ttrain-rmse:3.88238\n",
      "[971]\teval-rmse:43168.02344\ttrain-rmse:3.85142\n",
      "[972]\teval-rmse:43168.03906\ttrain-rmse:3.82741\n",
      "[973]\teval-rmse:43168.02734\ttrain-rmse:3.80692\n",
      "[974]\teval-rmse:43168.02344\ttrain-rmse:3.78414\n",
      "[975]\teval-rmse:43168.03906\ttrain-rmse:3.80712\n",
      "[976]\teval-rmse:43168.03516\ttrain-rmse:3.75885\n",
      "[977]\teval-rmse:43168.07031\ttrain-rmse:3.72819\n",
      "[978]\teval-rmse:43168.09375\ttrain-rmse:3.71241\n",
      "[979]\teval-rmse:43168.07812\ttrain-rmse:3.66756\n",
      "[980]\teval-rmse:43168.07812\ttrain-rmse:3.65573\n",
      "[981]\teval-rmse:43168.07812\ttrain-rmse:3.66487\n",
      "[982]\teval-rmse:43168.08203\ttrain-rmse:3.63027\n",
      "[983]\teval-rmse:43168.08594\ttrain-rmse:3.64870\n",
      "[984]\teval-rmse:43168.08984\ttrain-rmse:3.61547\n",
      "[985]\teval-rmse:43168.10547\ttrain-rmse:3.59346\n",
      "[986]\teval-rmse:43168.08594\ttrain-rmse:3.53768\n",
      "[987]\teval-rmse:43168.08984\ttrain-rmse:3.51282\n",
      "[988]\teval-rmse:43168.07422\ttrain-rmse:3.48501\n",
      "[989]\teval-rmse:43168.09375\ttrain-rmse:3.43328\n",
      "[990]\teval-rmse:43168.09766\ttrain-rmse:3.42080\n",
      "[991]\teval-rmse:43168.08594\ttrain-rmse:3.44474\n",
      "[992]\teval-rmse:43168.08984\ttrain-rmse:3.40136\n",
      "[993]\teval-rmse:43168.10156\ttrain-rmse:3.38970\n",
      "[994]\teval-rmse:43168.08984\ttrain-rmse:3.37466\n",
      "[995]\teval-rmse:43168.05469\ttrain-rmse:3.38045\n",
      "[996]\teval-rmse:43168.03906\ttrain-rmse:3.35319\n",
      "[997]\teval-rmse:43168.04688\ttrain-rmse:3.34621\n",
      "[998]\teval-rmse:43168.04297\ttrain-rmse:3.30415\n",
      "[999]\teval-rmse:43168.02344\ttrain-rmse:3.28742\n",
      "RMSE: 43168.024663\n",
      "\n",
      "\n",
      "R^2: 0.730159\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "#######################################\n",
    "# Now train model with the \"ideal\" parameters\n",
    "#####################################\n",
    "\n",
    "\n",
    "#   Specify parameters\n",
    "\n",
    "evallist = [(dtest, 'eval'), (dtrain, 'train')]\n",
    "\n",
    "evals_result = {}\n",
    "\n",
    "params = {\n",
    "    'booster': 'gbtree', # gbtree beats gblinear - don't feel forced to fit linear to this dataset\n",
    "    'objective': 'reg:squarederror',\n",
    "#    'colsample_bytree': 3,\n",
    "    'max_depth': 4,\n",
    "    'min_child_weight': 3,\n",
    "    'subsample': 0.9376570860012425,\n",
    "    'learning_rate': 1,\n",
    "#     'reg_alpha': 0,  #  L1 regularisation\n",
    "#     'reg_lambda': 0,  # L2 regularisation\n",
    "    'eval_metric': 'rmse',  # Used by Kaggle house price competition - this parameter is used for VALIDATION data\n",
    "#    'evals': evallist,\n",
    "    'verbose_eval': 1,  # If integer, value is printed every x boosting stages\n",
    "    'early_stopping_rounds': 10,\n",
    "    'verbosity': 1,  # 0 is silent, 3 is debug\n",
    "}\n",
    "\n",
    "#  Train the model\n",
    "num_round = 1000\n",
    "bst = xgb.train(params, dtrain, num_round, evallist)\n",
    "\n",
    "# Evaluate how good the model is\n",
    "y_pred = bst.predict(dtest)\n",
    "rmse = math.sqrt(mean_squared_error(y_test, y_pred))\n",
    "r2 = r2_score(y_test, y_pred)\n",
    "print(\"RMSE: %f\\n\\n\" % (rmse))\n",
    "print(\"R^2: %f\\n\\n\" % (r2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define objective function for hyperopt\n",
    "\n",
    "def objective(params):\n",
    "    model = xgb(**params)\n",
    "    num_round=10\n",
    "    model = xgb.train(\n",
    "        params,\n",
    "        dtrain,\n",
    "        num_boost_round=num_round,\n",
    "        evals=[(dtest, \"Test\")],\n",
    "        early_stopping_rounds=10\n",
    "        )\n",
    "    # model = XGBRegressor(**params)\n",
    "\n",
    "    # model.fit(X_train, y_train, eval_set=[(X_train, y_train), (X_test, y_test)],\n",
    "    #           verbose=False, early_stopping_rounds=10)\n",
    "\n",
    "    # y_pred = model.predict(X_test)\n",
    "    # score = math.sqrt(mean_squared_error(y_test, y_pred))\n",
    "    # print('RMSE: {:0.2f}'.format(score))\n",
    "    # return {'loss': score, 'status': STATUS_OK}\n",
    "\n",
    "\n",
    "    # num_round = 1000\n",
    "\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now run the actual pyperopt optimisation\n",
    "\n",
    "trials = Trials()\n",
    "best_params = fmin(\n",
    "    fn=objective, \n",
    "    space=space, \n",
    "    algo=tpe.suggest, \n",
    "    max_evals=2, #  Tune this: maximum no of evaluations\n",
    "    trials=trials #  trials object that keeps track of the sample results\n",
    "    )\n",
    "\n",
    "print(best_params)\n",
    "\n",
    "# Extract and plot the trials \n",
    "x = trials.vals['x']\n",
    "y = [x['loss'] for x in trials.results]\n",
    "plt.scatter(x, y)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param = {'max_depth': 2, 'eta': 1, 'objective': 'reg:squarederror'}\n",
    "param['nthread'] = 4\n",
    "param[\n",
    "    'eval_metric'] = 'rmse'  #  This is what the Kaggel house price competition uses to score\n",
    "\n",
    "#  Instantiate an XGBoost regressor object\n",
    "xg_reg = xgb(booster='gblinear',\n",
    "             objective='reg:squarederror',\n",
    "             colsample_bytree=0.3,\n",
    "             learning_rate=0.3,\n",
    "             max_depth=10,\n",
    "             alpha=0,\n",
    "             n_estimators=1000,\n",
    "             eval_metric='rmse')\n",
    "\n",
    "# num_round = 15\n",
    "# initial_trees = xgb.train(param, dtrain, num_round)\n",
    "\n",
    "xg_reg.fit(X_train, y_train)\n",
    "\n",
    "# dtest = xgb.DMatrix(X_test)\n",
    "# y_pred = bst.predict(dtest)\n",
    "\n",
    "y_pred = xg_reg.predict(X_test)\n",
    "\n",
    "math.sqrt(mean_squared_error(y_test, y_pred))\n",
    "\n",
    "#model = XGBRegressor(space)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit(X_train,\n",
    "          y_train,\n",
    "          eval_set=[(X_train, y_train), (X_test, y_test)],\n",
    "          verbose=False,\n",
    "          early_stopping_rounds=10)\n",
    "\n",
    "y_pred = model.predict(X_test)\n",
    "score = math.sqrt(mean_squared_error(y_test, y_pred))\n",
    "print('RMSE: {:0.2f}'.format(score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hyperopt.plotting import main_plot_history\n",
    "main_plot_history(trials)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model.predict(X_test)\n",
    "print(\"R^2\\n\\n\")\n",
    "r2_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "accuracies = cross_val_score(estimator=model, X=X_train, y=y_train, cv=10)\n",
    "print(\"Accuracy: {:.2f} %\".format(accuracies.mean() * 100))\n",
    "# print(\"Standard Deviation: {:.2f} %\".format(accuracies.std()*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "xgb.plot_tree(model, num_trees=0)\n",
    "plt.rcParams['figure.figsize'] = [500, 200]\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Another optimisation effort\n",
    "From https://towardsdatascience.com/how-to-get-started-on-kaggle-competitions-68b91e3e803a\n",
    "and https://www.kaggle.com/tilii7/hyperparameter-grid-search-with-xgboost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# parameters to be searched over\n",
    "param_grid = {\n",
    "    'model__n_estimators': [10, 50, 100, 200, 400, 600],\n",
    "    'model__max_depth': [2, 3, 5, 7, 10],\n",
    "    'model__min_child_weight': [0.0001, 0.001, 0.01],\n",
    "    'model__learning_rate': [0.01, 0.1, 0.5, 1]\n",
    "}\n",
    "\n",
    "xg_reg = xgb.XGBRegressor(booster='gblinear',\n",
    "                          objective='reg:squarederror',\n",
    "                          colsample_bytree=0.3,\n",
    "                          learning_rate=0.3,\n",
    "                          max_depth=10,\n",
    "                          alpha=0,\n",
    "                          n_estimators=1000,\n",
    "                          eval_metric='rmse')\n",
    "\n",
    "# find the best parameter\n",
    "kfold = KFold(shuffle=True, random_state=0)\n",
    "grid_search = GridSearchCV(xg_reg,\n",
    "                           param_grid,\n",
    "                           scoring='neg_root_mean_squared_error',\n",
    "                           cv=kfold,\n",
    "                           n_jobs=-1)\n",
    "grid_result = grid_search.fit(X_train, y_train)\n",
    "\n",
    "# folds = 3\n",
    "# param_comb = 5\n",
    "\n",
    "# skf = StratifiedKFold(n_splits=folds, shuffle = True, random_state = 1001)\n",
    "\n",
    "# random_search = RandomizedSearchCV(xgb, param_distributions=params, n_iter=param_comb, scoring='roc_auc', n_jobs=4, cv=skf.split(X,Y), verbose=3, random_state=1001 )\n",
    "\n",
    "# # Here we go\n",
    "# start_time = timer(None) # timing starts from this point for \"start_time\" variable\n",
    "# random_search.fit(X, Y)\n",
    "# timer(start_time) # timing ends here for \"start_time\" variable\n",
    "\n",
    "#  Print gridsearch results and save to file\n",
    "print('\\n All results:')\n",
    "print(grid_search.cv_results_)\n",
    "print('\\n Best estimator:')\n",
    "print(grid_search.best_estimator_)\n",
    "print(\n",
    "    '\\n Best normalized gini score for %d-fold search with %d parameter combinations:'\n",
    "    % (folds, param_comb))\n",
    "print(grid_search.best_score_ * 2 - 1)\n",
    "print('\\n Best hyperparameters:')\n",
    "print(grid_search.best_params_)\n",
    "results = pd.DataFrame(grid_search.cv_results_)\n",
    "results.to_csv('xgb-gridsearchcv-results-01.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#  Need to figure out how to extract the optimised model, then run the cells below to test how good it is\n",
    "######  Looks like it is in \"best estimator\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model.predict(X_test)\n",
    "print(\"R^2\\n\\n\")\n",
    "r2_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "accuracies = cross_val_score(estimator=model, X=X_train, y=y_train, cv=10)\n",
    "print(\"Accuracy: {:.2f} %\".format(accuracies.mean() * 100))\n",
    "# print(\"Standard Deviation: {:.2f} %\".format(accuracies.std()*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "xgb.plot_tree(model, num_trees=0)\n",
    "plt.rcParams['figure.figsize'] = [500, 200]\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "authorship_tag": "ABX9TyPuuFI9DETSuUNcfBKFNz89",
   "collapsed_sections": [],
   "name": "xg_boost.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6-final"
  },
  "toc-autonumbering": true
 },
 "nbformat": 4,
 "nbformat_minor": 4
}