{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1432970a",
   "metadata": {},
   "source": [
    "# 🤖 ML Pipeline HOÀN CHỈNH: Dự Đoán Số Lượng Nhân Sự\n",
    "\n",
    "**Mục tiêu:** Dự đoán số lượng nhân sự (`so_luong`) cần thiết cho mỗi ca làm việc\n",
    "\n",
    "**Dataset:** FINAL_DATASET_WITH_TEXT.xlsx (454 rows × 51 columns)\n",
    "\n",
    "**Features:** 47 features (không bao gồm 2 cột text)\n",
    "\n",
    "**Target:** `so_luong` (0-64 nhân sự)\n",
    "\n",
    "---\n",
    "\n",
    "## 📋 Pipeline Hoàn Chỉnh:\n",
    "\n",
    "1. ✅ **Import Libraries & Load Data** - Thiết lập môi trường\n",
    "2. ✅ **Data Cleaning & Preprocessing** - Làm sạch và chuẩn hóa dữ liệu\n",
    "3. ✅ **Feature Engineering** - Tạo features mới\n",
    "4. ✅ **Train/Val/Test Split** - Chia dữ liệu (60/20/20)\n",
    "5. ✅ **K-Fold Cross-Validation** - Đánh giá độ ổn định model\n",
    "6. ✅ **Feature Selection** - Tìm best features (RF + MI + Correlation)\n",
    "7. ✅ **Final Training** - Train model với best features\n",
    "8. ✅ **Test Evaluation** - Đánh giá trên test set\n",
    "9. ✅ **Save Model** - Lưu model + artifacts\n",
    "\n",
    "---\n",
    "\n",
    "**Version:** 2.0 - Complete Pipeline  \n",
    "**Date:** January 8, 2026"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "552b4010",
   "metadata": {},
   "source": [
    "# 1️⃣ Import Libraries & Load Data\n",
    "\n",
    "**Mục đích:** Thiết lập môi trường, import các thư viện cần thiết, và load dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "10157e9c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Libraries imported successfully!\n",
      "📅 Date: 2026-01-08 17:52:31\n"
     ]
    }
   ],
   "source": [
    "# Import essential libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import warnings\n",
    "import os\n",
    "import pickle\n",
    "import json\n",
    "from datetime import datetime\n",
    "\n",
    "# Machine Learning libraries\n",
    "from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV, KFold\n",
    "from sklearn.preprocessing import StandardScaler, LabelEncoder, OneHotEncoder\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor\n",
    "from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score\n",
    "from sklearn.feature_selection import SelectKBest, mutual_info_regression, RFE\n",
    "import xgboost as xgb\n",
    "import lightgbm as lgb\n",
    "\n",
    "# Configuration\n",
    "warnings.filterwarnings('ignore')\n",
    "plt.style.use('seaborn-v0_8-darkgrid')\n",
    "sns.set_palette(\"husl\")\n",
    "pd.set_option('display.max_columns', None)\n",
    "pd.set_option('display.max_rows', 100)\n",
    "\n",
    "# Random seed for reproducibility\n",
    "RANDOM_STATE = 42\n",
    "np.random.seed(RANDOM_STATE)\n",
    "\n",
    "# Create folders for outputs\n",
    "os.makedirs('data/cleaned', exist_ok=True)\n",
    "os.makedirs('data/splits', exist_ok=True)\n",
    "os.makedirs('models', exist_ok=True)\n",
    "os.makedirs('plots', exist_ok=True)\n",
    "\n",
    "print(\"✅ Libraries imported successfully!\")\n",
    "print(f\"📅 Date: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "8ecb54a2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "📂 Loading dataset...\n",
      "✅ Dataset loaded successfully!\n",
      "📊 Shape: (454, 51)\n",
      "📋 Columns: ['ma_dia_diem', 'all_task_normal', 'all_task_dinhky', 'loai_ca', 'bat_dau', 'ket_thuc', 'tong_gio_lam', 'so_ca_cua_toa', 'so_luong', 'num_tasks']... (showing first 10)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ma_dia_diem</th>\n",
       "      <th>all_task_normal</th>\n",
       "      <th>all_task_dinhky</th>\n",
       "      <th>loai_ca</th>\n",
       "      <th>bat_dau</th>\n",
       "      <th>ket_thuc</th>\n",
       "      <th>tong_gio_lam</th>\n",
       "      <th>so_ca_cua_toa</th>\n",
       "      <th>so_luong</th>\n",
       "      <th>num_tasks</th>\n",
       "      <th>num_cleaning_tasks</th>\n",
       "      <th>num_trash_collection_tasks</th>\n",
       "      <th>num_monitoring_tasks</th>\n",
       "      <th>num_room_cleaning_tasks</th>\n",
       "      <th>num_deep_cleaning_tasks</th>\n",
       "      <th>num_maintenance_tasks</th>\n",
       "      <th>num_support_tasks</th>\n",
       "      <th>num_other_tasks</th>\n",
       "      <th>num_wc_tasks</th>\n",
       "      <th>num_hallway_tasks</th>\n",
       "      <th>num_lobby_tasks</th>\n",
       "      <th>num_patient_room_tasks</th>\n",
       "      <th>num_clinic_room_tasks</th>\n",
       "      <th>num_surgery_room_tasks</th>\n",
       "      <th>num_outdoor_tasks</th>\n",
       "      <th>num_elevator_tasks</th>\n",
       "      <th>num_office_tasks</th>\n",
       "      <th>num_technical_room_tasks</th>\n",
       "      <th>cleaning_ratio</th>\n",
       "      <th>trash_collection_ratio</th>\n",
       "      <th>monitoring_ratio</th>\n",
       "      <th>room_cleaning_ratio</th>\n",
       "      <th>area_diversity</th>\n",
       "      <th>task_complexity_score</th>\n",
       "      <th>loai_hinh</th>\n",
       "      <th>ten_toa_thap</th>\n",
       "      <th>muc_do_luu_luong</th>\n",
       "      <th>so_tang</th>\n",
       "      <th>so_cua_thang_may</th>\n",
       "      <th>dien_tich_ngoai_canh</th>\n",
       "      <th>dien_tich_sanh</th>\n",
       "      <th>dien_tich_hanh_lang</th>\n",
       "      <th>dien_tich_wc</th>\n",
       "      <th>dien_tich_phong</th>\n",
       "      <th>dien_tich_tham</th>\n",
       "      <th>doc_ham</th>\n",
       "      <th>vien_phan_quang</th>\n",
       "      <th>op_tuong</th>\n",
       "      <th>op_chan_tuong</th>\n",
       "      <th>ranh_thoat_nuoc</th>\n",
       "      <th>dien_tich_kinh</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>115-2</td>\n",
       "      <td>Làm sạch toàn bộ phòng giao dịch tầng 1 (kể cả...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Part time</td>\n",
       "      <td>06:30:00</td>\n",
       "      <td>10:30:00</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>7.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.000</td>\n",
       "      <td>0.1429</td>\n",
       "      <td>0.2857</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.38</td>\n",
       "      <td>0</td>\n",
       "      <td>AGRIBANK CHI NHÁNH MỸ ĐÌNH</td>\n",
       "      <td>Trung bình (11–20 người)</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>290.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>20.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>115-5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Hành chính</td>\n",
       "      <td>06:30:00</td>\n",
       "      <td>16:00:00</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>AGRIBANK PGD SỐ 05</td>\n",
       "      <td>Trung bình (11–20 người)</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>300.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>30.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>101-1</td>\n",
       "      <td>Kiểm tra nhân sự các vị trí Thay rác, đẩy khô,...</td>\n",
       "      <td>Lau bảng biển, bình cứu hỏa , cây nước hành la...</td>\n",
       "      <td>Hành chính</td>\n",
       "      <td>06:30:00</td>\n",
       "      <td>16:00:00</td>\n",
       "      <td>7.5</td>\n",
       "      <td>6</td>\n",
       "      <td>24</td>\n",
       "      <td>441.0</td>\n",
       "      <td>258.0</td>\n",
       "      <td>145.0</td>\n",
       "      <td>134.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>62.0</td>\n",
       "      <td>57.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>89.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0.585</td>\n",
       "      <td>0.3288</td>\n",
       "      <td>0.3039</td>\n",
       "      <td>0.1474</td>\n",
       "      <td>10.0</td>\n",
       "      <td>10.00</td>\n",
       "      <td>0</td>\n",
       "      <td>Tòa 5 tầng</td>\n",
       "      <td>Rất cao (Trên 40 người)</td>\n",
       "      <td>10</td>\n",
       "      <td>18</td>\n",
       "      <td>1700.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2600.0</td>\n",
       "      <td>348.0</td>\n",
       "      <td>6825.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>9176.0</td>\n",
       "      <td>89.0</td>\n",
       "      <td>25</td>\n",
       "      <td>894.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>101-1</td>\n",
       "      <td>Kiểm tra nhân sự các vị trí Thay rác, đẩy khô,...</td>\n",
       "      <td>Lau bảng biển, bình cứu hỏa , cây nước hành la...</td>\n",
       "      <td>Ca sáng</td>\n",
       "      <td>06:00:00</td>\n",
       "      <td>14:00:00</td>\n",
       "      <td>8.0</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>441.0</td>\n",
       "      <td>258.0</td>\n",
       "      <td>145.0</td>\n",
       "      <td>134.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>62.0</td>\n",
       "      <td>57.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>89.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0.585</td>\n",
       "      <td>0.3288</td>\n",
       "      <td>0.3039</td>\n",
       "      <td>0.1474</td>\n",
       "      <td>10.0</td>\n",
       "      <td>10.00</td>\n",
       "      <td>0</td>\n",
       "      <td>Tòa 5 tầng</td>\n",
       "      <td>Rất cao (Trên 40 người)</td>\n",
       "      <td>10</td>\n",
       "      <td>18</td>\n",
       "      <td>1700.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2600.0</td>\n",
       "      <td>348.0</td>\n",
       "      <td>6825.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>9176.0</td>\n",
       "      <td>89.0</td>\n",
       "      <td>25</td>\n",
       "      <td>894.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>101-1</td>\n",
       "      <td>Kiểm tra nhân sự các vị trí Thay rác, đẩy khô,...</td>\n",
       "      <td>Lau bảng biển, bình cứu hỏa , cây nước hành la...</td>\n",
       "      <td>Ca chiều</td>\n",
       "      <td>14:00:00</td>\n",
       "      <td>22:00:00</td>\n",
       "      <td>8.0</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>441.0</td>\n",
       "      <td>258.0</td>\n",
       "      <td>145.0</td>\n",
       "      <td>134.0</td>\n",
       "      <td>65.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>62.0</td>\n",
       "      <td>57.0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>89.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>0.585</td>\n",
       "      <td>0.3288</td>\n",
       "      <td>0.3039</td>\n",
       "      <td>0.1474</td>\n",
       "      <td>10.0</td>\n",
       "      <td>10.00</td>\n",
       "      <td>0</td>\n",
       "      <td>Tòa 5 tầng</td>\n",
       "      <td>Rất cao (Trên 40 người)</td>\n",
       "      <td>10</td>\n",
       "      <td>18</td>\n",
       "      <td>1700.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2600.0</td>\n",
       "      <td>348.0</td>\n",
       "      <td>6825.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "      <td>9176.0</td>\n",
       "      <td>89.0</td>\n",
       "      <td>25</td>\n",
       "      <td>894.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  ma_dia_diem                                    all_task_normal  \\\n",
       "0       115-2  Làm sạch toàn bộ phòng giao dịch tầng 1 (kể cả...   \n",
       "1       115-5                                                NaN   \n",
       "2       101-1  Kiểm tra nhân sự các vị trí Thay rác, đẩy khô,...   \n",
       "3       101-1  Kiểm tra nhân sự các vị trí Thay rác, đẩy khô,...   \n",
       "4       101-1  Kiểm tra nhân sự các vị trí Thay rác, đẩy khô,...   \n",
       "\n",
       "                                     all_task_dinhky     loai_ca   bat_dau  \\\n",
       "0                                                NaN   Part time  06:30:00   \n",
       "1                                                NaN  Hành chính  06:30:00   \n",
       "2  Lau bảng biển, bình cứu hỏa , cây nước hành la...  Hành chính  06:30:00   \n",
       "3  Lau bảng biển, bình cứu hỏa , cây nước hành la...     Ca sáng  06:00:00   \n",
       "4  Lau bảng biển, bình cứu hỏa , cây nước hành la...    Ca chiều  14:00:00   \n",
       "\n",
       "   ket_thuc  tong_gio_lam  so_ca_cua_toa  so_luong  num_tasks  \\\n",
       "0  10:30:00           4.0              1         1        7.0   \n",
       "1  16:00:00           8.0              1         1        NaN   \n",
       "2  16:00:00           7.5              6        24      441.0   \n",
       "3  14:00:00           8.0              6         3      441.0   \n",
       "4  22:00:00           8.0              6         5      441.0   \n",
       "\n",
       "   num_cleaning_tasks  num_trash_collection_tasks  num_monitoring_tasks  \\\n",
       "0                 7.0                         1.0                   2.0   \n",
       "1                 NaN                         NaN                   NaN   \n",
       "2               258.0                       145.0                 134.0   \n",
       "3               258.0                       145.0                 134.0   \n",
       "4               258.0                       145.0                 134.0   \n",
       "\n",
       "   num_room_cleaning_tasks  num_deep_cleaning_tasks  num_maintenance_tasks  \\\n",
       "0                      0.0                      1.0                    1.0   \n",
       "1                      NaN                      NaN                    NaN   \n",
       "2                     65.0                     75.0                   62.0   \n",
       "3                     65.0                     75.0                   62.0   \n",
       "4                     65.0                     75.0                   62.0   \n",
       "\n",
       "   num_support_tasks  num_other_tasks  num_wc_tasks  num_hallway_tasks  \\\n",
       "0                0.0              0.0           4.0                0.0   \n",
       "1                NaN              NaN           NaN                NaN   \n",
       "2               57.0             45.0          89.0               90.0   \n",
       "3               57.0             45.0          89.0               90.0   \n",
       "4               57.0             45.0          89.0               90.0   \n",
       "\n",
       "   num_lobby_tasks  num_patient_room_tasks  num_clinic_room_tasks  \\\n",
       "0              0.0                     0.0                    0.0   \n",
       "1              NaN                     NaN                    NaN   \n",
       "2              5.0                    41.0                   25.0   \n",
       "3              5.0                    41.0                   25.0   \n",
       "4              5.0                    41.0                   25.0   \n",
       "\n",
       "   num_surgery_room_tasks  num_outdoor_tasks  num_elevator_tasks  \\\n",
       "0                     0.0                2.0                 3.0   \n",
       "1                     NaN                NaN                 NaN   \n",
       "2                    30.0                4.0                12.0   \n",
       "3                    30.0                4.0                12.0   \n",
       "4                    30.0                4.0                12.0   \n",
       "\n",
       "   num_office_tasks  num_technical_room_tasks  cleaning_ratio  \\\n",
       "0               1.0                       0.0           1.000   \n",
       "1               NaN                       NaN             NaN   \n",
       "2              39.0                      16.0           0.585   \n",
       "3              39.0                      16.0           0.585   \n",
       "4              39.0                      16.0           0.585   \n",
       "\n",
       "   trash_collection_ratio  monitoring_ratio  room_cleaning_ratio  \\\n",
       "0                  0.1429            0.2857               0.0000   \n",
       "1                     NaN               NaN                  NaN   \n",
       "2                  0.3288            0.3039               0.1474   \n",
       "3                  0.3288            0.3039               0.1474   \n",
       "4                  0.3288            0.3039               0.1474   \n",
       "\n",
       "   area_diversity  task_complexity_score  loai_hinh  \\\n",
       "0             4.0                   1.38          0   \n",
       "1             NaN                    NaN          0   \n",
       "2            10.0                  10.00          0   \n",
       "3            10.0                  10.00          0   \n",
       "4            10.0                  10.00          0   \n",
       "\n",
       "                 ten_toa_thap          muc_do_luu_luong  so_tang  \\\n",
       "0  AGRIBANK CHI NHÁNH MỸ ĐÌNH  Trung bình (11–20 người)        4   \n",
       "1          AGRIBANK PGD SỐ 05  Trung bình (11–20 người)        1   \n",
       "2                  Tòa 5 tầng   Rất cao (Trên 40 người)       10   \n",
       "3                  Tòa 5 tầng   Rất cao (Trên 40 người)       10   \n",
       "4                  Tòa 5 tầng   Rất cao (Trên 40 người)       10   \n",
       "\n",
       "   so_cua_thang_may  dien_tich_ngoai_canh  dien_tich_sanh  \\\n",
       "0                 0                   0.0             0.0   \n",
       "1                 0                   0.0             0.0   \n",
       "2                18                1700.0             0.0   \n",
       "3                18                1700.0             0.0   \n",
       "4                18                1700.0             0.0   \n",
       "\n",
       "   dien_tich_hanh_lang  dien_tich_wc  dien_tich_phong  dien_tich_tham  \\\n",
       "0                  0.0          15.0            290.0             0.0   \n",
       "1                  0.0          15.0            300.0             0.0   \n",
       "2               2600.0         348.0           6825.0             0.0   \n",
       "3               2600.0         348.0           6825.0             0.0   \n",
       "4               2600.0         348.0           6825.0             0.0   \n",
       "\n",
       "   doc_ham  vien_phan_quang  op_tuong  op_chan_tuong  ranh_thoat_nuoc  \\\n",
       "0        0                0       0.0            0.0                0   \n",
       "1        0                0       0.0            0.0                0   \n",
       "2       70                0    9176.0           89.0               25   \n",
       "3       70                0    9176.0           89.0               25   \n",
       "4       70                0    9176.0           89.0               25   \n",
       "\n",
       "   dien_tich_kinh  \n",
       "0            20.0  \n",
       "1            30.0  \n",
       "2           894.0  \n",
       "3           894.0  \n",
       "4           894.0  "
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load dataset\n",
    "print(\"📂 Loading dataset...\")\n",
    "df = pd.read_excel('FINAL_DATASET_WITH_TEXT_BACKUP_20260105_213507.xlsx')\n",
    "\n",
    "print(f\"✅ Dataset loaded successfully!\")\n",
    "print(f\"📊 Shape: {df.shape}\")\n",
    "print(f\"📋 Columns: {df.columns.tolist()[:10]}... (showing first 10)\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e49dd58",
   "metadata": {},
   "source": [
    "# 2️⃣ Exploratory Data Analysis (EDA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "b277092d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================================================================\n",
      "📊 EXPLORATORY DATA ANALYSIS\n",
      "================================================================================\n",
      "\n",
      "1. DATA INFO:\n",
      "   Shape: (454, 51)\n",
      "   Memory: 9.10 MB\n",
      "\n",
      "2. DATA TYPES:\n",
      "float64    35\n",
      "object      8\n",
      "int64       8\n",
      "Name: count, dtype: int64\n",
      "\n",
      "3. MISSING VALUES:\n",
      "                            Missing  Percentage\n",
      "all_task_dinhky                 168       37.00\n",
      "all_task_normal                  27        5.95\n",
      "num_patient_room_tasks           22        4.85\n",
      "area_diversity                   22        4.85\n",
      "room_cleaning_ratio              22        4.85\n",
      "monitoring_ratio                 22        4.85\n",
      "trash_collection_ratio           22        4.85\n",
      "cleaning_ratio                   22        4.85\n",
      "num_technical_room_tasks         22        4.85\n",
      "num_office_tasks                 22        4.85\n",
      "num_elevator_tasks               22        4.85\n",
      "num_outdoor_tasks                22        4.85\n",
      "num_surgery_room_tasks           22        4.85\n",
      "num_clinic_room_tasks            22        4.85\n",
      "num_lobby_tasks                  22        4.85\n",
      "num_hallway_tasks                22        4.85\n",
      "num_wc_tasks                     22        4.85\n",
      "num_other_tasks                  22        4.85\n",
      "num_support_tasks                22        4.85\n",
      "num_maintenance_tasks            22        4.85\n",
      "num_deep_cleaning_tasks          22        4.85\n",
      "num_room_cleaning_tasks          22        4.85\n",
      "num_monitoring_tasks             22        4.85\n",
      "num_trash_collection_tasks       22        4.85\n",
      "num_cleaning_tasks               22        4.85\n",
      "num_tasks                        22        4.85\n",
      "task_complexity_score            22        4.85\n",
      "\n",
      "4. TARGET VARIABLE (so_luong):\n",
      "count    454.000000\n",
      "mean       4.484581\n",
      "std        6.626822\n",
      "min        0.000000\n",
      "25%        1.000000\n",
      "50%        2.000000\n",
      "75%        5.000000\n",
      "max       64.000000\n",
      "Name: so_luong, dtype: float64\n",
      "   Zero values: 16\n",
      "   Skewness: 3.69\n",
      "   Kurtosis: 19.63\n"
     ]
    }
   ],
   "source": [
    "print(\"=\"*80)\n",
    "print(\"📊 EXPLORATORY DATA ANALYSIS\")\n",
    "print(\"=\"*80)\n",
    "\n",
    "print(\"\\n1. DATA INFO:\")\n",
    "print(f\"   Shape: {df.shape}\")\n",
    "print(f\"   Memory: {df.memory_usage(deep=True).sum() / 1024**2:.2f} MB\")\n",
    "\n",
    "print(\"\\n2. DATA TYPES:\")\n",
    "print(df.dtypes.value_counts())\n",
    "\n",
    "print(\"\\n3. MISSING VALUES:\")\n",
    "missing = df.isnull().sum()\n",
    "missing_pct = (missing / len(df) * 100).round(2)\n",
    "missing_df = pd.DataFrame({\n",
    "    'Missing': missing[missing > 0],\n",
    "    'Percentage': missing_pct[missing > 0]\n",
    "}).sort_values('Missing', ascending=False)\n",
    "print(missing_df)\n",
    "\n",
    "print(\"\\n4. TARGET VARIABLE (so_luong):\")\n",
    "print(df['so_luong'].describe())\n",
    "print(f\"   Zero values: {(df['so_luong'] == 0).sum()}\")\n",
    "print(f\"   Skewness: {df['so_luong'].skew():.2f}\")\n",
    "print(f\"   Kurtosis: {df['so_luong'].kurt():.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "234dfda5",
   "metadata": {},
   "source": [
    "# 4️⃣ Data Cleaning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "cda6f5fc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================================================================\n",
      "🧹 DATA CLEANING\n",
      "================================================================================\n",
      "\n",
      "1. Handling missing values...\n",
      "   Filled num_tasks with 0\n",
      "   Filled num_cleaning_tasks with 0\n",
      "   Filled num_trash_collection_tasks with 0\n",
      "   Filled num_monitoring_tasks with 0\n",
      "   Filled num_room_cleaning_tasks with 0\n",
      "   Filled num_deep_cleaning_tasks with 0\n",
      "   Filled num_maintenance_tasks with 0\n",
      "   Filled num_support_tasks with 0\n",
      "   Filled num_other_tasks with 0\n",
      "   Filled num_wc_tasks with 0\n",
      "   Filled num_hallway_tasks with 0\n",
      "   Filled num_lobby_tasks with 0\n",
      "   Filled num_patient_room_tasks with 0\n",
      "   Filled num_clinic_room_tasks with 0\n",
      "   Filled num_surgery_room_tasks with 0\n",
      "   Filled num_outdoor_tasks with 0\n",
      "   Filled num_elevator_tasks with 0\n",
      "   Filled num_office_tasks with 0\n",
      "   Filled num_technical_room_tasks with 0\n",
      "   Filled cleaning_ratio with 0\n",
      "   Filled trash_collection_ratio with 0\n",
      "   Filled monitoring_ratio with 0\n",
      "   Filled room_cleaning_ratio with 0\n",
      "   Filled area_diversity with 0\n",
      "   Filled task_complexity_score with 0\n",
      "\n",
      "2. Analyzing target outliers...\n",
      "   Zero values: 16\n",
      "   Decision: KEEP all values (including zeros)\n",
      "\n",
      "3. Checking duplicates...\n",
      "   Duplicates found: 9\n",
      "\n",
      "4. Dropping unnecessary columns...\n",
      "   Dropped: ['ma_dia_diem', 'all_task_normal', 'all_task_dinhky', 'ten_toa_thap']\n",
      "\n",
      "5. Validation after cleaning:\n",
      "   Shape: (445, 47)\n",
      "   Missing values: 0\n",
      "\n",
      "✅ Saved: data/cleaned/CLEANED_DATA.csv\n"
     ]
    }
   ],
   "source": [
    "print(\"=\"*80)\n",
    "print(\"🧹 DATA CLEANING\")\n",
    "print(\"=\"*80)\n",
    "\n",
    "df_clean = df.copy()\n",
    "\n",
    "# 1. Handle missing values\n",
    "print(\"\\n1. Handling missing values...\")\n",
    "\n",
    "# Task features: fill with 0\n",
    "task_features = [col for col in df_clean.columns if col.startswith('num_') or col.endswith('_ratio') or col in ['area_diversity', 'task_complexity_score']]\n",
    "for col in task_features:\n",
    "    if df_clean[col].isnull().sum() > 0:\n",
    "        df_clean[col] = df_clean[col].fillna(0)\n",
    "        print(f\"   Filled {col} with 0\")\n",
    "\n",
    "# Building features: fill with median\n",
    "building_features = [col for col in df_clean.columns if col.startswith('dien_tich_') or col in ['so_tang', 'so_cua_thang_may']]\n",
    "for col in building_features:\n",
    "    if df_clean[col].isnull().sum() > 0:\n",
    "        median_val = df_clean[col].median()\n",
    "        df_clean[col] = df_clean[col].fillna(median_val)\n",
    "        print(f\"   Filled {col} with median={median_val}\")\n",
    "\n",
    "# Categorical: fill with mode\n",
    "cat_features = ['loai_hinh', 'muc_do_luu_luong']\n",
    "for col in cat_features:\n",
    "    if col in df_clean.columns and df_clean[col].isnull().sum() > 0:\n",
    "        mode_val = df_clean[col].mode()[0]\n",
    "        df_clean[col] = df_clean[col].fillna(mode_val)\n",
    "        print(f\"   Filled {col} with mode={mode_val}\")\n",
    "\n",
    "# 2. Handle outliers (keep all for now)\n",
    "print(\"\\n2. Analyzing target outliers...\")\n",
    "print(f\"   Zero values: {(df_clean['so_luong'] == 0).sum()}\")\n",
    "print(\"   Decision: KEEP all values (including zeros)\")\n",
    "\n",
    "# 3. Remove duplicates\n",
    "print(\"\\n3. Checking duplicates...\")\n",
    "dups = df_clean.duplicated().sum()\n",
    "print(f\"   Duplicates found: {dups}\")\n",
    "if dups > 0:\n",
    "    df_clean = df_clean.drop_duplicates()\n",
    "\n",
    "# 4. Drop unnecessary columns\n",
    "print(\"\\n4. Dropping unnecessary columns...\")\n",
    "cols_to_drop = ['ma_dia_diem', 'all_task_normal', 'all_task_dinhky', 'ten_toa_thap']\n",
    "cols_to_drop = [col for col in cols_to_drop if col in df_clean.columns]\n",
    "df_clean = df_clean.drop(columns=cols_to_drop)\n",
    "print(f\"   Dropped: {cols_to_drop}\")\n",
    "\n",
    "# 5. Validate\n",
    "print(\"\\n5. Validation after cleaning:\")\n",
    "print(f\"   Shape: {df_clean.shape}\")\n",
    "print(f\"   Missing values: {df_clean.isnull().sum().sum()}\")\n",
    "\n",
    "# Save\n",
    "df_clean.to_csv('data/cleaned/CLEANED_DATA.csv', index=False)\n",
    "print(\"\\n✅ Saved: data/cleaned/CLEANED_DATA.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff3cc30f",
   "metadata": {},
   "source": [
    "# 5️⃣ Feature Engineering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "83c1bbba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================================================================\n",
      "⚙️ FEATURE ENGINEERING\n",
      "================================================================================\n",
      "\n",
      "1. Creating time features...\n",
      "   ✅ Created time features\n",
      "\n",
      "2. Creating interaction features...\n",
      "   ✅ Created interaction features\n",
      "\n",
      "3. Creating aggregation features...\n",
      "   ✅ Created aggregation features\n",
      "\n",
      "4. Encoding categorical features...\n",
      "   ✅ One-hot encoded loai_ca\n",
      "   ✅ Label encoded loai_hinh\n",
      "   ✅ Label encoded muc_do_luu_luong\n",
      "\n",
      "✅ Feature engineering complete! Shape: (445, 64)\n",
      "✅ Saved: data/cleaned/ENGINEERED_DATA.csv\n"
     ]
    }
   ],
   "source": [
    "print(\"=\"*80)\n",
    "print(\"⚙️ FEATURE ENGINEERING\")\n",
    "print(\"=\"*80)\n",
    "\n",
    "df_eng = df_clean.copy()\n",
    "\n",
    "# A. TIME FEATURES\n",
    "print(\"\\n1. Creating time features...\")\n",
    "\n",
    "def parse_time(time_val):\n",
    "    if pd.isna(time_val):\n",
    "        return 0\n",
    "    \n",
    "    # If it's already a datetime object\n",
    "    if isinstance(time_val, pd.Timestamp) or hasattr(time_val, 'hour'):\n",
    "        return time_val.hour\n",
    "    \n",
    "    # If it's a string\n",
    "    if isinstance(time_val, str):\n",
    "        time_val = time_val.strip()\n",
    "        \n",
    "        # Format: \"2025-01-01 22:00:00\" or \"2025-01-01 22\"\n",
    "        if ' ' in time_val:\n",
    "            # Split by space and get the time part\n",
    "            parts = time_val.split(' ')\n",
    "            if len(parts) >= 2:\n",
    "                time_part = parts[1]  # Get time part after space\n",
    "                if ':' in time_part:\n",
    "                    return int(time_part.split(':')[0])\n",
    "                else:\n",
    "                    return int(time_part)\n",
    "        \n",
    "        # Format: \"22:00:00\" or \"22:00\"\n",
    "        elif ':' in time_val:\n",
    "            return int(time_val.split(':')[0])\n",
    "        \n",
    "        # Try to parse as integer\n",
    "        try:\n",
    "            return int(float(time_val))\n",
    "        except:\n",
    "            return 0\n",
    "    \n",
    "    # If it's a number\n",
    "    if isinstance(time_val, (int, float)):\n",
    "        return int(time_val) if not np.isnan(time_val) else 0\n",
    "    \n",
    "    return 0\n",
    "\n",
    "df_eng['hour_start'] = df_eng['bat_dau'].apply(parse_time)\n",
    "df_eng['hour_end'] = df_eng['ket_thuc'].apply(parse_time)\n",
    "\n",
    "def parse_work_hours(time_str):\n",
    "    if pd.isna(time_str):\n",
    "        return 8.0\n",
    "    if isinstance(time_str, str) and ':' in time_str:\n",
    "        parts = time_str.split(':')\n",
    "        return float(parts[0]) + float(parts[1])/60 if len(parts) > 1 else float(parts[0])\n",
    "    return 8.0\n",
    "\n",
    "df_eng['work_hours_numeric'] = df_eng['tong_gio_lam'].apply(parse_work_hours)\n",
    "\n",
    "# Shift indicators\n",
    "df_eng['is_morning_shift'] = ((df_eng['hour_start'] >= 6) & (df_eng['hour_start'] < 12)).astype(int)\n",
    "df_eng['is_afternoon_shift'] = ((df_eng['hour_start'] >= 12) & (df_eng['hour_start'] < 18)).astype(int)\n",
    "df_eng['is_evening_shift'] = ((df_eng['hour_start'] >= 18) & (df_eng['hour_start'] < 24)).astype(int)\n",
    "df_eng['is_night_shift'] = ((df_eng['hour_start'] >= 0) & (df_eng['hour_start'] < 6)).astype(int)\n",
    "\n",
    "print(\"   ✅ Created time features\")\n",
    "\n",
    "# B. INTERACTION FEATURES\n",
    "print(\"\\n2. Creating interaction features...\")\n",
    "\n",
    "df_eng['tasks_per_hour'] = df_eng['num_tasks'] / (df_eng['work_hours_numeric'] + 0.1)\n",
    "df_eng['tasks_per_floor'] = df_eng['num_tasks'] / (df_eng['so_tang'] + 1)\n",
    "df_eng['wc_per_floor'] = df_eng['num_wc_tasks'] / (df_eng['so_tang'] + 1)\n",
    "df_eng['cleaning_workload'] = df_eng['num_cleaning_tasks'] * df_eng['area_diversity']\n",
    "\n",
    "print(\"   ✅ Created interaction features\")\n",
    "\n",
    "# C. AGGREGATION FEATURES\n",
    "print(\"\\n3. Creating aggregation features...\")\n",
    "\n",
    "area_cols = [col for col in df_eng.columns if col.startswith('dien_tich_')]\n",
    "df_eng['total_area'] = df_eng[area_cols].sum(axis=1)\n",
    "df_eng['area_per_floor'] = df_eng['total_area'] / (df_eng['so_tang'] + 1)\n",
    "\n",
    "special_cols = ['num_patient_room_tasks', 'num_surgery_room_tasks', 'num_clinic_room_tasks']\n",
    "df_eng['has_special_areas'] = (df_eng[special_cols].sum(axis=1) > 0).astype(int)\n",
    "\n",
    "print(\"   ✅ Created aggregation features\")\n",
    "\n",
    "# D. CATEGORICAL ENCODING\n",
    "print(\"\\n4. Encoding categorical features...\")\n",
    "\n",
    "# One-hot for loai_ca\n",
    "if 'loai_ca' in df_eng.columns:\n",
    "    loai_ca_dummies = pd.get_dummies(df_eng['loai_ca'], prefix='loai_ca', drop_first=True)\n",
    "    df_eng = pd.concat([df_eng, loai_ca_dummies], axis=1)\n",
    "    df_eng = df_eng.drop('loai_ca', axis=1)\n",
    "    print(f\"   ✅ One-hot encoded loai_ca\")\n",
    "\n",
    "# Label encoding for ordinal\n",
    "ordinal_map = {\n",
    "    'loai_hinh': {'Văn phòng': 0, 'Bệnh viện': 1, 'Trung tâm thương mại': 2, 'Khác': 3},\n",
    "    'muc_do_luu_luong': {'Thấp': 0, 'Trung bình': 1, 'Cao': 2}\n",
    "}\n",
    "\n",
    "for col, mapping in ordinal_map.items():\n",
    "    if col in df_eng.columns:\n",
    "        df_eng[col] = df_eng[col].map(mapping).fillna(0).astype(int)\n",
    "        print(f\"   ✅ Label encoded {col}\")\n",
    "\n",
    "# Drop original time columns\n",
    "time_cols = ['bat_dau', 'ket_thuc', 'tong_gio_lam']\n",
    "df_eng = df_eng.drop(columns=[c for c in time_cols if c in df_eng.columns])\n",
    "\n",
    "print(f\"\\n✅ Feature engineering complete! Shape: {df_eng.shape}\")\n",
    "\n",
    "# Save\n",
    "df_eng.to_csv('data/cleaned/ENGINEERED_DATA.csv', index=False)\n",
    "print(\"✅ Saved: data/cleaned/ENGINEERED_DATA.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a1031a3",
   "metadata": {},
   "source": [
    "# 6️⃣ Train/Val/Test Split\n",
    "\n",
    "**Mục đích:** Chia dữ liệu thành 3 tập: Train (60%), Validation (20%), Test (20%)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "5edc1e37",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================================================================\n",
      "✂️ TRAIN/VAL/TEST SPLIT\n",
      "================================================================================\n",
      "\n",
      "📊 Dataset:\n",
      "   Total samples: 445\n",
      "   Features: 63\n",
      "   Target range: [0, 64]\n",
      "\n",
      "✅ Split completed:\n",
      "   Train: 267 samples (60.0%)\n",
      "   Val:   89 samples (20.0%)\n",
      "   Test:  89 samples (20.0%)\n",
      "\n",
      "✅ Saved splits to data/splits/\n"
     ]
    }
   ],
   "source": [
    "print(\"=\"*80)\n",
    "print(\"✂️ TRAIN/VAL/TEST SPLIT\")\n",
    "print(\"=\"*80)\n",
    "\n",
    "# Separate features and target\n",
    "y = df_eng['so_luong']\n",
    "X = df_eng.drop('so_luong', axis=1)\n",
    "\n",
    "print(f\"\\n📊 Dataset:\")\n",
    "print(f\"   Total samples: {len(X)}\")\n",
    "print(f\"   Features: {X.shape[1]}\")\n",
    "print(f\"   Target range: [{y.min()}, {y.max()}]\")\n",
    "\n",
    "# Split train+val (80%) and test (20%)\n",
    "X_temp, X_test, y_temp, y_test = train_test_split(\n",
    "    X, y, test_size=0.2, random_state=RANDOM_STATE\n",
    ")\n",
    "\n",
    "# Split train (75% of 80% = 60%) and val (25% of 80% = 20%)\n",
    "X_train, X_val, y_train, y_val = train_test_split(\n",
    "    X_temp, y_temp, test_size=0.25, random_state=RANDOM_STATE\n",
    ")\n",
    "\n",
    "print(f\"\\n✅ Split completed:\")\n",
    "print(f\"   Train: {X_train.shape[0]} samples ({X_train.shape[0]/len(X)*100:.1f}%)\")\n",
    "print(f\"   Val:   {X_val.shape[0]} samples ({X_val.shape[0]/len(X)*100:.1f}%)\")\n",
    "print(f\"   Test:  {X_test.shape[0]} samples ({X_test.shape[0]/len(X)*100:.1f}%)\")\n",
    "\n",
    "# Save splits\n",
    "X_train.to_csv('data/splits/X_train.csv', index=False)\n",
    "X_val.to_csv('data/splits/X_val.csv', index=False)\n",
    "X_test.to_csv('data/splits/X_test.csv', index=False)\n",
    "y_train.to_csv('data/splits/y_train.csv', index=False)\n",
    "y_val.to_csv('data/splits/y_val.csv', index=False)\n",
    "y_test.to_csv('data/splits/y_test.csv', index=False)\n",
    "\n",
    "print(\"\\n✅ Saved splits to data/splits/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ed0b48c",
   "metadata": {},
   "source": [
    "# 7️⃣ K-Fold Cross-Validation\n",
    "\n",
    "**Mục đích:** Đánh giá độ ổn định của các models bằng 5-Fold CV trên training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "d1d120e4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================================================================\n",
      "🔄 K-FOLD CROSS-VALIDATION\n",
      "================================================================================\n",
      "\n",
      "📏 Preparing data for K-Fold CV...\n",
      "   ⚠️  Dropping 14 non-numeric columns: ['loai_hinh', 'muc_do_luu_luong', 'is_morning_shift', 'is_afternoon_shift', 'is_evening_shift', 'is_night_shift', 'has_special_areas', 'loai_ca_24/24', 'loai_ca_Ca chiều', 'loai_ca_Ca gãy', 'loai_ca_Ca sáng', 'loai_ca_Ca đêm', 'loai_ca_Hành chính', 'loai_ca_Part time']\n",
      "   Binary columns (no scaling): 0\n",
      "   One-hot columns (no scaling): 0\n",
      "   Numeric columns (will scale): 49\n",
      "   ✅ Scaled 49 numeric features\n",
      "\n",
      "🤖 Running 5-Fold CV...\n",
      "\n",
      "   Testing Linear Regression...\n",
      "      MAE: 3.7654 ± 0.8861\n",
      "      R²:  -0.1780 ± 0.5737\n",
      "\n",
      "   Testing Decision Tree...\n",
      "      MAE: 3.8242 ± 0.8261\n",
      "      R²:  -0.7565 ± 1.4466\n",
      "\n",
      "   Testing Random Forest...\n",
      "      MAE: 3.3523 ± 0.8554\n",
      "      R²:  -0.0652 ± 0.3507\n",
      "\n",
      "   Testing Gradient Boosting...\n",
      "      MAE: 3.3027 ± 0.8887\n",
      "      R²:  -0.1023 ± 0.5154\n",
      "\n",
      "   Testing XGBoost...\n",
      "      MAE: 3.3237 ± 0.7237\n",
      "      R²:  -0.1420 ± 0.5722\n",
      "\n",
      "   Testing LightGBM...\n",
      "      MAE: 3.4016 ± 0.8549\n",
      "      R²:  -0.0064 ± 0.2486\n",
      "\n",
      "================================================================================\n",
      "📊 K-FOLD CV RESULTS SUMMARY\n",
      "================================================================================\n",
      "            Model  Mean_MAE  Std_MAE   Mean_R2   Std_R2\n",
      "Gradient Boosting  3.302737 0.888736 -0.102337 0.515448\n",
      "          XGBoost  3.323695 0.723653 -0.142027 0.572242\n",
      "    Random Forest  3.352265 0.855390 -0.065234 0.350683\n",
      "         LightGBM  3.401601 0.854854 -0.006448 0.248615\n",
      "Linear Regression  3.765361 0.886077 -0.177970 0.573717\n",
      "    Decision Tree  3.824216 0.826131 -0.756537 1.446579\n",
      "\n",
      "🏆 Best Model by K-Fold CV: Gradient Boosting\n",
      "   MAE: 3.3027 ± 0.8887\n",
      "   R²:  -0.1023 ± 0.5154\n",
      "\n",
      "✅ Saved: models/kfold_cv_results.csv\n"
     ]
    }
   ],
   "source": [
    "print(\"=\"*80)\n",
    "print(\"🔄 K-FOLD CROSS-VALIDATION\")\n",
    "print(\"=\"*80)\n",
    "\n",
    "# Prepare data for K-Fold CV\n",
    "print(\"\\n📏 Preparing data for K-Fold CV...\")\n",
    "\n",
    "# Check for non-numeric columns\n",
    "non_numeric_cols = X_train.select_dtypes(exclude=['int64', 'float64']).columns.tolist()\n",
    "if non_numeric_cols:\n",
    "    print(f\"   ⚠️  Dropping {len(non_numeric_cols)} non-numeric columns: {non_numeric_cols}\")\n",
    "    X_train_cv = X_train.drop(columns=non_numeric_cols).copy()\n",
    "else:\n",
    "    X_train_cv = X_train.copy()\n",
    "\n",
    "# Identify columns to scale\n",
    "binary_cols = [col for col in X_train_cv.columns if X_train_cv[col].nunique() == 2 and set(X_train_cv[col].unique()).issubset({0, 1})]\n",
    "onehot_cols = [col for col in X_train_cv.columns if col.startswith('loai_ca_')]\n",
    "cols_to_scale = [col for col in X_train_cv.columns if col not in binary_cols and col not in onehot_cols]\n",
    "\n",
    "print(f\"   Binary columns (no scaling): {len(binary_cols)}\")\n",
    "print(f\"   One-hot columns (no scaling): {len(onehot_cols)}\")\n",
    "print(f\"   Numeric columns (will scale): {len(cols_to_scale)}\")\n",
    "\n",
    "# Scale numeric columns\n",
    "scaler_cv = StandardScaler()\n",
    "if cols_to_scale:\n",
    "    X_train_cv[cols_to_scale] = scaler_cv.fit_transform(X_train_cv[cols_to_scale])\n",
    "    print(f\"   ✅ Scaled {len(cols_to_scale)} numeric features\")\n",
    "\n",
    "# Initialize KFold\n",
    "kf = KFold(n_splits=5, shuffle=True, random_state=RANDOM_STATE)\n",
    "\n",
    "# Models to test\n",
    "cv_models = {\n",
    "    'Linear Regression': LinearRegression(),\n",
    "    'Decision Tree': DecisionTreeRegressor(max_depth=10, min_samples_split=5, random_state=RANDOM_STATE),\n",
    "    'Random Forest': RandomForestRegressor(n_estimators=100, max_depth=15, min_samples_split=5, random_state=RANDOM_STATE, n_jobs=-1),\n",
    "    'Gradient Boosting': GradientBoostingRegressor(n_estimators=100, learning_rate=0.1, max_depth=5, random_state=RANDOM_STATE),\n",
    "    'XGBoost': xgb.XGBRegressor(n_estimators=100, learning_rate=0.1, max_depth=5, random_state=RANDOM_STATE, n_jobs=-1),\n",
    "    'LightGBM': lgb.LGBMRegressor(n_estimators=100, learning_rate=0.1, max_depth=5, num_leaves=31, random_state=RANDOM_STATE, n_jobs=-1, verbose=-1)\n",
    "}\n",
    "\n",
    "# Run K-Fold CV\n",
    "cv_results = []\n",
    "print(\"\\n🤖 Running 5-Fold CV...\")\n",
    "\n",
    "for model_name, model in cv_models.items():\n",
    "    print(f\"\\n   Testing {model_name}...\")\n",
    "    \n",
    "    fold_maes = []\n",
    "    fold_r2s = []\n",
    "    \n",
    "    for fold, (train_idx, val_idx) in enumerate(kf.split(X_train_cv), 1):\n",
    "        X_fold_train = X_train_cv.iloc[train_idx]\n",
    "        y_fold_train = y_train.iloc[train_idx]\n",
    "        X_fold_val = X_train_cv.iloc[val_idx]\n",
    "        y_fold_val = y_train.iloc[val_idx]\n",
    "        \n",
    "        model.fit(X_fold_train, y_fold_train)\n",
    "        y_pred = model.predict(X_fold_val)\n",
    "        \n",
    "        mae = mean_absolute_error(y_fold_val, y_pred)\n",
    "        r2 = r2_score(y_fold_val, y_pred)\n",
    "        \n",
    "        fold_maes.append(mae)\n",
    "        fold_r2s.append(r2)\n",
    "    \n",
    "    mean_mae = np.mean(fold_maes)\n",
    "    std_mae = np.std(fold_maes)\n",
    "    mean_r2 = np.mean(fold_r2s)\n",
    "    std_r2 = np.std(fold_r2s)\n",
    "    \n",
    "    cv_results.append({\n",
    "        'Model': model_name,\n",
    "        'Mean_MAE': mean_mae,\n",
    "        'Std_MAE': std_mae,\n",
    "        'Mean_R2': mean_r2,\n",
    "        'Std_R2': std_r2\n",
    "    })\n",
    "    \n",
    "    print(f\"      MAE: {mean_mae:.4f} ± {std_mae:.4f}\")\n",
    "    print(f\"      R²:  {mean_r2:.4f} ± {std_r2:.4f}\")\n",
    "\n",
    "# Create results DataFrame\n",
    "cv_results_df = pd.DataFrame(cv_results).sort_values('Mean_MAE')\n",
    "\n",
    "print(\"\\n\" + \"=\"*80)\n",
    "print(\"📊 K-FOLD CV RESULTS SUMMARY\")\n",
    "print(\"=\"*80)\n",
    "print(cv_results_df.to_string(index=False))\n",
    "\n",
    "best_cv_model = cv_results_df.iloc[0]['Model']\n",
    "print(f\"\\n🏆 Best Model by K-Fold CV: {best_cv_model}\")\n",
    "print(f\"   MAE: {cv_results_df.iloc[0]['Mean_MAE']:.4f} ± {cv_results_df.iloc[0]['Std_MAE']:.4f}\")\n",
    "print(f\"   R²:  {cv_results_df.iloc[0]['Mean_R2']:.4f} ± {cv_results_df.iloc[0]['Std_R2']:.4f}\")\n",
    "\n",
    "# Save results\n",
    "cv_results_df.to_csv('models/kfold_cv_results.csv', index=False)\n",
    "print(\"\\n✅ Saved: models/kfold_cv_results.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d69b6c4e",
   "metadata": {},
   "source": [
    "# 8️⃣ Feature Selection\n",
    "\n",
    "**Mục đích:** Tìm best features sử dụng 3 phương pháp: Random Forest, Mutual Information, Correlation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "e017e7db",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================================================================\n",
      "🔬 FEATURE SELECTION\n",
      "================================================================================\n",
      "\n",
      "📏 Preparing data for feature selection...\n",
      "   Dropping 14 non-numeric columns: ['loai_hinh', 'muc_do_luu_luong', 'is_morning_shift', 'is_afternoon_shift', 'is_evening_shift', 'is_night_shift', 'has_special_areas', 'loai_ca_24/24', 'loai_ca_Ca chiều', 'loai_ca_Ca gãy', 'loai_ca_Ca sáng', 'loai_ca_Ca đêm', 'loai_ca_Hành chính', 'loai_ca_Part time']\n",
      "   ✅ Scaled 49 numeric features\n",
      "\n",
      "🌲 Method 1: Random Forest Feature Importance\n",
      "   Top 10 features:\n",
      "      hour_end                                : 0.1723\n",
      "      hour_start                              : 0.0724\n",
      "      num_other_tasks                         : 0.0590\n",
      "      num_trash_collection_tasks              : 0.0461\n",
      "      tasks_per_hour                          : 0.0430\n",
      "      cleaning_ratio                          : 0.0391\n",
      "      tasks_per_floor                         : 0.0354\n",
      "      dien_tich_wc                            : 0.0352\n",
      "      num_tasks                               : 0.0332\n",
      "      total_area                              : 0.0308\n",
      "\n",
      "📊 Method 2: Mutual Information\n",
      "   Top 10 features:\n",
      "      cleaning_workload                       : 0.2850\n",
      "      num_cleaning_tasks                      : 0.2775\n",
      "      num_trash_collection_tasks              : 0.2717\n",
      "      tasks_per_hour                          : 0.2396\n",
      "      num_tasks                               : 0.2319\n",
      "      num_other_tasks                         : 0.2167\n",
      "      task_complexity_score                   : 0.2143\n",
      "      hour_start                              : 0.2017\n",
      "      total_area                              : 0.1944\n",
      "      num_support_tasks                       : 0.1777\n",
      "\n",
      "📈 Method 3: Correlation with Target\n",
      "   Top 10 features:\n",
      "      num_trash_collection_tasks              : 0.3996\n",
      "      tasks_per_hour                          : 0.3622\n",
      "      num_tasks                               : 0.3622\n",
      "      cleaning_workload                       : 0.3603\n",
      "      num_cleaning_tasks                      : 0.3483\n",
      "      num_monitoring_tasks                    : 0.3214\n",
      "      num_technical_room_tasks                : 0.3189\n",
      "      num_support_tasks                       : 0.3113\n",
      "      task_complexity_score                   : 0.3106\n",
      "      num_other_tasks                         : 0.3059\n",
      "\n",
      "🎯 Aggregating Feature Scores...\n",
      "\n",
      "📋 Top 20 Features (by average score):\n",
      "   dien_tich_phong                         : 0.7402\n",
      "   tasks_per_hour                          : 0.6650\n",
      "   num_trash_collection_tasks              : 0.6648\n",
      "   num_surgery_room_tasks                  : 0.6385\n",
      "   dien_tich_hanh_lang                     : 0.6369\n",
      "   area_diversity                          : 0.6214\n",
      "   num_lobby_tasks                         : 0.6026\n",
      "   hour_end                                : 0.5416\n",
      "   dien_tich_ngoai_canh                    : 0.4777\n",
      "   monitoring_ratio                        : 0.4716\n",
      "   num_patient_room_tasks                  : 0.4620\n",
      "   num_monitoring_tasks                    : 0.4545\n",
      "   tasks_per_floor                         : 0.4460\n",
      "   work_hours_numeric                      : 0.4020\n",
      "   trash_collection_ratio                  : 0.3944\n",
      "   dien_tich_tham                          : 0.3927\n",
      "   so_tang                                 : 0.3817\n",
      "   so_ca_cua_toa                           : 0.3730\n",
      "   dien_tich_wc                            : 0.3634\n",
      "   num_wc_tasks                            : 0.3597\n",
      "\n",
      "================================================================================\n",
      "🔬 TESTING WITH DIFFERENT NUMBERS OF FEATURES\n",
      "================================================================================\n",
      "\n",
      "📌 BASELINE (All 49 Features):\n",
      "   Val MAE: 3.6346, R²: -0.0129\n",
      "\n",
      "================================================================================\n",
      "🔬 Testing with Top 20 Features\n",
      "================================================================================\n",
      "\n",
      "📋 Selected features:\n",
      "    1. dien_tich_phong                          (score: 0.7402)\n",
      "    2. tasks_per_hour                           (score: 0.6650)\n",
      "    3. num_trash_collection_tasks               (score: 0.6648)\n",
      "    4. num_surgery_room_tasks                   (score: 0.6385)\n",
      "    5. dien_tich_hanh_lang                      (score: 0.6369)\n",
      "    6. area_diversity                           (score: 0.6214)\n",
      "    7. num_lobby_tasks                          (score: 0.6026)\n",
      "    8. hour_end                                 (score: 0.5416)\n",
      "    9. dien_tich_ngoai_canh                     (score: 0.4777)\n",
      "   10. monitoring_ratio                         (score: 0.4716)\n",
      "   ... and 10 more\n",
      "\n",
      "🤖 Training models...\n",
      "   Random Forest       : MAE=3.6661 (-0.0315), R²=0.0374\n",
      "   Gradient Boosting   : MAE=3.7772 (-0.1426), R²=-0.0601\n",
      "   XGBoost             : MAE=3.6416 (-0.0071), R²=0.0158\n",
      "   LightGBM            : MAE=3.7484 (-0.1138), R²=0.0495\n",
      "\n",
      "================================================================================\n",
      "🔬 Testing with Top 25 Features\n",
      "================================================================================\n",
      "\n",
      "📋 Selected features:\n",
      "    1. dien_tich_phong                          (score: 0.7402)\n",
      "    2. tasks_per_hour                           (score: 0.6650)\n",
      "    3. num_trash_collection_tasks               (score: 0.6648)\n",
      "    4. num_surgery_room_tasks                   (score: 0.6385)\n",
      "    5. dien_tich_hanh_lang                      (score: 0.6369)\n",
      "    6. area_diversity                           (score: 0.6214)\n",
      "    7. num_lobby_tasks                          (score: 0.6026)\n",
      "    8. hour_end                                 (score: 0.5416)\n",
      "    9. dien_tich_ngoai_canh                     (score: 0.4777)\n",
      "   10. monitoring_ratio                         (score: 0.4716)\n",
      "   ... and 15 more\n",
      "\n",
      "🤖 Training models...\n",
      "   Random Forest       : MAE=3.6149 (+0.0197), R²=0.0681\n",
      "   Gradient Boosting   : MAE=3.7803 (-0.1457), R²=-0.0520\n",
      "   XGBoost             : MAE=3.7601 (-0.1255), R²=-0.0180\n",
      "   LightGBM            : MAE=3.7434 (-0.1088), R²=0.0423\n",
      "\n",
      "================================================================================\n",
      "🔬 Testing with Top 30 Features\n",
      "================================================================================\n",
      "\n",
      "📋 Selected features:\n",
      "    1. dien_tich_phong                          (score: 0.7402)\n",
      "    2. tasks_per_hour                           (score: 0.6650)\n",
      "    3. num_trash_collection_tasks               (score: 0.6648)\n",
      "    4. num_surgery_room_tasks                   (score: 0.6385)\n",
      "    5. dien_tich_hanh_lang                      (score: 0.6369)\n",
      "    6. area_diversity                           (score: 0.6214)\n",
      "    7. num_lobby_tasks                          (score: 0.6026)\n",
      "    8. hour_end                                 (score: 0.5416)\n",
      "    9. dien_tich_ngoai_canh                     (score: 0.4777)\n",
      "   10. monitoring_ratio                         (score: 0.4716)\n",
      "   ... and 20 more\n",
      "\n",
      "🤖 Training models...\n",
      "   Random Forest       : MAE=3.6134 (+0.0212), R²=0.0712\n",
      "   Gradient Boosting   : MAE=3.8167 (-0.1821), R²=-0.0845\n",
      "   XGBoost             : MAE=3.8264 (-0.1918), R²=-0.0368\n",
      "   LightGBM            : MAE=3.6914 (-0.0568), R²=0.0629\n",
      "\n",
      "================================================================================\n",
      "🔬 Testing with Top 35 Features\n",
      "================================================================================\n",
      "\n",
      "📋 Selected features:\n",
      "    1. dien_tich_phong                          (score: 0.7402)\n",
      "    2. tasks_per_hour                           (score: 0.6650)\n",
      "    3. num_trash_collection_tasks               (score: 0.6648)\n",
      "    4. num_surgery_room_tasks                   (score: 0.6385)\n",
      "    5. dien_tich_hanh_lang                      (score: 0.6369)\n",
      "    6. area_diversity                           (score: 0.6214)\n",
      "    7. num_lobby_tasks                          (score: 0.6026)\n",
      "    8. hour_end                                 (score: 0.5416)\n",
      "    9. dien_tich_ngoai_canh                     (score: 0.4777)\n",
      "   10. monitoring_ratio                         (score: 0.4716)\n",
      "   ... and 25 more\n",
      "\n",
      "🤖 Training models...\n",
      "   Random Forest       : MAE=3.6404 (-0.0058), R²=0.0730\n",
      "   Gradient Boosting   : MAE=3.9151 (-0.2805), R²=-0.1125\n",
      "   XGBoost             : MAE=3.8845 (-0.2499), R²=-0.0521\n",
      "   LightGBM            : MAE=3.8490 (-0.2144), R²=0.0009\n",
      "\n",
      "================================================================================\n",
      "🔬 Testing with Top 40 Features\n",
      "================================================================================\n",
      "\n",
      "📋 Selected features:\n",
      "    1. dien_tich_phong                          (score: 0.7402)\n",
      "    2. tasks_per_hour                           (score: 0.6650)\n",
      "    3. num_trash_collection_tasks               (score: 0.6648)\n",
      "    4. num_surgery_room_tasks                   (score: 0.6385)\n",
      "    5. dien_tich_hanh_lang                      (score: 0.6369)\n",
      "    6. area_diversity                           (score: 0.6214)\n",
      "    7. num_lobby_tasks                          (score: 0.6026)\n",
      "    8. hour_end                                 (score: 0.5416)\n",
      "    9. dien_tich_ngoai_canh                     (score: 0.4777)\n",
      "   10. monitoring_ratio                         (score: 0.4716)\n",
      "   ... and 30 more\n",
      "\n",
      "🤖 Training models...\n",
      "   Random Forest       : MAE=3.6265 (+0.0081), R²=0.0785\n",
      "   Gradient Boosting   : MAE=3.8842 (-0.2496), R²=-0.0892\n",
      "   XGBoost             : MAE=3.9312 (-0.2966), R²=-0.0725\n",
      "   LightGBM            : MAE=3.7473 (-0.1127), R²=0.0294\n",
      "\n",
      "================================================================================\n",
      "🏆 BEST FEATURE SELECTION RESULT\n",
      "================================================================================\n",
      "   🤖 Model: Random Forest\n",
      "   📋 Features: 30 selected\n",
      "\n",
      "   📈 Best Model Performance:\n",
      "      Val MAE: 3.6134\n",
      "      Val R²:  0.0712\n",
      "\n",
      "   📊 Comparison with Baseline:\n",
      "      Baseline MAE: 3.6346 | R²: -0.0129\n",
      "      Best MAE:     3.6134 | R²: 0.0712\n",
      "      MAE Improvement: 0.0212 (0.58%)\n",
      "      R² Improvement:  0.0842 (+8.42%)\n",
      "\n",
      "✅ Saved: models/feature_selection_results.csv\n"
     ]
    }
   ],
   "source": [
    "print(\"=\"*80)\n",
    "print(\"🔬 FEATURE SELECTION\")\n",
    "print(\"=\"*80)\n",
    "\n",
    "# Prepare data\n",
    "print(\"\\n📏 Preparing data for feature selection...\")\n",
    "\n",
    "# Drop non-numeric columns\n",
    "non_numeric_cols_fs = X_train.select_dtypes(exclude=['int64', 'float64']).columns.tolist()\n",
    "if non_numeric_cols_fs:\n",
    "    print(f\"   Dropping {len(non_numeric_cols_fs)} non-numeric columns: {non_numeric_cols_fs}\")\n",
    "    X_train_fs = X_train.drop(columns=non_numeric_cols_fs).copy()\n",
    "    X_val_fs_temp = X_val.drop(columns=non_numeric_cols_fs).copy()\n",
    "else:\n",
    "    X_train_fs = X_train.copy()\n",
    "    X_val_fs_temp = X_val.copy()\n",
    "\n",
    "# Identify columns to scale\n",
    "binary_cols_fs = [col for col in X_train_fs.columns if X_train_fs[col].nunique() == 2 and set(X_train_fs[col].unique()).issubset({0, 1})]\n",
    "onehot_cols_fs = [col for col in X_train_fs.columns if col.startswith('loai_ca_')]\n",
    "cols_to_scale_fs = [col for col in X_train_fs.columns if col not in binary_cols_fs and col not in onehot_cols_fs]\n",
    "\n",
    "# Scale\n",
    "scaler_fs = StandardScaler()\n",
    "if cols_to_scale_fs:\n",
    "    X_train_fs[cols_to_scale_fs] = scaler_fs.fit_transform(X_train_fs[cols_to_scale_fs])\n",
    "    X_val_fs_temp[cols_to_scale_fs] = scaler_fs.transform(X_val_fs_temp[cols_to_scale_fs])\n",
    "    print(f\"   ✅ Scaled {len(cols_to_scale_fs)} numeric features\")\n",
    "\n",
    "# Method 1: Random Forest Feature Importance\n",
    "print(\"\\n🌲 Method 1: Random Forest Feature Importance\")\n",
    "rf_selector = RandomForestRegressor(n_estimators=100, max_depth=15, random_state=RANDOM_STATE, n_jobs=-1)\n",
    "rf_selector.fit(X_train_fs, y_train)\n",
    "\n",
    "rf_importance_df = pd.DataFrame({\n",
    "    'feature': X_train_fs.columns,\n",
    "    'importance': rf_selector.feature_importances_\n",
    "}).sort_values('importance', ascending=False)\n",
    "\n",
    "print(f\"   Top 10 features:\")\n",
    "for i, row in rf_importance_df.head(10).iterrows():\n",
    "    print(f\"      {row['feature']:40s}: {row['importance']:.4f}\")\n",
    "\n",
    "# Method 2: Mutual Information\n",
    "print(\"\\n📊 Method 2: Mutual Information\")\n",
    "mi_scores = mutual_info_regression(X_train_fs, y_train, random_state=RANDOM_STATE)\n",
    "mi_importance_df = pd.DataFrame({\n",
    "    'feature': X_train_fs.columns,\n",
    "    'importance': mi_scores\n",
    "}).sort_values('importance', ascending=False)\n",
    "\n",
    "print(f\"   Top 10 features:\")\n",
    "for i, row in mi_importance_df.head(10).iterrows():\n",
    "    print(f\"      {row['feature']:40s}: {row['importance']:.4f}\")\n",
    "\n",
    "# Method 3: Correlation with target\n",
    "print(\"\\n📈 Method 3: Correlation with Target\")\n",
    "correlations = X_train_fs.corrwith(y_train).abs().sort_values(ascending=False)\n",
    "corr_df = pd.DataFrame({\n",
    "    'feature': correlations.index,\n",
    "    'importance': correlations.values\n",
    "})\n",
    "\n",
    "print(f\"   Top 10 features:\")\n",
    "for i, row in corr_df.head(10).iterrows():\n",
    "    print(f\"      {row['feature']:40s}: {row['importance']:.4f}\")\n",
    "\n",
    "# Aggregate scores\n",
    "print(\"\\n🎯 Aggregating Feature Scores...\")\n",
    "aggregated = pd.DataFrame({\n",
    "    'feature': X_train_fs.columns,\n",
    "    'rf_score': rf_importance_df.set_index('feature')['importance'],\n",
    "    'mi_score': mi_importance_df.set_index('feature')['importance'],\n",
    "    'corr_score': corr_df.set_index('feature')['importance']\n",
    "})\n",
    "\n",
    "# Normalize scores to [0,1]\n",
    "for col in ['rf_score', 'mi_score', 'corr_score']:\n",
    "    aggregated[col] = (aggregated[col] - aggregated[col].min()) / (aggregated[col].max() - aggregated[col].min())\n",
    "\n",
    "aggregated['avg_score'] = aggregated[['rf_score', 'mi_score', 'corr_score']].mean(axis=1)\n",
    "aggregated = aggregated.sort_values('avg_score', ascending=False)\n",
    "\n",
    "print(f\"\\n📋 Top 20 Features (by average score):\")\n",
    "for i, row in aggregated.head(20).iterrows():\n",
    "    print(f\"   {row['feature']:40s}: {row['avg_score']:.4f}\")\n",
    "\n",
    "# Test with different numbers of features\n",
    "print(\"\\n\" + \"=\"*80)\n",
    "print(\"🔬 TESTING WITH DIFFERENT NUMBERS OF FEATURES\")\n",
    "print(\"=\"*80)\n",
    "\n",
    "X_val_fs = X_val_fs_temp.copy()\n",
    "\n",
    "# Test feature counts\n",
    "top_k_values = [20, 25, 30, 35, 40, 50]\n",
    "feature_selection_results = []\n",
    "\n",
    "# Baseline: all features\n",
    "baseline_model = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1, max_depth=5, random_state=RANDOM_STATE)\n",
    "baseline_model.fit(X_train_fs, y_train)\n",
    "baseline_pred = baseline_model.predict(X_val_fs)\n",
    "baseline_mae = mean_absolute_error(y_val, baseline_pred)\n",
    "baseline_r2 = r2_score(y_val, baseline_pred)\n",
    "\n",
    "feature_selection_results.append({\n",
    "    'num_features': X_train_fs.shape[1],\n",
    "    'feature_set': 'All Features',\n",
    "    'model': 'Gradient Boosting',\n",
    "    'val_mae': baseline_mae,\n",
    "    'val_r2': baseline_r2\n",
    "})\n",
    "\n",
    "print(f\"\\n📌 BASELINE (All {X_train_fs.shape[1]} Features):\")\n",
    "print(f\"   Val MAE: {baseline_mae:.4f}, R²: {baseline_r2:.4f}\")\n",
    "\n",
    "# Test with top-k features\n",
    "for k in top_k_values:\n",
    "    if k >= X_train_fs.shape[1]:\n",
    "        continue\n",
    "        \n",
    "    print(f\"\\n{'='*80}\")\n",
    "    print(f\"🔬 Testing with Top {k} Features\")\n",
    "    print(f\"{'='*80}\")\n",
    "    \n",
    "    selected_features = aggregated.head(k)['feature'].tolist()\n",
    "    X_train_selected = X_train_fs[selected_features]\n",
    "    X_val_selected = X_val_fs[selected_features]\n",
    "    \n",
    "    print(f\"\\n📋 Selected features:\")\n",
    "    for i, feat in enumerate(selected_features[:10], 1):\n",
    "        score = aggregated[aggregated['feature'] == feat]['avg_score'].values[0]\n",
    "        print(f\"   {i:2d}. {feat:40s} (score: {score:.4f})\")\n",
    "    if k > 10:\n",
    "        print(f\"   ... and {k-10} more\")\n",
    "    \n",
    "    # Test multiple models\n",
    "    models_to_test = {\n",
    "        'Random Forest': RandomForestRegressor(n_estimators=100, max_depth=15, min_samples_split=5, random_state=RANDOM_STATE, n_jobs=-1),\n",
    "        'Gradient Boosting': GradientBoostingRegressor(n_estimators=100, learning_rate=0.1, max_depth=5, random_state=RANDOM_STATE),\n",
    "        'XGBoost': xgb.XGBRegressor(n_estimators=100, learning_rate=0.1, max_depth=5, random_state=RANDOM_STATE, n_jobs=-1),\n",
    "        'LightGBM': lgb.LGBMRegressor(n_estimators=100, learning_rate=0.1, max_depth=5, num_leaves=31, random_state=RANDOM_STATE, n_jobs=-1, verbose=-1)\n",
    "    }\n",
    "    \n",
    "    print(\"\\n🤖 Training models...\")\n",
    "    for model_name, model in models_to_test.items():\n",
    "        model.fit(X_train_selected, y_train)\n",
    "        y_val_pred = model.predict(X_val_selected)\n",
    "        \n",
    "        val_mae = mean_absolute_error(y_val, y_val_pred)\n",
    "        val_r2 = r2_score(y_val, y_val_pred)\n",
    "        \n",
    "        feature_selection_results.append({\n",
    "            'num_features': k,\n",
    "            'feature_set': f'Top {k}',\n",
    "            'model': model_name,\n",
    "            'val_mae': val_mae,\n",
    "            'val_r2': val_r2\n",
    "        })\n",
    "        \n",
    "        improvement = baseline_mae - val_mae\n",
    "        print(f\"   {model_name:20s}: MAE={val_mae:.4f} ({'+' if improvement > 0 else ''}{improvement:.4f}), R²={val_r2:.4f}\")\n",
    "\n",
    "# Find best result\n",
    "fs_results_df = pd.DataFrame(feature_selection_results)\n",
    "best_idx = fs_results_df['val_mae'].idxmin()\n",
    "best_result = fs_results_df.loc[best_idx]\n",
    "\n",
    "print(\"\\n\" + \"=\"*80)\n",
    "print(\"🏆 BEST FEATURE SELECTION RESULT\")\n",
    "print(\"=\"*80)\n",
    "print(f\"   🤖 Model: {best_result['model']}\")\n",
    "print(f\"   📋 Features: {int(best_result['num_features'])} selected\")\n",
    "print(f\"\\n   📈 Best Model Performance:\")\n",
    "print(f\"      Val MAE: {best_result['val_mae']:.4f}\")\n",
    "print(f\"      Val R²:  {best_result['val_r2']:.4f}\")\n",
    "print(f\"\\n   📊 Comparison with Baseline:\")\n",
    "print(f\"      Baseline MAE: {baseline_mae:.4f} | R²: {baseline_r2:.4f}\")\n",
    "print(f\"      Best MAE:     {best_result['val_mae']:.4f} | R²: {best_result['val_r2']:.4f}\")\n",
    "print(f\"      MAE Improvement: {baseline_mae - best_result['val_mae']:.4f} ({(baseline_mae - best_result['val_mae'])/baseline_mae*100:.2f}%)\")\n",
    "print(f\"      R² Improvement:  {best_result['val_r2'] - baseline_r2:.4f} ({'+' if best_result['val_r2'] > baseline_r2 else ''}{(best_result['val_r2'] - baseline_r2)*100:.2f}%)\")\n",
    "\n",
    "# Save results\n",
    "fs_results_df.to_csv('models/feature_selection_results.csv', index=False)\n",
    "print(\"\\n✅ Saved: models/feature_selection_results.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27452d46",
   "metadata": {},
   "source": [
    "# 9️⃣ Final Training với Best Features\n",
    "\n",
    "**Mục đích:** Train model cuối cùng với best model + best features. Combine train+val data để train, evaluate trên test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08e1918d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================================================================\n",
      "🎯 FINAL TRAINING WITH BEST FEATURES\n",
      "================================================================================\n",
      "\n",
      "📋 Best Configuration:\n",
      "   Model: Random Forest\n",
      "   Number of Features: 30\n",
      "\n",
      "📏 Preparing scaled data...\n",
      "   Dropping 14 non-numeric columns\n",
      "   ✅ Scaled 49 numeric features\n",
      "\n",
      "📋 Using Top 30 Selected Features:\n",
      "    1. dien_tich_phong                          (score: 0.7402)\n",
      "    2. tasks_per_hour                           (score: 0.6650)\n",
      "    3. num_trash_collection_tasks               (score: 0.6648)\n",
      "    4. num_surgery_room_tasks                   (score: 0.6385)\n",
      "    5. dien_tich_hanh_lang                      (score: 0.6369)\n",
      "    6. area_diversity                           (score: 0.6214)\n",
      "    7. num_lobby_tasks                          (score: 0.6026)\n",
      "    8. hour_end                                 (score: 0.5416)\n",
      "    9. dien_tich_ngoai_canh                     (score: 0.4777)\n",
      "   10. monitoring_ratio                         (score: 0.4716)\n",
      "   11. num_patient_room_tasks                   (score: 0.4620)\n",
      "   12. num_monitoring_tasks                     (score: 0.4545)\n",
      "   13. tasks_per_floor                          (score: 0.4460)\n",
      "   14. work_hours_numeric                       (score: 0.4020)\n",
      "   15. trash_collection_ratio                   (score: 0.3944)\n",
      "   ... and 15 more\n",
      "\n",
      "📊 Dataset Shapes:\n",
      "   Train+Val: (356, 30) (356 samples)\n",
      "   Test: (89, 30) (89 samples)\n",
      "\n",
      "🤖 Initializing Random Forest...\n",
      "🔄 Training on Train+Val data...\n",
      "   ✅ Training completed!\n",
      "\n",
      "🔮 Predicting on Test set...\n",
      "\n",
      "================================================================================\n",
      "📊 FINAL TEST SET RESULTS\n",
      "================================================================================\n",
      "\n",
      "🎯 Model: Random Forest\n",
      "📋 Features: 30 selected features\n",
      "\n",
      "📈 Test Set Performance:\n",
      "   MAE:  2.9597\n",
      "   MSE:  22.5281\n",
      "   RMSE: 4.7464\n",
      "   R²:   0.4147\n",
      "\n",
      "📊 Comparison with Baseline:\n",
      "   ⚪ Baseline (All features):   MAE=3.6346 | R²=-0.0129\n",
      "   🎯 Final Model (30 features): MAE=2.9597 | R²=0.4147\n",
      "\n",
      "   📈 Improvement:\n",
      "      MAE giảm: 0.6749 (18.57%)\n",
      "      R² tăng:  0.4276 (+42.76%)\n",
      "\n",
      "📊 Creating visualizations...\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 173084 characters]",
      "text/plain": [
       "<Figure size 1500x1200 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Saved: plts/final_model_test_results.png\n"
     ]
    }
   ],
   "source": [
    "print(\"=\"*80)\n",
    "print(\"🎯 FINAL TRAINING WITH BEST FEATURES\")\n",
    "print(\"=\"*80)\n",
    "\n",
    "# Get best configuration\n",
    "best_num_features = int(best_result['num_features'])\n",
    "best_model_name = best_result['model']\n",
    "\n",
    "print(f\"\\n📋 Best Configuration:\")\n",
    "print(f\"   Model: {best_model_name}\")\n",
    "print(f\"   Number of Features: {best_num_features}\")\n",
    "\n",
    "# Prepare scaled data for all sets\n",
    "print(\"\\n📏 Preparing scaled data...\")\n",
    "\n",
    "# Drop non-numeric columns\n",
    "non_numeric_cols_final = X_train.select_dtypes(exclude=['int64', 'float64']).columns.tolist()\n",
    "if non_numeric_cols_final:\n",
    "    print(f\"   Dropping {len(non_numeric_cols_final)} non-numeric columns\")\n",
    "    X_train_final_scaled = X_train.drop(columns=non_numeric_cols_final).copy()\n",
    "    X_val_final_scaled = X_val.drop(columns=non_numeric_cols_final).copy()\n",
    "    X_test_final_scaled = X_test.drop(columns=non_numeric_cols_final).copy()\n",
    "else:\n",
    "    X_train_final_scaled = X_train.copy()\n",
    "    X_val_final_scaled = X_val.copy()\n",
    "    X_test_final_scaled = X_test.copy()\n",
    "\n",
    "# Identify columns to scale\n",
    "binary_cols_final = [col for col in X_train_final_scaled.columns if X_train_final_scaled[col].nunique() == 2 and set(X_train_final_scaled[col].unique()).issubset({0, 1})]\n",
    "onehot_cols_final = [col for col in X_train_final_scaled.columns if col.startswith('loai_ca_')]\n",
    "cols_to_scale_final = [col for col in X_train_final_scaled.columns if col not in binary_cols_final and col not in onehot_cols_final]\n",
    "\n",
    "scaler_final = StandardScaler()\n",
    "if cols_to_scale_final:\n",
    "    X_train_final_scaled[cols_to_scale_final] = scaler_final.fit_transform(X_train_final_scaled[cols_to_scale_final])\n",
    "    X_val_final_scaled[cols_to_scale_final] = scaler_final.transform(X_val_final_scaled[cols_to_scale_final])\n",
    "    X_test_final_scaled[cols_to_scale_final] = scaler_final.transform(X_test_final_scaled[cols_to_scale_final])\n",
    "    print(f\"   ✅ Scaled {len(cols_to_scale_final)} numeric features\")\n",
    "\n",
    "# Select best features\n",
    "if best_num_features < X_train_final_scaled.shape[1]:\n",
    "    selected_features = aggregated.head(best_num_features)['feature'].tolist()\n",
    "    \n",
    "    print(f\"\\n📋 Using Top {best_num_features} Selected Features\")\n",
    "    print(f\"   Top 5: {', '.join(selected_features[:5])}\")\n",
    "    if best_num_features > 5:\n",
    "        print(f\"   ... and {best_num_features - 5} more features\")\n",
    "    \n",
    "    X_train_final = X_train_final_scaled[selected_features]\n",
    "    X_val_final = X_val_final_scaled[selected_features]\n",
    "    X_test_final = X_test_final_scaled[selected_features]\n",
    "else:\n",
    "    print(f\"\\n📋 Using ALL {X_train_final_scaled.shape[1]} Features\")\n",
    "    selected_features = X_train_final_scaled.columns.tolist()\n",
    "    X_train_final = X_train_final_scaled\n",
    "    X_val_final = X_val_final_scaled\n",
    "    X_test_final = X_test_final_scaled\n",
    "\n",
    "# Combine train + validation for final training\n",
    "X_train_val = pd.concat([X_train_final, X_val_final])\n",
    "y_train_val = pd.concat([y_train, y_val])\n",
    "\n",
    "# Initialize best model\n",
    "if best_model_name == 'Random Forest':\n",
    "    final_model = RandomForestRegressor(n_estimators=150, max_depth=20, min_samples_split=5, random_state=RANDOM_STATE, n_jobs=-1)\n",
    "elif best_model_name == 'Gradient Boosting':\n",
    "    final_model = GradientBoostingRegressor(n_estimators=150, learning_rate=0.05, max_depth=7, random_state=RANDOM_STATE)\n",
    "elif best_model_name == 'XGBoost':\n",
    "    final_model = xgb.XGBRegressor(n_estimators=150, learning_rate=0.05, max_depth=7, random_state=RANDOM_STATE, n_jobs=-1)\n",
    "elif best_model_name == 'LightGBM':\n",
    "    final_model = lgb.LGBMRegressor(n_estimators=150, learning_rate=0.05, max_depth=7, num_leaves=31, random_state=RANDOM_STATE, n_jobs=-1, verbose=-1)\n",
    "else:\n",
    "    final_model = GradientBoostingRegressor(n_estimators=150, learning_rate=0.05, max_depth=7, random_state=RANDOM_STATE)\n",
    "\n",
    "# Train on combined train+val\n",
    "print(f\"\\n🔄 Training {best_model_name} on {len(X_train_val)} samples...\")\n",
    "final_model.fit(X_train_val, y_train_val)\n",
    "\n",
    "# Predict on test set\n",
    "print(f\"🔮 Predicting on {len(X_test_final)} test samples...\")\n",
    "y_test_pred = final_model.predict(X_test_final)\n",
    "\n",
    "# Calculate metrics\n",
    "test_mae = mean_absolute_error(y_test, y_test_pred)\n",
    "test_mse = mean_squared_error(y_test, y_test_pred)\n",
    "test_rmse = np.sqrt(test_mse)\n",
    "test_r2 = r2_score(y_test, y_test_pred)\n",
    "\n",
    "print(\"\\n\" + \"=\"*80)\n",
    "print(\"📊 FINAL TEST SET RESULTS\")\n",
    "print(\"=\"*80)\n",
    "print(f\"🎯 Model: {best_model_name} | Features: {best_num_features}\")\n",
    "print(f\"\\n📈 Test Performance: MAE={test_mae:.4f} | RMSE={test_rmse:.4f} | R²={test_r2:.4f}\")\n",
    "print(f\"📊 Baseline (All):   MAE={baseline_mae:.4f} | R²={baseline_r2:.4f}\")\n",
    "print(f\"📈 Improvement:      MAE giảm {baseline_mae - test_mae:.4f} ({(baseline_mae - test_mae)/baseline_mae*100:.1f}%) | R² tăng {test_r2 - baseline_r2:.4f} ({(test_r2 - baseline_r2)*100:.1f}%)\")\n",
    "\n",
    "# Visualizations\n",
    "print(f\"\\n📊 Creating visualizations...\")\n",
    "\n",
    "fig, axes = plt.subplots(2, 2, figsize=(15, 12))\n",
    "\n",
    "# 1. Predicted vs Actual\n",
    "axes[0, 0].scatter(y_test, y_test_pred, alpha=0.6, edgecolors='k', s=50)\n",
    "axes[0, 0].plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], 'r--', lw=2)\n",
    "axes[0, 0].set_xlabel('Actual Values', fontsize=11)\n",
    "axes[0, 0].set_ylabel('Predicted Values', fontsize=11)\n",
    "axes[0, 0].set_title(f'{best_model_name} - Test Set\\nPredicted vs Actual', fontsize=12, fontweight='bold')\n",
    "axes[0, 0].grid(True, alpha=0.3)\n",
    "axes[0, 0].text(0.05, 0.95, f'R² = {test_r2:.4f}\\nMAE = {test_mae:.4f}',\n",
    "                transform=axes[0, 0].transAxes, fontsize=10, verticalalignment='top',\n",
    "                bbox=dict(boxstyle='round', facecolor='wheat', alpha=0.5))\n",
    "\n",
    "# 2. Residuals plot\n",
    "residuals = y_test - y_test_pred\n",
    "axes[0, 1].scatter(y_test_pred, residuals, alpha=0.6, edgecolors='k', s=50)\n",
    "axes[0, 1].axhline(y=0, color='r', linestyle='--', lw=2)\n",
    "axes[0, 1].set_xlabel('Predicted Values', fontsize=11)\n",
    "axes[0, 1].set_ylabel('Residuals', fontsize=11)\n",
    "axes[0, 1].set_title('Residual Plot', fontsize=12, fontweight='bold')\n",
    "axes[0, 1].grid(True, alpha=0.3)\n",
    "\n",
    "# 3. Residuals distribution\n",
    "axes[1, 0].hist(residuals, bins=30, edgecolor='black', alpha=0.7, color='skyblue')\n",
    "axes[1, 0].axvline(x=0, color='r', linestyle='--', lw=2)\n",
    "axes[1, 0].set_xlabel('Residuals', fontsize=11)\n",
    "axes[1, 0].set_ylabel('Frequency', fontsize=11)\n",
    "axes[1, 0].set_title('Residual Distribution', fontsize=12, fontweight='bold')\n",
    "axes[1, 0].grid(True, alpha=0.3)\n",
    "\n",
    "# 4. Feature importance (if available)\n",
    "if hasattr(final_model, 'feature_importances_'):\n",
    "    feature_importance = pd.DataFrame({\n",
    "        'feature': selected_features,\n",
    "        'importance': final_model.feature_importances_\n",
    "    }).sort_values('importance', ascending=False).head(15)\n",
    "    \n",
    "    axes[1, 1].barh(range(len(feature_importance)), feature_importance['importance'], alpha=0.7)\n",
    "    axes[1, 1].set_yticks(range(len(feature_importance)))\n",
    "    axes[1, 1].set_yticklabels(feature_importance['feature'], fontsize=9)\n",
    "    axes[1, 1].set_xlabel('Importance', fontsize=11)\n",
    "    axes[1, 1].set_title('Top 15 Feature Importance', fontsize=12, fontweight='bold')\n",
    "    axes[1, 1].invert_yaxis()\n",
    "    axes[1, 1].grid(True, alpha=0.3, axis='x')\n",
    "else:\n",
    "    axes[1, 1].text(0.5, 0.5, f'Test MAE: {test_mae:.4f}\\nTest R²: {test_r2:.4f}\\n{best_num_features} features used',\n",
    "                    transform=axes[1, 1].transAxes, fontsize=14, ha='center', va='center',\n",
    "                    bbox=dict(boxstyle='round', facecolor='lightblue', alpha=0.3))\n",
    "    axes[1, 1].axis('off')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.savefig('plots/final_model_test_results.png', dpi=300, bbox_inches='tight')\n",
    "plt.show()\n",
    "\n",
    "print(\"✅ Visualization saved!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61aaae99",
   "metadata": {},
   "source": [
    "# 🔟 Save Model & Artifacts\n",
    "\n",
    "**Mục đích:** Lưu model, scaler, selected features list, và training report để sử dụng sau này."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "bc3da137",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================================================================\n",
      "💾 SAVING MODEL & ARTIFACTS\n",
      "================================================================================\n",
      "\n",
      "✅ Saved model: models/final_model_random_forest_top15.pkl\n",
      "✅ Saved scaler: models/scaler_final.pkl\n",
      "✅ Saved features list: models/selected_features_top15.txt\n",
      "✅ Saved columns to scale: models/columns_to_scale.txt\n",
      "✅ Saved training report: models/training_report_final.json\n",
      "\n",
      "================================================================================\n",
      "📋 TRAINING SUMMARY\n",
      "================================================================================\n",
      "\n",
      "🎯 Model Configuration:\n",
      "   Model Type: Random Forest\n",
      "   Features Used: 15/57\n",
      "   Feature Reduction: 73.7%\n",
      "\n",
      "📊 Dataset:\n",
      "   Training Samples: 356 (Train+Val combined)\n",
      "   Test Samples: 89\n",
      "\n",
      "📈 Performance:\n",
      "   Test MAE:  3.0054\n",
      "   Test RMSE: 5.0646\n",
      "   Test R²:   0.3336\n",
      "\n",
      "🎯 Improvement:\n",
      "   Baseline Val MAE: 4.5372\n",
      "   Final Test MAE:   3.0054\n",
      "   Improvement:      1.5318 (33.76%)\n",
      "\n",
      "💾 Saved Files:\n",
      "   • models/final_model_random_forest_top15.pkl\n",
      "   • models/scaler_final.pkl\n",
      "   • models/selected_features_top15.txt\n",
      "   • models/columns_to_scale.txt\n",
      "   • models/training_report_final.json\n",
      "   • plots/final_model_test_results.png\n",
      "   • models/feature_selection_results.csv\n",
      "\n",
      "================================================================================\n",
      "✅ PIPELINE COMPLETED SUCCESSFULLY!\n",
      "================================================================================\n",
      "\n",
      "💡 Next Steps:\n",
      "   1. Review test set predictions and error analysis\n",
      "   2. Try hyperparameter tuning for further improvement\n",
      "   3. Deploy model to production\n",
      "   4. Monitor model performance over time\n"
     ]
    }
   ],
   "source": [
    "print(\"=\"*80)\n",
    "print(\"💾 SAVING MODEL & ARTIFACTS\")\n",
    "print(\"=\"*80)\n",
    "\n",
    "# 1. Save the trained model\n",
    "model_filename = f'models/final_model_{best_model_name.lower().replace(\" \", \"_\")}_top{best_num_features}.pkl'\n",
    "with open(model_filename, 'wb') as f:\n",
    "    pickle.dump(final_model, f)\n",
    "print(f\"\\n✅ Saved model: {model_filename}\")\n",
    "\n",
    "# 2. Save the scaler\n",
    "scaler_filename = 'models/scaler_final.pkl'\n",
    "with open(scaler_filename, 'wb') as f:\n",
    "    pickle.dump(scaler_final, f)\n",
    "print(f\"✅ Saved scaler: {scaler_filename}\")\n",
    "\n",
    "# 3. Save selected features list\n",
    "features_filename = f'models/selected_features_top{best_num_features}.txt'\n",
    "with open(features_filename, 'w', encoding='utf-8') as f:\n",
    "    f.write('\\n'.join(selected_features))\n",
    "print(f\"✅ Saved features list: {features_filename}\")\n",
    "\n",
    "# 4. Save columns to scale list\n",
    "cols_to_scale_filename = 'models/columns_to_scale.txt'\n",
    "with open(cols_to_scale_filename, 'w', encoding='utf-8') as f:\n",
    "    f.write('\\n'.join(cols_to_scale_final))\n",
    "print(f\"✅ Saved columns to scale: {cols_to_scale_filename}\")\n",
    "\n",
    "# 5. Save training report\n",
    "report = {\n",
    "    'timestamp': datetime.now().strftime('%Y-%m-%d %H:%M:%S'),\n",
    "    'model_name': best_model_name,\n",
    "    'num_features': best_num_features,\n",
    "    'total_features_available': X_train_final_scaled.shape[1],\n",
    "    'training_samples': len(X_train_val),\n",
    "    'test_samples': len(y_test),\n",
    "    'test_mae': float(test_mae),\n",
    "    'test_mse': float(test_mse),\n",
    "    'test_rmse': float(test_rmse),\n",
    "    'test_r2': float(test_r2),\n",
    "    'baseline_val_mae': float(baseline_mae),\n",
    "    'improvement_vs_baseline': float(baseline_mae - test_mae),\n",
    "    'improvement_pct': float((baseline_mae - test_mae) / baseline_mae * 100) if baseline_mae > 0 else 0,\n",
    "    'k_fold_cv': 'Completed (5-fold)',\n",
    "    'feature_selection_methods': ['Random Forest', 'Mutual Information', 'Correlation'],\n",
    "    'selected_features': selected_features,\n",
    "    'cols_to_scale': cols_to_scale_final\n",
    "}\n",
    "\n",
    "report_filename = 'models/training_report_final.json'\n",
    "with open(report_filename, 'w', encoding='utf-8') as f:\n",
    "    json.dump(report, f, indent=4, ensure_ascii=False)\n",
    "print(f\"✅ Saved training report: {report_filename}\")\n",
    "\n",
    "# Print summary\n",
    "print(\"\\n\" + \"=\"*80)\n",
    "print(\"📋 TRAINING SUMMARY\")\n",
    "print(\"=\"*80)\n",
    "print(f\"\\n🎯 Model Configuration:\")\n",
    "print(f\"   Model Type: {best_model_name}\")\n",
    "print(f\"   Features Used: {best_num_features}/{X_train_final_scaled.shape[1]}\")\n",
    "\n",
    "print(f\"\\n📊 Dataset:\")\n",
    "print(f\"   Training Samples: {len(X_train_val)} (Train+Val combined)\")\n",
    "print(f\"   Test Samples: {len(y_test)}\")\n",
    "\n",
    "print(f\"\\n📈 Performance:\")\n",
    "print(f\"   Test MAE:  {test_mae:.4f}\")\n",
    "print(f\"   Test RMSE: {test_rmse:.4f}\")\n",
    "print(f\"   Test R²:   {test_r2:.4f}\")\n",
    "\n",
    "print(f\"\\n🎯 Improvement:\")\n",
    "print(f\"   Baseline Val MAE: {baseline_mae:.4f}\")\n",
    "print(f\"   Final Test MAE:   {test_mae:.4f}\")\n",
    "if baseline_mae > 0:\n",
    "    print(f\"   Improvement:      {baseline_mae - test_mae:.4f} ({(baseline_mae - test_mae)/baseline_mae*100:.2f}%)\")\n",
    "\n",
    "print(f\"\\n💾 Saved Files:\")\n",
    "print(f\"   • {model_filename}\")\n",
    "print(f\"   • {scaler_filename}\")\n",
    "print(f\"   • {features_filename}\")\n",
    "print(f\"   • {cols_to_scale_filename}\")\n",
    "print(f\"   • {report_filename}\")\n",
    "print(f\"   • plots/final_model_test_results.png\")\n",
    "print(f\"   • models/feature_selection_results.csv\")\n",
    "print(f\"   • models/kfold_cv_results.csv\")\n",
    "\n",
    "print(\"\\n\" + \"=\"*80)\n",
    "print(\"✅ PIPELINE COMPLETED SUCCESSFULLY!\")\n",
    "print(\"=\"*80)\n",
    "print(\"\\n💡 Next Steps:\")\n",
    "print(\"   1. Review test set predictions and error analysis\")\n",
    "print(\"   2. Try hyperparameter tuning with GridSearchCV\")\n",
    "print(\"   3. Consider ensemble methods (Voting/Stacking)\")\n",
    "print(\"   4. Deploy model to production\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8af7c09d",
   "metadata": {},
   "source": [
    "# 🎉 Pipeline Hoàn Thành!\n",
    "\n",
    "---\n",
    "\n",
    "## ✅ Đã Hoàn Thành:\n",
    "\n",
    "1. ✅ **Data Loading & Cleaning** - Load và làm sạch dữ liệu\n",
    "2. ✅ **Feature Engineering** - Tạo features mới, encode categorical\n",
    "3. ✅ **Train/Val/Test Split** - Chia dữ liệu 60/20/20\n",
    "4. ✅ **K-Fold Cross-Validation** - Test độ ổn định với 5-fold CV\n",
    "5. ✅ **Feature Selection** - Tìm best features với 3 phương pháp\n",
    "6. ✅ **Final Training** - Train model với best configuration\n",
    "7. ✅ **Test Evaluation** - Đánh giá trên test set\n",
    "8. ✅ **Save Artifacts** - Lưu model, scaler, features, report\n",
    "\n",
    "---\n",
    "\n",
    "## 📊 Kết Quả Cuối Cùng:\n",
    "\n",
    "- **Model:** Best model được chọn qua feature selection\n",
    "- **Features:** Best number of features được tối ưu hóa\n",
    "- **Performance:** Test MAE, RMSE, R² được đo trên test set\n",
    "- **Improvement:** So sánh với baseline model\n",
    "\n",
    "---\n",
    "\n",
    "## 💾 Files Đã Tạo:\n",
    "\n",
    "- `models/final_model_*.pkl` - Trained model\n",
    "- `models/scaler_final.pkl` - Feature scaler\n",
    "- `models/selected_features_top*.txt` - Danh sách features được chọn\n",
    "- `models/columns_to_scale.txt` - Danh sách columns cần scale\n",
    "- `models/training_report_final.json` - Báo cáo chi tiết\n",
    "- `models/feature_selection_results.csv` - Kết quả feature selection\n",
    "- `plots/final_model_test_results.png` - Visualizations\n",
    "\n",
    "---\n",
    "\n",
    "## 🚀 Cách Sử Dụng Model:\n",
    "\n",
    "```python\n",
    "import pickle\n",
    "import pandas as pd\n",
    "\n",
    "# Load model và artifacts\n",
    "with open('models/final_model_*.pkl', 'rb') as f:\n",
    "    model = pickle.load(f)\n",
    "\n",
    "with open('models/scaler_final.pkl', 'rb') as f:\n",
    "    scaler = pickle.load(f)\n",
    "\n",
    "# Load selected features\n",
    "with open('models/selected_features_top*.txt', 'r') as f:\n",
    "    selected_features = [line.strip() for line in f]\n",
    "\n",
    "# Load columns to scale\n",
    "with open('models/columns_to_scale.txt', 'r') as f:\n",
    "    cols_to_scale = [line.strip() for line in f]\n",
    "\n",
    "# Prepare new data\n",
    "new_data = pd.DataFrame(...)  # Your new data\n",
    "new_data_scaled = new_data.copy()\n",
    "new_data_scaled[cols_to_scale] = scaler.transform(new_data[cols_to_scale])\n",
    "new_data_selected = new_data_scaled[selected_features]\n",
    "\n",
    "# Predict\n",
    "predictions = model.predict(new_data_selected)\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## 💡 Recommendations:\n",
    "\n",
    "1. **Hyperparameter Tuning:** Sử dụng GridSearchCV hoặc RandomizedSearchCV\n",
    "2. **Ensemble Methods:** Thử kết hợp nhiều models (Voting, Stacking)\n",
    "3. **Error Analysis:** Phân tích các predictions sai nhiều nhất\n",
    "4. **Feature Engineering:** Thêm features mới từ domain knowledge\n",
    "5. **Data Collection:** Thu thập thêm data để cải thiện model\n",
    "\n",
    "---\n",
    "\n",
    "**Version:** 2.0 - Complete Pipeline  \n",
    "**Created:** January 8, 2026  \n",
    "**Status:** ✅ COMPLETED"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
