{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing Feature Ranking Methods\n",
    "Different methods can rank features differently. This notebook compares multi-pass permutation importance, ALE variance-based ranking, SHAP-based ranking, and grouped permutation importance to provide a comprehensive view of feature relevance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import skexplain\n",
    "import plotting_config\n",
    "import shap\n",
    "from skexplain.common.utils import shap_values_to_importance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimators = skexplain.load_models()\n",
    "X, y = skexplain.load_data()\n",
    "X = X.astype({'urban': 'category', 'rural': 'category'})\n",
    "\n",
    "print(f'X Shape : {X.shape}')\n",
    "print(f'y Skew : {y.mean()*100:.1f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "explainer = skexplain.ExplainToolkit(estimators=estimators, X=X, y=y)\n",
    "\n",
    "explainer.set_plotting_config(\n",
    "    display_feature_names=plotting_config.display_feature_names,\n",
    "    display_units=plotting_config.display_units,\n",
    "    feature_colors=plotting_config.color_dict,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multi-Pass Permutation Importance\n",
    "The standard backward multi-pass method progressively permutes features and measures the drop in model performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "perm_results = explainer.permutation_importance(\n",
    "    n_vars=10,\n",
    "    evaluation_fn='norm_aupdc',\n",
    "    n_permute=5,\n",
    "    subsample=0.1,\n",
    "    n_jobs=8,\n",
    "    verbose=True,\n",
    "    random_seed=42,\n",
    "    direction='backward',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ALE Variance-Based Ranking\n",
    "Instead of measuring performance loss, we can rank features by the standard deviation of their 1D Accumulated Local Effects (ALE). Features with higher ALE variance have a larger range of contribution to the model's predictions. This approach is inspired by Greenwell et al. (2018)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ale_1d_ds = explainer.ale(features='all', n_bootstrap=1, subsample=1000, n_jobs=8, n_bins=20)\n",
    "ale_var_1d = explainer.ale_variance(ale=ale_1d_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SHAP-Based Ranking\n",
    "SHAP values provide per-example feature attributions. By summarizing the absolute SHAP values across examples, we get another importance ranking. Here we use `local_attributions(method='shap')` and convert the results to an importance dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use a subset for SHAP computation (can be expensive)\n",
    "X_subset = shap.sample(X, 10, random_state=22)\n",
    "background_dataset = shap.sample(X, 100)\n",
    "\n",
    "shap_explainer = skexplain.ExplainToolkit(estimators[-1], X=X_subset)\n",
    "shap_explainer.set_plotting_config(\n",
    "    display_feature_names=plotting_config.display_feature_names,\n",
    "    display_units=plotting_config.display_units,\n",
    "    feature_colors=plotting_config.color_dict,\n",
    ")\n",
    "\n",
    "shap_results = shap_explainer.local_attributions(\n",
    "    method='shap',\n",
    "    shap_kws={'masker': background_dataset, 'algorithm': 'auto'},\n",
    ")\n",
    "\n",
    "shap_values = shap_results['shap_values__Logistic Regression'].values\n",
    "shap_rank = shap_values_to_importance(\n",
    "    shap_values,\n",
    "    estimator_name='Logistic Regression',\n",
    "    feature_names=X.columns,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grouped Permutation Importance\n",
    "When features are correlated, grouping them reveals their joint importance. The Group Only Permutation Feature Importance (GOPFI) method from Au et al. (2021) clusters correlated features and evaluates each group together. The feature groups are stored in `.attrs[\"feature_groups\"]` of the returned dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grouped_results = explainer.grouped_permutation_importance(\n",
    "    perm_method='grouped_only',\n",
    "    evaluation_fn='norm_aupdc',\n",
    "    n_permute=5,\n",
    "    n_jobs=8,\n",
    "    subsample=0.1,\n",
    "    clustering_kwargs={'n_clusters': 10},\n",
    ")\n",
    "\n",
    "# Inspect the automatically determined feature groups\n",
    "print('Feature groups:', grouped_results.attrs['feature_groups'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparing All Methods\n",
    "We can plot results from different ranking methods side-by-side using `plot_importance` with multiple panels. This lets us see where different methods agree or disagree on feature rankings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [perm_results, perm_results, ale_var_1d, shap_rank]\n",
    "panels = [\n",
    "    ('backward_multipass', 'Logistic Regression'),\n",
    "    ('backward_singlepass', 'Logistic Regression'),\n",
    "    ('ale_variance', 'Logistic Regression'),\n",
    "    ('shap', 'Logistic Regression'),\n",
    "]\n",
    "\n",
    "fig = explainer.plot_importance(\n",
    "    data=data,\n",
    "    panels=panels,\n",
    "    num_vars_to_plot=10,\n",
    "    figsize=(14, 4),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analysis\n",
    "Comparing the rankings across methods helps triangulate which features are truly important. Features that rank highly across multiple methods are more likely to be robustly important, while disagreements may indicate sensitivity to inter-feature correlations or the specific way each method defines \"importance.\" Grouped importance is particularly useful when many features are correlated, as it avoids double-counting redundant information."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}