.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "generated/gallery/examples/plot_loci.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_generated_gallery_examples_plot_loci.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_generated_gallery_examples_plot_loci.py:


Leave-One-Covariate-In (LOCI) feature importance with different regression models
==================================================================================

This example demonstrates how to compare LOCI feature importance [:footcite:t:`Williamson_General_2023`] across different
predictive models on the same regression dataset. LOCI is model-agnostic and can be
applied to any predictive model. Here, we use a linear model, a random forest, a neural
network, and a support vector machine. We compare the models based on their predictive
performance (R2 score) and the LOCI feature importance they yield.

.. GENERATED FROM PYTHON SOURCE LINES 13-22

Loading and preparing the data
------------------------------
We begin by simulating a regression dataset with 10 features, 5 of which
are in the support set, meaning they contribute to generating the outcome. In this example,
we use a simulated dataset to have access to the true support set of features and
evaluate how well the different models identify these important features.
The data is then split into training and test sets. These sets are used both to fit
the predictive models and within the LOCI procedure, which refits models on subsets
of features that exclude the feature of interest.

.. GENERATED FROM PYTHON SOURCE LINES 22-47

.. code-block:: Python


    from sklearn.datasets import make_regression
    from sklearn.model_selection import train_test_split

    X, y, beta = make_regression(
        n_samples=300,
        n_features=10,
        n_informative=5,
        random_state=0,
        coef=True,
        noise=4.0,
    )

    # We convert the coefficients of the data-generating process into a binary array
    # indicating the true support set of features.
    beta = beta != 0

    X_train, X_test, y_train, y_test = train_test_split(
        X,
        y,
        test_size=0.2,
        random_state=0,
        shuffle=True,
    )








.. GENERATED FROM PYTHON SOURCE LINES 48-56

Fitting models and computing LOCI feature importance
----------------------------------------------------
We define a list of predictive models to compare. We use RidgeCV for linear
regression, RandomForestRegressor for a tree-based model, MLPRegressor for a
neural network, and SVR for a support vector machine, with RBF kernel. We then fit
each model on the training data, compute the LOCI feature importance on the test
data, and store the results in a DataFrame for comparison.
Beware on the choice of hyperparameters of certain models to avoid over-fitting.

.. GENERATED FROM PYTHON SOURCE LINES 56-102

.. code-block:: Python


    import pandas as pd
    from sklearn.ensemble import RandomForestRegressor
    from sklearn.linear_model import RidgeCV
    from sklearn.metrics import mean_squared_error
    from sklearn.neural_network import MLPRegressor
    from sklearn.svm import SVR

    from hidimstat import LOCI

    models_list = [
        RidgeCV(),
        MLPRegressor(
            hidden_layer_sizes=(8),
            random_state=0,
            max_iter=500,
            learning_rate_init=0.1,
        ),
        SVR(kernel="linear"),
        RandomForestRegressor(
            n_estimators=50, max_depth=3, min_samples_leaf=10, random_state=0
        ),
    ]

    df_list = []
    for model in models_list:
        # Fit the full model
        model = model.fit(X_train, y_train)
        loci = LOCI(model, method="predict", loss=mean_squared_error)
        # Refit the model on a single feature / group of feature, and compute LOCI
        # importance. This process is repeated for all features / groups of features to assess
        # their individual contributions.
        loci.fit(X_train, y_train)
        importances = loci.importance(X_test, y_test)
        df_list.append(
            pd.DataFrame(
                {
                    "feature": list(range(X.shape[1])),
                    "importance": importances,
                    "model": model.__class__.__name__,
                    "R2 score": model.score(X_test, y_test),
                }
            )
        )









.. GENERATED FROM PYTHON SOURCE LINES 103-106

The predictive performance of the models can be compared using their R2 scores.
This helps assess how effectively each model captures the underlying data-generating
process.

.. GENERATED FROM PYTHON SOURCE LINES 106-110

.. code-block:: Python


    df_plot = pd.concat(df_list)
    df_plot.groupby("model").mean()["R2 score"].to_frame()






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>R2 score</th>
        </tr>
        <tr>
          <th>model</th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>MLPRegressor</th>
          <td>0.998202</td>
        </tr>
        <tr>
          <th>RandomForestRegressor</th>
          <td>0.632126</td>
        </tr>
        <tr>
          <th>RidgeCV</th>
          <td>0.999342</td>
        </tr>
        <tr>
          <th>SVR</th>
          <td>0.997806</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 111-116

Visualization of LOCI feature importance
----------------------------------------
Finally, we visualize the LOCI feature importance for each model using a horizontal
bar plot. The true support features are highlighted in the plot with a green shaded
background.

.. GENERATED FROM PYTHON SOURCE LINES 116-143

.. code-block:: Python


    import matplotlib.pyplot as plt
    import seaborn as sns

    ax = sns.barplot(
        data=df_plot,
        y="feature",
        x="importance",
        hue="model",
        palette="muted",
        orient="h",
    )
    sns.despine()

    for i, support in enumerate(beta):
        if support != 0:
            ax.axhspan(
                i - 0.45,
                i + 0.45,
                color="tab:olive",
                alpha=0.3,
                zorder=-1,
                label="True Support" if i == 1 else None,
            )
    ax.legend()
    plt.show()




.. image-sg:: /generated/gallery/examples/images/sphx_glr_plot_loci_001.png
   :alt: plot loci
   :srcset: /generated/gallery/examples/images/sphx_glr_plot_loci_001.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 144-149

The plot shows that the different models all identify the true support features and
assign them higher importance scores. However, the magnitude of the importance scores
varies across models. It can be observed that models with a greater predictive
performance (higher R2 score) tend to assign higher importance scores to the true
support features.

.. GENERATED FROM PYTHON SOURCE LINES 151-158

Conclusion
----------
LOCI is a simple and easily-interpretable method that can be used
in parallel to LOCO to obtain a different interpretation of variable importance.
Please keep in mind that models are fit on single features, which can lead to
overfitting depending on the hyperparameters. One way to counteract this is to
use LOCICV which can reduce noise for high-variance or unregularized models.

.. GENERATED FROM PYTHON SOURCE LINES 160-163

References
----------
.. footbibliography::


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 2.495 seconds)

**Estimated memory usage:**  220 MB


.. _sphx_glr_download_generated_gallery_examples_plot_loci.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_loci.ipynb <plot_loci.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_loci.py <plot_loci.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: plot_loci.zip <plot_loci.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_