
.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/over-sampling/plot_comparison_over_sampling.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_over-sampling_plot_comparison_over_sampling.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_over-sampling_plot_comparison_over_sampling.py:


==============================
Compare over-sampling samplers
==============================

The following example attends to make a qualitative comparison between the
different over-sampling algorithms available in the imbalanced-learn package.

.. GENERATED FROM PYTHON SOURCE LINES 9-13

.. code-block:: default


    # Authors: Guillaume Lemaitre <g.lemaitre58@gmail.com>
    # License: MIT








.. GENERATED FROM PYTHON SOURCE LINES 14-21

.. code-block:: default

    print(__doc__)

    import matplotlib.pyplot as plt
    import seaborn as sns

    sns.set_context("poster")








.. GENERATED FROM PYTHON SOURCE LINES 22-25

The following function will be used to create toy dataset. It uses the
:func:`~sklearn.datasets.make_classification` from scikit-learn but fixing
some parameters.

.. GENERATED FROM PYTHON SOURCE LINES 28-52

.. code-block:: default

    from sklearn.datasets import make_classification


    def create_dataset(
        n_samples=1000,
        weights=(0.01, 0.01, 0.98),
        n_classes=3,
        class_sep=0.8,
        n_clusters=1,
    ):
        return make_classification(
            n_samples=n_samples,
            n_features=2,
            n_informative=2,
            n_redundant=0,
            n_repeated=0,
            n_classes=n_classes,
            n_clusters_per_class=n_clusters,
            weights=list(weights),
            class_sep=class_sep,
            random_state=0,
        )









.. GENERATED FROM PYTHON SOURCE LINES 53-55

The following function will be used to plot the sample space after resampling
to illustrate the specificities of an algorithm.

.. GENERATED FROM PYTHON SOURCE LINES 58-67

.. code-block:: default

    def plot_resampling(X, y, sampler, ax, title=None):
        X_res, y_res = sampler.fit_resample(X, y)
        ax.scatter(X_res[:, 0], X_res[:, 1], c=y_res, alpha=0.8, edgecolor="k")
        if title is None:
            title = f"Resampling with {sampler.__class__.__name__}"
        ax.set_title(title)
        sns.despine(ax=ax, offset=10)









.. GENERATED FROM PYTHON SOURCE LINES 68-70

The following function will be used to plot the decision function of a
classifier given some data.

.. GENERATED FROM PYTHON SOURCE LINES 73-92

.. code-block:: default

    import numpy as np


    def plot_decision_function(X, y, clf, ax, title=None):
        plot_step = 0.02
        x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
        y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
        xx, yy = np.meshgrid(
            np.arange(x_min, x_max, plot_step), np.arange(y_min, y_max, plot_step)
        )

        Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
        Z = Z.reshape(xx.shape)
        ax.contourf(xx, yy, Z, alpha=0.4)
        ax.scatter(X[:, 0], X[:, 1], alpha=0.8, c=y, edgecolor="k")
        if title is not None:
            ax.set_title(title)









.. GENERATED FROM PYTHON SOURCE LINES 93-98

Illustration of the influence of the balancing ratio
----------------------------------------------------

We will first illustrate the influence of the balancing ratio on some toy
data using a logistic regression classifier which is a linear model.

.. GENERATED FROM PYTHON SOURCE LINES 100-104

.. code-block:: default

    from sklearn.linear_model import LogisticRegression

    clf = LogisticRegression()








.. GENERATED FROM PYTHON SOURCE LINES 105-107

We will fit and show the decision boundary model to illustrate the impact of
dealing with imbalanced classes.

.. GENERATED FROM PYTHON SOURCE LINES 109-124

.. code-block:: default

    fig, axs = plt.subplots(nrows=2, ncols=2, figsize=(15, 12))

    weights_arr = (
        (0.01, 0.01, 0.98),
        (0.01, 0.05, 0.94),
        (0.2, 0.1, 0.7),
        (0.33, 0.33, 0.33),
    )
    for ax, weights in zip(axs.ravel(), weights_arr):
        X, y = create_dataset(n_samples=300, weights=weights)
        clf.fit(X, y)
        plot_decision_function(X, y, clf, ax, title=f"weight={weights}")
        fig.suptitle(f"Decision function of {clf.__class__.__name__}")
    fig.tight_layout()




.. image-sg:: /auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_001.png
   :alt: Decision function of LogisticRegression, weight=(0.01, 0.01, 0.98), weight=(0.01, 0.05, 0.94), weight=(0.2, 0.1, 0.7), weight=(0.33, 0.33, 0.33)
   :srcset: /auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_001.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 125-136

Greater is the difference between the number of samples in each class, poorer
are the classification results.

Random over-sampling to balance the data set
--------------------------------------------

Random over-sampling can be used to repeat some samples and balance the
number of samples between the dataset. It can be seen that with this trivial
approach the boundary decision is already less biased toward the majority
class. The class :class:`~imblearn.over_sampling.RandomOverSampler`
implements such of a strategy.

.. GENERATED FROM PYTHON SOURCE LINES 136-139

.. code-block:: default


    from imblearn.over_sampling import RandomOverSampler








.. GENERATED FROM PYTHON SOURCE LINES 140-156

.. code-block:: default

    from imblearn.pipeline import make_pipeline

    X, y = create_dataset(n_samples=100, weights=(0.05, 0.25, 0.7))

    fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(15, 7))

    clf.fit(X, y)
    plot_decision_function(X, y, clf, axs[0], title="Without resampling")

    sampler = RandomOverSampler(random_state=0)
    model = make_pipeline(sampler, clf).fit(X, y)
    plot_decision_function(X, y, model, axs[1], f"Using {model[0].__class__.__name__}")

    fig.suptitle(f"Decision function of {clf.__class__.__name__}")
    fig.tight_layout()




.. image-sg:: /auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_002.png
   :alt: Decision function of LogisticRegression, Without resampling, Using RandomOverSampler
   :srcset: /auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_002.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 157-161

By default, random over-sampling generates a bootstrap. The parameter
`shrinkage` allows adding a small perturbation to the generated data
to generate a smoothed bootstrap instead. The plot below shows the difference
between the two data generation strategies.

.. GENERATED FROM PYTHON SOURCE LINES 163-174

.. code-block:: default

    fig, axs = plt.subplots(nrows=1, ncols=2, figsize=(15, 7))

    sampler.set_params(shrinkage=None)
    plot_resampling(X, y, sampler, ax=axs[0], title="Normal bootstrap")

    sampler.set_params(shrinkage=0.3)
    plot_resampling(X, y, sampler, ax=axs[1], title="Smoothed bootstrap")

    fig.suptitle(f"Resampling with {sampler.__class__.__name__}")
    fig.tight_layout()




.. image-sg:: /auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_003.png
   :alt: Resampling with RandomOverSampler, Normal bootstrap, Smoothed bootstrap
   :srcset: /auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_003.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 175-186

It looks like more samples are generated with smoothed bootstrap. This is due
to the fact that the samples generated are not superimposing with the
original samples.

More advanced over-sampling using ADASYN and SMOTE
--------------------------------------------------

Instead of repeating the same samples when over-sampling or perturbating the
generated bootstrap samples, one can use some specific heuristic instead.
:class:`~imblearn.over_sampling.ADASYN` and
:class:`~imblearn.over_sampling.SMOTE` can be used in this case.

.. GENERATED FROM PYTHON SOURCE LINES 188-207

.. code-block:: default

    from imblearn import FunctionSampler  # to use a idendity sampler
    from imblearn.over_sampling import ADASYN, SMOTE

    X, y = create_dataset(n_samples=150, weights=(0.1, 0.2, 0.7))

    fig, axs = plt.subplots(nrows=2, ncols=2, figsize=(15, 15))

    samplers = [
        FunctionSampler(),
        RandomOverSampler(random_state=0),
        SMOTE(random_state=0),
        ADASYN(random_state=0),
    ]

    for ax, sampler in zip(axs.ravel(), samplers):
        title = "Original dataset" if isinstance(sampler, FunctionSampler) else None
        plot_resampling(X, y, sampler, ax, title=title)
    fig.tight_layout()




.. image-sg:: /auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_004.png
   :alt: Original dataset, Resampling with RandomOverSampler, Resampling with SMOTE, Resampling with ADASYN
   :srcset: /auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_004.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 208-215

The following plot illustrates the difference between
:class:`~imblearn.over_sampling.ADASYN` and
:class:`~imblearn.over_sampling.SMOTE`.
:class:`~imblearn.over_sampling.ADASYN` will focus on the samples which are
difficult to classify with a nearest-neighbors rule while regular
:class:`~imblearn.over_sampling.SMOTE` will not make any distinction.
Therefore, the decision function depending of the algorithm.

.. GENERATED FROM PYTHON SOURCE LINES 215-233

.. code-block:: default


    X, y = create_dataset(n_samples=150, weights=(0.05, 0.25, 0.7))

    fig, axs = plt.subplots(nrows=1, ncols=3, figsize=(20, 6))

    models = {
        "Without sampler": clf,
        "ADASYN sampler": make_pipeline(ADASYN(random_state=0), clf),
        "SMOTE sampler": make_pipeline(SMOTE(random_state=0), clf),
    }

    for ax, (title, model) in zip(axs, models.items()):
        model.fit(X, y)
        plot_decision_function(X, y, model, ax=ax, title=title)

    fig.suptitle(f"Decision function using a {clf.__class__.__name__}")
    fig.tight_layout()




.. image-sg:: /auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_005.png
   :alt: Decision function using a LogisticRegression, Without sampler, ADASYN sampler, SMOTE sampler
   :srcset: /auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_005.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 234-236

Due to those sampling particularities, it can give rise to some specific
issues as illustrated below.

.. GENERATED FROM PYTHON SOURCE LINES 238-253

.. code-block:: default

    X, y = create_dataset(n_samples=5000, weights=(0.01, 0.05, 0.94), class_sep=0.8)

    samplers = [SMOTE(random_state=0), ADASYN(random_state=0)]

    fig, axs = plt.subplots(nrows=2, ncols=2, figsize=(15, 15))
    for ax, sampler in zip(axs, samplers):
        model = make_pipeline(sampler, clf).fit(X, y)
        plot_decision_function(
            X, y, clf, ax[0], title=f"Decision function with {sampler.__class__.__name__}"
        )
        plot_resampling(X, y, sampler, ax[1])

    fig.suptitle("Particularities of over-sampling with SMOTE and ADASYN")
    fig.tight_layout()




.. image-sg:: /auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_006.png
   :alt: Particularities of over-sampling with SMOTE and ADASYN, Decision function with SMOTE, Resampling with SMOTE, Decision function with ADASYN, Resampling with ADASYN
   :srcset: /auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_006.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 254-263

SMOTE proposes several variants by identifying specific samples to consider
during the resampling. The borderline version
(:class:`~imblearn.over_sampling.BorderlineSMOTE`) will detect which point to
select which are in the border between two classes. The SVM version
(:class:`~imblearn.over_sampling.SVMSMOTE`) will use the support vectors
found using an SVM algorithm to create new sample while the KMeans version
(:class:`~imblearn.over_sampling.KMeansSMOTE`) will make a clustering before
to generate samples in each cluster independently depending each cluster
density.

.. GENERATED FROM PYTHON SOURCE LINES 265-294

.. code-block:: default

    from sklearn.cluster import MiniBatchKMeans

    from imblearn.over_sampling import SVMSMOTE, BorderlineSMOTE, KMeansSMOTE

    X, y = create_dataset(n_samples=5000, weights=(0.01, 0.05, 0.94), class_sep=0.8)

    fig, axs = plt.subplots(5, 2, figsize=(15, 30))

    samplers = [
        SMOTE(random_state=0),
        BorderlineSMOTE(random_state=0, kind="borderline-1"),
        BorderlineSMOTE(random_state=0, kind="borderline-2"),
        KMeansSMOTE(
            kmeans_estimator=MiniBatchKMeans(n_clusters=10, n_init=1, random_state=0),
            random_state=0,
        ),
        SVMSMOTE(random_state=0),
    ]

    for ax, sampler in zip(axs, samplers):
        model = make_pipeline(sampler, clf).fit(X, y)
        plot_decision_function(
            X, y, clf, ax[0], title=f"Decision function for {sampler.__class__.__name__}"
        )
        plot_resampling(X, y, sampler, ax[1])

    fig.suptitle("Decision function and resampling using SMOTE variants")
    fig.tight_layout()




.. image-sg:: /auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_007.png
   :alt: Decision function and resampling using SMOTE variants, Decision function for SMOTE, Resampling with SMOTE, Decision function for BorderlineSMOTE, Resampling with BorderlineSMOTE, Decision function for BorderlineSMOTE, Resampling with BorderlineSMOTE, Decision function for KMeansSMOTE, Resampling with KMeansSMOTE, Decision function for SVMSMOTE, Resampling with SVMSMOTE
   :srcset: /auto_examples/over-sampling/images/sphx_glr_plot_comparison_over_sampling_007.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 295-298

When dealing with a mixed of continuous and categorical features,
:class:`~imblearn.over_sampling.SMOTENC` is the only method which can handle
this case.

.. GENERATED FROM PYTHON SOURCE LINES 300-329

.. code-block:: default

    from collections import Counter

    from imblearn.over_sampling import SMOTENC

    rng = np.random.RandomState(42)
    n_samples = 50
    # Create a dataset of a mix of numerical and categorical data
    X = np.empty((n_samples, 3), dtype=object)
    X[:, 0] = rng.choice(["A", "B", "C"], size=n_samples).astype(object)
    X[:, 1] = rng.randn(n_samples)
    X[:, 2] = rng.randint(3, size=n_samples)
    y = np.array([0] * 20 + [1] * 30)

    print("The original imbalanced dataset")
    print(sorted(Counter(y).items()))
    print()
    print("The first and last columns are containing categorical features:")
    print(X[:5])
    print()

    smote_nc = SMOTENC(categorical_features=[0, 2], random_state=0)
    X_resampled, y_resampled = smote_nc.fit_resample(X, y)
    print("Dataset after resampling:")
    print(sorted(Counter(y_resampled).items()))
    print()
    print("SMOTE-NC will generate categories for the categorical features:")
    print(X_resampled[-5:])
    print()





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    The original imbalanced dataset
    [(0, 20), (1, 30)]

    The first and last columns are containing categorical features:
    [['C' -0.14021849735700803 2]
     ['A' -0.033193400066544886 2]
     ['C' -0.7490765234433554 1]
     ['C' -0.7783820070908942 2]
     ['A' 0.9488428577190162 2]]

    Dataset after resampling:
    [(0, 30), (1, 30)]

    SMOTE-NC will generate categories for the categorical features:
    [['A' 0.5246469549655819 2]
     ['B' -0.3657680728116921 2]
     ['B' 0.9344237230779995 2]
     ['B' 0.371089161882461 2]
     ['B' 0.3327240726719727 2]]





.. GENERATED FROM PYTHON SOURCE LINES 330-332

However, if the dataset is composed of only categorical features then one
should use :class:`~imblearn.over_sampling.SMOTEN`.

.. GENERATED FROM PYTHON SOURCE LINES 334-351

.. code-block:: default

    from imblearn.over_sampling import SMOTEN

    # Generate only categorical data
    X = np.array(["A"] * 10 + ["B"] * 20 + ["C"] * 30, dtype=object).reshape(-1, 1)
    y = np.array([0] * 20 + [1] * 40, dtype=np.int32)

    print(f"Original class counts: {Counter(y)}")
    print()
    print(X[:5])
    print()

    sampler = SMOTEN(random_state=0)
    X_res, y_res = sampler.fit_resample(X, y)
    print(f"Class counts after resampling {Counter(y_res)}")
    print()
    print(X_res[-5:])
    print()




.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Original class counts: Counter({1: 40, 0: 20})

    [['A']
     ['A']
     ['A']
     ['A']
     ['A']]

    Class counts after resampling Counter({0: 40, 1: 40})

    [['B']
     ['B']
     ['A']
     ['B']
     ['A']]






.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 0 minutes  3.798 seconds)


.. _sphx_glr_download_auto_examples_over-sampling_plot_comparison_over_sampling.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example




    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_comparison_over_sampling.py <plot_comparison_over_sampling.py>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_comparison_over_sampling.ipynb <plot_comparison_over_sampling.ipynb>`


.. include:: plot_comparison_over_sampling.recommendations


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_
