Amirhessam Tahmassebi

Machine Learning Algorithms From Scratch: Logistic Regression

2021-02-26T00:00:00+00:00

Introduction

Logistic regression is a classification algorithm used to assign observations to a discrete set of classes. Unlike linear regression which outputs continuous number values, logistic regression transforms its output using the logistic sigmoid function to return a probability value which can then be mapped to two or more discrete classes.

In order to map predicted values to probabilities, we use the sigmoid function. The function maps any real value into another value between 0 and 1. In machine learning, we use sigmoid to map predictions to probabilities.

Sigmoid Function: $f(x) = \frac{1}{1 + exp(-x)}$

# loading libraries
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
import seaborn as sns

from IPython.core.display import display, HTML
import warnings

sns.set_style("ticks")
mpl.rcParams["axes.linewidth"] = 2
mpl.rcParams["lines.linewidth"] = 2
warnings.filterwarnings("ignore")
display(HTML("<style>.container { width:95% !important; }</style>"))


def sigmoid(x):
    """
    Returns sigmoid(x)
    """
    return 1.0/(1 + np.exp(-x))

def plot_sigmoid(x):
    """
    Plots sigmoid(x)
    """
    fig, ax = plt.subplots(figsize=(5,5))
    ax.plot(x, sigmoid(x), lw=2, c="navy", label="Sigmoid")
    ax.axhline(0.5, lw=2, ls="--", c="red", label="Decision Bound")
    ax.set(title="Sigmoid Function",
           xlabel="x",
           ylabel="Sigmoid(x)")
    ax.grid(True)
    ax.legend(loc=0)
    plt.plot()

# define x to plot sigmoid(x)
x = np.linspace(-5, 5, 100)
plot_sigmoid(x)

The main algorithm to develop Logistic Regression is based on updating weights using Gradient Descent approach iteratively.

- Define linear model: $z = wx + b$

- Define prediction: $\hat{y} = sigmoid(z) = sigmoid(wx + b) = \frac{1}{1 + \exp({-(wx+b)})}$

- Update weights: $w = w - lr \times dw$ where $dw = \frac{1}{N} \Sigma_{i=1}^{N}(2x_i(\hat{y_i} - y_i))$

- Update bias $b = b - lr \times db$ where $db = \frac{1}{N} \Sigma_{i=1}^{N}(\hat{y_i} - y_i)$

class LogisticRegression:
    """
    Logistic Regression based on Gradient Descent
    z = wx + b
    y_hat = sigmoid(z) = sigmoid(wx + b) = 1/(1 + exp[-(wx+b)])
    update rules: w = w - lr * dw
                  b = b - lr * db
    where dw = 1/N sum(2x(y_hat - y))
          db = 1/N sum(2(y_hat - y))
    Parameters
    ----------
    lr: float, optional (default=0.001)
        Learning rate used of updating weights/bias
    n_iters: int, optional (default=1000)
        Maximum number of iterations to update weights/bias
    weights: numpy.array, optional (default=None)
        Weights array of shape (n_features, ) where
        will be initialized as zeros
    bias: float, optional (default=None)
        Bias value where will be initialized as zero
    """

    def __init__(self, learning_rate=0.001, n_iters=1000, weights=None, bias=None):
        self.learning_rate = learning_rate
        self.n_iters = n_iters
        self.weights = None
        self.bias = None

    def fit(self, X_train, y_train):
        """
        Train model using iterative gradient descent
        Parameters
        ----------
        X_train: numpy.array
            Training feature matrix
        y_train: 1D numpy.array or list
            Training binary class labels [0, 1]
        """
        # unpack the shape of X_train
        n_samples, n_features = X_train.shape

        # initialize weights and bias with zeros
        self.weights = np.zeros(n_features)
        self.bias = 0.0

        # main loop
        # self.loss = []
        for _ in range(self.n_iters):
            z = np.dot(X_train, self.weights) + self.bias
            y_hat = self._sigmoid(z)

            # update weights + bias
            dw = (1.0 / n_samples) * 2 * np.dot(X_train.T, (y_hat - y_train))
            db = (1.0 / n_samples) * 2 * np.sum(y_hat - y_train)

            self.weights -= self.learning_rate * dw
            self.bias -= self.learning_rate * db

        return None

    def predict_proba(self, X_test):
        """
        Prediction probability of the test samples
        Parameters
        ----------
        X_test: numpy.array
            Testing feature matrix
        """
        z = np.dot(X_test, self.weights) + self.bias
        y_pred_proba = self._sigmoid(z)

        return y_pred_proba

    def predict(self, X_test, threshold=0.5):
        """
        Prediction class of the test samples
        Parameters
        ----------
        X_test: numpy.array
            Testing feature matrix
        threshold: float, optional (default=0.5)
            Threshold to define predicted classes
        """
        y_pred_proba = self.predict_proba(X_test)
        y_pred = [1 if x >= threshold else 0 for x in y_pred_proba]

        return y_pred

    def plot_decision_boundary(self, X_test, y_test, threshold=0.5, figsize=(8, 6)):
        """
        Plot the decision boundary
        Parameters
        ----------
        X_test: numpy.array
            Testing feature matrix
        y_test: 1D numpy.array or list
            Testing targets
        threshold: float, optional (default=0.5)
            Threshold to define predicted classes
        figsize: tuple, optional (default=(8,6))
            Figure size
        """
        # calc pred_proba
        y_pred_proba = self.predict_proba(X_test)

        # define positive and negative classes
        pos_class = []
        neg_class = []
        for i in range(len(y_pred_proba)):
            if y_test[i] == 1:
                pos_class.append(y_pred_proba[i])
            else:
                neg_class.append(y_pred_proba[i])

        # plotting
        fig, ax = plt.subplots(figsize=figsize)
        ax.scatter(
            range(len(pos_class)),
            pos_class,
            s=25,
            color="navy",
            marker="o",
            label="Positive Class",
        )
        ax.scatter(
            range(len(neg_class)),
            neg_class,
            s=25,
            color="red",
            marker="s",
            label="Negative Class",
        )
        ax.axhline(
            threshold,
            lw=2,
            ls="--",
            color="black",
            label=f"Decision Bound = {threshold}",
        )
        ax.set_title("Decision Boundary")
        ax.set_xlabel("# of Samples", fontsize=12)
        ax.set_ylabel("Predicted Probability", fontsize=12)
        ax.tick_params(axis="both", which="major", labelsize=12)
        ax.legend(bbox_to_anchor=(1.2, 0.5), loc="center", ncol=1, framealpha=0.0)
        plt.show()

    def _sigmoid(self, z):
        """
        Sigmoid Function
        f(z) = 1/(1 + exp(-z))
        """

        return 1.0 / (1 + np.exp(-z))

This is part of the ML-Algs library which can be downloaded via PIP or from Git.

# you can simply do
!pip install mlalgs

# or cloning the project directly from GitHub
!git clone https://github.com/amirhessam88/ml-algs.git

# change directory to the project
%cd ml-algs

# installing requirements
!pip install -r requirements.txt

# loading libraries
from mlalgs.logistic_regression import LogisticRegression
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

Sample breast cancer data for binary classification task. I have used stratifed train/test splits where test size is the 20\% of the data.

# loading data
data = datasets.load_breast_cancer()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                    test_size=0.2,
                                                    shuffle=True,
                                                    random_state=1367,
                                                    stratify=y)

Now, train the model using our algorithm.

# define model
clf = LogisticRegression()
# train model
clf.fit(X_train, y_train)
# predicted class
y_pred = clf.predict(X_test)
# predicted proba
y_pred_proba = clf.predict_proba(X_test)

# accuracy
accuracy_score(y_test, y_pred)

0.9298245614035088

We can also plot the decision boundary and the predicted probabilities in each class.

clf.plot_decision_boundary(X_test, y_test)

We can also calculate all the binary classification metrics using SlickML library.

# simply install slickml via pip
!pip install slickml

# loading BinaryClassificationMetrics
from slickml.metrics import BinaryClassificationMetrics

# initialize the metric object
clf_metrics = BinaryClassificationMetrics(y_test, y_pred_proba)

	Accuracy	Balanced Accuracy	ROC AUC	PR AUC	Precision	Recall	Average Precision	F-1 Score	F-2 Score	F-0.50 Score	Threat Score	TP	TN	FP	FN
Threshold = 0.500 \| Average = Binary	0.930000	0.915000	0.946000	0.969000	0.921000	0.972000	0.943000	0.946000	0.962000	0.931000	0.897000	70	36	6	2

# plot roc, precision-recall with different thresholds calculations
clf_metrics.plot()

Updates on Regression Error Characteristics Curves

2021-01-10T00:00:00+00:00

Updates on Regression Error Characteristics Curve

Introduction

In machine learning, a Receiver Operating Characteristic (ROC) curve visualizes the performance of a classifier applied on a binary class problem across all possible trade-offs between the false positive rates and the true positive rates. A graph consists of multiple ROC curves of different models characterizes the performance of the models on a binary problem and makes the comparison process of the models easier by visualization. Additionally, the area under the ROC curve (AUC) represents the expected performance of the classification model as a single scalar value.

Although ROC curves are limited to classification problems, Regression Error Characteristic (REC) curves can be used to visualize the performance of the regressor models. REC illustrates the absolute deviation tolerance versus the fraction of the exemplars predicted correctly within the tolerance interval. The resulting curve estimates the cumulative distribution function of the error. The area over the REC curve (AOC), which can be calculated via the area under the REC curve (AOC = 1 - AUC) is a biased estimate of the expected error.

Furthermore, the coefficient of determination $R^2$ can be also calculated with respect to the AOC. Likewise the ROC curve, the shape of the REC curve can also be used as a guidance for the users to reveal additional information about the data modeling. The REC curve was implemented in Python.

Previously, I have implemented a simple version of REC LINK. Here, is an overview of the objected oriented implementation of REC and its updates.

You can clone the project from my GitHub Repo LINK or run the following cell.

# cloning the project
!git clone https://github.com/amirhessam88/Regression-Error-Characteristic-Curve.git

# change directory to the project
%cd Regression-Error-Characteristic-Curve

# installing requirements
!pip install -r requirements.txt

Or you can follow the class below which is the content of `src/rec.py`

# Loading Packages
import numpy as np
import scipy as scp
import seaborn as sns
import matplotlib as mpl
import matplotlib.pyplot as plt

# class definition
class RegressionErrorCharacteristic:
    """Regression Error Characteristics (REC).
    This is wrapper to implement the REC algorithm.
    REC is implemented based on the following paper:
    Bi, J., & Bennett, K. P. (2003). Regression error characteristic curves.
    In Proceedings of the 20th international conference on machine learning (ICML-03) (pp. 43-50).
    https://www.aaai.org/Papers/ICML/2003/ICML03-009.pdf
    Parameters
    ----------
    y_true: numpy.array[int] or list[float]
        List of ground truth target (response) values
    y_pred: numpy.array[float] or list[float]
        List of predicted target values list[float]
    Attributes
    ----------
    auc_rec: float value between 0. and 1
        Area under REC curve.
    deviation:  numpy.array[float] or list[float]
        List of deviations to plot REC curve.
    accuracy:  numpy.array[float] or list[float]
        Calculated accuracy at each deviation to plot REC curve.
    plotting_dict: dict()
        Plotting object as a dictionary consists of all
        calculated metrics which was used to plot curves
    plot_rec(): Func
        Function to plot the REC curve.
    """

    def __init__(self, y_true, y_pred):
        if not isinstance(y_true, np.ndarray):
            self.y_true = np.array(y_true)
        else:
            self.y_true = y_true
        if not isinstance(y_pred, np.ndarray):
            self.y_pred = np.array(y_pred)
        else:
            self.y_pred = y_pred
        self.deviation, self.accuracy, self.auc_rec = self._rec_curve()

    def _rec_curve(self):
        """
        Function to calculate the rec curve elements: deviation, accuracy, auc.
        Simpson method is used as the integral method to calculate the area under
        regression error characteristics (REC).
        REC is implemented based on the following paper:
        Bi, J., & Bennett, K. P. (2003). Regression error characteristic curves.
        In Proceedings of the 20th international conference on machine learning (ICML-03) (pp. 43-50).
        https://www.aaai.org/Papers/ICML/2003/ICML03-009.pdf
        """
        begin = 0.0
        end = 1.0
        interval = 0.01
        accuracy = []
        deviation = np.arange(begin, end, interval)

        # main loop to calculate norm and compare with each deviation
        for i in range(len(deviation)):
            count = 0.0
            for j in range(len(self.y_true)):
                calc_norm = np.linalg.norm(self.y_true[j] - self.y_pred[j]) / np.sqrt(
                    np.linalg.norm(self.y_true[j]) ** 2
                    + np.linalg.norm(self.y_pred[j]) ** 2
                )
                if calc_norm < deviation[i]:
                    count += 1
            accuracy.append(count / len(self.y_true))

        auc_rec = scp.integrate.simps(accuracy, deviation) / end

        return deviation, accuracy, auc_rec

    def plot_rec(self, figsize=None, color=None, linestyle=None, fontsize=None):
        """Function to plot REC curve.
        Parameters
        ----------
        figsize: tuple, optional, (default=(8, 5))
            Figure size
        color: str, optional, (default="navy")
            Color of the curve.
        linestyle: str, optional, (default="--")
        fontsize: int or float, optional, (default=15)
            Fontsize for xlabel and ylabel, and ticks parameters
        """
        sns.set_style("ticks")
        mpl.rcParams["axes.linewidth"] = 3
        mpl.rcParams["lines.linewidth"] = 3

        # initializing figsize
        if figsize is None:
            figsize = (8, 5)
        elif isinstance(figsize, list) or isinstance(figsize, tuple):
            figsize = figsize
        else:
            raise TypeError("Only tuple and list types are allowed for figsize.")

        # initializing color
        if color is None:
            color = "navy"
        elif isinstance(color, str):
            color = color
        else:
            raise TypeError("Only str type is allowed for color.")

        # initializing linestyle
        if linestyle is None:
            linestyle = "--"
        elif isinstance(linestyle, str):
            linestyle = linestyle
        else:
            raise TypeError("Only str type is allowed for linestyle.")

        # initializing fontsize
        if fontsize is None:
            fontsize = 15
        elif isinstance(fontsize, float) or isinstance(fontsize, int):
            fontsize = fontsize
        else:
            raise TypeError("Only int and float types are allowed for fontsize.")

        fig, ax = plt.subplots(figsize=figsize)

        ax.plot(
            self.deviation,
            self.accuracy,
            color=color,
            linestyle=linestyle,
            label=f"AUC = {self.auc_rec:.3f}",
        )

        ax.set(
            xlim=[-0.01, 1.01],
            ylim=[-0.01, 1.01],
        )
        ax.set_xlabel("Deviation", fontsize=fontsize)
        ax.set_ylabel("Accuracy", fontsize=fontsize)
        ax.set_title("REC Curve", fontsize=fontsize)

        ax.tick_params(axis="both", which="major", labelsize=14)
        ax.legend(prop={"size": fontsize}, loc=4, framealpha=0.0)

        plt.show()

Load Data and Train a Regression Model

# load data, train model
from sklearn import linear_model, datasets
from sklearn.model_selection import cross_val_predict

# loading a sample regression dataset
boston = datasets.load_boston()
X = boston.data
y_true = boston.target

# defining a simple linear regression model
model = linear_model.LinearRegression()

# predicting using 4-folds cross-validation
y_pred = cross_val_predict(model, X, y_true, cv=4)

Let’s try to plot REC curve

# initiate the RegressionErrorCharacteristic with y_pred and y_true
myREC = RegressionErrorCharacteristic(y_true, y_pred)

# now we can use the class function plot_rec()
myREC.plot_rec()

Currently, REC is also implemented in my Machine Learning Python Library, SlickML which can be easily installed via pip.

!pip install slickml

# plot regression metrics
from slickml.metrics import RegressionMetrics
reg_metrics = RegressionMetrics(y_true, y_pred)
reg_metrics.plot()

	R2 Score	Explained Variance Score	Mean Absolute Error	Mean Squared Error	Mean Squared Log Error	Mean Absolute Percentage Error	REC AUC	Coeff. of Variation	Mean of Variation
Metrics	0.499000	0.509000	4.646000	42.299000	None	0.254000	0.835000	0.342000	1.110000

Extra notes about the REC class

# list of attributes of the REC object
dir(myREC)

['__class__',
 '__delattr__',
 '__dict__',
 '__dir__',
 '__doc__',
 '__eq__',
 '__format__',
 '__ge__',
 '__getattribute__',
 '__gt__',
 '__hash__',
 '__init__',
 '__init_subclass__',
 '__le__',
 '__lt__',
 '__module__',
 '__ne__',
 '__new__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__setattr__',
 '__sizeof__',
 '__str__',
 '__subclasshook__',
 '__weakref__',
 '_rec_curve',
 'accuracy',
 'auc_rec',
 'deviation',
 'plot_rec',
 'y_pred',
 'y_true']

# dict of attributes values of the REC object
myREC.__dict__

{'y_true': array([24. , 21.6, 34.7, 33.4, 36.2, 28.7, 22.9, 27.1, 16.5, 18.9, 15. ,
        18.9, 21.7, 20.4, 18.2, 19.9, 23.1, 17.5, 20.2, 18.2, 13.6, 19.6,
        15.2, 14.5, 15.6, 13.9, 16.6, 14.8, 18.4, 21. , 12.7, 14.5, 13.2,
        13.1, 13.5, 18.9, 20. , 21. , 24.7, 30.8, 34.9, 26.6, 25.3, 24.7,
        21.2, 19.3, 20. , 16.6, 14.4, 19.4, 19.7, 20.5, 25. , 23.4, 18.9,
        35.4, 24.7, 31.6, 23.3, 19.6, 18.7, 16. , 22.2, 25. , 33. , 23.5,
        19.4, 22. , 17.4, 20.9, 24.2, 21.7, 22.8, 23.4, 24.1, 21.4, 20. ,
        20.8, 21.2, 20.3, 28. , 23.9, 24.8, 22.9, 23.9, 26.6, 22.5, 22.2,
        23.6, 28.7, 22.6, 22. , 22.9, 25. , 20.6, 28.4, 21.4, 38.7, 43.8,
        33.2, 27.5, 26.5, 18.6, 19.3, 20.1, 19.5, 19.5, 20.4, 19.8, 19.4,
        21.7, 22.8, 18.8, 18.7, 18.5, 18.3, 21.2, 19.2, 20.4, 19.3, 22. ,
        20.3, 20.5, 17.3, 18.8, 21.4, 15.7, 16.2, 18. , 14.3, 19.2, 19.6,
        23. , 18.4, 15.6, 18.1, 17.4, 17.1, 13.3, 17.8, 14. , 14.4, 13.4,
        15.6, 11.8, 13.8, 15.6, 14.6, 17.8, 15.4, 21.5, 19.6, 15.3, 19.4,
        17. , 15.6, 13.1, 41.3, 24.3, 23.3, 27. , 50. , 50. , 50. , 22.7,
        25. , 50. , 23.8, 23.8, 22.3, 17.4, 19.1, 23.1, 23.6, 22.6, 29.4,
        23.2, 24.6, 29.9, 37.2, 39.8, 36.2, 37.9, 32.5, 26.4, 29.6, 50. ,
        32. , 29.8, 34.9, 37. , 30.5, 36.4, 31.1, 29.1, 50. , 33.3, 30.3,
        34.6, 34.9, 32.9, 24.1, 42.3, 48.5, 50. , 22.6, 24.4, 22.5, 24.4,
        20. , 21.7, 19.3, 22.4, 28.1, 23.7, 25. , 23.3, 28.7, 21.5, 23. ,
        26.7, 21.7, 27.5, 30.1, 44.8, 50. , 37.6, 31.6, 46.7, 31.5, 24.3,
        31.7, 41.7, 48.3, 29. , 24. , 25.1, 31.5, 23.7, 23.3, 22. , 20.1,
        22.2, 23.7, 17.6, 18.5, 24.3, 20.5, 24.5, 26.2, 24.4, 24.8, 29.6,
        42.8, 21.9, 20.9, 44. , 50. , 36. , 30.1, 33.8, 43.1, 48.8, 31. ,
        36.5, 22.8, 30.7, 50. , 43.5, 20.7, 21.1, 25.2, 24.4, 35.2, 32.4,
        32. , 33.2, 33.1, 29.1, 35.1, 45.4, 35.4, 46. , 50. , 32.2, 22. ,
        20.1, 23.2, 22.3, 24.8, 28.5, 37.3, 27.9, 23.9, 21.7, 28.6, 27.1,
        20.3, 22.5, 29. , 24.8, 22. , 26.4, 33.1, 36.1, 28.4, 33.4, 28.2,
        22.8, 20.3, 16.1, 22.1, 19.4, 21.6, 23.8, 16.2, 17.8, 19.8, 23.1,
        21. , 23.8, 23.1, 20.4, 18.5, 25. , 24.6, 23. , 22.2, 19.3, 22.6,
        19.8, 17.1, 19.4, 22.2, 20.7, 21.1, 19.5, 18.5, 20.6, 19. , 18.7,
        32.7, 16.5, 23.9, 31.2, 17.5, 17.2, 23.1, 24.5, 26.6, 22.9, 24.1,
        18.6, 30.1, 18.2, 20.6, 17.8, 21.7, 22.7, 22.6, 25. , 19.9, 20.8,
        16.8, 21.9, 27.5, 21.9, 23.1, 50. , 50. , 50. , 50. , 50. , 13.8,
        13.8, 15. , 13.9, 13.3, 13.1, 10.2, 10.4, 10.9, 11.3, 12.3,  8.8,
         7.2, 10.5,  7.4, 10.2, 11.5, 15.1, 23.2,  9.7, 13.8, 12.7, 13.1,
        12.5,  8.5,  5. ,  6.3,  5.6,  7.2, 12.1,  8.3,  8.5,  5. , 11.9,
        27.9, 17.2, 27.5, 15. , 17.2, 17.9, 16.3,  7. ,  7.2,  7.5, 10.4,
         8.8,  8.4, 16.7, 14.2, 20.8, 13.4, 11.7,  8.3, 10.2, 10.9, 11. ,
         9.5, 14.5, 14.1, 16.1, 14.3, 11.7, 13.4,  9.6,  8.7,  8.4, 12.8,
        10.5, 17.1, 18.4, 15.4, 10.8, 11.8, 14.9, 12.6, 14.1, 13. , 13.4,
        15.2, 16.1, 17.8, 14.9, 14.1, 12.7, 13.5, 14.9, 20. , 16.4, 17.7,
        19.5, 20.2, 21.4, 19.9, 19. , 19.1, 19.1, 20.1, 19.9, 19.6, 23.2,
        29.8, 13.8, 13.3, 16.7, 12. , 14.6, 21.4, 23. , 23.7, 25. , 21.8,
        20.6, 21.2, 19.1, 20.6, 15.2,  7. ,  8.1, 13.6, 20.1, 21.8, 24.5,
        23.1, 19.7, 18.3, 21.2, 17.5, 16.8, 22.4, 20.6, 23.9, 22. , 11.9]),
 'y_pred': array([30.5267484 , 25.0863876 , 30.50798059, 28.18447014, 27.27659833,
        25.0206824 , 22.97649705, 18.83349203, 10.41056667, 18.22035867,
        17.95015824, 21.25976346, 20.80047342, 19.60016143, 19.196703  ,
        19.47833086, 20.78071079, 16.6822844 , 16.65544328, 18.72117678,
        12.29985737, 17.59449166, 15.35105412, 13.40048216, 15.32683646,
        13.20887423, 15.17435384, 14.21411649, 19.05918486, 20.41568379,
        10.89254337, 17.90439239,  7.89185461, 14.15801366, 13.24116244,
        24.47046953, 22.94182224, 23.67632915, 23.36460532, 32.39402215,
        35.14808049, 27.98190352, 25.45873917, 24.73521528, 23.00170805,
        22.57211674, 20.58753997, 17.52214917,  8.13490741, 16.97684727,
        21.06815411, 23.91508564, 27.64691619, 24.14485615, 15.56029526,
        30.79882223, 24.80264078, 33.4623118 , 21.30659952, 20.90728705,
        17.44043906, 17.9999192 , 23.53676251, 21.48419631, 21.78225019,
        31.51422368, 26.62239223, 21.4259965 , 17.5785947 , 21.04346836,
        25.64100972, 22.2476999 , 25.28041972, 24.53308237, 26.43625821,
        24.64444619, 23.58527409, 24.18580195, 21.5659332 , 23.34277551,
        28.76031545, 27.27313482, 26.55259368, 25.59918469, 24.89705822,
        28.07627352, 22.42719907, 26.60952109, 31.01227403, 31.12757221,
        27.72734566, 28.06284015, 30.0598584 , 30.47238708, 28.12557027,
        29.29080253, 25.36109703, 35.73204957, 35.27318261, 32.47046941,
        24.93842677, 26.00700668, 20.31952914, 20.78433732, 22.06589527,
        19.2069564 , 17.68898542, 21.40525709, 23.09291626, 20.11633401,
        21.10974578, 27.00546521, 21.45765484, 21.16538012, 26.0590909 ,
        21.10224572, 24.0431348 , 24.54699762, 21.06883648, 21.61550914,
        22.7970042 , 23.29465489, 21.21031219, 16.65846995, 21.31241829,
        23.35272473, 15.00484047, 13.98216192, 16.31224834, 12.82892742,
        17.67100485, 17.35169438, 18.06590466, 14.54231082, 12.43429351,
        14.70629958, 14.13498891, 16.76691574, 11.86766115, 14.10364074,
        10.48750529,  2.60155375, 17.25421879, 10.95127839,  8.45772726,
         9.92604258, 16.03249641,  8.13668368,  8.93992704, 13.84872187,
        19.46012801, 18.81839351, 25.07151207, 17.06680828, 24.76523955,
        22.84078468, 14.8100259 , 32.08569446, 29.75556827, 24.11794019,
        36.71250211, 34.44106735, 41.40717071, 41.14172393, 25.14455214,
        25.88779219, 33.83456259, 23.53734156, 26.16880192, 26.08817423,
        22.99156864, 24.85059583, 24.84904356, 29.49513872, 27.7346177 ,
        29.86929462, 25.88565372, 29.83332418, 30.83380806, 31.60060406,
        32.04371601, 27.97667106, 33.12994507, 31.59105041, 24.50879627,
        24.84504636, 32.59723405, 32.6452962 , 32.13317928, 32.94471956,
        29.48976084, 29.76312024, 31.45418695, 30.53869113, 30.8009377 ,
        38.04946061, 34.94816639, 31.63807099, 33.42340016, 28.38581989,
        28.58556215, 29.92318278, 34.22091216, 38.91580814, 39.72884181,
        21.62558428, 22.49795701, 17.75037721, 26.49671976, 21.82975475,
        26.21074138, 21.32617731, 25.95677617, 23.52170299,  8.98474282,
        23.49843966, 29.84060044, 27.35266917, 28.57173841, 33.16259161,
        36.11656764, 27.36112851, 34.95044465, 29.70227353, 34.88411689,
        35.37154272, 34.9656523 , 31.30723951, 31.67098863, 30.16041348,
        25.22307605, 31.68482563, 34.49607873, 33.59011098, 34.88609777,
        25.65542526, 33.56085549, 31.12769825, 27.71172655, 27.9700297 ,
        26.07987213, 24.53909622, 24.22949914, 26.79548935, 18.36775975,
        14.56236196, 20.0211073 , 20.77494205, 20.94318096, 22.45082716,
        23.00302431, 24.13854883, 22.8773354 , 24.94523676, 23.43062427,
        21.9220964 , 37.75348485, 46.0387449 , 36.29956397, 33.42227994,
        35.63121347, 37.73550614, 43.15871658, 35.54582198, 35.78581939,
        26.2145508 , 33.04359663, 44.15984014, 40.10445186, 23.67097031,
        22.83333215, 27.75872967, 28.30030813, 35.97207441, 33.5821157 ,
        33.00259849, 34.53104554, 33.03762186, 30.33610549, 35.79537384,
        40.14368337, 35.13832636, 39.72628001, 44.66430225, 33.53476984,
        28.39893868, 21.44108838, 26.46426956, 26.45036422, 28.19254823,
        32.28714563, 34.00450932, 30.73048779, 27.58136754, 25.50416679,
        30.41717428, 28.5537616 , 21.26499536, 28.56613136, 33.32854451,
        30.70616356, 29.65558785, 29.99829605, 33.95640231, 34.03666941,
        29.92228892, 35.26268183, 31.4684171 , 27.27185199, 22.45919981,
        17.35290013, 25.78745656, 22.18307636, 24.27488685, 25.22185983,
        19.3730908 , 18.75693281, 18.96546422, 24.70680454, 22.25319816,
        24.90064011, 24.58899763, 22.27170985, 18.29406617, 25.31675015,
        25.74825622, 24.24006005, 20.74247194, 22.2724639 , 25.99697212,
        22.95074899, 21.08849735, 24.02165592, 22.27198713, 21.97337322,
        20.83605635, 19.88580193, 18.8456278 , 22.22297757, 21.17604193,
        20.66020263, 32.18750342, 23.77228515, 27.10826844, 28.88233483,
        17.79339712, 16.23249967, 25.02233521, 27.40697378, 24.01081973,
        20.9085694 , 20.94202826, 17.03969642, 25.0247316 , 13.01837226,
        15.28516465, 16.05874373, 18.43463798, 17.16282305, 16.50646587,
        18.9234259 , 16.53289546, 12.3301473 , 14.91260929, 36.53029657,
         3.44934894,  9.86726338,  2.15755822, 14.51490398, 25.64604461,
        28.14171927, 20.55676673, 18.99700387,  6.29338716,  0.36723264,
        25.21231157, 19.00722091, 20.84373786, 16.83178745, 16.52858787,
         6.94732752, 23.14935994, 18.00555339, 17.87839155,  5.58726281,
        11.43561647,  7.11854835,  7.71483181,  9.85102228, 18.58218928,
        22.06088402, 23.23262238, 13.50474107, 25.30921032, 21.85118929,
        25.93778574, 25.4333105 , 21.25055767,  6.49120345, 16.70422006,
        14.62619407, 22.47437656, 23.96833655, 14.36796361,  7.2210479 ,
         3.00634116,  8.63434   , 23.73687785, 19.24233528, 26.55582952,
        14.69932053, 24.6247661 ,  6.53985825, 13.3906666 , -4.21497838,
        17.08348653, 22.44850424, 10.47410586,  2.99281815, 23.29672629,
        25.22113115, 23.89095978, 22.77551145, 21.75192822, 21.64393502,
        16.85421874, 22.9314145 , 17.00701415, 22.25307386, 21.39011639,
        25.78774037, 26.91686181, 29.16977078, 25.4209194 , 22.48929316,
        21.52325279, 22.11480045, 16.51672591, 12.5907749 , 17.84338686,
        15.60712236, 23.04596371, 24.61591619, 23.81288272, 17.27815149,
        20.49211538, 24.31173475, 23.4049838 , 22.79344004, 23.91122798,
        25.99821758, 26.340751  , 24.80431406, 29.81887588, 24.31283927,
        25.00306443, 21.45560985, 20.94064609, 23.81165456, 23.93028539,
        26.86069153, 26.55911716, 25.52133615, 28.37067106, 25.58182393,
        23.86249106, 22.9542643 , 23.23084127, 20.85348155, 22.14460316,
        25.95941808, 28.56570542, 28.66950016, 32.40424752, 20.80900555,
        22.63860862, 26.82479237, 15.10173037, 24.19086321, 25.83217571,
        28.72040665, 32.90390596, 34.77066018, 26.30591365, 25.34381775,
        28.23758999, 25.17944017, 26.4705689 , 11.10717431,  8.17635637,
         3.73550246, 13.8301605 , 15.69551789, 20.29222007, 20.71648521,
        17.02842287, 13.84795979, 19.07576995, 21.45547232, 18.09944532,
        20.72111835, 23.33408351, 21.35832828, 27.58215819, 25.92833448,
        21.13637498]),
 'deviation': array([0.  , 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1 ,
        0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.2 , 0.21,
        0.22, 0.23, 0.24, 0.25, 0.26, 0.27, 0.28, 0.29, 0.3 , 0.31, 0.32,
        0.33, 0.34, 0.35, 0.36, 0.37, 0.38, 0.39, 0.4 , 0.41, 0.42, 0.43,
        0.44, 0.45, 0.46, 0.47, 0.48, 0.49, 0.5 , 0.51, 0.52, 0.53, 0.54,
        0.55, 0.56, 0.57, 0.58, 0.59, 0.6 , 0.61, 0.62, 0.63, 0.64, 0.65,
        0.66, 0.67, 0.68, 0.69, 0.7 , 0.71, 0.72, 0.73, 0.74, 0.75, 0.76,
        0.77, 0.78, 0.79, 0.8 , 0.81, 0.82, 0.83, 0.84, 0.85, 0.86, 0.87,
        0.88, 0.89, 0.9 , 0.91, 0.92, 0.93, 0.94, 0.95, 0.96, 0.97, 0.98,
        0.99]),
 'accuracy': [0.0,
  0.04940711462450593,
  0.10869565217391304,
  0.17391304347826086,
  0.22924901185770752,
  0.274703557312253,
  0.3201581027667984,
  0.3616600790513834,
  0.40711462450592883,
  0.4407114624505929,
  0.4881422924901186,
  0.5276679841897233,
  0.5612648221343873,
  0.5849802371541502,
  0.616600790513834,
  0.6442687747035574,
  0.6600790513833992,
  0.6818181818181818,
  0.7055335968379447,
  0.717391304347826,
  0.7351778656126482,
  0.7509881422924901,
  0.7648221343873518,
  0.7766798418972332,
  0.7865612648221344,
  0.7924901185770751,
  0.808300395256917,
  0.8181818181818182,
  0.8260869565217391,
  0.8320158102766798,
  0.8399209486166008,
  0.8458498023715415,
  0.8557312252964426,
  0.8675889328063241,
  0.8774703557312253,
  0.8893280632411067,
  0.8952569169960475,
  0.9051383399209486,
  0.9090909090909091,
  0.9189723320158103,
  0.9229249011857708,
  0.9288537549407114,
  0.9308300395256917,
  0.9347826086956522,
  0.9387351778656127,
  0.9446640316205533,
  0.950592885375494,
  0.950592885375494,
  0.9525691699604744,
  0.9545454545454546,
  0.9565217391304348,
  0.9624505928853755,
  0.9644268774703557,
  0.9644268774703557,
  0.9664031620553359,
  0.9683794466403162,
  0.9703557312252964,
  0.9703557312252964,
  0.974308300395257,
  0.9782608695652174,
  0.9802371541501976,
  0.9822134387351779,
  0.9822134387351779,
  0.9841897233201581,
  0.9861660079051383,
  0.9881422924901185,
  0.9881422924901185,
  0.9881422924901185,
  0.9881422924901185,
  0.9901185770750988,
  0.9901185770750988,
  0.9901185770750988,
  0.9901185770750988,
  0.9901185770750988,
  0.9901185770750988,
  0.9901185770750988,
  0.9901185770750988,
  0.9901185770750988,
  0.9901185770750988,
  0.9901185770750988,
  0.9901185770750988,
  0.9920948616600791,
  0.9920948616600791,
  0.9920948616600791,
  0.9920948616600791,
  0.9920948616600791,
  0.9920948616600791,
  0.9940711462450593,
  0.9940711462450593,
  0.9940711462450593,
  0.9940711462450593,
  0.9960474308300395,
  0.9960474308300395,
  0.9960474308300395,
  0.9960474308300395,
  0.9960474308300395,
  0.9960474308300395,
  0.9960474308300395,
  0.9980237154150198,
  0.9980237154150198],
 'auc_rec': 0.8348534255599472}

ROC vs Precision-Recall Curves with N-Folds Cross-Validation

2019-09-17T00:00:00+00:00

Using n-folds Cross Validation is a stapled piece to any problems for the sake of training. In this post, I have presented the ROC curves and Precision-Recall curves with n-folds Cross-Validation using XGBoost. The ROC one comes from Scikit-Learn documentation and I have customized it for Precision-Recall accordingly.

Loading Libraries

# Loading libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib as mpl
from scipy import interp
from sklearn.preprocessing import scale
from sklearn.metrics import roc_auc_score, classification_report, accuracy_score, roc_curve, confusion_matrix, average_precision_score, precision_recall_curve
from sklearn.model_selection import cross_val_score, KFold, StratifiedKFold, train_test_split
from xgboost import XGBClassifier
import itertools
import glmnet
import xgboost as xgb

import seaborn as sns
sns.set_style("ticks")
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] = 2
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:95% !important; }</style>"))
import warnings 
warnings.filterwarnings("ignore")
%matplotlib inline

Functions

def _plot_pr_nfolds_xgboost(df_X, Y,
                                     n_folds = 10,
                                     n_estimators = 100,
                                     learning_rate = 0.05,
                                     max_depth = 3,
                                     min_child_weight = 5.0,
                                     gamma = 0.5,
                                     reg_alpha = 0.0,
                                     reg_lambda = 1.0,
                                     subsample = 0.9,
                                     objective = "binary:logistic",
                                     scale_pos_weight = 1.0,
                                     shuffle = True,
                                     random_state = 1367,
                                     saveFigPath = None
                            ):
    """
    a function to plot k-fold cv Precision-Recall (PR) using xgboost
    input parameters:
                     df_X : features : pandas dataframe (numpy array will be built inside the function)
                     Y : targets
                     n_folds : number of cv folds (default = 10)
                     n_estimators = number of trees (default = 100)
                     learning_rate : step size of xgboost (default = 0.05)
                     max_depth : maximum tree depth for xgboost (default = 3)
                     min_child_weight : (default = 5.0)
                     gamma : (default = 0.5)
                     reg_alpha : lasso penalty (L1) (default = 0.0)
                     reg_lambda : ridge penalty (L2) (default = 1.0)
                     subsample : subsample fraction (default = 0.9)
                     objective : objective function for ML (default = "binary:logistic" for classification)
                     scale_pos_weight : (default = 1.0)
                     shuffle : shuffle flag for cv (default = True)
                     random_state : (default = 1367
                     saveFigPath: name of the file for saving the figure as PDF format. (default = None))
    
    """
    
    import numpy as np
    from scipy import interp
    import matplotlib.pyplot as plt
    from sklearn.metrics import roc_curve, auc, precision_recall_curve
    from sklearn.model_selection import StratifiedKFold
    from xgboost import XGBClassifier
    from sklearn.preprocessing import scale
    import matplotlib as mpl

    mpl.rcParams['axes.linewidth'] = 3 
    mpl.rcParams['lines.linewidth'] =7


    # Defining the data + scaling
    if isinstance(df_X, pd.DataFrame):
        X = scale(df_X.values)
    else:
        X = scale(df_X)
        
    y = Y
    n_samples, n_features = X.shape


    # #############################################################################
    # Classification and PR analysis

    # Run classifier with cross-validation and plot PR curves
    cv = StratifiedKFold(n_splits = n_folds , shuffle = shuffle , random_state = random_state)
    classifier = XGBClassifier(learning_rate = learning_rate,
                               n_estimators = n_estimators,
                               max_depth = max_depth,
                               min_child_weight = min_child_weight,
                               gamma = gamma,
                               reg_alpha = reg_alpha,
                               reg_lambda = reg_lambda,
                               subsample = subsample,
                               objective = objective,
                               nthread = 4,
                               scale_pos_weight = 1.,
                               base_score = np.mean(y),
                               seed = random_state,
                               random_state = random_state)
    

    # defining the lists
    prs = []
    aucs = []
    mean_recall = np.linspace(0, 1, 100)
    
    plt.figure(figsize=(18 , 13))
    i = 0
    for train, test in cv.split(X, y):
        probas_ = classifier.fit(X[train], y[train]).predict_proba(X[test])
        # Compute PR curve and area the curve
        precision, recall, thresholds = precision_recall_curve(y[test], probas_[:, 1])
        prs.append(interp(mean_recall, precision, recall))
        pr_auc = auc(recall, precision)
        aucs.append(pr_auc)
        plt.plot(recall, precision, lw=3, alpha=0.5, label='Fold %d (AUCPR = %0.2f)' % (i+1, pr_auc))
        i += 1
    
    plt.plot([0, 1], [1, 0], linestyle='--', lw=3, color='k', label='Luck', alpha=.8)
    mean_precision = np.mean(prs, axis=0)
    mean_auc = auc(mean_recall, mean_precision)
    std_auc = np.std(aucs)
    plt.plot(mean_precision, mean_recall, color='navy',
             label=r'Mean (AUCPR = %0.3f $\pm$ %0.2f)' % (mean_auc, std_auc),
             lw=4)
    

    plt.xlim([-0.05, 1.05])
    plt.ylim([-0.05, 1.05])
    plt.xlabel('Recall' ,  fontweight = "bold" , fontsize=30)
    plt.ylabel('Precision',fontweight = "bold" , fontsize=30)
    plt.tick_params(axis='both', which='major', labelsize=20)
    plt.legend( prop={'size':20} , loc = 0)
    if saveFigPath:
        plt.savefig(F"./{saveFigPath}.pdf" ,bbox_inches='tight')
    plt.show()

 
# --------------------------------------------------------------------------------------

def _plot_roc_nfolds_xgboost(df_X, Y,
                                     n_folds = 10,
                                     n_estimators = 100,
                                     learning_rate = 0.05,
                                     max_depth = 3,
                                     min_child_weight = 5.0,
                                     gamma = 0.5,
                                     reg_alpha = 0.0,
                                     reg_lambda = 1.0,
                                     subsample = 0.9,
                                     objective = "binary:logistic",
                                     scale_pos_weight = 1.0,
                                     shuffle = True,
                                     random_state = 1367,
                                     saveFigPath = None
                            ):
    """
    a function to plot k-fold cv ROC using xgboost
    input parameters:
                     df_X : features : pandas dataframe (numpy array will be built inside the function)
                     Y : targets
                     n_folds : number of cv folds (default = 10)
                     n_estimators = number of trees (default = 100)
                     learning_rate : step size of xgboost (default = 0.05)
                     max_depth : maximum tree depth for xgboost (default = 3)
                     min_child_weight : (default = 5.0)
                     gamma : (default = 0.5)
                     reg_alpha : lasso penalty (L1) (default = 0.0)
                     reg_lambda : ridge penalty (L2) (default = 1.0)
                     subsample : subsample fraction (default = 0.9)
                     objective : objective function for ML (default = "binary:logistic" for classification)
                     scale_pos_weight : (default = 1.0)
                     shuffle : shuffle flag for cv (default = True)
                     random_state : (default = 1367
                     saveFigPath: name of the file for saving the figure as PDF format. (default = None))
    
    """
    
    import numpy as np
    from scipy import interp
    import matplotlib.pyplot as plt
    from sklearn.metrics import roc_curve, auc
    from sklearn.model_selection import StratifiedKFold
    from xgboost import XGBClassifier
    from sklearn.preprocessing import scale
    import matplotlib as mpl

    mpl.rcParams['axes.linewidth'] = 3 
    mpl.rcParams['lines.linewidth'] =7


    # Defining the data + scaling
    if isinstance(df_X, pd.DataFrame):
        X = scale(df_X.values)
    else:
        X = scale(df_X)
        
    y = Y
    n_samples, n_features = X.shape


    # #############################################################################
    # Classification and ROC analysis

    # Run classifier with cross-validation and plot ROC curves
    cv = StratifiedKFold(n_splits = n_folds , shuffle = shuffle , random_state = random_state)
    classifier = XGBClassifier(learning_rate = learning_rate,
                               n_estimators = n_estimators,
                               max_depth = max_depth,
                               min_child_weight = min_child_weight,
                               gamma = gamma,
                               reg_alpha = reg_alpha,
                               reg_lambda = reg_lambda,
                               subsample = subsample,
                               objective = objective,
                               nthread = 4,
                               scale_pos_weight = 1.,
                               base_score = np.mean(y),
                               seed = random_state,
                               random_state = random_state)
    

    tprs = []
    aucs = []
    mean_fpr = np.linspace(0, 1, 100)
    plt.figure(figsize=(18 , 13))
    i = 0
    for train, test in cv.split(X, y):
        probas_ = classifier.fit(X[train], y[train]).predict_proba(X[test])
        # Compute ROC curve and area the curve
        fpr, tpr, thresholds = roc_curve(y[test], probas_[:, 1])
        tprs.append(interp(mean_fpr, fpr, tpr))
        tprs[-1][0] = 0.0
        roc_auc = auc(fpr, tpr)
        aucs.append(roc_auc)
        plt.plot(fpr, tpr, lw=3, alpha=0.5, 
                 label='ROC Fold %d (AUC = %0.2f)' % (i+1, roc_auc))

        i += 1
    plt.plot([0, 1], [0, 1], linestyle='--', lw=3, color='k',
             label='Luck', alpha=.8)

    mean_tpr = np.mean(tprs, axis=0)
    mean_tpr[-1] = 1.0
    mean_auc = auc(mean_fpr, mean_tpr)
    std_auc = np.std(aucs)
    plt.plot(mean_fpr, mean_tpr, color='navy',
             label=r'Mean ROC (AUC = %0.3f $\pm$ %0.2f)' % (mean_auc, std_auc),
             lw=4)

    std_tpr = np.std(tprs, axis=0)
    tprs_upper = np.minimum(mean_tpr + std_tpr, 1)
    tprs_lower = np.maximum(mean_tpr - std_tpr, 0)
    plt.fill_between(mean_fpr, tprs_lower, tprs_upper, color='grey', alpha=.4,
                     label=r'$\pm$ 1 Standard Deviation')

    plt.xlim([-0.05, 1.05])
    plt.ylim([-0.05, 1.05])
    plt.xlabel('False Positive Rate' ,  fontweight = "bold" , fontsize=30)
    plt.ylabel('True Positive Rate',fontweight = "bold" , fontsize=30)
    plt.tick_params(axis='both', which='major', labelsize=20)
    plt.legend( prop={'size':20} , loc = 4)
    if saveFigPath:
        plt.savefig(F"./{saveFigPath}.pdf" ,bbox_inches='tight')
    plt.show()   

KDD CUP 99 Data

from sklearn.datasets import fetch_kddcup99
data = fetch_kddcup99()
df = pd.DataFrame(data.data)
df.drop([1,2,3], axis = 1, inplace = True)
df = df.astype(np.float32)
# df = df.select_dtypes(exclude=['object'])
y = data.target

df.dtypes

   float32
   float32
   float32
   float32
   float32
   float32
   float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
  float32
dtype: object

Adding noise to features to not have perfect problem

dimensions = df.shape #to get the dimesion of the data
noise = np.random.rand(dimensions[0], dimensions[1])
noisy_df = df + df * noise # to add noise the existing data 

Checking the targets

pd.Series(y).value_counts()

b'smurf.'              280790
b'neptune.'            107201
b'normal.'              97278
b'back.'                 2203
b'satan.'                1589
b'ipsweep.'              1247
b'portsweep.'            1040
b'warezclient.'          1020
b'teardrop.'              979
b'pod.'                   264
b'nmap.'                  231
b'guess_passwd.'           53
b'buffer_overflow.'        30
b'land.'                   21
b'warezmaster.'            20
b'imap.'                   12
b'rootkit.'                10
b'loadmodule.'              9
b'ftp_write.'               8
b'multihop.'                7
b'phf.'                     4
b'perl.'                    3
b'spy.'                     2
dtype: int64

Building binary flag for ‘back’ or ‘normal’ class

yy = np.where(((y == b'back.')|(y == b'normal.')), 1, 0)
print(F"Shape: {df.shape}")
print(F"""Prevalence = {yy.sum()/len(yy)* 100:.3f}%""")

Shape: (494021, 38)
Prevalence = 20.137%

pd.Series(yy).value_counts().plot(kind ="pie", autopct="%1.2f%%")
plt.show()

noisy_df.head()

	4	5	11	...	31	32	33	35
0	210.804036	6844.436574	1.888843	...	12.502868	11.185510	1.111647	0.182288
1	240.040258	755.161401	1.927321	...	31.836429	36.460213	1.217937	0.098349
2	359.574976	2614.731730	1.506819	...	48.112809	30.087684	1.936690	0.054079
3	324.864701	2027.837786	1.845100	...	52.215076	69.985597	1.183668	0.031935
4	340.145440	2828.912572	1.280997	...	71.912794	93.026926	1.645168	0.034165

5 rows × 38 columns

ROC Curve + 10-folds CV with XGBOOST

_plot_roc_nfolds_xgboost(noisy_df.loc[:, [ 11, 40]], yy, n_folds = 10, n_estimators = 3)

Precision Curve + 10-folds CV with XGBOOST

_plot_pr_nfolds_xgboost(noisy_df.loc[:, [11, 40]], yy, n_folds = 10, n_estimators = 3)

Finding Thresholds in Imbalanced Binary Classification

2019-08-16T00:00:00+00:00

Classification with imbalanced class distributions is a major problem in machine learning. Researchers have given considerable attention to the applications in many real-world scenarios. Although several works have utilized the area under the receiver operating characteristic (ROC) curve to select potentially optimal classifiers in imbalanced classifications, limited studies have been devoted to finding the classification threshold for testing or unknown datasets. In general, the classification threshold is simply set to 0.5, which is usually unsuitable for an imbalanced classification. Here I showed, thresholds based on FPR-TPR, Precision-Recall-F1score, and Youden J-index in Python

Loading Libraries

# Loading libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib as mpl
from scipy import interp
from sklearn.preprocessing import scale
from sklearn.metrics import roc_auc_score, classification_report, accuracy_score, roc_curve, confusion_matrix, average_precision_score, precision_recall_curve
from sklearn.model_selection import cross_val_score, KFold, StratifiedKFold, train_test_split
from xgboost import XGBClassifier
import itertools
import glmnet
import xgboost as xgb

import seaborn as sns
sns.set_style("ticks")
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] = 2
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:95% !important; }</style>"))
import warnings 
warnings.filterwarnings("ignore")
%matplotlib inline

Functions

#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
# Function: _model

def _clf_train(X_train, y_train, X_test, y_test,
                 learning_rate = 0.05,
                 n_estimators = 100,
                 max_depth = 3,
                 min_child_weight = 5.0,
                 gamma = 1,
                 reg_alpha = 0.0,
                 reg_lambda = 1.0,
                 subsample = 0.9,
                 colsample_bytree = 0.9,
                 objective = "binary:logistic",
                 nthread = 4,
                 scale_pos_weight = 1.0,
                 seed = 1367,
                 random_state = 1367):
    """
    an xgboost model for training
    """

    clf = XGBClassifier(learning_rate = learning_rate,
                        n_estimators = n_estimators,
                        max_depth = max_depth,
                        min_child_weight = min_child_weight,
                        gamma = gamma,
                        reg_alpha = reg_alpha,
                        reg_lambda = reg_lambda,
                        subsample = subsample,
                        colsample_bytree = colsample_bytree,
                        objective = objective,
                        nthread = nthread,
                        scale_pos_weight = scale_pos_weight,
                        seed = seed,
                        random_state = random_state)
    
    clf.fit(X_train, y_train, eval_metric = "auc", early_stopping_rounds = 20, verbose = True, eval_set = [(X_test, y_test)])

    return clf

#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
# Function: Finding thresholds
def _threshold_finder(model, X, y_true):
    """
    a function to find the optimal threshold for binary classification
    model: a trained model object (such as xgboost, glmnet, ...)
    X: the test set of features (pandas dataframe or numpy array)
    y_true: the true class labels (list or array of 0's and 1's)    
    """
    
    y_predict_proba = model.predict_proba(X)[:, 1]
    fpr, tpr, thresholds = roc_curve(y_true, y_predict_proba)
    auc = roc_auc_score(y_true, y_predict_proba)
    precision, recall, thresholds2 = precision_recall_curve(y_true, y_predict_proba)
    
    class_names = [0, 1]
    youden_idx = np.argmax(np.abs(tpr - fpr))
    youden_threshold = thresholds[youden_idx]
    y_pred_youden = (y_predict_proba > youden_threshold).astype(int)
    cnf_matrix = confusion_matrix(y_true, y_pred_youden)
    np.set_printoptions(precision=2)
    
    f1 = []
    for i in range(len(precision)):
        f1.append(2 * (precision[i] * recall[i]) / (precision[i] + recall[i]))
        
    queue_rate = []
    for thr in thresholds2:
        queue_rate.append((y_predict_proba >= thr).mean()) 

    plt.figure(figsize = (10, 5))
    plt.subplot(1,2,1)
    plt.plot(fpr, tpr, color = "red", label = F"AUC = {auc:.3f}")
    plt.plot(fpr[youden_idx], tpr[youden_idx], marker = "o", color = "navy", ms =10, label =F"Youden Threshold = {youden_threshold:.2f}" )
    plt.axvline(x = fpr[youden_idx], ymin = fpr[youden_idx], ymax = tpr[youden_idx], color = "navy", ls = "--")
    plt.plot([0,1], [0,1] , color = "black", ls = "--")
    plt.xlim([-0.01, 1.01])
    plt.ylim([-0.01, 1.01])
    plt.xlabel('1 - Specificity' , fontsize=12)
    plt.ylabel('Sensitivity' , fontsize=12)
    plt.tick_params(axis='both', which='major', labelsize=12)
    plt.legend( prop={'size':12} , loc = 4)

    plt.subplot(1,2,2)
    _plot_confusion_matrix(cnf_matrix, classes=class_names, normalize = False, cmap=plt.cm.Reds, title = F"Youden Threshold = {youden_threshold:.2f}\nAccuracy = {accuracy_score(y_true, y_pred_youden)*100:.2f}%")
    plt.show()
    
    plt.figure(figsize = (12, 5))
    plt.subplot(1,2,1)
    plt.plot(thresholds, 1-fpr, label = "1 - Specificity")
    plt.plot(thresholds, tpr, label = "Sensitivity")
    plt.xlabel("Threshold", fontsize = 12)
    plt.ylabel("Score", fontsize = 12)
    plt.legend(loc = 0)
    plt.xlim([0.025, thresholds[np.argmin(abs(tpr + fpr - 1))]+0.2])
    plt.axvline(thresholds[np.argmin(abs(tpr + fpr - 1))], color="k", ls = "--")
    plt.title(F"Threshold = {thresholds[np.argmin(abs(tpr + fpr - 1))]:.3f}", fontsize = 12)
    
    plt.subplot(1,2,2)
    plt.plot(thresholds2, precision[1:], label = "Precision")
    plt.plot(thresholds2, recall[1:], label = "Recall")
    plt.plot(thresholds2, f1[1:], label = "F1-Score")
    plt.plot(thresholds2, queue_rate, label = "Queue Rate")
    plt.legend(loc = 0)
    plt.xlim([0.025, thresholds2[np.argmin(abs(precision-recall))] + 0.2])
    plt.xlabel("Threshold", fontsize = 12)
    plt.ylabel("Score", fontsize = 12)
    plt.axvline(thresholds2[np.argmin(abs(precision-recall))], color="k", ls = "--")
    plt.title(label = F"Threshold = {thresholds2[np.argmin(abs(precision-recall))]:.3f}", fontsize = 12)
    plt.show()

#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
# Function: Plotting Confusion Matrix
def _plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Greens):
    from sklearn.metrics import precision_score, recall_score, roc_auc_score, accuracy_score, roc_curve, auc, confusion_matrix
    import itertools
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]

    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title, fontsize = 14)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes)
    plt.yticks(tick_marks, classes)

    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="black")

    plt.ylabel('True Class', fontsize = 14)
    plt.xlabel('Predicted Class', fontsize = 14)

    plt.tick_params(axis='both', which='major', labelsize=14)
    plt.tight_layout()

Device Failure Data

First, I loaded the data into a pandas dataframe to get some idea.

# readin the data into a dataframe
dateparser = lambda x: pd.datetime.strptime(x, "%Y-%m-%d")
df_raw = pd.read_csv("./device_failure.csv",
                     parse_dates = ["date"],
                     date_parser = dateparser,
                     encoding = "cp1252")

print("Shape: {}".format(df_raw.shape))
print("Prevalence = {:.3f}%".format(df_raw["failure"].sum()/df_raw.shape[0] * 100))

Shape: (124494, 12)
Prevalence = 0.085%

df_raw.head()

	date	device	attribute1	attribute2	attribute3	attribute4	attribute5	attribute6	attribute9
0	2015-01-01	S1F01085	215630672	56	0	52	6	407438	7
1	2015-01-01	S1F0166B	61370680	0	3	0	6	403174	0
2	2015-01-01	S1F01E6Y	173295968	0	0	0	12	237394	0
3	2015-01-01	S1F01JE0	79694024	0	0	0	6	410186	0
4	2015-01-01	S1F01R2B	135970480	0	0	0	15	313173	3

Preprocessing and builing X, Y training/testing sets.

target = "failure"
to_drop = ["date", "device", "attribute8", "failure"]
y = df_raw[target].values
X = df_raw.drop(to_drop, axis = 1)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.20, shuffle = True, random_state = 1367, stratify = y)
print(F"Train Size = {X_train.shape}")
print(F"Test Size = {X_test.shape}")

Train Size = (99595, 8)
Test Size = (24899, 8)

Printing the first 5 rows of the sclaed features.

X_train.head()

	attribute1	attribute3	attribute4	attribute5	attribute6	attribute9
7043	154658576	0	6	7	211825	1
115494	139749056	0	0	8	300480	0
42429	150809064	318	0	6	310486	18
35289	136377576	0	0	12	237584	0
57282	60144304	0	1	7	307632	0

Train a model

model = _clf_train(X_train, y_train, X_test, y_test)

[0]	validation_0-auc:0.594635
Will train until validation_0-auc hasn't improved in 20 rounds.
[1]	validation_0-auc:0.594635
[2]	validation_0-auc:0.618237
[3]	validation_0-auc:0.618237
[4]	validation_0-auc:0.618271
[5]	validation_0-auc:0.618271
[6]	validation_0-auc:0.618271
[7]	validation_0-auc:0.618271
[8]	validation_0-auc:0.617054
[9]	validation_0-auc:0.617054
[10]	validation_0-auc:0.662072
[11]	validation_0-auc:0.662072
[12]	validation_0-auc:0.662072
[13]	validation_0-auc:0.685412
[14]	validation_0-auc:0.685173
[15]	validation_0-auc:0.685164
[16]	validation_0-auc:0.685183
[17]	validation_0-auc:0.772991
[18]	validation_0-auc:0.773219
[19]	validation_0-auc:0.773221
[20]	validation_0-auc:0.773208
[21]	validation_0-auc:0.773261
[22]	validation_0-auc:0.773641
[23]	validation_0-auc:0.773637
[24]	validation_0-auc:0.773637
[25]	validation_0-auc:0.773635
[26]	validation_0-auc:0.773796
[27]	validation_0-auc:0.774048
[28]	validation_0-auc:0.773964
[29]	validation_0-auc:0.773948
[30]	validation_0-auc:0.773976
[31]	validation_0-auc:0.773968
[32]	validation_0-auc:0.773887
[33]	validation_0-auc:0.773855
[34]	validation_0-auc:0.773727
[35]	validation_0-auc:0.773725
[36]	validation_0-auc:0.773794
[37]	validation_0-auc:0.774106
[38]	validation_0-auc:0.774033
[39]	validation_0-auc:0.774095
[40]	validation_0-auc:0.77406
[41]	validation_0-auc:0.773846
[42]	validation_0-auc:0.773844
[43]	validation_0-auc:0.774055
[44]	validation_0-auc:0.773927
[45]	validation_0-auc:0.773927
[46]	validation_0-auc:0.773969
[47]	validation_0-auc:0.77399
[48]	validation_0-auc:0.774048
[49]	validation_0-auc:0.77426
[50]	validation_0-auc:0.814616
[51]	validation_0-auc:0.814617
[52]	validation_0-auc:0.814366
[53]	validation_0-auc:0.814277
[54]	validation_0-auc:0.814324
[55]	validation_0-auc:0.814311
[56]	validation_0-auc:0.814347
[57]	validation_0-auc:0.814238
[58]	validation_0-auc:0.814154
[59]	validation_0-auc:0.813977
[60]	validation_0-auc:0.813904
[61]	validation_0-auc:0.813735
[62]	validation_0-auc:0.811469
[63]	validation_0-auc:0.811915
[64]	validation_0-auc:0.811839
[65]	validation_0-auc:0.811803
[66]	validation_0-auc:0.811862
[67]	validation_0-auc:0.811875
[68]	validation_0-auc:0.811767
[69]	validation_0-auc:0.811832
[70]	validation_0-auc:0.81184
[71]	validation_0-auc:0.811859
Stopping. Best iteration:
[51]	validation_0-auc:0.814617

Visualization of Thresholds

_threshold_finder(model = model, X = X_test, y_true = y_test)

MNIST DATA

from sklearn.datasets import fetch_openml
mnist = fetch_openml("mnist_784", version = 1)
mnist.keys()

dict_keys(['data', 'target', 'feature_names', 'DESCR', 'details', 'categories', 'url'])

X, y = mnist["data"], mnist["target"]
X.shape

(70000, 784)

plt.figure()
plt.imshow(X[1000].reshape(28, 28), cmap = "gray")
plt.show()

As you know, minist contains 10 classes. So, we need to turn this multi-class data to a binary class.

X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]

y_train_0 = np.where(y_train == "0", 1, 0)
y_test_0 = np.where(y_test == "0", 1, 0)

model = _clf_train(X_train, y_train_0, X_test, y_test_0)

[0]	validation_0-auc:0.973597
Will train until validation_0-auc hasn't improved in 20 rounds.
[1]	validation_0-auc:0.97854
[2]	validation_0-auc:0.985121
[3]	validation_0-auc:0.984864
[4]	validation_0-auc:0.984705
[5]	validation_0-auc:0.986008
[6]	validation_0-auc:0.987206
[7]	validation_0-auc:0.988978
[8]	validation_0-auc:0.989422
[9]	validation_0-auc:0.989856
[10]	validation_0-auc:0.990381
[11]	validation_0-auc:0.990555
[12]	validation_0-auc:0.991208
[13]	validation_0-auc:0.991901
[14]	validation_0-auc:0.992248
[15]	validation_0-auc:0.993116
[16]	validation_0-auc:0.993295
[17]	validation_0-auc:0.99356
[18]	validation_0-auc:0.994208
[19]	validation_0-auc:0.994625
[20]	validation_0-auc:0.994797
[21]	validation_0-auc:0.994938
[22]	validation_0-auc:0.994881
[23]	validation_0-auc:0.995083
[24]	validation_0-auc:0.995208
[25]	validation_0-auc:0.995286
[26]	validation_0-auc:0.995402
[27]	validation_0-auc:0.995571
[28]	validation_0-auc:0.995561
[29]	validation_0-auc:0.995693
[30]	validation_0-auc:0.995979
[31]	validation_0-auc:0.996092
[32]	validation_0-auc:0.996214
[33]	validation_0-auc:0.996322
[34]	validation_0-auc:0.996352
[35]	validation_0-auc:0.99641
[36]	validation_0-auc:0.996402
[37]	validation_0-auc:0.996489
[38]	validation_0-auc:0.996557
[39]	validation_0-auc:0.996579
[40]	validation_0-auc:0.996637
[41]	validation_0-auc:0.996672
[42]	validation_0-auc:0.99685
[43]	validation_0-auc:0.996861
[44]	validation_0-auc:0.996935
[45]	validation_0-auc:0.996909
[46]	validation_0-auc:0.99693
[47]	validation_0-auc:0.996938
[48]	validation_0-auc:0.996977
[49]	validation_0-auc:0.997013
[50]	validation_0-auc:0.997113
[51]	validation_0-auc:0.99714
[52]	validation_0-auc:0.997175
[53]	validation_0-auc:0.997218
[54]	validation_0-auc:0.997237
[55]	validation_0-auc:0.997279
[56]	validation_0-auc:0.997314
[57]	validation_0-auc:0.997326
[58]	validation_0-auc:0.997386
[59]	validation_0-auc:0.997412
[60]	validation_0-auc:0.997424
[61]	validation_0-auc:0.997437
[62]	validation_0-auc:0.997439
[63]	validation_0-auc:0.997506
[64]	validation_0-auc:0.997521
[65]	validation_0-auc:0.997557
[66]	validation_0-auc:0.997573
[67]	validation_0-auc:0.997619
[68]	validation_0-auc:0.997634
[69]	validation_0-auc:0.997656
[70]	validation_0-auc:0.9977
[71]	validation_0-auc:0.997758
[72]	validation_0-auc:0.997782
[73]	validation_0-auc:0.997816
[74]	validation_0-auc:0.997846
[75]	validation_0-auc:0.997868
[76]	validation_0-auc:0.997896
[77]	validation_0-auc:0.997908
[78]	validation_0-auc:0.997916
[79]	validation_0-auc:0.997942
[80]	validation_0-auc:0.997937
[81]	validation_0-auc:0.997975
[82]	validation_0-auc:0.997996
[83]	validation_0-auc:0.998036
[84]	validation_0-auc:0.998106
[85]	validation_0-auc:0.998123
[86]	validation_0-auc:0.99815
[87]	validation_0-auc:0.998165
[88]	validation_0-auc:0.99818
[89]	validation_0-auc:0.998211
[90]	validation_0-auc:0.998225
[91]	validation_0-auc:0.998296
[92]	validation_0-auc:0.998316
[93]	validation_0-auc:0.998319
[94]	validation_0-auc:0.998359
[95]	validation_0-auc:0.998413
[96]	validation_0-auc:0.998415
[97]	validation_0-auc:0.998418
[98]	validation_0-auc:0.998432
[99]	validation_0-auc:0.99847

_threshold_finder(model = model, X = X_test, y_true = y_test_0)

Web Scraping using BeautifulSoup

2019-08-12T00:00:00+00:00

Web Scraping (also termed Screen Scraping, Web Data Extraction, Web Harvesting etc.) is a technique employed to extract large amounts of data from websites whereby the data is extracted and saved to a local file in your computer or to a database in table (spreadsheet) format.

# Loading libraries
import numpy as np
import pandas as pd
import requests
from bs4 import BeautifulSoup

Here is the link to Trump’s lies artcile from NY-Times: LINK

Here is the structure of the first lie:

Jan. 21 “I wasn’t a fan of Iraq. I didn’t want to go into Iraq.” (He was for an invasion before he was against it.)

link = "https://www.nytimes.com/interactive/2017/06/23/opinion/trumps-lies.html"
response = requests.get(link)

Collecting all the records

soup = BeautifulSoup(response.text, "html.parser")

results = soup.find_all(name = "span", attrs = {"class" : "short-desc"})

len(results)

results[0]

<span class="short-desc"><strong>Jan. 21 </strong>“I wasn't a fan of Iraq. I didn't want to go into Iraq.” <span class="short-truth"><a href="https://www.buzzfeed.com/andrewkaczynski/in-2002-donald-trump-said-he-supported-invading-iraq-on-the" target="_blank">(He was for an invasion before he was against it.)</a></span></span>

Parse the first lie into 4 structured columns:

Date
Lie
Description
Link

The first result to see the pattern of the html

r = results[0]

<span class="short-desc"><strong>Jan. 21 </strong>“I wasn't a fan of Iraq. I didn't want to go into Iraq.” <span class="short-truth"><a href="https://www.buzzfeed.com/andrewkaczynski/in-2002-donald-trump-said-he-supported-invading-iraq-on-the" target="_blank">(He was for an invasion before he was against it.)</a></span></span>

Parsing date by looking at the strong tag

date = r.find("strong").text[:-1] + ", 2017"

date

'Jan. 21, 2017'

Parsing lie by parsing the content out

lie = r.contents[1][1:-2]

lie

"I wasn't a fan of Iraq. I didn't want to go into Iraq."

Parsing explanation by looking at the a tag

explanation = r.find("a").text[1:-1]

explanation

'He was for an invasion before he was against it.'

Parsing link out by looking at the href key of the a tag

link = r.find("a")["href"]

link

'https://www.buzzfeed.com/andrewkaczynski/in-2002-donald-trump-said-he-supported-invading-iraq-on-the'

Now, iterating over the whole records and put it into a dataframe

rows = []
for r in results:
    date = r.find("strong").text[:-1] + ", 2017"
    lie = r.contents[1][1:-2]
    explanation = r.find("a").text[1:-1]
    link = r.find("a")["href"]
    rows.append( (date, lie, explanation, link) )

df = pd.DataFrame(data = rows, columns=["Date", "Lie", "Explanation", "Link"])
df["Date"] = pd.to_datetime(df["Date"])

df.head(20)

	Date	Lie	Explanation	Link
0	2017-01-21	I wasn't a fan of Iraq. I didn't want to go in...	He was for an invasion before he was against it.	https://www.buzzfeed.com/andrewkaczynski/in-20...
1	2017-01-21	A reporter for Time magazine — and I have been...	Trump was on the cover 11 times and Nixon appe...	http://nation.time.com/2013/11/06/10-things-yo...
2	2017-01-23	Between 3 million and 5 million illegal votes ...	There's no evidence of illegal voting.	https://www.nytimes.com/2017/01/23/us/politics...
3	2017-01-25	Now, the audience was the biggest ever. But th...	Official aerial photos show Obama's 2009 inaug...	https://www.nytimes.com/2017/01/21/us/politics...
4	2017-01-25	Take a look at the Pew reports (which show vot...	The report never mentioned voter fraud.	https://www.nytimes.com/2017/01/24/us/politics...
5	2017-01-25	You had millions of people that now aren't ins...	The real number is less than 1 million, accord...	https://www.nytimes.com/2017/03/13/us/politics...
6	2017-01-25	So, look, when President Obama was there two w...	There were no gun homicide victims in Chicago ...	https://www.dnainfo.com/chicago/2017-chicago-m...
7	2017-01-26	We've taken in tens of thousands of people. We...	Vetting lasts up to two years.	https://www.nytimes.com/interactive/2017/01/29...
8	2017-01-26	I cut off hundreds of millions of dollars off ...	Most of the cuts were already planned.	https://www.washingtonpost.com/news/fact-check...
9	2017-01-28	The coverage about me in the @nytimes and the ...	It never apologized.	https://www.nytimes.com/2016/11/13/us/election...
10	2017-01-29	The Cuban-Americans, I got 84 percent of that ...	There is no support for this.	http://www.pewresearch.org/fact-tank/2016/11/1...
11	2017-01-30	Only 109 people out of 325,000 were detained a...	At least 746 people were detained and processe...	http://markets.on.nytimes.com/research/stocks/...
12	2017-02-03	Professional anarchists, thugs and paid protes...	There is no evidence of paid protesters.	https://www.nytimes.com/2017/01/28/nyregion/jf...
13	2017-02-04	After being forced to apologize for its bad an...	It never apologized.	https://www.nytimes.com/2016/11/13/us/election...
14	2017-02-05	We had 109 people out of hundreds of thousands...	About 60,000 people were affected.	http://www.politifact.com/truth-o-meter/statem...
15	2017-02-06	I have already saved more than $700 million wh...	Much of the price drop was projected before Tr...	https://www.washingtonpost.com/news/fact-check...
16	2017-02-06	It's gotten to a point where it is not even be...	Terrorism has been reported on, often in detail.	https://www.nytimes.com/2017/02/07/us/politics...
17	2017-02-06	The failing @nytimes was forced to apologize t...	It didn't apologize.	https://www.nytimes.com/2016/11/13/us/election...
18	2017-02-06	And the previous administration allowed it to ...	The group’s origins date to 2004.	https://www.nytimes.com/2015/11/19/world/middl...
19	2017-02-07	And yet the murder rate in our country is the ...	It was higher in the 1980s and '90s.	http://www.politifact.com/truth-o-meter/statem...

Bayesian Optimization For XGBoost

2019-06-14T00:00:00+00:00

XGBoost has many hyper-paramters which need to be tuned to have an optimum model. There are many ways to find these tuned parameters such as grid-search or random search. However, bayesian optimization makes it easier and faster for us.

Gaussian processes (GPs) provide a principled, practical, and probabilistic approach in machine learning. GPs simply have an essential assumption that similar inputs give similar outputs. This simple and weak prior are actually very sensible for the effects of hyper-parameters. GPs are able to learn for each input dimension what the appropriate scale is for measuring similarity. GPs predict a Gaussian distribution of values rather than just predicting a single value. Bayesian optimization is a constrained global optimization approach built upon Bayesian inference and Gaussian process models to find the maximum value of an unknown function in the most efficient ways (less iterations).

# Loading libraries
from bayes_opt import BayesianOptimization as bo
import xgboost as xgb
import numpy as np
import pandas as pd 
import matplotlib.pyplot as plt
from sklearn.preprocessing import scale
import warnings
warnings.simplefilter("ignore")
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:95% !important; }</style>"))
import seaborn as sns
sns.set_style("ticks")
%matplotlib inline

Bayesian Optimization

def _my_bayesian_optimization(X, Y, n_iter = 5,
                                       init_points = 5,
                                       acq = "ei",
                                       num_boost_round = 1000,
                                       nfold = 10,
                                       stratified = True,
                                       metrics = ("auc"),
                                       early_stopping_rounds = 20,
                                       seed = 1367,
                                       shuffle = True,
                                       show_stdv = False,
                                       pbounds = None,
                                       importance_type = "total_gain",
                                       callbacks = False,
                                       verbose_eval = False):
    """
    a function to run bayesian optimization for xgboost
    input parameters:
                    X: features (pandas dataframe or numpy array)
                    Y: targets (1D array or list)
                    n_iter: total number of bayesian iterations (default = 5)
                    init_points: total initial points of optimization (default = 5)
                    acq
                    num_boost_rounds: max number of boosting rounds, (default = 1000)
                    stratified: stratificaiton of the targets (default = True)
                    metrics: classification/regression metrics (default = ("auc))
                    early_stopping_rounds: the criteria for stopping if the test metric is not improved (default = 20)
                    seed: random seed (default = 1367)
                    shuffle: shuffling the data (default = True)
                    show_stdv = showing standard deviation of cv results (default = False)
                    pbounds = set of parameters for bayesian optimization of xgboost cv
                            (default_params = {
                                               "eval_metric" : "auc",
                                               "tree_method": "hist",
                                               "objective" : "binary:logistic",
                                               "learning_rate" : 0.05,
                                               "max_depth": 2,
                                               "min_child_weight": 1,
                                               "gamma" : 0.0,
                                               "reg_alpha" : 0.0,
                                               "reg_lambda" : 1.0,
                                               "subsample" : 0.9,
                                               "max_delta_step": 1,
                                               "silent" : 1,
                                               "nthread" : 4,
                                               "scale_pos_weight" : 1
                                               }
                            )
                    importance_type = importance type of xgboost as string (default = "total_gain")
                                      the other options will be "weight", "gain", "cover", and "total_cover"
                    callbacks = printing callbacks for xgboost cv
                                (defaults = False, if True: [xgb.callback.print_evaluation(show_stdv = show_stdv),
                                                             xgb.callback.early_stop(early_stopping_rounds)])
                    verbose_eval : a flag to show the result during train on train/test sets (default = False)
    outputs:
            df_res: the parameters related to the best performance
            xgb_params: a dictionary of the best parameters of xgboost                
    
    
    """

    
    # callback flag
    if(callbacks == True):
        callbacks = [xgb.callback.print_evaluation(show_stdv = show_stdv),
                     xgb.callback.early_stop(early_stopping_rounds)]
    else:
        callbacks = None    
    
    # pbounds
    default_pbounds = {"max_depth" : (2, 5),
                       "learning_rate" : (0, 1), 
                       "min_child_weight" : (1, 20),
                       "subsample" : (0.1, 1),
                       "gamma": (0, 1),
                       "colsample_bytree": (0.1, 1.0)
                      }
    
    # updating the default parameters of the pbounds
    if pbounds is not None:
        for key, val in pbounds.items():
            default_pbounds[key] = val
    
    
    def __xgb_eval(learning_rate,
                   max_depth,
                   gamma,
                   colsample_bytree,
                   min_child_weight,
                   subsample):

        params = {"eval_metric" : "auc",
                  "tree_method": "hist",
                  "objective" : "binary:logistic",
                  "max_delta_step": 1,
                  "silent" : 1,
                  "nthread" : 4,
                  "scale_pos_weight" : 1,
                  "reg_alpha" : 0.0,
                  "reg_lambda" : 1.0,
                  "learning_rate" : learning_rate,
                  "max_depth": int(max_depth),
                  "min_child_weight": min_child_weight,
                  "gamma" : gamma,
                  "subsample" : subsample,
                  "colsample_bytree" : colsample_bytree 
                 }
        dtrain = xgb.DMatrix(data = X, label = Y)
        cv_result = xgb.cv(params = params,
                           dtrain = dtrain,
                           num_boost_round = num_boost_round,
                           nfold = nfold,
                           stratified = stratified,
                           metrics = metrics,
                           early_stopping_rounds = early_stopping_rounds,
                           seed = seed,
                           verbose_eval = verbose_eval,
                           shuffle = shuffle,
                           callbacks = callbacks)

        return cv_result.iloc[-1][2]
    

    xgb_bo = bo(__xgb_eval, default_pbounds, random_state = seed, verbose = 2)
    xgb_bo.maximize(init_points = init_points, n_iter = n_iter, acq = acq)
    
    
    targets = []
    for i, rs in enumerate(xgb_bo.res):
        targets.append(rs["target"])
    best_params = xgb_bo.res[targets.index(max(targets))]["params"]
    best_params["max_depth"] = int(best_params["max_depth"])
    
    xgb_params = {"eval_metric" : "auc",
                  "tree_method": "hist",
                  "objective" : "binary:logistic",
                  "max_delta_step": 1,
                  "silent" : 1,
                  "nthread" : 4,
                  "scale_pos_weight" : 1,
                  "reg_alpha" : 0.0,
                  "reg_lambda" : 1.0,
                  "learning_rate" : 0.05,
                  "max_depth": 2,
                  "min_child_weight": 5,
                  "gamma" : 0.0,
                  "subsample" : 1.0,
                  "colsample_bytree" : 0.9 
                 }
    for key, val in best_params.items():
        xgb_params[key] = val
    
    dtrain = xgb.DMatrix(data = X, label = Y)
    bst = xgb.train(params = xgb_params,
                    dtrain = dtrain,
                    num_boost_round = num_boost_round)
    
    # build results dataframe
    frames = []
    for idx, res in enumerate(xgb_bo.res):
        d = res['params']
        d[metrics] = res["target"]
        frames.append(pd.DataFrame(data = d, index = [idx]))
    
    res_df = pd.concat(frames)
   
    print(F"-*-*-*-*-*-* Optimization Results -*-*-*-*-*-*")
    display(res_df)
    
    # Plotting
    import matplotlib as mpl

    mpl.rcParams['axes.linewidth'] = 3 
    mpl.rcParams['lines.linewidth'] = 3
    cols = [col for col in res_df.columns.tolist() if col != "auc"]
    ip = 1
    plt.figure(figsize = (22, 10))
    colors = ["navy", "lavender", "lightblue", "cyan", "cadetblue", "slateblue"]
    for col in cols:
        res_df.sort_values(by = col, inplace=True)
        plt.subplot(2,3,ip)
        plt.plot(res_df.loc[:, col], res_df.loc[:, metrics], color = colors[ip-1])
        plt.xlabel(F"{col}", fontsize = 20)
        plt.ylabel(F"{metrics}", fontsize = 20)
        plt.tick_params(axis='both', which='major', labelsize = 12)
        ip += 1
    plt.show()
    
    print(F"-*-*-*-*-*-* Best Performance -*-*-*-*-*-*")
    display(res_df.loc[res_df[metrics] == res_df[metrics].max(), :])
    
    from xgboost import plot_importance
    from pylab import rcParams
    rcParams['figure.figsize'] = (10,10)
    plot_importance(bst, importance_type = importance_type, color = "skyblue", xlabel = importance_type)
    plt.show()   
    
    return res_df, xgb_params


def _x_y_maker(df, target, to_drop):
    """
    A function to receive the dataframe and make the X, and Y matrices:
    ----------
    parameters
    
    input: dataframe df
           string target (class label)
           list[string] to_drop (list of columns need to be dropped)
           
    output: dataframe Feature matrix (df_X)
            Target vector (Y)
    """    
    Targets = df[target]
    to_drop.append(target)
    Features = df.drop(to_drop, axis = 1)
    Scaled_Features = scale(Features.values)
    df_X = pd.DataFrame(data = Scaled_Features, columns = Features.columns.tolist())
    Y = Targets.values
    
    return df_X, Y

Loading Data

dateparser = lambda x: pd.datetime.strptime(x, "%Y-%m-%d")
df_raw = pd.read_csv("./device_failure.csv",
                     parse_dates = ["date"],
                     date_parser = dateparser,
                     encoding = "cp1252")

print("Shape: {}".format(df_raw.shape))
print("Prevalence = {:.3f}%".format(df_raw["failure"].sum()/df_raw.shape[0] * 100))

Shape: (124494, 12)
Prevalence = 0.085%

df_raw.head()

	date	device	attribute1	attribute2	attribute3	attribute4	attribute5	attribute6	attribute9
0	2015-01-01	S1F01085	215630672	56	0	52	6	407438	7
1	2015-01-01	S1F0166B	61370680	0	3	0	6	403174	0
2	2015-01-01	S1F01E6Y	173295968	0	0	0	12	237394	0
3	2015-01-01	S1F01JE0	79694024	0	0	0	6	410186	0
4	2015-01-01	S1F01R2B	135970480	0	0	0	15	313173	3

df = df_raw.copy()
target = "failure"
to_drop = ["date", "device", "attribute8"]
df_X, Y = _x_y_maker(df, target, to_drop)

df_res, params = _my_bayesian_optimization(df_X, Y)

|   iter    |  target   | colsam... |   gamma   | learni... | max_depth | min_ch... | subsample |
-------------------------------------------------------------------------------------------------
| [0m 1       [0m | [0m 0.8795  [0m | [0m 0.8975  [0m | [0m 0.04571 [0m | [0m 0.6628  [0m | [0m 3.343   [0m | [0m 1.436   [0m | [0m 0.3758  [0m |
| [0m 2       [0m | [0m 0.844   [0m | [0m 0.7422  [0m | [0m 0.1034  [0m | [0m 0.7671  [0m | [0m 3.934   [0m | [0m 18.39   [0m | [0m 0.3402  [0m |
| [0m 3       [0m | [0m 0.8697  [0m | [0m 0.206   [0m | [0m 0.7772  [0m | [0m 0.269   [0m | [0m 4.909   [0m | [0m 16.83   [0m | [0m 0.6439  [0m |
| [95m 4       [0m | [95m 0.8894  [0m | [95m 0.7187  [0m | [95m 0.9302  [0m | [95m 0.7736  [0m | [95m 2.183   [0m | [95m 10.72   [0m | [95m 0.6848  [0m |
| [95m 5       [0m | [95m 0.8912  [0m | [95m 0.7297  [0m | [95m 0.8513  [0m | [95m 0.4627  [0m | [95m 3.654   [0m | [95m 4.965   [0m | [95m 0.9395  [0m |
| [0m 6       [0m | [0m 0.5     [0m | [0m 0.1     [0m | [0m 1.0     [0m | [0m 0.0     [0m | [0m 2.0     [0m | [0m 3.543   [0m | [0m 0.1     [0m |
| [0m 7       [0m | [0m 0.8746  [0m | [0m 1.0     [0m | [0m 0.0     [0m | [0m 1.0     [0m | [0m 5.0     [0m | [0m 8.494   [0m | [0m 1.0     [0m |
| [0m 8       [0m | [0m 0.8899  [0m | [0m 1.0     [0m | [0m 0.0     [0m | [0m 1.0     [0m | [0m 5.0     [0m | [0m 1.0     [0m | [0m 1.0     [0m |
| [0m 9       [0m | [0m 0.5     [0m | [0m 1.0     [0m | [0m 1.0     [0m | [0m 0.0     [0m | [0m 2.0     [0m | [0m 15.37   [0m | [0m 1.0     [0m |
| [0m 10      [0m | [0m 0.5     [0m | [0m 0.1     [0m | [0m 1.0     [0m | [0m 1.0     [0m | [0m 5.0     [0m | [0m 20.0    [0m | [0m 0.1     [0m |
=================================================================================================
-*-*-*-*-*-* Optimization Results -*-*-*-*-*-*

	colsample_bytree	gamma	learning_rate	max_depth	min_child_weight	subsample	auc
0	0.897530	0.045712	0.662807	3.343081	1.435660	0.375782	0.879458
1	0.742226	0.103395	0.767115	3.934126	18.388615	0.340209	0.844032
2	0.206024	0.777161	0.269010	4.908585	16.829513	0.643933	0.869726
3	0.718692	0.930205	0.773597	2.183002	10.716203	0.684811	0.889447
4	0.729727	0.851274	0.462704	3.654198	4.964748	0.939488	0.891218
5	0.100000	1.000000	0.000000	2.000000	3.542604	0.100000	0.500000
6	1.000000	0.000000	1.000000	5.000000	8.493812	1.000000	0.874567
7	1.000000	0.000000	1.000000	5.000000	1.000000	1.000000	0.889880
8	1.000000	1.000000	0.000000	2.000000	15.368715	1.000000	0.500000
9	0.100000	1.000000	1.000000	5.000000	20.000000	0.100000	0.500000

-*-*-*-*-*-* Best Performance -*-*-*-*-*-*

	colsample_bytree	gamma	learning_rate	max_depth	min_child_weight	subsample	auc
4	0.729727	0.851274	0.462704	3.654198	4.964748	0.939488	0.891218

Best XGBoost Parameters

params

{'eval_metric': 'auc',
 'tree_method': 'hist',
 'objective': 'binary:logistic',
 'max_delta_step': 1,
 'silent': 1,
 'nthread': 4,
 'scale_pos_weight': 1,
 'reg_alpha': 0.0,
 'reg_lambda': 1.0,
 'learning_rate': 0.46270355802290863,
 'max_depth': 3,
 'min_child_weight': 4.964747561175287,
 'gamma': 0.8512735546087907,
 'subsample': 0.9394881696342923,
 'colsample_bytree': 0.7297272989171456}

Adaptive Learning Optimizers For Neural Networks

2019-06-08T00:00:00+00:00

# Loading Libraries
import time
import matplotlib.pyplot as plt
import numpy as np
from keras.models import Sequential
from keras.layers.core import Dense, Activation
from keras.optimizers import SGD
from keras.utils import np_utils
import matplotlib as mpl
import seaborn as sns
sns.set_style("ticks", {"xtick.direction": u"in", "ytick.direction": u"in"})
np.random.seed(1367)

Using TensorFlow backend.

from keras.datasets import mnist
(train_features, train_labels), (test_features, test_labels) = mnist.load_data()
_, img_rows, img_cols =  train_features.shape
num_classes = len(np.unique(train_labels))
num_input_nodes = img_rows*img_cols
print "Number of training samples: %d"%train_features.shape[0]
print "Number of test samples: %d"%test_features.shape[0]
print "Image rows: %d"%train_features.shape[1]
print "Image columns: %d"%train_features.shape[2]
print "Number of classes: %d"%num_classes

Number of training samples: 60000
Number of test samples: 10000
Image rows: 28
Image columns: 28
Number of classes: 10

fig = plt.figure(figsize=(10,5))
for i in range(num_classes):
    ax = fig.add_subplot(2, 5, 1 + i, xticks=[], yticks=[])
    features_idx = train_features[train_labels[:]==i,:]
    ax.set_title("Class: " + str(i) , fontsize = 20)
    plt.imshow(features_idx[1], cmap="gray")
plt.savefig("./mnist.png" , bbox_inches='tight')
plt.show()

# reshape images to column vectors
train_features = train_features.reshape(train_features.shape[0], img_rows*img_cols)
test_features = test_features.reshape(test_features.shape[0], img_rows*img_cols)
# convert class labels to binary class labels
train_labels = np_utils.to_categorical(train_labels, num_classes)
test_labels = np_utils.to_categorical(test_labels, num_classes)

def simple_nn():
    # initialize model
    model = Sequential()
    # add an input layer and a hidden layer
    model.add(Dense(100, input_dim = num_input_nodes))
    # add activation layer to add non-linearity
    model.add(Activation('sigmoid'))
    # to add ReLu instead of sigmoid: model.add(Activation('relu'))
    # combine above 2 layers: model.add(Dense(100, input_dim=784),Activation('sigmoid'))
    # add output layer
    model.add(Dense(num_classes))
    # add softmax layer 
    model.add(Activation('softmax'))
    return model

def plot_model_history(model_history):
    fig, axs = plt.subplots(1,2,figsize=(15,5))
    # summarize history for accuracy
    axs[0].plot(range(1,len(model_history.history['acc'])+1),model_history.history['acc'])
    axs[0].plot(range(1,len(model_history.history['val_acc'])+1),model_history.history['val_acc'])
    axs[0].set_title('Model Accuracy')
    axs[0].set_ylabel('Accuracy')
    axs[0].set_xlabel('Epoch')
    axs[0].set_xticks(np.arange(1,len(model_history.history['acc'])+1),len(model_history.history['acc'])/10)
    axs[0].legend(['train', 'val'], loc='best')
    # summarize history for loss
    axs[1].plot(range(1,len(model_history.history['loss'])+1),model_history.history['loss'])
    axs[1].plot(range(1,len(model_history.history['val_loss'])+1),model_history.history['val_loss'])
    axs[1].set_title('Model Loss')
    axs[1].set_ylabel('Loss')
    axs[1].set_xlabel('Epoch')
    axs[1].set_xticks(np.arange(1,len(model_history.history['loss'])+1),len(model_history.history['loss'])/10)
    axs[1].legend(['train', 'val'], loc='best')
    plt.show()

def accuracy(test_x, test_y, model):
    result = model.predict(test_x)
    predicted_class = np.argmax(result, axis=1)
    true_class = np.argmax(test_y, axis=1)
    num_correct = np.sum(predicted_class == true_class) 
    accuracy = float(num_correct)/result.shape[0]
    return (accuracy * 100)

# decrease the learning rate
# define model
model_sgd = simple_nn()
model_sgd.compile(optimizer="SGD", loss='categorical_crossentropy', metrics=['accuracy'])
start = time.time()
model_info_sgd = model_sgd.fit(train_features, train_labels, batch_size=128, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_sgd)
print "SGD Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "SGD Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model)

SGD Model took 102.65 seconds to train
SGD Accuracy on test data is: 94.28

# decrease the learning rate
# define model
model_adagrad = simple_nn()
model_adagrad.compile(optimizer="Adagrad", loss='categorical_crossentropy', metrics=['accuracy'])
start = time.time()
model_info_adagrad = model_adagrad.fit(train_features, train_labels, batch_size=128, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adagrad)
print "Adagrad Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "adagrad Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model)

Adagrad Model took 166.58 seconds to train
adagrad Accuracy on test data is: 94.28

# decrease the learning rate
# define model
model_adadelta = simple_nn()
model_adadelta.compile(optimizer="Adadelta", loss='categorical_crossentropy', metrics=['accuracy'])
start = time.time()
model_info_adadelta = model_adadelta.fit(train_features, train_labels, batch_size=128, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adadelta)
print "Adadelta Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "adadelta Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model)

Adadelta Model took 176.11 seconds to train
adadelta Accuracy on test data is: 94.28

# decrease the learning rate
# define model
model_adam = simple_nn()
model_adam.compile(optimizer="Adam", loss='categorical_crossentropy', metrics=['accuracy'])
start = time.time()
model_info_adam = model_adam.fit(train_features, train_labels, batch_size=128, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adam)
print "Adam Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Adam Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model)

Adam Model took 189.46 seconds to train
Adam Accuracy on test data is: 94.28

# decrease the learning rate
# define model
model_nadam = simple_nn()
model_nadam.compile(optimizer="Nadam", loss='categorical_crossentropy', metrics=['accuracy'])
start = time.time()
model_info_nadam = model_nadam.fit(train_features, train_labels, batch_size=128, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_nadam)
print "Nadam Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Nadam Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_nadam)

Nadam Model took 260.45 seconds to train
Nadam Accuracy on test data is: 94.52

# decrease the learning rate
# define model
model_adamax = simple_nn()
model_adamax.compile(optimizer="Adamax", loss='categorical_crossentropy', metrics=['accuracy'])
start = time.time()
model_info_adamax = model_adamax.fit(train_features, train_labels, batch_size=128, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adamax)
print "Nadam Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Adamax Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_adamax)

Nadam Model took 184.97 seconds to train
Adamax Accuracy on test data is: 96.61

# decrease the learning rate
# define model
model_rmsprop = simple_nn()
model_rmsprop.compile(optimizer="RMSprop", loss='categorical_crossentropy', metrics=['accuracy'])
start = time.time()
model_info_rmsprop = model_rmsprop.fit(train_features, train_labels, batch_size=128, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_rmsprop)
print "RMSprop Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "RMSprop Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_rmsprop)

RMSprop Model took 172.51 seconds to train
RMSprop Accuracy on test data is: 95.61

# Plot with Error bars
plt.figure(figsize=(20, 7))

plt.subplot(1,2,1)
plt.errorbar(range(1,len(model_info_sgd.history['acc'])+1),model_info_sgd.history['acc'] ,color = "DarkGreen", linestyle = "--" , marker = "v" , label = "SGD")
plt.errorbar(range(1,len(model_info_adam.history['acc'])+1),model_info_adam.history['acc'] ,color = "navy", linestyle = "--" , marker = "^" , label = "Adam")
plt.errorbar(range(1,len(model_info_nadam.history['acc'])+1),model_info_nadam.history['acc'] ,color = "red", linestyle = "--" , marker = "s" , label = "Nadam")
plt.errorbar(range(1,len(model_info_adamax.history['acc'])+1),model_info_adamax.history['acc'] ,color = "cyan", linestyle = "--" , marker = "h" , label = "Adamax")
plt.errorbar(range(1,len(model_info_adadelta.history['acc'])+1),model_info_adadelta.history['acc'] ,color = "yellow", linestyle = "--" , marker = ">" , label = "Adadelta")
plt.errorbar(range(1,len(model_info_adagrad.history['acc'])+1),model_info_adagrad.history['acc'],color = "purple", linestyle = "--" , marker = "<" , label = "Adagrad")
plt.errorbar(range(1,len(model_info_rmsprop.history['acc'])+1),model_info_rmsprop.history['acc'] ,color = "darkorange", linestyle = "--" , marker = "o" , label = "RMSprop")

plt.title("Training" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
plt.ylabel("Accuracy" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 4)


plt.subplot(1,2,2)
plt.errorbar(range(1,len(model_info_sgd.history['val_acc'])+1),model_info_sgd.history['val_acc'] ,color = "DarkGreen", linestyle = "--" , marker = "v" , label = "SGD")
plt.errorbar(range(1,len(model_info_adam.history['val_acc'])+1),model_info_adam.history['val_acc'] ,color = "navy", linestyle = "--" , marker = "^" , label = "Adam")
plt.errorbar(range(1,len(model_info_nadam.history['val_acc'])+1),model_info_nadam.history['val_acc']  ,color = "red", linestyle = "--" , marker = "s" , label = "Nadam")
plt.errorbar(range(1,len(model_info_adamax.history['val_acc'])+1),model_info_adamax.history['val_acc']  ,color = "cyan", linestyle = "--" , marker = "h" , label = "Adamax")
plt.errorbar(range(1,len(model_info_adadelta.history['val_acc'])+1),model_info_adadelta.history['val_acc'],color = "yellow", linestyle = "--" , marker = ">" , label = "Adadelta")
plt.errorbar(range(1,len(model_info_adagrad.history['val_acc'])+1),model_info_adagrad.history['val_acc'],color = "purple", linestyle = "--" , marker = "<" , label = "Adagrad")
plt.errorbar(range(1,len(model_info_rmsprop.history['val_acc'])+1),model_info_rmsprop.history['val_acc'],color = "darkorange", linestyle = "--" , marker = "o" , label = "RMSprop")

plt.title("Testing" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
plt.ylabel("Accuracy" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 4)

plt.savefig("./score_all.png" , bbox_inches='tight')
plt.show()

# Plot with Error bars
plt.figure(figsize=(20, 7))

plt.subplot(1,2,1)
plt.errorbar(range(1,len(model_info_sgd.history['loss'])+1),model_info_sgd.history['loss'] ,color = "DarkGreen", linestyle = "--" , marker = "v" , label = "SGD")
plt.errorbar(range(1,len(model_info_adam.history['loss'])+1),model_info_adam.history['loss'] ,color = "navy", linestyle = "--" , marker = "^" , label = "Adam")
plt.errorbar(range(1,len(model_info_nadam.history['loss'])+1),model_info_nadam.history['loss'],color = "red", linestyle = "--" , marker = "s" , label = "Nadam")
plt.errorbar(range(1,len(model_info_adamax.history['loss'])+1),model_info_adamax.history['loss'] ,color = "cyan", linestyle = "--" , marker = "h" , label = "Adamax")
plt.errorbar(range(1,len(model_info_adadelta.history['loss'])+1),model_info_adadelta.history['loss'] ,color = "yellow", linestyle = "--" , marker = ">" , label = "Adadelta")
plt.errorbar(range(1,len(model_info_adagrad.history['loss'])+1),model_info_adagrad.history['loss'],color = "purple", linestyle = "--" , marker = "<" , label = "Adagrad")
plt.errorbar(range(1,len(model_info_rmsprop.history['loss'])+1),model_info_rmsprop.history['loss'],color = "darkorange", linestyle = "--" , marker = "o" , label = "RMSprop")

plt.title("Training" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
plt.ylabel("Loss" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 1)

plt.subplot(1,2,2) 
plt.errorbar(range(1,len(model_info_sgd.history['val_loss'])+1),model_info_sgd.history['val_loss'] ,color = "DarkGreen", linestyle = "--" , marker = "v" , label = "SGD")
plt.errorbar(range(1,len(model_info_adam.history['val_loss'])+1),model_info_adam.history['val_loss'] ,color = "navy", linestyle = "--" , marker = "^" , label = "Adam")
plt.errorbar(range(1,len(model_info_nadam.history['val_loss'])+1),model_info_nadam.history['val_loss'],color = "red", linestyle = "--" , marker = "s" , label = "Nadam")
plt.errorbar(range(1,len(model_info_adamax.history['val_loss'])+1),model_info_adamax.history['val_loss'],color = "cyan", linestyle = "--" , marker = "h" , label = "Adamax")
plt.errorbar(range(1,len(model_info_adadelta.history['val_loss'])+1),model_info_adadelta.history['val_loss'],color = "yellow", linestyle = "--" , marker = ">" , label = "Adadelta")
plt.errorbar(range(1,len(model_info_adagrad.history['val_loss'])+1),model_info_adagrad.history['val_loss'],color = "purple", linestyle = "--" , marker = "<" , label = "Adagrad")
plt.errorbar(range(1,len(model_info_rmsprop.history['val_loss'])+1),model_info_rmsprop.history['val_loss'],color = "darkorange", linestyle = "--" , marker = "o" , label = "RMSprop")

plt.title("Testing" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
plt.ylabel("Loss" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 1)

plt.savefig("./loss_all.png" ,bbox_inches='tight')
plt.show()

# decrease the learning rate
# define model
import keras
adamax_lr1 = keras.optimizers.Adamax(lr=0.1)
model_adamax_lr01 = simple_nn()
model_adamax_lr01.compile(optimizer=adamax_lr1, loss='mae', metrics=['accuracy'])
start = time.time()
model_info_adamax_lr01 = model_adamax_lr01.fit(train_features, train_labels, batch_size=128, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adamax_lr01)
print " Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_adamax_lr01)

 Model took 277.18 seconds to train
Accuracy on test data is: 74.45

# decrease the learning rate
# define model
import keras
adamax_lr2 = keras.optimizers.Adamax(lr=0.01)
model_adamax_lr2 = simple_nn()
model_adamax_lr2.compile(optimizer=adamax_lr2, loss='mae', metrics=['accuracy'])
start = time.time()
model_info_adamax_lr2 = model_adamax_lr2.fit(train_features, train_labels, batch_size=128, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adamax_lr2)
print " Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_adamax_lr2)

 Model took 216.88 seconds to train
Accuracy on test data is: 95.15

# decrease the learning rate
# define model
import keras
adamax_lr3 = keras.optimizers.Adamax(lr=0.001)
model_adamax_lr3 = simple_nn()
model_adamax_lr3.compile(optimizer=adamax_lr3, loss='mae', metrics=['accuracy'])
start = time.time()
model_info_adamax_lr3 = model_adamax_lr3.fit(train_features, train_labels, batch_size=128, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adamax_lr3)
print " Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_adamax_lr3)

 Model took 161.09 seconds to train
Accuracy on test data is: 95.71

# decrease the learning rate
# define model
import keras
adamax_lr4 = keras.optimizers.Adamax(lr=0.1)
model_adamax_lr4 = simple_nn()
model_adamax_lr4.compile(optimizer=adamax_lr4, loss='mse', metrics=['accuracy'])
start = time.time()
model_info_adamax_lr4 = model_adamax_lr4.fit(train_features, train_labels, batch_size=128, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adamax_lr4)
print " Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_adamax_lr4)

 Model took 327.13 seconds to train
Accuracy on test data is: 83.28

# decrease the learning rate
# define model
import keras
adamax_lr5 = keras.optimizers.Adamax(lr=0.01)
model_adamax_lr5 = simple_nn()
model_adamax_lr5.compile(optimizer=adamax_lr5, loss='mse', metrics=['accuracy'])
start = time.time()
model_info_adamax_lr5 = model_adamax_lr5.fit(train_features, train_labels, batch_size=128, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adamax_lr5)
print " Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_adamax_lr5)

 Model took 208.68 seconds to train
Accuracy on test data is: 95.77

# decrease the learning rate
# define model
import keras
adamax_lr6 = keras.optimizers.Adamax(lr=0.001)
model_adamax_lr6 = simple_nn()
model_adamax_lr6.compile(optimizer=adamax_lr6, loss='mse', metrics=['accuracy'])
start = time.time()
model_info_adamax_lr6 = model_adamax_lr6.fit(train_features, train_labels, batch_size=128, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adamax_lr6)
print " Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_adamax_lr6)

 Model took 148.24 seconds to train
Accuracy on test data is: 96.00

# decrease the learning rate
# define model
import keras
adamax_lr7 = keras.optimizers.Adamax(lr=0.1)
model_adamax_lr7 = simple_nn()
model_adamax_lr7.compile(optimizer=adamax_lr7, loss='categorical_crossentropy', metrics=['accuracy'])
start = time.time()
model_info_adamax_lr7 = model_adamax_lr7.fit(train_features, train_labels, batch_size=128, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adamax_lr7)
print " Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_adamax_lr7)

 Model took 280.57 seconds to train
Accuracy on test data is: 93.28

# decrease the learning rate
# define model
import keras
adamax_lr8 = keras.optimizers.Adamax(lr=0.01)
model_adamax_lr8 = simple_nn()
model_adamax_lr8.compile(optimizer=adamax_lr8, loss='categorical_crossentropy', metrics=['accuracy'])
start = time.time()
model_info_adamax_lr8 = model_adamax_lr8.fit(train_features, train_labels, batch_size=128, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adamax_lr8)
print " Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_adamax_lr8)

 Model took 183.22 seconds to train
Accuracy on test data is: 95.96

# decrease the learning rate
# define model
import keras
adamax_lr9 = keras.optimizers.Adamax(lr=0.001)
model_adamax_lr9 = simple_nn()
model_adamax_lr9.compile(optimizer=adamax_lr9, loss='categorical_crossentropy', metrics=['accuracy'])
start = time.time()
model_info_adamax_lr9 = model_adamax_lr9.fit(train_features, train_labels, batch_size=128, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adamax_lr9)
print " Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_adamax_lr9)

 Model took 165.88 seconds to train
Accuracy on test data is: 95.82

# learning effect - epochs
plt.figure(figsize=(25, 15))


plt.subplot(3,3,1)
plt.errorbar(range(1,len(model_info_adamax_lr01.history['loss'])+1),model_info_adamax_lr01.history['loss'] ,color = "cyan", linestyle = "-" , marker = "v" , label = "Training")
plt.errorbar(range(1,len(model_info_adamax_lr01.history['val_loss'])+1),model_info_adamax_lr01.history['val_loss'] ,color = "navy", linestyle = "-" , marker = "^" , label = "Testing")

plt.title(r"$\eta = 0.1$" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
# plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
plt.ylabel("MAE" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 1)


plt.subplot(3,3,2) 
plt.errorbar(range(1,len(model_info_adamax_lr2.history['loss'])+1),model_info_adamax_lr2.history['loss'] ,color = "cyan", linestyle = "-" , marker = "v" , label = "Training")
plt.errorbar(range(1,len(model_info_adamax_lr2.history['val_loss'])+1),model_info_adamax_lr2.history['val_loss'] ,color = "navy", linestyle = "-" , marker = "^" , label = "Testing")

plt.title(r"$\eta = 0.01$" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
# plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
# plt.ylabel("Loss" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 1)

plt.subplot(3,3,3) 
plt.errorbar(range(1,len(model_info_adamax_lr3.history['loss'])+1),model_info_adamax_lr3.history['loss'] ,color = "cyan", linestyle = "-" , marker = "v" , label = "Training")
plt.errorbar(range(1,len(model_info_adamax_lr3.history['val_loss'])+1),model_info_adamax_lr3.history['val_loss'] ,color = "navy", linestyle = "-" , marker = "^" , label = "Testing")

plt.title(r"$\eta = 0.001$" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
# plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
# plt.ylabel("Loss" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 1)

plt.subplot(3,3,4) 
plt.errorbar(range(1,len(model_info_adamax_lr4.history['loss'])+1),model_info_adamax_lr4.history['loss'] ,color = "cyan", linestyle = "-" , marker = "v" , label = "Training")
plt.errorbar(range(1,len(model_info_adamax_lr4.history['val_loss'])+1),model_info_adamax_lr4.history['val_loss'] ,color = "navy", linestyle = "-" , marker = "^" , label = "Testing")

# plt.title("Testing" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
# plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
plt.ylabel("MSE" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 1)


plt.subplot(3,3,5) 
plt.errorbar(range(1,len(model_info_adamax_lr5.history['loss'])+1),model_info_adamax_lr5.history['loss'] ,color = "cyan", linestyle = "-" , marker = "v" , label = "Training")
plt.errorbar(range(1,len(model_info_adamax_lr5.history['val_loss'])+1),model_info_adamax_lr5.history['val_loss'] ,color = "navy", linestyle = "-" , marker = "^" , label = "Testing")

# plt.title("Testing" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
# plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
# plt.ylabel("Loss" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 1)



plt.subplot(3,3,6) 
plt.errorbar(range(1,len(model_info_adamax_lr6.history['loss'])+1),model_info_adamax_lr6.history['loss'] ,color = "cyan", linestyle = "-" , marker = "v" , label = "Training")
plt.errorbar(range(1,len(model_info_adamax_lr6.history['val_loss'])+1),model_info_adamax_lr6.history['val_loss'] ,color = "navy", linestyle = "-" , marker = "^" , label = "Testing")

# plt.title("Testing" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
# plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
# plt.ylabel("Loss" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 1)



plt.subplot(3,3,7) 
plt.errorbar(range(1,len(model_info_adamax_lr7.history['loss'])+1),model_info_adamax_lr7.history['loss'] ,color = "cyan", linestyle = "-" , marker = "v" , label = "Training")
plt.errorbar(range(1,len(model_info_adamax_lr7.history['val_loss'])+1),model_info_adamax_lr7.history['val_loss'] ,color = "navy", linestyle = "-" , marker = "^" , label = "Testing")

# plt.title("Testing" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
plt.ylabel("CCE" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 1)


plt.subplot(3,3,8) 
plt.errorbar(range(1,len(model_info_adamax_lr8.history['loss'])+1),model_info_adamax_lr8.history['loss'] ,color = "cyan", linestyle = "-" , marker = "v" , label = "Training")
plt.errorbar(range(1,len(model_info_adamax_lr8.history['val_loss'])+1),model_info_adamax_lr8.history['val_loss'] ,color = "navy", linestyle = "-" , marker = "^" , label = "Testing")

# plt.title("Testing" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
# plt.ylabel("Loss" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 1)



plt.subplot(3,3,9) 
plt.errorbar(range(1,len(model_info_adamax_lr9.history['loss'])+1),model_info_adamax_lr9.history['loss'] ,color = "cyan", linestyle = "-" , marker = "v" , label = "Training")
plt.errorbar(range(1,len(model_info_adamax_lr9.history['val_loss'])+1),model_info_adamax_lr9.history['val_loss'] ,color = "navy", linestyle = "-" , marker = "^" , label = "Testing")

# plt.title("Testing" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
# plt.ylabel("Loss" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 1)

plt.savefig("./LR.png" ,bbox_inches='tight')
plt.show()# Plot with Error bars 

# decrease the learning rate
# define model
import keras
adamax_b1 = keras.optimizers.Adamax(lr=0.001)
model_adamax_b1 = simple_nn()
model_adamax_b1.compile(optimizer=adamax_b1, loss='mae', metrics=['accuracy'])
start = time.time()
model_info_adamax_b1 = model_adamax_b1.fit(train_features, train_labels, batch_size=16, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adamax_b1)
print " Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_adamax_b1)

 Model took 526.42 seconds to train
Accuracy on test data is: 96.13

# decrease the learning rate
# define model
import keras
adamax_b2 = keras.optimizers.Adamax(lr=0.001)
model_adamax_b2 = simple_nn()
model_adamax_b2.compile(optimizer=adamax_b2, loss='mae', metrics=['accuracy'])
start = time.time()
model_info_adamax_b2 = model_adamax_b2.fit(train_features, train_labels, batch_size=64, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adamax_b2)
print " Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_adamax_b2)

 Model took 207.72 seconds to train
Accuracy on test data is: 95.56

# decrease the learning rate
# define model
import keras
adamax_b3 = keras.optimizers.Adamax(lr=0.001)
model_adamax_b3 = simple_nn()
model_adamax_b3.compile(optimizer=adamax_b3, loss='mae', metrics=['accuracy'])
start = time.time()
model_info_adamax_b3 = model_adamax_b3.fit(train_features, train_labels, batch_size=256, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adamax_b3)
print " Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_adamax_b3)

 Model took 135.38 seconds to train
Accuracy on test data is: 95.17

# decrease the learning rate
# define model
import keras
adamax_b4 = keras.optimizers.Adamax(lr=0.001)
model_adamax_b4 = simple_nn()
model_adamax_b4.compile(optimizer=adamax_b4, loss='mse', metrics=['accuracy'])
start = time.time()
model_info_adamax_b4 = model_adamax_b4.fit(train_features, train_labels, batch_size=16, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adamax_b4)
print " Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_adamax_b4)

 Model took 518.54 seconds to train
Accuracy on test data is: 96.21

# decrease the learning rate
# define model
import keras
adamax_b5 = keras.optimizers.Adamax(lr=0.001)
model_adamax_b5 = simple_nn()
model_adamax_b5.compile(optimizer=adamax_b5, loss='mse', metrics=['accuracy'])
start = time.time()
model_info_adamax_b5 = model_adamax_b5.fit(train_features, train_labels, batch_size=64, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adamax_b5)
print " Model took %0.2f seconds to train"%(end - start)
# compute test accuracy# decrease the learning rate
# define model
import keras
adamax_b5 = keras.optimizers.Adamax(lr=0.001)
model_adamax_b5 = simple_nn()
model_adamax_b5.compile(optimizer=adamax_b5, loss='mse', metrics=['accuracy'])
start = time.time()
model_info_adamax_b5 = model_adamax_b5.fit(train_features, train_labels, batch_size=64, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adamax_b5)
print " Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_adamax_b5)
print "Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_adamax_b5)

 Model took 201.08 seconds to train
Accuracy on test data is: 95.77

# decrease the learning rate
# define model
import keras
adamax_b6 = keras.optimizers.Adamax(lr=0.001)
model_adamax_b6 = simple_nn()
model_adamax_b6.compile(optimizer=adamax_b6, loss='mse', metrics=['accuracy'])
start = time.time()
model_info_adamax_b6 = model_adamax_b6.fit(train_features, train_labels, batch_size=256, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adamax_b6)
print " Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_adamax_b6)

 Model took 122.64 seconds to train
Accuracy on test data is: 95.55

# decrease the learning rate
# define model
import keras
adamax_b7 = keras.optimizers.Adamax(lr=0.001)
model_adamax_b7 = simple_nn()
model_adamax_b7.compile(optimizer=adamax_b7, loss='categorical_crossentropy', metrics=['accuracy'])
start = time.time()
model_info_adamax_b7 = model_adamax_b7.fit(train_features, train_labels, batch_size=16, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adamax_b7)
print " Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_adamax_b7)

 Model took 1142.46 seconds to train
Accuracy on test data is: 96.42

# decrease the learning rate
# define model
import keras
adamax_b8 = keras.optimizers.Adamax(lr=0.001)
model_adamax_b8 = simple_nn()
model_adamax_b8.compile(optimizer=adamax_b8, loss='categorical_crossentropy', metrics=['accuracy'])
start = time.time()
model_info_adamax_b8 = model_adamax_b8.fit(train_features, train_labels, batch_size=64, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adamax_b8)
print " Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_adamax_b8)

 Model took 611.61 seconds to train
Accuracy on test data is: 95.84

# decrease the learning rate
# define model
import keras
adamax_b9 = keras.optimizers.Adamax(lr=0.001)
model_adamax_b9 = simple_nn()
model_adamax_b9.compile(optimizer=adamax_b9, loss='categorical_crossentropy', metrics=['accuracy'])
start = time.time()
model_info_adamax_b9 = model_adamax_b9.fit(train_features, train_labels, batch_size=256, \
                       nb_epoch=100, verbose=0, validation_data=(test_features , test_labels))
end = time.time()
# plot model history
plot_model_history(model_info_adamax_b9)
print " Model took %0.2f seconds to train"%(end - start)
# compute test accuracy
print "Accuracy on test data is: %0.2f"%accuracy(test_features, test_labels, model_adamax_b9)

 Model took 332.20 seconds to train
Accuracy on test data is: 95.86

# batch size vs epochs
plt.figure(figsize=(25, 15))


plt.subplot(3,3,1)
plt.errorbar(range(1,len(model_info_adamax_b1.history['loss'])+1),model_info_adamax_b1.history['loss'] ,color = "cyan", linestyle = "-" , marker = "v" , label = "Training")
plt.errorbar(range(1,len(model_info_adamax_b1.history['val_loss'])+1),model_info_adamax_b1.history['val_loss'] ,color = "navy", linestyle = "-" , marker = "^" , label = "Testing")

plt.title("$Batch = 16$" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
# plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
plt.ylabel("MAE" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 1)


plt.subplot(3,3,2) 
plt.errorbar(range(1,len(model_info_adamax_b2.history['loss'])+1),model_info_adamax_b2.history['loss'] ,color = "cyan", linestyle = "-" , marker = "v" , label = "Training")
plt.errorbar(range(1,len(model_info_adamax_b2.history['val_loss'])+1),model_info_adamax_b2.history['val_loss'] ,color = "navy", linestyle = "-" , marker = "^" , label = "Testing")

plt.title("$Batch = 64$" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
# plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
# plt.ylabel("Loss" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 1)


plt.subplot(3,3,3) 
plt.errorbar(range(1,len(model_info_adamax_b3.history['loss'])+1),model_info_adamax_b3.history['loss'] ,color = "cyan", linestyle = "-" , marker = "v" , label = "Training")
plt.errorbar(range(1,len(model_info_adamax_b3.history['val_loss'])+1),model_info_adamax_b3.history['val_loss'] ,color = "navy", linestyle = "-" , marker = "^" , label = "Testing")

plt.title("$Batch = 256$" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
# plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
# plt.ylabel("Loss" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 1)


plt.subplot(3,3,4) 
plt.errorbar(range(1,len(model_info_adamax_b4.history['loss'])+1),model_info_adamax_b4.history['loss'] ,color = "cyan", linestyle = "-" , marker = "v" , label = "Training")
plt.errorbar(range(1,len(model_info_adamax_b4.history['val_loss'])+1),model_info_adamax_b4.history['val_loss'] ,color = "navy", linestyle = "-" , marker = "^" , label = "Testing")

# plt.title("Testing" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
# plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
plt.ylabel("MSE" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 1)


plt.subplot(3,3,5) 
plt.errorbar(range(1,len(model_info_adamax_b5.history['loss'])+1),model_info_adamax_b5.history['loss'] ,color = "cyan", linestyle = "-" , marker = "v" , label = "Training")
plt.errorbar(range(1,len(model_info_adamax_b5.history['val_loss'])+1),model_info_adamax_b5.history['val_loss'] ,color = "navy", linestyle = "-" , marker = "^" , label = "Testing")

# plt.title("Testing" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
# plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
# plt.ylabel("Loss" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 1)



plt.subplot(3,3,6) 
plt.errorbar(range(1,len(model_info_adamax_b6.history['loss'])+1),model_info_adamax_b6.history['loss'] ,color = "cyan", linestyle = "-" , marker = "v" , label = "Training")
plt.errorbar(range(1,len(model_info_adamax_b6.history['val_loss'])+1),model_info_adamax_b6.history['val_loss'] ,color = "navy", linestyle = "-" , marker = "^" , label = "Testing")

# plt.title("Testing" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
# plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
# plt.ylabel("Loss" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 1)



plt.subplot(3,3,7) 
plt.errorbar(range(1,len(model_info_adamax_b7.history['loss'])+1),model_info_adamax_b7.history['loss'] ,color = "cyan", linestyle = "-" , marker = "v" , label = "Training")
plt.errorbar(range(1,len(model_info_adamax_b7.history['val_loss'])+1),model_info_adamax_b7.history['val_loss'] ,color = "navy", linestyle = "-" , marker = "^" , label = "Testing")

# plt.title("Testing" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
plt.ylabel("CCE" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 1)


plt.subplot(3,3,8) 
plt.errorbar(range(1,len(model_info_adamax_b8.history['loss'])+1),model_info_adamax_b8.history['loss'] ,color = "cyan", linestyle = "-" , marker = "v" , label = "Training")
plt.errorbar(range(1,len(model_info_adamax_b8.history['val_loss'])+1),model_info_adamax_b8.history['val_loss'] ,color = "navy", linestyle = "-" , marker = "^" , label = "Testing")

# plt.title("Testing" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
# plt.ylabel("Loss" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 1)



plt.subplot(3,3,9) 
plt.errorbar(range(1,len(model_info_adamax_b9.history['loss'])+1),model_info_adamax_b9.history['loss'] ,color = "cyan", linestyle = "-" , marker = "v" , label = "Training")
plt.errorbar(range(1,len(model_info_adamax_b9.history['val_loss'])+1),model_info_adamax_b9.history['val_loss'] ,color = "navy", linestyle = "-" , marker = "^" , label = "Testing")

# plt.title("Testing" ,  fontweight = "bold" , fontsize=30)

mpl.rcParams['lines.markersize'] = 1
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =3
plt.xlabel("Number of Epochs", fontweight = "bold" , fontsize=30)
# plt.ylabel("Loss" , fontweight = "bold" , fontsize=30)
plt.tick_params(axis='both', which='major', labelsize=20)
# plt.ylim([0.75,1])
plt.xlim([0,101])
plt.legend( prop={'size':18} , loc = 1)

plt.savefig("./Batch.png" ,bbox_inches='tight')
plt.show()# Plot with Error bars 

Imbalanced Classification

2019-06-01T00:00:00+00:00

Imbalanced classification (or classification problems with low prevalence (low number of instances in one of the classes) can be challenging. In this post, I have discussed how we can model a problem with prevalence of 0.09% for positive class using gradient boosting and generalized linear model.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib as mpl
from scipy import interp
from sklearn.preprocessing import scale
from sklearn.metrics import precision_score, recall_score, roc_auc_score, accuracy_score, roc_curve, auc, confusion_matrix
from sklearn.model_selection import cross_val_score, KFold, StratifiedKFold, train_test_split
from xgboost import XGBClassifier
import itertools
import glmnet
import xgboost as xgb
import shap
import seaborn as sns
sns.set_style("ticks")
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =7
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:95% !important; }</style>"))
import warnings 
warnings.filterwarnings("ignore")
%matplotlib inline

# Unused Libraries here
# import keras
# from bayes_opt import BayesianOptimization
# from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
# from fancyimpute import SoftImpute, IterativeImputer

Functions

#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
# Function: _x_y_maker

def _x_y_maker(df, target, to_drop):
    """
    A function to receive the dataframe and make the X, and Y matrices:
    ----------
    parameters
    
    input: dataframe df
           string target (class label)
           list[string] to_drop (list of columns need to be dropped)
           
    output: dataframe Feature matrix (df_X)
            Target vector (Y)
    """    
    Targets = df[target]
    to_drop.append(target)
    Features = df.drop(to_drop, axis = 1)
    Scaled_Features = scale(Features.values)
    df_X = pd.DataFrame(data = Scaled_Features, columns = Features.columns.tolist())
    Y = Targets.values
    
    return df_X, Y

#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
# Function: _plot_roc_nfolds_xgboost

def _plot_roc_nfolds_xgboost(df_X, Y,
                                     n_folds = 10,
                                     n_estimators = 100,
                                     learning_rate = 0.05,
                                     max_depth = 3,
                                     min_child_weight = 5.0,
                                     gamma = 0.5,
                                     reg_alpha = 0.0,
                                     reg_lambda = 1.0,
                                     subsample = 0.9,
                                     objective = "binary:logistic",
                                     scale_pos_weight = 1.0,
                                     shuffle = True,
                                     random_state = 1367
                            ):
    """
    a function to plot k-fold cv ROC using xgboost
    input parameters:
                     df_X : features : pandas dataframe (numpy array will be built inside the function)
                     Y : targets
                     n_folds : number of cv folds (default = 10)
                     n_estimators = number of trees (default = 100)
                     learning_rate : step size of xgboost (default = 0.05)
                     max_depth : maximum tree depth for xgboost (default = 3)
                     min_child_weight : (default = 5.0)
                     gamma : (default = 0.5)
                     reg_alpha : lasso penalty (L1) (default = 0.0)
                     reg_lambda : ridge penalty (L2) (default = 1.0)
                     subsample : subsample fraction (default = 0.9)
                     objective : objective function for ML (default = "binary:logistic" for classification)
                     scale_pos_weight : (default = 1.0)
                     shuffle : shuffle flag for cv (default = True)
                     random_state : (default = 1367)
    
    """

    # Defining the data
    X = scale(df_X.values)
    y = Y
    n_samples, n_features = X.shape

    # Run classifier with cross-validation and plot ROC curves
    cv = StratifiedKFold(n_splits = n_folds , shuffle = shuffle , random_state = random_state)
    classifier = XGBClassifier(learning_rate = learning_rate,
                               n_estimators = n_estimators,
                               max_depth = max_depth,
                               min_child_weight = min_child_weight,
                               gamma = gamma,
                               reg_alpha = reg_alpha,
                               reg_lambda = reg_lambda,
                               subsample = subsample,
                               objective = objective,
                               nthread = 4,
                               scale_pos_weight = 1.,
                               base_score = np.mean(y),
                               seed = random_state,
                               random_state = random_state)
    

    tprs = []
    aucs = []
    mean_fpr = np.linspace(0, 1, 100)
    plt.figure(figsize=(18 , 13))
    i = 0
    for train, test in cv.split(X, y):
        probas_ = classifier.fit(X[train], y[train]).predict_proba(X[test])
        # Compute ROC curve and area the curve
        fpr, tpr, thresholds = roc_curve(y[test], probas_[:, 1])
        tprs.append(interp(mean_fpr, fpr, tpr))
        tprs[-1][0] = 0.0
        roc_auc = auc(fpr, tpr)
        aucs.append(roc_auc)
        plt.plot(fpr, tpr, lw=3, alpha=0.5, 
                 label="ROC Fold %d (AUC = %0.2f)" % (i+1, roc_auc))

        i += 1
    plt.plot([0, 1], [0, 1], linestyle="--", lw=3, color="k",
             label="Luck", alpha=.8)

    mean_tpr = np.mean(tprs, axis=0)
    mean_tpr[-1] = 1.0
    mean_auc = auc(mean_fpr, mean_tpr)
    std_auc = np.std(aucs)
    plt.plot(mean_fpr, mean_tpr, color="navy",
             label=r"Mean ROC (AUC = %0.2f $\pm$ %0.2f)" % (mean_auc, std_auc),
             lw=4)

    std_tpr = np.std(tprs, axis=0)
    tprs_upper = np.minimum(mean_tpr + std_tpr, 1)
    tprs_lower = np.maximum(mean_tpr - std_tpr, 0)
    plt.fill_between(mean_fpr, tprs_lower, tprs_upper, color="grey", alpha=.4,
                     label=r"$\pm$ 1 Standard Deviation")

    plt.xlim([-0.05, 1.05])
    plt.ylim([-0.05, 1.05])
    plt.xlabel("False Positive Rate" ,  fontweight = "bold" , fontsize=30)
    plt.ylabel("True Positive Rate",fontweight = "bold" , fontsize=30)
    plt.tick_params(axis="both", which="major", labelsize=20)
    plt.legend( prop={"size":20} , loc = 4)
    # plt.savefig("./roc_xgboost.pdf" ,bbox_inches="tight")
    plt.show()

#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
# Function: _permute

def _permute(df, random_state):
    """
    a funtion to permute the rows of features and add them
    to the dataframe as noisy features to explore stability
    input parameters:
                    df : pandas dataframe
                    random_state : random seed for noisy features
    """
    normal_df = df.copy().reset_index(drop = True)
    noisy_df = df.copy()
    noisy_df.rename(columns = {col : "noisy_" + col for col in noisy_df.columns}, inplace = True)
    np.random.seed(seed = random_state)
    noisy_df = noisy_df.reindex(np.random.permutation(noisy_df.index))
    merged_df = pd.concat([normal_df, noisy_df.reset_index(drop = True)] , axis = 1)
    
    return merged_df
    
#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
# Function: _my_feature_selector

def _my_feature_selector(X, Y, n_iter = 1,
                                  num_boost_round = 1000,
                                  nfold = 10,
                                  stratified = True,
                                  metrics = ("auc"),
                                  early_stopping_rounds = 50,
                                  seed = 1367,
                                  shuffle = True,
                                  show_stdv = False,
                                  params = None,
                                  importance_type = "total_gain",
                                  callbacks = False,
                                  verbose_eval = False):
    """
    a function to run xgboost kfolds cv and train the model based on the best boosting round of each iteration.
    for different number of iterations. at each iteration noisy features are added as well. at each iteration
    internal and external cv calculated.
    NOTE: it is recommended to use ad-hoc parameters to make sure the model is under-fitted for the sake of feature selection
    input parameters:
                    X: features (pandas dataframe or numpy array)
                    Y: targets (1D array or list)
                    n_iter: total number of iterations (default = 1)
                    num_boost_rounds: max number of boosting rounds, (default = 1000)
                    stratified: stratificaiton of the targets (default = True)
                    metrics: classification/regression metrics (default = ("auc))
                    early_stopping_rounds: the criteria for stopping if the test metric is not improved (default = 20)
                    seed: random seed (default = 1367)
                    shuffle: shuffling the data (default = True)
                    show_stdv = showing standard deviation of cv results (default = False)
                    params = set of parameters for xgboost cv
                            (default_params = {
                                               "eval_metric" : "auc",
                                               "tree_method": "hist",
                                               "objective" : "binary:logistic",
                                               "learning_rate" : 0.05,
                                               "max_depth": 2,
                                               "min_child_weight": 1,
                                               "gamma" : 0.0,
                                               "reg_alpha" : 0.0,
                                               "reg_lambda" : 1.0,
                                               "subsample" : 0.9,
                                               "max_delta_step": 1,
                                               "silent" : 1,
                                               "nthread" : 4,
                                               "scale_pos_weight" : 1
                                               }
                            )
                    importance_type = importance type of xgboost as string (default = "total_gain")
                                      the other options will be "weight", "gain", "cover", and "total_cover"
                    callbacks = printing callbacks for xgboost cv
                                (defaults = False, if True: [xgb.callback.print_evaluation(show_stdv = show_stdv),
                                                             xgb.callback.early_stop(early_stopping_rounds)])
                    verbose_eval : a flag to show the result during train on train/test sets (default = False)
    outputs:
            outputs_dict: a dict contains the fitted models, internal and external cv results for train/test sets
            df_selected: a dataframe contains feature names with their frequency of being selected at each run of folds
    """
    # callback flag
    if(callbacks == True):
        callbacks = [xgb.callback.print_evaluation(show_stdv = show_stdv),
                     xgb.callback.early_stop(early_stopping_rounds)]
    else:
        callbacks = None
    
    # params
    default_params = {
                      "eval_metric" : "auc",
                      "tree_method": "hist",
                      "objective" : "binary:logistic",
                      "learning_rate" : 0.05,
                      "max_depth": 2,
                      "min_child_weight": 1,
                      "gamma" : 0.0,
                      "reg_alpha" : 0.0,
                      "reg_lambda" : 1.0,
                      "subsample" : 0.9,
                      "max_delta_step": 1,
                      "silent" : 1,
                      "nthread" : 4,
                      "scale_pos_weight" : 1,
                      "base_score" : np.mean(Y)
                     }
    # updating the default parameters with the input
    if params is not None:
        for key, val in params.items():
            default_params[key] = val
            
    # total results list
    int_cv_train = []
    int_cv_test = []
    ext_cv_train = []
    ext_cv_test = []
    bst_list = []
    
    # main loop
    for iteration in range(n_iter):
        
        # results list at iteration
        int_cv_train2 = []
        int_cv_test2 = []
        ext_cv_train2 = []
        ext_cv_test2 = []
        
        # update random state 
        random_state = seed * iteration 
    
        # adding noise to data
        X_permuted = _permute(df = X, random_state = random_state)
        cols = X_permuted.columns.tolist()
        Xval = X_permuted.values

        # building DMatrix for training/testing + kfolds cv
        cv = StratifiedKFold(n_splits = nfold , shuffle = shuffle , random_state = random_state)

        for train_index, test_index in cv.split(Xval, Y):
            Xval_train = Xval[train_index]
            Xval_test = Xval[test_index]
            Y_train = Y[train_index]
            Y_test = Y[test_index]

            X_train = pd.DataFrame(data = Xval_train, columns = cols)
            X_test = pd.DataFrame(data = Xval_test, columns = cols)
            
            dtrain = xgb.DMatrix(data = X_train, label = Y_train)
            dtest = xgb.DMatrix(data = X_test, label = Y_test)

            # watchlist during final training
            watchlist = [(dtrain,"train"), (dtest,"eval")]
            
            # a dict to store training results
            evals_result = {}
            
            # xgb cv
            cv_results = xgb.cv(params = default_params,
                                dtrain = dtrain,
                                num_boost_round = num_boost_round,
                                nfold = nfold,
                                stratified = stratified,
                                metrics = metrics,
                                early_stopping_rounds = early_stopping_rounds,
                                seed = random_state,
                                verbose_eval = verbose_eval,
                                shuffle = shuffle,
                                callbacks = callbacks)
            int_cv_train.append(cv_results.iloc[-1][0])
            int_cv_test.append(cv_results.iloc[-1][2])
            int_cv_train2.append(cv_results.iloc[-1][0])
            int_cv_test2.append(cv_results.iloc[-1][2])
            
            # xgb train
            bst = xgb.train(params = default_params,
                            dtrain = dtrain,
                            num_boost_round = len(cv_results) - 1,
                            evals = watchlist,
                            evals_result = evals_result,
                            verbose_eval = verbose_eval)
            # appending outputs
            bst_list.append(bst)
            ext_cv_train.append(evals_result["train"]["auc"][-1])
            ext_cv_test.append(evals_result["eval"]["auc"][-1])
            ext_cv_train2.append(evals_result["train"]["auc"][-1])
            ext_cv_test2.append(evals_result["eval"]["auc"][-1])
        
        print(F"-*-*-*-*-*-*-*-*- Iteration {iteration + 1} -*-*-*-*-*-*-*-*")
        print(F"-*-*- Internal-{nfold} Folds-CV-Train = {np.mean(int_cv_train2):.3} +/- {np.std(int_cv_train2):.3} -*-*- Internal-{nfold} Folds-CV-Test = {np.mean(int_cv_test2):.3} +/- {np.std(int_cv_test2):.3} -*-*")
        print(F"-*-*- External-{nfold} Folds-CV-Train = {np.mean(ext_cv_train2):.3} +/- {np.std(ext_cv_train2):.3} -*-*- External-{nfold} Folds-CV-Test = {np.mean(ext_cv_test2):.3} +/- {np.std(ext_cv_test2):.3} -*-*")
    
    # putting together the outputs in one dict
    outputs = {}
    outputs["bst"] = bst_list
    outputs["int_cv_train"] = int_cv_train
    outputs["int_cv_test"] = int_cv_test
    outputs["ext_cv_train"] = ext_cv_train
    outputs["ext_cv_test"] = ext_cv_test
    

    pruned_features = []
    for bst_ in outputs["bst"]:
        features_gain = bst_.get_score(importance_type = importance_type)
        for key, value in features_gain.items():
            pruned_features.append(key)

    unique_elements, counts_elements = np.unique(pruned_features, return_counts = True)
    counts_elements = [float(i) for i in list(counts_elements)]
    df_features = pd.DataFrame(data = {"columns" : list(unique_elements) , "count" : counts_elements})
    df_features.sort_values(by = ["count"], ascending = False, inplace = True)

    # plotting
    plt.figure()
    df_features.sort_values(by = ["count"]).plot.barh(x = "columns", y = "count", color = "pink" , figsize = (8,8))
    plt.show()

    plt.figure(figsize = (16,12))

    plt.subplot(2,2,1)
    plt.title(F"Internal {nfold} CV {metrics.upper()} - Train", fontsize = 20)
    plt.hist(outputs["int_cv_train"], bins = 20, color = "lightgreen")

    plt.subplot(2,2,2)
    plt.title(F"Internal {nfold} CV {metrics.upper()} - Test", fontsize = 20)
    plt.hist(outputs["int_cv_test"], bins = 20, color = "lightgreen")

    plt.subplot(2,2,3)
    plt.title(F"External {nfold} CV {metrics.upper()} - Train", fontsize = 20)
    plt.hist(outputs["ext_cv_train"], bins = 20, color = "lightblue")

    plt.subplot(2,2,4)
    plt.title(F"External {nfold} CV {metrics.upper()} - Test", fontsize = 20)
    plt.hist(outputs["ext_cv_test"], bins = 20, color = "lightblue")

    plt.show()
    
    return outputs, df_features
#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
# Function: Confidence Interval calculator for 95% significance

def _ConfidenceInterval(score, n):
    """
    confidence interval for 95% significance level
    input: score such as accuracy, auc
           the size of test set n
    """
    
    CI_range = 1.96 * np.sqrt( (score * (1 - score)) / n)
    CI_pos = score + CI_range
    CI_neg = score - CI_range
    if(CI_neg < 0):
        CI_neg = 0
        
    return CI_neg, CI_pos

#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
# Function: _bst_shap

def _bst_shap(X, Y, num_boost_round = 1000,
                                    nfold = 10,
                                    stratified = True,
                                    metrics = ("auc"),
                                    early_stopping_rounds = 50,
                                    seed = 1367,
                                    shuffle = True,
                                    show_stdv = False,
                                    params = None,
                                    callbacks = False):
    """
    a function to run xgboost kfolds cv with the params and train the model based on best boosting rounds
    input parameters:
                    X: features
                    Y: targets
                    num_boost_rounds: max number of boosting rounds, (default = 1000)
                    stratified: stratificaiton of the targets (default = True)
                    metrics: classification/regression metrics (default = ("auc))
                    early_stopping_rounds: the criteria for stopping if the test metric is not improved (default = 20)
                    seed: random seed (default = 1367)
                    shuffle: shuffling the data (default = True)
                    show_stdv = showing standard deviation of cv results (default = False)
                    params = set of parameters for xgboost cv ( default_params = {
                                                                                  "eval_metric" : "auc",
                                                                                  "tree_method": "hist",
                                                                                  "objective" : "binary:logistic",
                                                                                  "learning_rate" : 0.05,
                                                                                  "max_depth": 2,
                                                                                  "min_child_weight": 5,
                                                                                  "gamma" : 0.0,
                                                                                  "reg_alpha" : 0.0,
                                                                                  "reg_lambda" : 1.0,
                                                                                  "subsample" : 0.9,
                                                                                  "silent" : 1,
                                                                                  "nthread" : 4,
                                                                                  "scale_pos_weight" : 1
                                                                                 }
                                                                    )
                   callbacks = printing callbacks for xgboost cv (defaults: [xgb.callback.print_evaluation(show_stdv = show_stdv),
                                                                             xgb.callback.early_stop(early_stopping_rounds)])
    outputs:
            bst: the trained model based on optimum number of boosting rounds
            cv_results: a dataframe contains the cv-results (train/test + std of train/test) for metric
    """
    # callback flag
    if(callbacks == True):
        callbacks = [xgb.callback.print_evaluation(show_stdv = show_stdv),
                     xgb.callback.early_stop(early_stopping_rounds)]
    else:
        callbacks = None
    
    # params
    default_params = {
                      "eval_metric" : "auc",
                      "tree_method": "hist",
                      "objective" : "binary:logistic",
                      "learning_rate" : 0.05,
                      "max_depth": 2,
                      "min_child_weight": 1,
                      "gamma" : 0.0,
                      "reg_alpha" : 0.0,
                      "reg_lambda" : 1.0,
                      "subsample" : 0.9,
                      "max_delta_step": 1,
                      "silent" : 1,
                      "nthread" : 4
                     }
    # updating the default parameters with the input
    if params is not None:
        for key, val in params.items():
            default_params[key] = val

    # building DMatrix for training
    dtrain = xgb.DMatrix(data = X, label = Y)
    print("-*-*-*-*-*-*-* CV Started *-*-*-*-*-*-*-")
    cv_results = xgb.cv(params = default_params,
                        dtrain = dtrain,
                        num_boost_round = num_boost_round,
                        nfold = nfold,
                        stratified = stratified,
                        metrics = metrics,
                        early_stopping_rounds = early_stopping_rounds,
                        seed = seed,
                        shuffle = shuffle,
                        callbacks = callbacks)
    print(F"*-*- Boosting Round = {len(cv_results) - 1} -*-*- Train = {cv_results.iloc[-1][0]:.3} -*-*- Test = {cv_results.iloc[-1][2]:.3} -*-*")
    print("-*-*-*-*-*-*-* Train Started *-*-*-*-*-*-*-")
    bst = xgb.train(params = default_params,
                    dtrain = dtrain,
                    num_boost_round = len(cv_results) - 1,
                   )
    
    return bst, cv_results

#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
# Function: _plot_xgboost_cv_score

def _plot_xgboost_cv_score(cv_results):
    """
    a function to plot train/test cv score for 
    """

    import matplotlib as mpl
    mpl.rcParams['axes.linewidth'] = 3 
    mpl.rcParams['lines.linewidth'] = 2
    plt.figure(figsize=(10,8))
    plt.errorbar(range(cv_results.shape[0]), cv_results["train-auc-mean"],
                yerr=cv_results["train-auc-std"], fmt = "--", ecolor="lightgreen", c = "navy", label = "Train (9-Folds)")

    plt.errorbar(range(cv_results.shape[0]), cv_results["test-auc-mean"],
                yerr=cv_results["test-auc-std"], fmt = "--", ecolor="lightblue", c = "red", label = "Test (1-Fold)")


    plt.xlabel("# of Boosting Rounds" ,  fontweight = "bold" , fontsize=30)
    plt.ylabel("AUC",fontweight = "bold" , fontsize=30)
    plt.tick_params(axis='both', which='major', labelsize=20)
    plt.legend(loc = 4, prop={'size': 20})
    plt.show()
    
#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
# Function: _plot_importance

def _plot_importance(bst, importance_type = "total_gain", color = "pink", figsize = (10,10)):
    """
    a function to plot feature importance in xgboost
    """
    from xgboost import plot_importance
    from pylab import rcParams
    rcParams['figure.figsize'] = figsize
    plot_importance(bst, importance_type = importance_type, color = color, xlabel = importance_type)
    plt.show()

#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
# Function: _plot_confusion_matrix
def _plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Greens):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title, fontsize = 20)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes)
    plt.yticks(tick_marks, classes)

    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.ylabel('True label', fontsize = 20)
    plt.xlabel('Predicted label', fontsize = 20)
    plt.tick_params(axis='both', which='major', labelsize=20)
    plt.tight_layout()

#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
# Function: _clf_xgboost

def _clf_xgboost(df_X, Y, test_size, n_folds):
    X = df_X.values
    clf = XGBClassifier(learning_rate = 0.05,
                                n_estimators = 100,
                                max_depth = 3,
                                min_child_weight = 5.,
                               # max_delta_step= 1,
                                gamma = 0.5,
                                reg_alpha = 0.0,
                                reg_lambda = 1.0,
                                subsample = 0.9,
                                colsample_bytree = 0.9,
                                objective = "binary:logistic",
                                nthread = 4,
                                #scale_pos_weight = (len(Y) - sum(Y)) / sum(Y),
                                base_score = np.mean(Y),
                                seed = 1367,
                                random_state = 1367)
    
    X_train, X_test, y_train, y_test = train_test_split(df_X, Y, test_size = test_size, shuffle = True, random_state = 1367, stratify = Y)
    clf.fit(X_train, y_train, eval_metric="auc", early_stopping_rounds = 50, verbose = False, eval_set=[(X_test, y_test)])
    accuracy_score = clf.score(X_test, y_test)
    cv_roc_score = cross_val_score(clf, X, Y, cv = n_folds, scoring = "roc_auc")
    y_pred_prob = clf.predict_proba(X_test)
    roc_score = roc_auc_score(y_test, y_pred_prob[:,1])
    fpr, tpr, thresholds = roc_curve(y_test, y_pred_prob[:,1])
    optimal_idx = np.argmax(np.abs(tpr - fpr))
    optimal_threshold = thresholds[optimal_idx]
    
    return accuracy_score, roc_score, cv_roc_score, y_pred_prob, X_test, y_test, optimal_threshold

#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
# Function: _xgboost_auc_hist
def _xgboost_auc_hist(df_X, Y, n_iterations = 100, alpha = 0.95):
    """
    plot histogram of ROC AUC for N stratified classification for 95% significance level
    """

    n_size = int(len(Y) * 0.30)

    # Main Loop
    scores = list()
    for i in range(n_iterations):
        # split the data into train/test sets
        X_train, X_test, y_train, y_test = train_test_split(df_X, Y, test_size = n_size, shuffle = True, random_state = 1367 * i , stratify = Y)
        # fit the xgboost classifier
        clf = XGBClassifier(learning_rate = 0.05,
                                    n_estimators = 100,
                                    max_depth = 3,
                                    min_child_weight = 5.,
                                   # max_delta_step= 1,
                                    gamma = 0.5,
                                    reg_alpha = 0.0,
                                    reg_lambda = 1.0,
                                    subsample = 0.9,
                                    colsample_bytree = 0.9,
                                    objective = "binary:logistic",
                                    nthread = 4,
                                    #scale_pos_weight = (len(Y) - sum(Y)) / sum(Y),
                                    base_score = np.mean(Y),
                                    seed = 1367 * i ,
                                    random_state = 1367 * i)
        clf.fit(X_train, y_train, eval_metric = "auc", early_stopping_rounds = 50, verbose = False, eval_set = [(X_test, y_test)])
        # evaluate model
        predictions = clf.predict_proba(X_test)[:,1]
        pred_auc = roc_auc_score(y_test, predictions)
        scores.append(pred_auc)

    # Ploting the scores
    plt.figure(figsize=(10,5))
    plt.hist(scores, color = "pink")
    plt.xlabel("ROC AUC", fontsize = 30)
    plt.tick_params(axis = "both", which = "major", labelsize = 15)
    plt.show()

    # confidence intervals
    alpha = 0.95
    p = ((1.0 - alpha)/2.0) * 100
    lower = max(0.0, np.percentile(scores, p))
    p = (alpha + ((1.0 - alpha)/2.0)) * 100
    upper = min(1.0, np.percentile(scores, p))
    print(F"{alpha*100}% Significance Level - ROC AUC Confidence Interval= [{lower:.3f} , {upper:.3f}]")
    
#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
# Function: _glmnet

def _glmnet(X, Y, alpha = 0.1, n_splits = 10, scoring = "roc_auc"):
    """
    a function for standard glmnet
    input parameters: pandas dataframe X
                      class labels Y
                      alpha = 0.1
                      n_splits = 10
                      scoring = "roc_auc"
    output: glmnet model 

    """  
    model = glmnet.LogitNet(alpha = alpha,
                     n_lambda = 100,
                     n_splits = n_splits,
                     cut_point = 1.0,
                     scoring = scoring,
                     n_jobs = -1,
                     random_state = 1367)
    model.fit(X, Y)
    
    return model

#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
# Function: _plot_glmnet_cv_score

def _plot_glmnet_cv_score(model):
    """
    a function to plot cv scores vs lambda
    parameters:
               model : a fitted glmnet object
    """
    mpl.rcParams['axes.linewidth'] = 3 
    mpl.rcParams['lines.linewidth'] =2
    plt.figure(figsize=(10,6))
    plt.errorbar(-np.log(model.lambda_path_), model.cv_mean_score_, yerr=model.cv_standard_error_ , c = "r", ecolor="k", marker = "o" )
    plt.vlines(-np.log(model.lambda_best_), ymin = min(model.cv_mean_score_) - 0.05 , ymax = max(model.cv_mean_score_) + 0.05, lw = 3, linestyles = "--", colors = "b" ,label = "best $\lambda$")
    plt.vlines(-np.log(model.lambda_max_), ymin = min(model.cv_mean_score_) - 0.05 , ymax = max(model.cv_mean_score_) + 0.05, lw = 3, linestyles = "--", colors = "c" ,label = "max $\lambda$")
    plt.tick_params(axis='both', which='major', labelsize = 12)
    plt.grid(True)
    plt.ylim([min(model.cv_mean_score_) - 0.05, max(model.cv_mean_score_) + 0.05])
    plt.legend(loc = 4, prop={'size': 20})
    plt.xlabel("$-Log(\lambda)$" , fontsize = 20)
    plt.ylabel(F"Mean {model.n_splits} Folds CV {(model.scoring).upper()}", fontsize = 20)
    plt.title(F"Best $\lambda$ = {model.lambda_best_[0]:.2} with {len(np.nonzero(  model.coef_)[1])} Features" , fontsize = 20)
    plt.show()

#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
# Function: _plot_glmnet_coeff_path

def _plot_glmnet_coeff_path(model, df):
    """
    a function to plot coefficients vs lambda
    parameters:
               model : a fitted glmnet object
               df: in case that the input is the pandas dataframe,
                   the column names of the coeff. will appear as a legend
    """
    mpl.rcParams['axes.linewidth'] = 3 
    mpl.rcParams['lines.linewidth'] =2
    plt.figure(figsize=(10,6))
    if not df.empty:
        for i in list(np.nonzero(np.reshape(model.coef_, (1,-1)))[1]):
            plt.plot(-np.log(model.lambda_path_) ,(model.coef_path_.reshape(-1,model.coef_path_.shape[-1]))[i,:], label = df.columns.values.tolist()[i]);
        plt.legend(loc= "right", bbox_to_anchor=(1.2 , .5), ncol=1, fancybox=True, shadow=True)

    else:
        for i in list(np.nonzero(np.reshape(model.coef_, (1,-1)))[1]):
            plt.plot(-np.log(model.lambda_path_) ,(model.coef_path_.reshape(-1, model.coef_path_.shape[-1]))[i,:]);
            
    plt.tick_params(axis='both', which='major', labelsize = 12)    
    plt.ylabel("Coefficients", fontsize = 20)
    plt.xlabel("-$Log(\lambda)$", fontsize = 20)
    plt.title(F"Best $\lambda$ = {model.lambda_best_[0]:.2} with {len(np.nonzero(model.coef_)[1])} Features" , fontsize = 20)
    plt.grid(True)
    plt.show()
    
#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
# Function: _df_glmnet_coeff_path

def _df_glmnet_coeff_path(model, df):
    """
    a function to build a dataframe for nonzero coeff.
    parameters:
               model : a fitted glmnet object
               df: in case that the input is the pandas dataframe,
                   the column names of the coeff. will appear as a legend
                   
    """
    idx = list(np.nonzero(np.reshape(model.coef_, (1,-1)))[1])
    dct = dict( zip([df.columns.tolist()[i] for i in idx], [model.coef_[0][i] for i in idx]))
        
    return pd.DataFrame(data = dct.items() , columns = ["Features", "Coeffs"]).sort_values(by = "Coeffs", ascending = False).reset_index(drop = True)

#*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
# Function: _glmnet_best_score_alpha

def _glmnet_best_score_alpha(X, Y):
    """
    a function to run glmnet for different alpha
    """
    
    def logitnet_nonzero_coef(g):
        idx = np.where(g.coef_[0] != 0)[0]
        return pd.DataFrame({"idx": idx, "coef": g.coef_[0][idx]})

    alpha_list = np.arange(0.0, 0.5, 0.05)
    mean_auc = []
    std_auc = []
    n_features = []
    for alpha in alpha_list:
        model = _glmnet(X, Y, alpha = alpha)
        mean_auc.append(model.cv_mean_score_[model.lambda_best_inx_])
        std_auc.append(model.cv_standard_error_[model.lambda_best_inx_])
        fe = logitnet_nonzero_coef(model)
        n_features.append(fe.shape[0])
        
    mpl.rcParams['axes.linewidth'] = 3 
    mpl.rcParams['lines.linewidth'] =2
    plt.figure(figsize=(12,6))

    plt.subplot(1,2,1)
    plt.errorbar(alpha_list, mean_auc, yerr = std_auc, c = "r", ecolor="k", marker = "o" )
    plt.tick_params(axis='both', which='major', labelsize = 12)
    plt.xlabel(r"$\alpha$" , fontsize = 20)
    plt.ylabel("Mean 10 Folds CV AUC", fontsize = 20)
    plt.grid(True)

    plt.subplot(1,2,2)
    plt.plot(alpha_list, n_features, c = "r",  marker = "o" )
    plt.tick_params(axis='both', which='major', labelsize = 12)
    plt.xlabel(r"$\alpha$" , fontsize = 20)
    plt.ylabel("Number of Features", fontsize = 20)
    plt.grid(True)

    plt.show()

   
    

First, I loaded the data into a pandas dataframe to get some idea.

# readin the data into a dataframe
dateparser = lambda x: pd.datetime.strptime(x, "%Y-%m-%d")
df_raw = pd.read_csv("./device_failure.csv",
                     parse_dates = ["date"],
                     date_parser = dateparser,
                     encoding = "cp1252")

print("Shape: {}".format(df_raw.shape))
print("Prevalence = {:.3f}%".format(df_raw["failure"].sum()/df_raw.shape[0] * 100))

Shape: (124494, 12)
Prevalence = 0.085%

df_raw.head()

	date	device	attribute1	attribute2	attribute3	attribute4	attribute5	attribute6	attribute9
0	2015-01-01	S1F01085	215630672	56	0	52	6	407438	7
1	2015-01-01	S1F0166B	61370680	0	3	0	6	403174	0
2	2015-01-01	S1F01E6Y	173295968	0	0	0	12	237394	0
3	2015-01-01	S1F01JE0	79694024	0	0	0	6	410186	0
4	2015-01-01	S1F01R2B	135970480	0	0	0	15	313173	3

Printing out the statistical description of the data. It would help us to have idea about the variation of the each column in the data.

df_raw.describe()

	failure	attribute1	attribute2	attribute3	attribute4	attribute5	attribute6	attribute7	attribute8	attribute9
count	124494.000000	1.244940e+05	124494.000000	124494.000000	124494.000000	124494.000000	124494.000000	124494.000000	124494.000000	124494.000000
mean	0.000851	1.223881e+08	159.484762	9.940455	1.741120	14.222669	260172.657726	0.292528	0.292528	12.451524
std	0.029167	7.045933e+07	2179.657730	185.747321	22.908507	15.943028	99151.078547	7.436924	7.436924	191.425623
min	0.000000	0.000000e+00	0.000000	0.000000	0.000000	1.000000	8.000000	0.000000	0.000000	0.000000
25%	0.000000	6.128476e+07	0.000000	0.000000	0.000000	8.000000	221452.000000	0.000000	0.000000	0.000000
50%	0.000000	1.227974e+08	0.000000	0.000000	0.000000	10.000000	249799.500000	0.000000	0.000000	0.000000
75%	0.000000	1.833096e+08	0.000000	0.000000	0.000000	12.000000	310266.000000	0.000000	0.000000	0.000000
max	1.000000	2.441405e+08	64968.000000	24929.000000	1666.000000	98.000000	689161.000000	832.000000	832.000000	18701.000000

Printing out the number of Null values in each column. These numbers should be imputed before feeding into the classifier.

df_raw.isnull().sum()

date          0
device        0
failure       0
attribute1    0
attribute2    0
attribute3    0
attribute4    0
attribute5    0
attribute6    0
attribute7    0
attribute8    0
attribute9    0
dtype: int64

Checking the data types of the columns to make sure all the columns have continuous values as it is mentioned in the meta data, except date and device

df_raw.dtypes

date          datetime64[ns]
device                object
failure                int64
attribute1             int64
attribute2             int64
attribute3             int64
attribute4             int64
attribute5             int64
attribute6             int64
attribute7             int64
attribute8             int64
attribute9             int64
dtype: object

Checking the pair-wise correlation between features

methods = {‘pearson’, ‘kendall’, ‘spearman’}

df_raw.corr(method = "kendall")

	failure	attribute1	attribute2	attribute3	attribute4	attribute5	attribute6	attribute7	attribute8	attribute9
failure	1.000000	0.001611	0.053361	0.003208	0.057295	0.003475	0.002337	0.096926	0.096926	0.004633
attribute1	0.001611	1.000000	-0.001004	0.001960	0.001289	-0.003713	-0.001779	-0.002030	-0.002030	-0.002662
attribute2	0.053361	-0.001004	1.000000	-0.018322	0.219415	-0.022125	-0.062126	0.107491	0.107491	-0.027759
attribute3	0.003208	0.001960	-0.018322	1.000000	0.117459	0.089091	0.056736	-0.009622	-0.009622	0.369092
attribute4	0.057295	0.001289	0.219415	0.117459	1.000000	-0.017980	0.009706	0.160216	0.160216	0.046189
attribute5	0.003475	-0.003713	-0.022125	0.089091	-0.017980	1.000000	0.057295	-0.016603	-0.016603	0.026998
attribute6	0.002337	-0.001779	-0.062126	0.056736	0.009706	0.057295	1.000000	-0.013186	-0.013186	0.069585
attribute7	0.096926	-0.002030	0.107491	-0.009622	0.160216	-0.016603	-0.013186	1.000000	1.000000	-0.017097
attribute8	0.096926	-0.002030	0.107491	-0.009622	0.160216	-0.016603	-0.013186	1.000000	1.000000	-0.017097
attribute9	0.004633	-0.002662	-0.027759	0.369092	0.046189	0.026998	0.069585	-0.017097	-0.017097	1.000000

# Number of unique devices: this can be used as feauture based on the demand
df_raw["device"].nunique()

Preprocessing: attribute7 and attribute8 are totally correlated (the same feature). For now, I am gonna drop date and device as well. Device has 1169 unique values that can be used as features.

df = df_raw.copy()
target = "failure"
to_drop = ["date", "device", "attribute8"]
df_X, Y = _x_y_maker(df, target, to_drop)

Printing the first 5 rows of the sclaed features.

df_X.head()

	attribute1	attribute2	attribute3	attribute4	attribute5	attribute6	attribute7	attribute9
0	1.323358	-0.047478	-0.053516	2.193905	-0.515755	1.485268	-0.039335	-0.028479
1	-0.865998	-0.073170	-0.037365	-0.076004	-0.515755	1.442263	-0.039335	-0.065047
2	0.722517	-0.073170	-0.053516	-0.076004	-0.139414	-0.229738	-0.039335	-0.065047
3	-0.605942	-0.073170	-0.053516	-0.076004	-0.515755	1.512983	-0.039335	-0.065047
4	0.192770	-0.073170	-0.053516	-0.076004	0.048757	0.534543	-0.039335	-0.049375

Checking the correlation again.

df_X.corr()

	attribute1	attribute2	attribute3	attribute4	attribute5	attribute6	attribute7	attribute9
attribute1	1.000000	-0.004250	0.003701	0.001836	-0.003376	-0.001522	0.000151	0.001121
attribute2	-0.004250	1.000000	-0.002617	0.146593	-0.013999	-0.026350	0.141367	-0.002736
attribute3	0.003701	-0.002617	1.000000	0.097452	-0.006696	0.009027	-0.001884	0.532366
attribute4	0.001836	0.146593	0.097452	1.000000	-0.009773	0.024870	0.045631	0.036069
attribute5	-0.003376	-0.013999	-0.006696	-0.009773	1.000000	-0.017049	-0.009384	0.005949
attribute6	-0.001522	-0.026350	0.009027	0.024870	-0.017049	1.000000	-0.012207	0.021152
attribute7	0.000151	0.141367	-0.001884	0.045631	-0.009384	-0.012207	1.000000	0.006861
attribute9	0.001121	-0.002736	0.532366	0.036069	0.005949	0.021152	0.006861	1.000000

Running my feature selector along with noise with 10-folds CV for multiple iterations to make sure about the selected features. This would help to have a simpler model with the same performance

# choosing params to under-fit the model for feature selection + noisy features. This would help to have simpler model
params = {
          "learning_rate" : 0.05,
          "max_depth": 2,
          "min_child_weight": 5,
          "gamma" : 0.5
          
         }
o_, d_ = _my_feature_selector(df_X, Y, n_iter = 10,
                                       params = params,
                                       seed = 1367,
                                       callbacks = False)

-*-*-*-*-*-*-*-*- Iteration 1 -*-*-*-*-*-*-*-*
-*-*- Internal-10 Folds-CV-Train = 0.935 +/- 0.00485 -*-*- Internal-10 Folds-CV-Test = 0.897 +/- 0.00849 -*-*
-*-*- External-10 Folds-CV-Train = 0.933 +/- 0.005 -*-*- External-10 Folds-CV-Test = 0.898 +/- 0.0435 -*-*
-*-*-*-*-*-*-*-*- Iteration 2 -*-*-*-*-*-*-*-*
-*-*- Internal-10 Folds-CV-Train = 0.932 +/- 0.00891 -*-*- Internal-10 Folds-CV-Test = 0.896 +/- 0.00948 -*-*
-*-*- External-10 Folds-CV-Train = 0.931 +/- 0.00924 -*-*- External-10 Folds-CV-Test = 0.892 +/- 0.0413 -*-*
-*-*-*-*-*-*-*-*- Iteration 3 -*-*-*-*-*-*-*-*
-*-*- Internal-10 Folds-CV-Train = 0.934 +/- 0.0065 -*-*- Internal-10 Folds-CV-Test = 0.896 +/- 0.00591 -*-*
-*-*- External-10 Folds-CV-Train = 0.933 +/- 0.00634 -*-*- External-10 Folds-CV-Test = 0.892 +/- 0.0571 -*-*
-*-*-*-*-*-*-*-*- Iteration 4 -*-*-*-*-*-*-*-*
-*-*- Internal-10 Folds-CV-Train = 0.935 +/- 0.00669 -*-*- Internal-10 Folds-CV-Test = 0.897 +/- 0.00905 -*-*
-*-*- External-10 Folds-CV-Train = 0.933 +/- 0.00776 -*-*- External-10 Folds-CV-Test = 0.894 +/- 0.0548 -*-*
-*-*-*-*-*-*-*-*- Iteration 5 -*-*-*-*-*-*-*-*
-*-*- Internal-10 Folds-CV-Train = 0.934 +/- 0.00684 -*-*- Internal-10 Folds-CV-Test = 0.897 +/- 0.00607 -*-*
-*-*- External-10 Folds-CV-Train = 0.932 +/- 0.00653 -*-*- External-10 Folds-CV-Test = 0.894 +/- 0.052 -*-*
-*-*-*-*-*-*-*-*- Iteration 6 -*-*-*-*-*-*-*-*
-*-*- Internal-10 Folds-CV-Train = 0.932 +/- 0.0068 -*-*- Internal-10 Folds-CV-Test = 0.898 +/- 0.0082 -*-*
-*-*- External-10 Folds-CV-Train = 0.931 +/- 0.00678 -*-*- External-10 Folds-CV-Test = 0.892 +/- 0.0493 -*-*
-*-*-*-*-*-*-*-*- Iteration 7 -*-*-*-*-*-*-*-*
-*-*- Internal-10 Folds-CV-Train = 0.931 +/- 0.0071 -*-*- Internal-10 Folds-CV-Test = 0.896 +/- 0.00643 -*-*
-*-*- External-10 Folds-CV-Train = 0.93 +/- 0.00858 -*-*- External-10 Folds-CV-Test = 0.892 +/- 0.0339 -*-*
-*-*-*-*-*-*-*-*- Iteration 8 -*-*-*-*-*-*-*-*
-*-*- Internal-10 Folds-CV-Train = 0.933 +/- 0.00951 -*-*- Internal-10 Folds-CV-Test = 0.898 +/- 0.00782 -*-*
-*-*- External-10 Folds-CV-Train = 0.932 +/- 0.00995 -*-*- External-10 Folds-CV-Test = 0.892 +/- 0.0766 -*-*
-*-*-*-*-*-*-*-*- Iteration 9 -*-*-*-*-*-*-*-*
-*-*- Internal-10 Folds-CV-Train = 0.929 +/- 0.00658 -*-*- Internal-10 Folds-CV-Test = 0.895 +/- 0.00973 -*-*
-*-*- External-10 Folds-CV-Train = 0.928 +/- 0.00671 -*-*- External-10 Folds-CV-Test = 0.896 +/- 0.0523 -*-*
-*-*-*-*-*-*-*-*- Iteration 10 -*-*-*-*-*-*-*-*
-*-*- Internal-10 Folds-CV-Train = 0.935 +/- 0.00538 -*-*- Internal-10 Folds-CV-Test = 0.898 +/- 0.00774 -*-*
-*-*- External-10 Folds-CV-Train = 0.933 +/- 0.0058 -*-*- External-10 Folds-CV-Test = 0.883 +/- 0.0507 -*-*




<Figure size 432x288 with 0 Axes>

params = {
          "learning_rate" : 0.05,
          "max_depth": 2,
          "min_child_weight": 5.,
          "gamma" : 0.5
         }
bst, cvr = _bst_shap(df_X, Y, params = params, callbacks = True)

-*-*-*-*-*-*-* CV Started *-*-*-*-*-*-*-
[22:09:38] Tree method is selected to be 'hist', which uses a single updater grow_fast_histmaker.
[22:09:38] Tree method is selected to be 'hist', which uses a single updater grow_fast_histmaker.
[22:09:38] Tree method is selected to be 'hist', which uses a single updater grow_fast_histmaker.
[22:09:39] Tree method is selected to be 'hist', which uses a single updater grow_fast_histmaker.
[22:09:39] Tree method is selected to be 'hist', which uses a single updater grow_fast_histmaker.
[22:09:39] Tree method is selected to be 'hist', which uses a single updater grow_fast_histmaker.
[22:09:39] Tree method is selected to be 'hist', which uses a single updater grow_fast_histmaker.
[22:09:39] Tree method is selected to be 'hist', which uses a single updater grow_fast_histmaker.
[22:09:39] Tree method is selected to be 'hist', which uses a single updater grow_fast_histmaker.
[22:09:39] Tree method is selected to be 'hist', which uses a single updater grow_fast_histmaker.
[0]	train-auc:0.5	test-auc:0.5
Multiple eval metrics have been passed: 'test-auc' will be used for early stopping.

Will train until test-auc hasn't improved in 50 rounds.
[1]	train-auc:0.5	test-auc:0.5
[2]	train-auc:0.5	test-auc:0.5
[3]	train-auc:0.5	test-auc:0.5
[4]	train-auc:0.5	test-auc:0.5
[5]	train-auc:0.5	test-auc:0.5
[6]	train-auc:0.5	test-auc:0.5
[7]	train-auc:0.5	test-auc:0.5
[8]	train-auc:0.5	test-auc:0.5
[9]	train-auc:0.5	test-auc:0.5
[10]	train-auc:0.5	test-auc:0.5
[11]	train-auc:0.5	test-auc:0.5
[12]	train-auc:0.5	test-auc:0.5
[13]	train-auc:0.5	test-auc:0.5
[14]	train-auc:0.5	test-auc:0.5
[15]	train-auc:0.5	test-auc:0.5
[16]	train-auc:0.5	test-auc:0.5
[17]	train-auc:0.5	test-auc:0.5
[18]	train-auc:0.502591	test-auc:0.499988
[19]	train-auc:0.507799	test-auc:0.499972
[20]	train-auc:0.515655	test-auc:0.499924
[21]	train-auc:0.515655	test-auc:0.499924
[22]	train-auc:0.521354	test-auc:0.504899
[23]	train-auc:0.527633	test-auc:0.509397
[24]	train-auc:0.544239	test-auc:0.51922
[25]	train-auc:0.547894	test-auc:0.519192
[26]	train-auc:0.562889	test-auc:0.528576
[27]	train-auc:0.562888	test-auc:0.528573
[28]	train-auc:0.575345	test-auc:0.551115
[29]	train-auc:0.589815	test-auc:0.570037
[30]	train-auc:0.589813	test-auc:0.570032
[31]	train-auc:0.589815	test-auc:0.570038
[32]	train-auc:0.594991	test-auc:0.579028
[33]	train-auc:0.603815	test-auc:0.588018
[34]	train-auc:0.607991	test-auc:0.587983
[35]	train-auc:0.607994	test-auc:0.587977
[36]	train-auc:0.609034	test-auc:0.587952
[37]	train-auc:0.611603	test-auc:0.58792
[38]	train-auc:0.611606	test-auc:0.58792
[39]	train-auc:0.621348	test-auc:0.592728
[40]	train-auc:0.630541	test-auc:0.59751
[41]	train-auc:0.641352	test-auc:0.610898
[42]	train-auc:0.645456	test-auc:0.619846
[43]	train-auc:0.649023	test-auc:0.624774
[44]	train-auc:0.649023	test-auc:0.629293
[45]	train-auc:0.652603	test-auc:0.638277
[46]	train-auc:0.656164	test-auc:0.638207
[47]	train-auc:0.656163	test-auc:0.638208
[48]	train-auc:0.660153	test-auc:0.638036
[49]	train-auc:0.660154	test-auc:0.638033
[50]	train-auc:0.668212	test-auc:0.637722
[51]	train-auc:0.683198	test-auc:0.651949
[52]	train-auc:0.68676	test-auc:0.65183
[53]	train-auc:0.702173	test-auc:0.664918
[54]	train-auc:0.705216	test-auc:0.673904
[55]	train-auc:0.716932	test-auc:0.688166
[56]	train-auc:0.746947	test-auc:0.705966
[57]	train-auc:0.766238	test-auc:0.732417
[58]	train-auc:0.76625	test-auc:0.732301
[59]	train-auc:0.781952	test-auc:0.749171
[60]	train-auc:0.785332	test-auc:0.753368
[61]	train-auc:0.788794	test-auc:0.753104
[62]	train-auc:0.788767	test-auc:0.753255
[63]	train-auc:0.789759	test-auc:0.757569
[64]	train-auc:0.789792	test-auc:0.757407
[65]	train-auc:0.791717	test-auc:0.762459
[66]	train-auc:0.791741	test-auc:0.762543
[67]	train-auc:0.794139	test-auc:0.771179
[68]	train-auc:0.79414	test-auc:0.771128
[69]	train-auc:0.798393	test-auc:0.78537
[70]	train-auc:0.799311	test-auc:0.790212
[71]	train-auc:0.799362	test-auc:0.790179
[72]	train-auc:0.799436	test-auc:0.790156
[73]	train-auc:0.804061	test-auc:0.789724
[74]	train-auc:0.809716	test-auc:0.793472
[75]	train-auc:0.81023	test-auc:0.793571
[76]	train-auc:0.810291	test-auc:0.793676
[77]	train-auc:0.810273	test-auc:0.794129
[78]	train-auc:0.815368	test-auc:0.793779
[79]	train-auc:0.818557	test-auc:0.798271
[80]	train-auc:0.818553	test-auc:0.798253
[81]	train-auc:0.818556	test-auc:0.798244
[82]	train-auc:0.819048	test-auc:0.798204
[83]	train-auc:0.819046	test-auc:0.798256
[84]	train-auc:0.819084	test-auc:0.798343
[85]	train-auc:0.819094	test-auc:0.798331
[86]	train-auc:0.826427	test-auc:0.810713
[87]	train-auc:0.827297	test-auc:0.815134
[88]	train-auc:0.827317	test-auc:0.820203
[89]	train-auc:0.82735	test-auc:0.820224
[90]	train-auc:0.835594	test-auc:0.836472
[91]	train-auc:0.839335	test-auc:0.83642
[92]	train-auc:0.839334	test-auc:0.836343
[93]	train-auc:0.844798	test-auc:0.840614
[94]	train-auc:0.845237	test-auc:0.840554
[95]	train-auc:0.845206	test-auc:0.840589
[96]	train-auc:0.845202	test-auc:0.840541
[97]	train-auc:0.845634	test-auc:0.840411
[98]	train-auc:0.846109	test-auc:0.840341
[99]	train-auc:0.8461	test-auc:0.840213
[100]	train-auc:0.846579	test-auc:0.840174
[101]	train-auc:0.847455	test-auc:0.840033
[102]	train-auc:0.848847	test-auc:0.839937
[103]	train-auc:0.848889	test-auc:0.839934
[104]	train-auc:0.851638	test-auc:0.839058
[105]	train-auc:0.851658	test-auc:0.839094
[106]	train-auc:0.851696	test-auc:0.839025
[107]	train-auc:0.851761	test-auc:0.839107
[108]	train-auc:0.855548	test-auc:0.838203
[109]	train-auc:0.855952	test-auc:0.838216
[110]	train-auc:0.859843	test-auc:0.837271
[111]	train-auc:0.859896	test-auc:0.837218
[112]	train-auc:0.862772	test-auc:0.844197
[113]	train-auc:0.866681	test-auc:0.847177
[114]	train-auc:0.866765	test-auc:0.847206
[115]	train-auc:0.867214	test-auc:0.847115
[116]	train-auc:0.868626	test-auc:0.855585
[117]	train-auc:0.869564	test-auc:0.859661
[118]	train-auc:0.869588	test-auc:0.859604
[119]	train-auc:0.870008	test-auc:0.859496
[120]	train-auc:0.870083	test-auc:0.859439
[121]	train-auc:0.870515	test-auc:0.859276
[122]	train-auc:0.871057	test-auc:0.859007
[123]	train-auc:0.871131	test-auc:0.859277
[124]	train-auc:0.871211	test-auc:0.859541
[125]	train-auc:0.871218	test-auc:0.859699
[126]	train-auc:0.871588	test-auc:0.863687
[127]	train-auc:0.871767	test-auc:0.863805
[128]	train-auc:0.87185	test-auc:0.864008
[129]	train-auc:0.871903	test-auc:0.863906
[130]	train-auc:0.871946	test-auc:0.863755
[131]	train-auc:0.871992	test-auc:0.863533
[132]	train-auc:0.87198	test-auc:0.863675
[133]	train-auc:0.872139	test-auc:0.863535
[134]	train-auc:0.872082	test-auc:0.86365
[135]	train-auc:0.872188	test-auc:0.863516
[136]	train-auc:0.872232	test-auc:0.86354
[137]	train-auc:0.872228	test-auc:0.863652
[138]	train-auc:0.872266	test-auc:0.863696
[139]	train-auc:0.872255	test-auc:0.863683
[140]	train-auc:0.872303	test-auc:0.863636
[141]	train-auc:0.872336	test-auc:0.863547
[142]	train-auc:0.872402	test-auc:0.863523
[143]	train-auc:0.872441	test-auc:0.863512
[144]	train-auc:0.876281	test-auc:0.865899
[145]	train-auc:0.876307	test-auc:0.865908
[146]	train-auc:0.876324	test-auc:0.86593
[147]	train-auc:0.876275	test-auc:0.870426
[148]	train-auc:0.876311	test-auc:0.8704
[149]	train-auc:0.876318	test-auc:0.870282
[150]	train-auc:0.876344	test-auc:0.870302
[151]	train-auc:0.876323	test-auc:0.87028
[152]	train-auc:0.876347	test-auc:0.870184
[153]	train-auc:0.876402	test-auc:0.87025
[154]	train-auc:0.876392	test-auc:0.870155
[155]	train-auc:0.876437	test-auc:0.870074
[156]	train-auc:0.876463	test-auc:0.870078
[157]	train-auc:0.884578	test-auc:0.870697
[158]	train-auc:0.884583	test-auc:0.87077
[159]	train-auc:0.888683	test-auc:0.867303
[160]	train-auc:0.88869	test-auc:0.867468
[161]	train-auc:0.89149	test-auc:0.863448
[162]	train-auc:0.891448	test-auc:0.86335
[163]	train-auc:0.89525	test-auc:0.865754
[164]	train-auc:0.895239	test-auc:0.865826
[165]	train-auc:0.902117	test-auc:0.873953
[166]	train-auc:0.902968	test-auc:0.872901
[167]	train-auc:0.906869	test-auc:0.881087
[168]	train-auc:0.908143	test-auc:0.880633
[169]	train-auc:0.910264	test-auc:0.890346
[170]	train-auc:0.911123	test-auc:0.888702
[171]	train-auc:0.912856	test-auc:0.890256
[172]	train-auc:0.912859	test-auc:0.890457
[173]	train-auc:0.914933	test-auc:0.885244
[174]	train-auc:0.917162	test-auc:0.888691
[175]	train-auc:0.919827	test-auc:0.893754
[176]	train-auc:0.921293	test-auc:0.898306
[177]	train-auc:0.921269	test-auc:0.898513
[178]	train-auc:0.922025	test-auc:0.900192
[179]	train-auc:0.922283	test-auc:0.898478
[180]	train-auc:0.922729	test-auc:0.896203
[181]	train-auc:0.922026	test-auc:0.89636
[182]	train-auc:0.922845	test-auc:0.89517
[183]	train-auc:0.922994	test-auc:0.89481
[184]	train-auc:0.923685	test-auc:0.893755
[185]	train-auc:0.924463	test-auc:0.89445
[186]	train-auc:0.925612	test-auc:0.895708
[187]	train-auc:0.925986	test-auc:0.896097
[188]	train-auc:0.926308	test-auc:0.898049
[189]	train-auc:0.926701	test-auc:0.89994
[190]	train-auc:0.926804	test-auc:0.900076
[191]	train-auc:0.927169	test-auc:0.899289
[192]	train-auc:0.927667	test-auc:0.898406
[193]	train-auc:0.927906	test-auc:0.898717
[194]	train-auc:0.928189	test-auc:0.897986
[195]	train-auc:0.928687	test-auc:0.897934
[196]	train-auc:0.928905	test-auc:0.897401
[197]	train-auc:0.929011	test-auc:0.899055
[198]	train-auc:0.929382	test-auc:0.898587
[199]	train-auc:0.929621	test-auc:0.897067
[200]	train-auc:0.930354	test-auc:0.897016
[201]	train-auc:0.930419	test-auc:0.895062
[202]	train-auc:0.930577	test-auc:0.894952
[203]	train-auc:0.93098	test-auc:0.89376
[204]	train-auc:0.930982	test-auc:0.893814
[205]	train-auc:0.931073	test-auc:0.893492
[206]	train-auc:0.931204	test-auc:0.893421
[207]	train-auc:0.931199	test-auc:0.892662
[208]	train-auc:0.931229	test-auc:0.893241
[209]	train-auc:0.931645	test-auc:0.892992
[210]	train-auc:0.932143	test-auc:0.893321
[211]	train-auc:0.932168	test-auc:0.89339
[212]	train-auc:0.932059	test-auc:0.892893
[213]	train-auc:0.932261	test-auc:0.892952
[214]	train-auc:0.932471	test-auc:0.892118
[215]	train-auc:0.932655	test-auc:0.892261
[216]	train-auc:0.932859	test-auc:0.891813
[217]	train-auc:0.932999	test-auc:0.892246
[218]	train-auc:0.933072	test-auc:0.892004
[219]	train-auc:0.933236	test-auc:0.891923
[220]	train-auc:0.93351	test-auc:0.891635
[221]	train-auc:0.93367	test-auc:0.891286
[222]	train-auc:0.933816	test-auc:0.891612
[223]	train-auc:0.934	test-auc:0.891521
[224]	train-auc:0.934035	test-auc:0.890987
[225]	train-auc:0.934061	test-auc:0.890868
[226]	train-auc:0.934116	test-auc:0.890957
[227]	train-auc:0.934202	test-auc:0.890346
[228]	train-auc:0.934334	test-auc:0.890627
Stopping. Best iteration:
[178]	train-auc:0.922025+0.00538528	test-auc:0.900192+0.0420388

*-*- Boosting Round = 178 -*-*- Train = 0.922 -*-*- Test = 0.9 -*-*
-*-*-*-*-*-*-* Train Started *-*-*-*-*-*-*-
[22:09:58] Tree method is selected to be 'hist', which uses a single updater grow_fast_histmaker.

_plot_xgboost_cv_score(cvr)

_plot_importance(bst, figsize=(10,6), importance_type = "total_gain")

Tree_explainer = shap.TreeExplainer(bst)
shap_Treevalues = Tree_explainer.shap_values(df_X)
shap.summary_plot(shap_Treevalues, df_X, plot_type = "bar", color = "pink", max_display = 10)

As seen, attribute3 has not played an important role through the feature selection process. Therefore, I dropped it and build the final model based on that.

df_pruned = df_X.drop(["attribute3"], axis = 1)

As seen, the total gain for attribute4, attirbute7, and attribute 2 are higher than the rest. This would say that, for linear models such as regulirized linear models including elastic net, these features will be pruned. Now, we can see, how these features perform:

_plot_roc_nfolds_xgboost(df_pruned, Y)

test_size = 0.3
n_folds = 10
accuracy_score, roc_score, cv_roc_score, y_pred_prob, X_test, y_test, optimal_threshold = _clf_xgboost(df_pruned, Y, test_size, n_folds)
CI_neg, CI_pos = _ConfidenceInterval(roc_score, X_test.shape[0])
print(F"Accuracy for test size {test_size}  = {accuracy_score:.5f}")
print(F"ROC for test size {test_size} = {roc_score:.3f}")
print(F"Mean ROC for {n_folds} folds CV = {cv_roc_score.mean():.3f}")
print(F"Confidence Interval Accuracy = [{CI_neg:.3f} , {CI_pos:.3f}]")

class_names = [0, 1]
#optimal threshold based on Youden Index
y_pred = (y_pred_prob[:,1] > optimal_threshold).astype(int)
# Compute confusion matrix
cnf_matrix = confusion_matrix(y_test, y_pred)
np.set_printoptions(precision=2)
# Plot normalized confusion matrix
plt.figure(figsize=(6,6))
_plot_confusion_matrix(cnf_matrix, classes=class_names, normalize = True)
plt.show()

Accuracy for test size 0.3  = 0.99914
ROC for test size 0.3 = 0.890
Mean ROC for 10 folds CV = 0.896
Confidence Interval Accuracy = [0.887 , 0.893]
Normalized confusion matrix
[[0.91 0.09]
 [0.25 0.75]]

Discussion:

As seen, the xgboost approach showed a good performance in terms of classification of the individuals in terms of Failure and Non-failure. The classification problem is an imbalanced problem (prevalence < 1%). Therefore, the classification accuracy by itself cannot be trusted and the other classification metrics such as area under ROC curve and Precision-Recalls or F-1 score which are robust to imbalanced classes should be used. The other challenge is the size of the dataset. In fact, we are looking for a model that can be generalized to a larger population. In this regard, it is also better to report the classification error with a confidence interval at a statistical significance level. For instance, to have 95% significance, we can calculate the classification error from the following formula using the Z-score of 1.96:

$ConfidenceInterval = Score \pm 1.96 \times \sqrt{ \frac{Score \times (1 - Score)}{N}}$

For instance, for the approach , we have N= 37349 (30% of the data) individuals in the testing set (unseen individuals) with an AUC of 0.89 which is within the calculated 95% confidence interval. Now, we can add the prediction pribabilities for both class 0 and 1 to the test data.

X_test["pred_proba_class_0"] = pd.Series(y_pred_prob[:,0], index = X_test.index)
X_test["pred_proba_class_1"] = pd.Series(y_pred_prob[:,1], index = X_test.index)

print(F"Test Size = {X_test.shape}")
X_test.head()

Test Size = (37349, 9)

	attribute1	attribute2	attribute4	attribute5	attribute6	attribute7	attribute9	pred_proba_class_0	pred_proba_class_1
22999	0.394775	-0.073170	-0.076004	-0.202137	-0.414557	-0.039335	-0.065047	0.999829	0.000171
26237	-1.097957	0.844409	-0.076004	0.613269	-2.623549	-0.039335	-0.065047	0.997224	0.002776
123435	-1.179217	-0.073170	-0.076004	-0.515755	0.573101	-0.039335	-0.065047	0.999710	0.000290
120834	0.404221	-0.073170	-0.076004	-0.515755	1.461849	-0.039335	-0.065047	0.999801	0.000199
31117	0.243279	-0.073170	-0.076004	1.428676	0.208837	-0.039335	-0.065047	0.999160	0.000840

Now we can repeat this process for N times (let’s do 100 times) and check the histograms of auc for 95% significance level.

_xgboost_auc_hist(df_pruned, Y, n_iterations = 100, alpha = 0.95)

95.0% Significance Level - ROC AUC Confidence Interval= [0.849 , 0.934]

Note that, in this example I have chosen the hyper-parameters based on rule of thumb and my experience. However, for finalizing the model, an exhaustive GridSearch or Bayesian Optimization is needed. I have not done it here due to my short time to finish this project. However, for all of my model that went to production so far, I do use hyper-parameter tuning.

2nd Approach: Regularized Linear Model

# Ridge
alpha = 0.0

glmnet_model = _glmnet(df_X, Y, alpha = alpha)
_plot_glmnet_coeff_path(glmnet_model, df_X)
_plot_glmnet_cv_score(glmnet_model)
_df_glmnet_coeff_path(glmnet_model, df_X)

	Features	Coeffs
0	attribute7	0.001592
1	attribute4	0.000901
2	attribute2	0.000707
3	attribute5	0.000030
4	attribute1	0.000027
5	attribute9	0.000022
6	attribute6	-0.000007
7	attribute3	-0.000013

_glmnet_best_score_alpha(df_X, Y)

# Elastic-Net
alpha = 0.1

glmnet_model = _glmnet(df_X, Y, alpha = alpha)
_plot_glmnet_coeff_path(glmnet_model, df_X)
_plot_glmnet_cv_score(glmnet_model)
_df_glmnet_coeff_path(glmnet_model, df_X)

	Features	Coeffs
0	attribute7	0.052871
1	attribute4	0.018768

As seen, glmnet has added more bias to the model (we lost around 4% AUC). However, the model is way more simpler. Generally, both methods along with each other would result in a more generalized model.

Notes could be considered:

Other than, hyper parameter optimization (using grid search or bayesian optimization), I have not implemented any neural network models here that could be used as well (neural network without hyper-parameter optimizaiton is technically useless). Another thing that could be used is the time at event. For example, any useful information such as some specific day-of-the-month that failure happened can be engineered as a new feature (attribute). In addition to this, the device include 1169 unique values where failure happened for 106 of them.

g = df_raw.groupby("device").agg({"failure" : "mean"}).sort_values(by = ["failure"], ascending = False)
g = g.reset_index(level=["device"])
print(F"Number of groups with non-zero failure rate = {g[g['failure'] > 0].shape[0]}")
g.head(10)

Number of groups with non-zero failure rate = 106

	device	failure
0	S1F0RRB1	0.200000
1	S1F10E6M	0.142857
2	S1F11MB0	0.142857
3	S1F0CTDN	0.142857
4	Z1F1AG5N	0.111111
5	W1F0PNA5	0.111111
6	W1F13SRV	0.076923
7	W1F03DP4	0.071429
8	W1F1230J	0.071429
9	W1F0T034	0.058824

For instance, the rows that include the device “S1F0RRB1” has 20% failure rate. In better words, these sets of devices (106) can be used as binary features. However, I was not sure that this is the goal or not. In fact, I am not sure, if would ruin the model since it brings the information from future with knowing the nature of the device. The other note would be for k-folds cross-validation. This devices can be used to group the data into stratified-group-k-folds for classificaiton. In better words, all the rows for specific groups go into the same folds.

Box-Cox Transformation

2018-03-14T00:00:00+00:00

Box-Cox tranformation is a simple generalization of the square root transformation and the log transformation.

\begin{equation} \hat{x} = \begin{cases} \frac{x^{\lambda} - 1}{\lambda} & if \lambda \neq 0
ln(x) & if \lambda = 0\ \end{cases} \end{equation}

The Box-Cox transformation does only work with positive data. Therefore, sometimes you need to shift the data with adding a fixed constant. The power of $\lambda$ needs to be tuned that the box-cox module in scipy package does this process for you.

# Loading Libraries
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
import matplotlib.pylab as pylab
import seaborn as sns
from scipy import stats
sns.set_style("ticks")



plot_params = {
               "legend.fontsize": "x-large",
               "figure.figsize": (10, 6),
               "axes.labelsize": "x-large",
               "axes.titlesize":"x-large",
               "xtick.labelsize":"x-large",
               "ytick.labelsize":"x-large",
               "axes.linewidth" : 3,
               "lines.linewidth" : 2
              }
pylab.rcParams.update(plot_params)

x = np.linspace(0.001 , 3., 100)
lambda_list = [0., 0.25, 0.50, 0.75, 1., 1.25, 1.50]
c = ["k", "royalblue", "plum", "aqua", "gold", "tomato", "darkkhaki"]

plt.figure(figsize=(10,6))
for l in lambda_list:
    plt.plot(x, stats.boxcox(x, lmbda = l), lw = 3, ls = "--", c = c[lambda_list.index(l)], label = F"$\lambda = {l}$")
plt.legend(loc = 0 , prop={'size': 14})
plt.xlabel("X", fontsize = 20)
plt.ylabel(r"$\hat{X}$", fontsize = 20)
plt.title(r"Box-Cox Transformation for different values of $\lambda$", fontsize = 20)
plt.show()
    

# using stats module in scipy
from scipy import stats

# non-normally distrubuted data
x = stats.loggamma.rvs(2, size = 1000) + 4
xt, lambda_optimal = stats.boxcox(x)

plt.figure(figsize = (10, 5))

plt.subplot(1,2,1)
plt.hist(x , color = "navy")
plt.title("X")


plt.subplot(1,2,2)
plt.hist(xt , color = "navy")
plt.title("X-transformed with Box-Cox")
plt.show()

fig = plt.figure(figsize=(8,6))
ax = fig.add_subplot(111)
stats.boxcox_normplot(x, -20,20, plot = ax, N = 200);
ax.axvline(lambda_optimal, lw = 2, color='r', label = F"$\lambda = {lambda_optimal:.4}$");
plt.legend(loc = 0 , prop={'size': 14})
plt.show()

fig = plt.figure(figsize=(8,6))
ax = fig.add_subplot(111)
res = stats.probplot(x, dist=stats.loggamma, sparams=(5,), plot=ax)
ax.set_title("Probplot for loggamma dist with shape parameter 5", fontsize = 14);

Regularized Linear Models

2018-02-28T00:00:00+00:00

# Loading Libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.linear_model import LinearRegression, Lasso, Ridge
import glmnet
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, scale
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:95% !important; }</style>"))
sns.set()
%matplotlib inline

Simple Linear Regression

# defining random number generator with a fixed seed for reproduing the results
rng = np.random.RandomState(seed = 1367)

# defining x and y
x = 10 * rng.rand(50)
y = 0.5 * x - 4 + rng.randn(50)

# scatter plot of x and y
plt.figure(figsize = (20,6))
plt.subplot(1,2,1)
plt.scatter(x, y, c = "navy", s = 30);

# defining the model and fitting the model to the data
model = LinearRegression(fit_intercept = True)
model.fit(x[:, np.newaxis], y)

# fitted line to the data
xfit = np.linspace(0, 10, 1000)
yfit = model.predict(xfit[:, np.newaxis])
# plotting the data and the line
plt.subplot(1,2,2)

plt.scatter(x, y, c = "navy", s = 30)
plt.plot(xfit, yfit, "r", lw = 3);

# Joint plot using SEABORN
plt.figure(figsize = (10,10))
sns.jointplot(x = x, y = y, kind = "reg", color = "navy")
plt.show()

Nonlinear Transformation

# (x1, x2) ---> (x1^2 , x2^2)
plt.figure(figsize=(13,5))
plt.subplot(1,2,1)
plt.scatter([0.1, 0.2, 0.4, 0.8, 0.9], [0.8, 0.9, 0.1, 0.2, 0.4] , marker="o", s = 100, c = "red")
plt.scatter([-1, -2, 3, 4, 5, -4, -5, 2, 3], [-3, -4, 2, 5, 1, 5, 1, -2, -3] , marker="x", s = 100, c = "blue")
plt.title(r"$(x_1, x_2)$", fontsize = 20)
plt.xlabel(r"$x_1$", fontsize = 20)
plt.ylabel(r"$x_2$", fontsize = 20)

plt.subplot(1,2,2)
plt.scatter([x**2 for x in [0.1, 0.2, 0.4, 0.8, 0.9]], [x**2 for x in [0.8, 0.9, 0.1, 0.2, 0.4]], marker="o", s = 100, c = "red")
plt.scatter([x**2 for x in [-1, -2, 3, 4, 5, -4, -5, 2, 3]], [x**2 for x in [-3, -4, 2, 5, 1, 5, 1, -2, -3]], marker="x", s = 100, c = "blue")
plt.title(r"$(x_1^2, x_2^2)$", fontsize = 20)
plt.xlabel(r"$x_1 ^2$", fontsize = 20)
plt.ylabel(r"$x_2 ^2$", fontsize = 20)
plt.show()

Polynomial basis

x = np.linspace(-1,1, 100)
plt.figure(figsize=(8,6))
for degree in [2,3, 4, 5, 6, 7, 8]:
    plt.plot(x, x**degree, label = F"degree {degree}")
    plt.legend(loc = 0)
plt.show()

Noisy Sin(x) using Polynomial basis functions and OLS

# data x and y
x = 10 * rng.rand(50)
y = np.sin(x) + 0.1 * rng.randn(50)

# generate points used to plot
x_plot = np.linspace(0, 10, 100)

# generate points and keep a subset of them
rng = np.random.RandomState(1367)

x = 10 * rng.rand(50)
y = np.sin( x) + 0.1 * rng.randn(50)


# create matrix versions of these arrays
X = x[:, np.newaxis]
X_plot = x_plot[:, np.newaxis]

colors = ["teal", "yellowgreen", "gold", "darkorange", "cyan", "red", "pink", "purple"]
lw = 2
plt.figure(figsize = (10,6))
plt.scatter(x, y, color="navy", s=30, marker="o", label="training points")

for count, degree in enumerate([1, 2, 3, 4, 5, 6, 7, 8]):
    model = make_pipeline(PolynomialFeatures(degree = degree), LinearRegression())
    model.fit(X, y)
    y_plot = model.predict(X_plot)
    plt.plot(x_plot, y_plot, color=colors[count], linewidth=lw,
             label="degree %d" % degree)
plt.legend(loc="lower left")
plt.title("Noisy Sin(x) with Polynomial Basis Functions", fontsize = 20)
plt.show()

Noisy Sin(x) using Polynomial degree 25 and LASSO

# data x and y
x = 10 * rng.rand(50)
y = np.sin(x) + 0.1 * rng.randn(50)

# generate points used to plot
x_plot = np.linspace(0, 10, 100)

# generate points and keep a subset of them
rng = np.random.RandomState(1367)

x = 10 * rng.rand(50)
y = np.sin( x) + 0.1 * rng.randn(50)


# create matrix versions of these arrays
X = x[:, np.newaxis]
X_plot = x_plot[:, np.newaxis]

colors = ["teal", "yellowgreen", "gold", "darkorange", "cyan", "red", "pink", "purple"]
lw = 2
plt.figure(figsize = (10,6))
plt.scatter(x, y, color="navy", s=30, marker="o", label="training points")

degree = 25
for count, alpha in enumerate([1.0, 1e-2, 1e-4]):
    model = make_pipeline(PolynomialFeatures(degree = degree), Lasso(alpha = alpha, max_iter=1e6))
    model.fit(X, y)
    y_plot = model.predict(X_plot)
    plt.plot(x_plot, y_plot, color=colors[count], linewidth=lw,
             label=r"$\lambda$ = %f" % alpha)

plt.legend(loc="lower left")
plt.title(F"Noisy Sin(x) with Polynomial Degree {degree}", fontsize = 20)
plt.show()

# Nonconvex REGION: criteria |w1|^0.5 + |w2|^0.5 <= t using Monte Carlo
# for (a,b) = (-1, 1) we have t = 1
n_iter = 5000
t = 1
w1_list = []
w2_list = []
for i in range(n_iter):
    w1 = 2.0 * np.random.random() - 1.0
    w2 = 2.0 * np.random.random() - 1.0
    if(np.abs(w1) ** 0.5 + np.abs(w2) **0.5 <= t):
        w1_list.append(w1)
        w2_list.append(w2)
        
plt.figure(figsize=(8, 8))        
plt.scatter(w1_list, w2_list, c="navy")
plt.title(r"Nonconvex Region ($L_1^ \frac{1}{2}$ Penalty): $\sqrt{|W_1|} + \sqrt{|W_2|} <= t = 1$")
plt.xlabel(r"$W_1$")
plt.ylabel(r"$W_2$")
plt.text(0.5, -.75, F"{n_iter} Points", fontsize = 12)
plt.show()

# LASSO REGION: criteria |w1| + |w2| <= t using Monte Carlo
# for (a,b) = (-1, 1) we have t = 1
n_iter = 5000
t = 1
w1_list = []
w2_list = []
for i in range(n_iter):
    w1 = 2.0 * np.random.random() - 1.0
    w2 = 2.0 * np.random.random() - 1.0
    if(np.abs(w1) + np.abs(w2) <= t):
        w1_list.append(w1)
        w2_list.append(w2)
        
plt.figure(figsize=(8, 8))        
plt.scatter(w1_list, w2_list, c="navy")
plt.title(r"LASSO Region ($L_1$ Penalty): $|W_1| + |W_2| <= t = 1$")
plt.xlabel(r"$W_1$")
plt.ylabel(r"$W_2$")
plt.text(0.7, -1, F"{n_iter} Points", fontsize = 12)
plt.show()

# RIDGE REGION: criteria w1^2 + w2^2 <= t using Monte Carlo
# for (a,b) = (-1, 1) we have t = 1
n_iter = 5000
t = 1
w1_list = []
w2_list = []
for i in range(n_iter):
    w1 = 2.0 * np.random.random() - 1.0
    w2 = 2.0 * np.random.random() - 1.0
    if(w1**2 + w2**2 <= t):
        w1_list.append(w1)
        w2_list.append(w2)
        
plt.figure(figsize=(8, 8))        
plt.scatter(w1_list, w2_list, c="navy")
plt.title(r"RIDGE Region ($L_2$ Penalty): $W_1^2 + W_2 ^2 <= t =1$")
plt.xlabel(r"$W_1$")
plt.ylabel(r"$W_2$")
plt.text(0.7, -1, F"{n_iter} Points", fontsize = 12)
plt.show()

# Elastic Net REGION: criteria (1-alpha)/2 using Monte Carlo
# for (a,b) = (-1, 1) we have t = 1
n_iter = 50000
t = 1
alpha = 0.7
w1_list = []
w2_list = []
for i in range(n_iter):
    w1 = 2.0 * np.random.random() - 1.0
    w2 = 2.0 * np.random.random() - 1.0
    if( (1-alpha)/2. * (w1**2 + w2**2) + alpha * (np.abs(w1) + np.abs(w2)) ) <= t:
        w1_list.append(w1)
        w2_list.append(w2)
        
plt.figure(figsize=(8, 8))        
plt.scatter(w1_list, w2_list, c="navy")
plt.title(r"ELASTIC-NET Region: $\frac{1-\alpha}{2} L_2 + \alpha L_1 <= t =1$")
plt.xlabel(r"$W_1$")
plt.ylabel(r"$W_2$")
plt.text(0.7, -1, F"{n_iter} Points", fontsize = 12)

plt.show()

def costfunction(X,y,theta):
    '''OLS cost function'''
    #Initialisation of useful values 
    m = np.size(y)
    
    #Cost function in vectorized form
    h = X @ theta
    J = float((1./(2*m)) * (h - y).T @ (h - y));    
    return J;


def closed_form_solution(X,y):
    '''Linear regression closed form solution'''
    return np.linalg.inv(X.T @ X) @ X.T @ y
    
def closed_form_reg_solution(X,y,lamda = 10): 
    '''Ridge regression closed form solution'''
    m,n = X.shape
    I = np.eye((n))
    return (np.linalg.inv(X.T @ X + lamda * I) @ X.T @ y)[:,0]

def cost_l2(x,y):
    '''L2 cost functiom'''
    return x**2 + y**2

def cost_l1(x,y):
    '''L1 cost function'''
    return np.abs(x) + np.abs(y)

##########################################
#Creating the dataset (as previously)
x = np.linspace(0,1,40)
noise = 1*np.random.uniform(  size = 40)
y = np.sin(x * 1.5 * np.pi ) 
y_noise = (y + noise).reshape(-1,1)

#Subtracting the mean so that the y's are centered
y_noise = y_noise - y_noise.mean()
X = np.vstack((2*x,x**2)).T

#Nornalizing the design matrix to facilitate visualization
X = X / np.linalg.norm(X,axis = 0)

#######################################
lambda_range = np.logspace(0,4,num = 100)/1000
w_0_list_reg_l2 = []
w_1_list_reg_l2 = []

for l in lambda_range:
    t0, t1 = closed_form_reg_solution(X,y_noise,l)
    w_0_list_reg_l2.append(t0)
    w_1_list_reg_l2.append(t1)
    
lambda_range = np.logspace(0,2,num = 100)/1000
w_0_list_reg_l1 = []
w_1_list_reg_l1 = []

for l in lambda_range:
    model_sk_reg = Lasso(alpha=l, fit_intercept=False)
    model_sk_reg.fit(X,y_noise)
    t0, t1 = model_sk_reg.coef_
    w_0_list_reg_l1.append(t0)
    w_1_list_reg_l1.append(t1)
###################################
#Setup of meshgrid of w values
xx, yy = np.meshgrid(np.linspace(-2,17,100),np.linspace(-17,3,100))

#Computing the cost function for each w combination
zz_l2 = np.array(  [cost_l2(xi, yi )for xi, yi in zip(np.ravel(xx), np.ravel(yy)) ] ) #L2 function

zz_l1 = np.array(  [cost_l1(xi, yi )for xi, yi in zip(np.ravel(xx), np.ravel(yy)) ] ) #L1 function

zz_ls = np.array(  [costfunction(X, y_noise.reshape(-1,1),np.array([t0,t1]).reshape(-1,1)) 
                     for t0, t1 in zip(np.ravel(xx), np.ravel(yy)) ] ) #least square cost function

#Reshaping the cost values    
Z_l2 = zz_l2.reshape(xx.shape)
Z_ls = zz_ls.reshape(xx.shape)
Z_l1 = zz_l1.reshape(xx.shape)

#Defining the global min of each function
min_ls = np.linalg.inv(X.T@X)@X.T@y_noise
min_l2 = np.array([0.,0.])
min_l1 = np.array([0.,0.])

#Plotting the contours - L2 
fig = plt.figure(figsize = (16,7))
ax = fig.add_subplot(1, 2, 1)
ax.contour(xx, yy, Z_l2, levels = [.5,1.5,3,6,9,15,30,60,100,150,250],  cmap = "gray")
ax.contour(xx, yy, Z_ls, levels = [.01,.06,.09,.11,.15], cmap = "coolwarm")
ax.set_xlabel(r"$w_1$")
ax.set_ylabel(r"$w_2$")
ax.set_title("Ridge solution as a function of $\\lambda$ - OLS and $L_2$ ")

#Plotting the minimum - L2 
ax.plot(min_ls[0],min_ls[1], marker = "*", color = "red", markersize = 20)
ax.plot(0,0, marker = "o", color = "black", markersize = 10)

#Plotting the path of L2 regularized minimum
ax.plot(w_0_list_reg_l2,w_1_list_reg_l2, linestyle = "none", marker = "o", color = "navy", alpha = .2)


#Plotting the contours - L1 
ax = fig.add_subplot(1, 2, 2)
ax.contour(xx, yy, Z_l1, levels = [.5,1,2,3,4,5,6,8,10,12,14],  cmap = "gray")
ax.contour(xx, yy, Z_ls, levels = [.01,.06,.09,.11,.15], cmap = "coolwarm")
ax.set_xlabel(r"$w_1$")
ax.set_ylabel(r"$w_2$")
ax.set_title("Lasso solution as a function of $\\lambda$ - OLS and $L_1$ ")

#Plotting the minimum - L1
ax.plot(min_ls[0],min_ls[1], marker = "*", color = "red", markersize = 20)
ax.plot(0,0, marker = "o", color = "black", markersize = 10)

#Plotting the path of L1 regularized minimum
ax.plot(w_0_list_reg_l1,w_1_list_reg_l1, linestyle = "none", marker = "o", color = "navy", alpha = .2)

plt.show()

Loading Diabetes Dataset

from sklearn.datasets import load_diabetes

diabetes_data = load_diabetes()
X = diabetes_data.data
y = diabetes_data.target
names = diabetes_data.feature_names
df_X = pd.DataFrame(data = X, columns=names)
df_X.head()

	age	sex	bmi	bp	s1	s2	s3	s4	s5	s6
0	0.038076	0.050680	0.061696	0.021872	-0.044223	-0.034821	-0.043401	-0.002592	0.019908	-0.017646
1	-0.001882	-0.044642	-0.051474	-0.026328	-0.008449	-0.019163	0.074412	-0.039493	-0.068330	-0.092204
2	0.085299	0.050680	0.044451	-0.005671	-0.045599	-0.034194	-0.032356	-0.002592	0.002864	-0.025930
3	-0.089063	-0.044642	-0.011595	-0.036656	0.012191	0.024991	-0.036038	0.034309	0.022692	-0.009362
4	0.005383	-0.044642	-0.036385	0.021872	0.003935	0.015596	0.008142	-0.002592	-0.031991	-0.046641

Ridge Regression

# Note that in scikit-learn they use alpha instead of lambda as a naming convention

n_alphas = 200
alphas = np.logspace(-10, 10, n_alphas)

ridge_coeff = np.zeros((n_alphas, X.shape[1]))

for i in range(len(alphas)):
    ridge = Ridge(alpha= alphas[i], fit_intercept=False)
    ridge.fit(X, y)
    ridge_coeff[i,:] = ridge.coef_

df_ridge_coeff = pd.DataFrame(data = ridge_coeff, columns= df_X.columns.tolist())
plt.figure(figsize=(10,6))
for col in df_ridge_coeff.columns.tolist():
    plt.semilogx(alphas, df_ridge_coeff[col].values, label =F"{col}")
plt.xlabel(r"$\lambda$", fontsize = 20)
plt.ylabel("Coefficients", fontsize = 20)
plt.legend(loc= "right", prop={'size': 20}, bbox_to_anchor=(1.25 , .5), ncol=1, fancybox=True, shadow=True)
plt.show()

df_ridge_coeff.tail()

	age	sex	bmi	bp	s1	s2	s3	s4	s5	s6
195	7.676179e-08	1.759294e-08	2.395937e-07	1.803679e-07	8.662161e-08	7.110945e-08	-1.612908e-07	1.758612e-07	2.311912e-07	1.562633e-07
196	6.090355e-08	1.395841e-08	1.900960e-07	1.431056e-07	6.872642e-08	5.641892e-08	-1.279697e-07	1.395299e-07	1.834293e-07	1.239808e-07
197	4.832146e-08	1.107474e-08	1.508240e-07	1.135414e-07	5.452821e-08	4.476332e-08	-1.015324e-07	1.107044e-07	1.455346e-07	9.836758e-08
198	3.833872e-08	8.786804e-09	1.196652e-07	9.008482e-08	4.326321e-08	3.551565e-08	-8.055678e-08	8.783395e-08	1.154686e-07	7.804579e-08
199	3.041831e-08	6.971536e-09	9.494353e-08	7.147416e-08	3.432545e-08	2.817846e-08	-6.391453e-08	6.968830e-08	9.161387e-08	6.192228e-08

LASSO Regression

n_alphas = 200
alphas = np.logspace(-5, 1, n_alphas)

lasso_coeff = np.zeros((n_alphas, X.shape[1]))

for i in range(len(alphas)):
    lasso = Lasso(alpha= alphas[i], fit_intercept=True, max_iter=100000)
    lasso.fit(X, y)
    lasso_coeff[i,:] = lasso.coef_

df_lasso_coeff = pd.DataFrame(data = lasso_coeff, columns= df_X.columns.tolist())

import matplotlib as mpl
mpl.rcParams['axes.linewidth'] = 3 
mpl.rcParams['lines.linewidth'] =2
plt.figure(figsize=(10,6))
for col in df_lasso_coeff.columns.tolist():
    plt.semilogx(alphas, df_lasso_coeff[col].values, label =F"{col}")
plt.xlabel(r"$\lambda$", fontsize = 20)
plt.ylabel("Coefficients", fontsize = 20)
plt.legend(loc= "right",prop={'size': 20}, bbox_to_anchor=(1.25 , .5), ncol=1, fancybox=True, shadow=True)
plt.show()

df_lasso_coeff.tail()

	age	sex	bmi	bp	s1	s2	s3	s4	s5	s6
195	0.0	0.0	0.0	0.0	0.0	0.0	-0.0	0.0	0.0	0.0
196	0.0	0.0	0.0	0.0	0.0	0.0	-0.0	0.0	0.0	0.0
197	0.0	0.0	0.0	0.0	0.0	0.0	-0.0	0.0	0.0	0.0
198	0.0	0.0	0.0	0.0	0.0	0.0	-0.0	0.0	0.0	0.0
199	0.0	0.0	0.0	0.0	0.0	0.0	-0.0	0.0	0.0	0.0

Elastic-Net Regression using GLM-NET

def _glmnetCV(X, Y, is_classification = True,
                   n_splits = 4,
                   alpha = 0.5,
                   n_lambda = 100,
                   cut_point = 1,
                   scoring = "roc",
                   random_state = 1367,
                   print_cv = True):
    """
    a function for running glmnet cv for both classification and regression
    
    parameters:
    
    X : set of features
    Y : set of labels/values
    is_classification: a flag which is by default is set to True.
                       For regression problems set it to False
    n_splits : number of cross-validation folds. By default set to 4.
    alpha : the elastic parameter[0,1]. alpha = 0 gives Lasso and alpha = 1 Ridge.
            By default is set to 0.5.
    n_lambda: number of penalty terms. By default is set to 100.
    cut_point : the number of standard error distance between best and maximum lambda
                to select the best lambda. The default is set to 1.
    scoring : a string to set the scoring method:
              for classification : "roc_auc",
                                   "accuracy",
                                   "average_precision",
                                   "precision",
                                   "recall"
              for classification, the defualt is set to "roc_auc".
              
              for regression : "r2",
                               "mean_squared_error",
                               "mean_absolute_error",
                               "median_absolute_error"
              for regression, the defualt is set to "r2".
    random_state : random seed. By default is set to 1367.
    print_cv : a flag to print the mean cv scores. By default is set to True.
               To ignore the print out of the results, set it to False.
    """
    if is_classification:
        model = glmnet.LogitNet(alpha = alpha,
                                n_lambda = n_lambda,
                                n_splits = n_splits,
                                cut_point = cut_point,
                                scoring = scoring,
                                n_jobs = -1,
                                random_state = random_state
                               )
    else:
        model = glmnet.ElasticNet(alpha = alpha,
                                  n_lambda = n_lambda,
                                  n_splits = n_splits,
                                  cut_point = cut_point,
                                  scoring = scoring,
                                  n_jobs = -1,
                                  random_state = random_state
                                 )

    fit = model.fit(X, Y)
    if print_cv:
        for ij in range(len(fit.cv_mean_score_)):
            print(F"{model.n_splits} Folds CV Mean {model.scoring} = {fit.cv_mean_score_[ij]:.2} +/- {fit.cv_standard_error_[ij]:.1}")
    
    return model

def _glmnet_plot_cv_score_lambda(model):
    """
    a function to plot cv scores vs lambda
    parameters:
               model : a fitted glmnet object
    """
    import matplotlib as mpl
    mpl.rcParams['axes.linewidth'] = 3 
    mpl.rcParams['lines.linewidth'] =2
    plt.figure(figsize=(10,6))
    plt.errorbar(-np.log(model.lambda_path_), model.cv_mean_score_, yerr=model.cv_standard_error_ , c = "r", ecolor="k", marker = "o", )
    plt.vlines(-np.log(model.lambda_best_), ymin = min(model.cv_mean_score_) - 0.05 , ymax = max(model.cv_mean_score_) + 0.05, lw = 3, linestyles = "--", colors = "b" ,label = "best $\lambda$")
    plt.vlines(-np.log(model.lambda_max_), ymin = min(model.cv_mean_score_) - 0.05 , ymax = max(model.cv_mean_score_) + 0.05, lw = 3, linestyles = "--", colors = "c" ,label = "max $\lambda$")
    plt.tick_params(axis='both', which='major', labelsize = 12)
    plt.grid(True)
    plt.ylim([min(model.cv_mean_score_) - 0.05, max(model.cv_mean_score_) + 0.05])
    plt.legend(loc = 4, prop={'size': 20})
    plt.xlabel("$-Log(\lambda)$" , fontsize = 20)
    plt.ylabel(F"Mean {model.n_splits} Folds CV {(model.scoring).upper()}", fontsize = 20)
    plt.title(F"Best $\lambda$ = {model.lambda_best_[0]:.2} with {len(np.nonzero(np.reshape(model.coef_, (1,-1)))[1])} Features, $\\alpha$ = {model.alpha}" , fontsize = 20)
    plt.show()

def _glmnet_plot_coeff_path(model, df):
    """
    a function to plot coefficients vs lambda
    parameters:
               model : a fitted glmnet object
               df: in case that the input is the pandas dataframe,
                   the column names of the coeff. will appear as a legend
    """
    import matplotlib as mpl
    mpl.rcParams['axes.linewidth'] = 3 
    mpl.rcParams['lines.linewidth'] =2
    plt.figure(figsize=(10,6))
    if not df.empty:
        for i in list(np.nonzero(np.reshape(model.coef_, (1,-1)))[1]):
            plt.plot(-np.log(model.lambda_path_) ,(model.coef_path_.reshape(-1,model.coef_path_.shape[-1]))[i,:], label = df.columns.values.tolist()[i]);
        plt.legend(loc= "right", bbox_to_anchor=(1.2 , .5), ncol=1, fancybox=True, shadow=True)

    else:
        for i in list(np.nonzero(np.reshape(model.coef_, (1,-1)))[1]):
            plt.plot(-np.log(model.lambda_path_) ,(model.coef_path_.reshape(-1,model.coef_path_.shape[-1]))[i,:]);
            
    plt.tick_params(axis='both', which='major', labelsize = 12)    
    plt.ylabel("Coefficients", fontsize = 20)
    plt.xlabel("-$Log(\lambda)$", fontsize = 20)
    plt.title(F"Best $\lambda$ = {model.lambda_best_[0]:.2} with {len(np.nonzero(np.reshape(model.coef_, (1,-1)))[1])} Features, $\\alpha$ = {model.alpha}" , fontsize = 20)
    plt.grid(True)
    plt.show()

def _glmnet_coeff(model, df):
    """
    a function to print the coefficients of glmnet model
    Input:
         model : fitted glmnet model
         df : dataframe to have access to column names

    """
    df_nonzero = df.iloc[:, np.nonzero(np.reshape(model.coef_, (-1,1)))[0]]
    df_coeff = pd.DataFrame(data = model.coef_[np.nonzero(np.reshape(model.coef_, (-1,1)))[0]],  index= df_nonzero.columns)
    df_coeff.sort_values(by = 0, ascending=False)
    
    return df_coeff
    

def _glmnet_plot_cv_score_alpha(X, Y, is_classification = True,
                                n_splits = 4,
                                n_lambda = 100,
                                cut_point = 1,
                                scoring = "roc",
                                random_state = 1367,
                                print_cv = True):
    """
    a function for running glmnet cv for both classification and regression for a range of alpha
    it outputs the best cv score with the number of non-zero coefficients
    parameters:
    
    X : set of features
    Y : set of labels/values
    is_classification: a flag which is by default is set to True.
                       For regression problems set it to False
    n_splits : number of cross-validation folds. By default set to 4.
    n_lambda: number of penalty terms. By default is set to 100.
    cut_point : the number of standard error distance between best and maximum lambda
                to select the best lambda. The default is set to 1.
    scoring : a string to set the scoring method:
              for classification : "roc_auc",
                                   "accuracy",
                                   "average_precision",
                                   "precision",
                                   "recall"
              for classification, the defualt is set to "roc_auc".
              
              for regression : "r2",
                               "mean_squared_error",
                               "mean_absolute_error",
                               "median_absolute_error"
              for regression, the defualt is set to "r2".
    random_state : random seed. By default is set to 1367.
    print_cv : a flag to print the mean cv scores. By default is set to True.
               To ignore the print out of the results, set it to False.
    """    
    alpha_list = np.arange(0.01, 1.0, 0.01)
    mean_score = []
    std_score = []
    n_features = []
    
    if is_classification:
        for alpha in alpha_list:
            model = glmnet.LogitNet(alpha = alpha,
                                    n_lambda = n_lambda,
                                    n_splits = n_splits,
                                    cut_point = cut_point,
                                    scoring = scoring,
                                    n_jobs = -1,
                                    random_state = random_state
                                   )
            model.fit(X, Y)
            mean_score.append(model.cv_mean_score_[model.lambda_best_inx_])
            std_score.append(model.cv_standard_error_[model.lambda_best_inx_])
            n_features.append(len(np.nonzero(np.reshape(model.coef_, (-1,1)))[0]))
    else:
        for alpha in alpha_list:
            model = glmnet.ElasticNet(alpha = alpha,
                                      n_lambda = n_lambda,
                                      n_splits = n_splits,
                                      cut_point = cut_point,
                                      scoring = scoring,
                                      n_jobs = -1,
                                      random_state = random_state
                                     )
            model.fit(X, Y)
            mean_score.append(model.cv_mean_score_[model.lambda_best_inx_])
            std_score.append(model.cv_standard_error_[model.lambda_best_inx_])
            n_features.append(len(np.nonzero(np.reshape(model.coef_, (-1,1)))[0]))
    
    # Plotting the results
    import matplotlib as mpl
    mpl.rcParams['axes.linewidth'] = 3 
    mpl.rcParams['lines.linewidth'] =2
    plt.figure(figsize=(30,6))

    plt.subplot(1,2,1)
    plt.errorbar(alpha_list, mean_score, yerr = std_score, c = "r", ecolor="k", marker = "o", )
    plt.tick_params(axis='both', which='major', labelsize = 12)
    plt.xlabel(r"$\alpha$" , fontsize = 20)
    plt.ylabel(F"Mean {model.n_splits} Folds CV {model.scoring}", fontsize = 20)
    plt.grid(True)

    plt.subplot(1,2,2)
    plt.plot(alpha_list, n_features, c = "r",  marker = "o", )
    plt.tick_params(axis='both', which='major', labelsize = 12)
    plt.xlabel(r"$\alpha$" , fontsize = 20)
    plt.ylabel("Number of Features", fontsize = 20)
    plt.grid(True)

    plt.show()

_glmnet_plot_cv_score_alpha(X, y, is_classification=False, scoring="r2")

glmnet_model = _glmnetCV(X, y, is_classification = False, scoring="r2", n_splits=10, alpha=0.5)

Folds CV Mean r2 = -0.034 +/- 0.02
Folds CV Mean r2 = 0.0079 +/- 0.02
Folds CV Mean r2 = 0.061 +/- 0.02
Folds CV Mean r2 = 0.11 +/- 0.02
Folds CV Mean r2 = 0.15 +/- 0.02
Folds CV Mean r2 = 0.19 +/- 0.03
Folds CV Mean r2 = 0.22 +/- 0.03
Folds CV Mean r2 = 0.26 +/- 0.03
Folds CV Mean r2 = 0.28 +/- 0.03
Folds CV Mean r2 = 0.31 +/- 0.03
Folds CV Mean r2 = 0.33 +/- 0.03
Folds CV Mean r2 = 0.35 +/- 0.03
Folds CV Mean r2 = 0.37 +/- 0.03
Folds CV Mean r2 = 0.38 +/- 0.03
Folds CV Mean r2 = 0.39 +/- 0.04
Folds CV Mean r2 = 0.4 +/- 0.04
Folds CV Mean r2 = 0.41 +/- 0.04
Folds CV Mean r2 = 0.42 +/- 0.04
Folds CV Mean r2 = 0.42 +/- 0.04
Folds CV Mean r2 = 0.43 +/- 0.04
Folds CV Mean r2 = 0.43 +/- 0.04
Folds CV Mean r2 = 0.43 +/- 0.04
Folds CV Mean r2 = 0.44 +/- 0.04
Folds CV Mean r2 = 0.44 +/- 0.04
Folds CV Mean r2 = 0.45 +/- 0.04
Folds CV Mean r2 = 0.45 +/- 0.04
Folds CV Mean r2 = 0.45 +/- 0.04
Folds CV Mean r2 = 0.45 +/- 0.04
Folds CV Mean r2 = 0.46 +/- 0.04
Folds CV Mean r2 = 0.46 +/- 0.04
Folds CV Mean r2 = 0.46 +/- 0.04
Folds CV Mean r2 = 0.46 +/- 0.04
Folds CV Mean r2 = 0.46 +/- 0.04
Folds CV Mean r2 = 0.46 +/- 0.04
Folds CV Mean r2 = 0.46 +/- 0.04
Folds CV Mean r2 = 0.46 +/- 0.04
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.47 +/- 0.05
Folds CV Mean r2 = 0.47 +/- 0.05
Folds CV Mean r2 = 0.47 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05
Folds CV Mean r2 = 0.46 +/- 0.05

_glmnet_plot_cv_score_lambda(glmnet_model)

_glmnet_plot_coeff_path(glmnet_model, df_X)

df_glment_coeff = _glmnet_coeff(glmnet_model, df_X)
df_glment_coeff

	0
bmi	444.032048
bp	171.000669
s3	-101.325339
s5	389.635151

Amirhessam Tahmassebi

Machine Learning Algorithms From Scratch: Logistic Regression

Introduction

In order to map predicted values to probabilities, we use the sigmoid function. The function maps any real value into another value between 0 and 1. In machine learning, we use sigmoid to map predictions to probabilities.

Sigmoid Function: $f(x) = \frac{1}{1 + exp(-x)}$

The main algorithm to develop Logistic Regression is based on updating weights using Gradient Descent approach iteratively.

- Define linear model: $z = wx + b$

- Define prediction: $\hat{y} = sigmoid(z) = sigmoid(wx + b) = \frac{1}{1 + \exp({-(wx+b)})}$

- Update weights: $w = w - lr \times dw$ where $dw = \frac{1}{N} \Sigma_{i=1}^{N}(2x_i(\hat{y_i} - y_i))$

- Update bias $b = b - lr \times db$ where $db = \frac{1}{N} \Sigma_{i=1}^{N}(\hat{y_i} - y_i)$

This is part of the ML-Algs library which can be downloaded via PIP or from Git.

Sample breast cancer data for binary classification task. I have used stratifed train/test splits where test size is the 20\% of the data.

Now, train the model using our algorithm.

We can also plot the decision boundary and the predicted probabilities in each class.

We can also calculate all the binary classification metrics using SlickML library.

Updates on Regression Error Characteristics Curves

Updates on Regression Error Characteristics Curve

Introduction

Furthermore, the coefficient of determination $R^2$ can be also calculated with respect to the AOC. Likewise the ROC curve, the shape of the REC curve can also be used as a guidance for the users to reveal additional information about the data modeling. The REC curve was implemented in Python.

Previously, I have implemented a simple version of REC LINK. Here, is an overview of the objected oriented implementation of REC and its updates.

You can clone the project from my GitHub Repo LINK or run the following cell.

Or you can follow the class below which is the content of src/rec.py

Load Data and Train a Regression Model

Let’s try to plot REC curve

Currently, REC is also implemented in my Machine Learning Python Library, SlickML which can be easily installed via pip.

Extra notes about the REC class

ROC vs Precision-Recall Curves with N-Folds Cross-Validation

Loading Libraries

Functions

KDD CUP 99 Data

Adding noise to features to not have perfect problem

Checking the targets

Building binary flag for ‘back’ or ‘normal’ class

ROC Curve + 10-folds CV with XGBOOST

Precision Curve + 10-folds CV with XGBOOST

Finding Thresholds in Imbalanced Binary Classification

Loading Libraries

Functions

Device Failure Data

First, I loaded the data into a pandas dataframe to get some idea.

Preprocessing and builing X, Y training/testing sets.

Printing the first 5 rows of the sclaed features.

Train a model

Visualization of Thresholds

MNIST DATA

As you know, minist contains 10 classes. So, we need to turn this multi-class data to a binary class.

Web Scraping using BeautifulSoup

Web Scraping (also termed Screen Scraping, Web Data Extraction, Web Harvesting etc.) is a technique employed to extract large amounts of data from websites whereby the data is extracted and saved to a local file in your computer or to a database in table (spreadsheet) format.

Here is the link to Trump’s lies artcile from NY-Times: LINK

Here is the structure of the first lie:

Collecting all the records

Parse the first lie into 4 structured columns:

The first result to see the pattern of the html

Parsing date by looking at the strong tag

Parsing lie by parsing the content out

Parsing explanation by looking at the a tag

Parsing link out by looking at the href key of the a tag

Now, iterating over the whole records and put it into a dataframe

Bayesian Optimization For XGBoost

XGBoost has many hyper-paramters which need to be tuned to have an optimum model. There are many ways to find these tuned parameters such as grid-search or random search. However, bayesian optimization makes it easier and faster for us.

Bayesian Optimization

Loading Data

Best XGBoost Parameters

Adaptive Learning Optimizers For Neural Networks

Imbalanced Classification

Imbalanced classification (or classification problems with low prevalence (low number of instances in one of the classes) can be challenging. In this post, I have discussed how we can model a problem with prevalence of 0.09% for positive class using gradient boosting and generalized linear model.

Functions

First, I loaded the data into a pandas dataframe to get some idea.

Printing out the statistical description of the data. It would help us to have idea about the variation of the each column in the data.

Printing out the number of Null values in each column. These numbers should be imputed before feeding into the classifier.

Checking the data types of the columns to make sure all the columns have continuous values as it is mentioned in the meta data, except date and device

Checking the pair-wise correlation between features

methods = {‘pearson’, ‘kendall’, ‘spearman’}

Preprocessing: attribute7 and attribute8 are totally correlated (the same feature). For now, I am gonna drop date and device as well. Device has 1169 unique values that can be used as features.

Printing the first 5 rows of the sclaed features.

Checking the correlation again.

Running my feature selector along with noise with 10-folds CV for multiple iterations to make sure about the selected features. This would help to have a simpler model with the same performance

As seen, attribute3 has not played an important role through the feature selection process. Therefore, I dropped it and build the final model based on that.

As seen, the total gain for attribute4, attirbute7, and attribute 2 are higher than the rest. This would say that, for linear models such as regulirized linear models including elastic net, these features will be pruned. Now, we can see, how these features perform:

Discussion:

Or you can follow the class below which is the content of `src/rec.py`