Week 3 of Andrew Ng’s Machine Learning Course focusses on logic regression. As before the the coding exercises have been completed in Python. Functions required for the assessment part of the assignment are below. The Git repository of the complete script is here.

Required modules

import numpy as py

Return inverse of the logit function

Uses the following equation:

\begin{align*}& g = {1 \over 1 + e^{-z}} \end{align*}

def sigmoid(z):
    g = 1 / (1 + (np.exp(-z)))
    return(g)

...
sigmoid(0) == 0.5 # evaluates True

Input values:

Return values are:

Cost function and gradient with optional regularisation

Uses the following equation:

Regularised Cost funtion: \begin{align*}& J_{reg}(\theta) = - \dfrac{1}{m} \left[\sum_{i=1}^{m} y^{(i)} \log(h_\theta(x^{(i)})) + (1 - y^{(i)}) \log(1-h_\theta(x^{(i)}))\right] + \frac{\lambda}{2m} \sum_{j=1}^{n} \theta_j^2 \end{align*}

Regularised gradient: \begin{align*} & \frac{\partial}{\partial \theta_j} J_{reg}(\theta) = \frac{1}{m} \sum\limits_{i=1}^{m} (h_\theta(x^{(i)}) - y^{(i)}) x_j^{(i)} \hspace{25pt} \text{for }j = 0 \end{align*}

\begin{align*} \frac{\partial}{\partial \theta_j} J_{reg}(\theta) = \frac{1}{m} \sum\limits_{i=1}^{m} (h_\theta(x^{(i)}) - y^{(i)}) x_j^{(i)} + \frac{\lambda}{m}\theta_j \hspace{25pt} \text{for }j \geq 1 \end{align*}

Note: If λ is set to 0 then the regularisation formula will evaluate to 0; therefore, no regularisation will be applied.

def cost_function(theta, X, y, lambda_var=0):
    m = y.shape[0]
    h = sigmoid(X @ theta)

    # cost function
    error = (-y * np.log(h)) - (1 - y) * np.log(1 - h)

    regularise = (lambda_var/(2*m)) * np.sum(theta[1:]**2)

    J = 1/m * np.sum(error) + regularise

    # gadient
    regularise = (lambda_var/m) * theta[1:]

    grad_0 = 1/m * (np.transpose(X) @ (h - y))[0]
    grad_1 = 1/m * (np.transpose(X) @ (h - y))[1:] + regularise
    grad = np.hstack([grad_0.reshape(1), grad_1])

    return(J, grad)

...
J, grad = cost_function(theta, X_1, y, lambda_ = 0)

Input values:

Return values are:

Predicting y and measuring accuracy

Uses the following equation:

\begin{align*} h_\theta(x) = \frac{1}{1 + e^{\theta^{\top} x}} \end{align*}

def predict_y(theta, X):
    y = np.round(sigmoid(X @ theta))
    return(y)

...
y_pred = predict_y(theta, X_1)
y_accuracy = np.mean(y_pred == y) * 100
print(f"Accuracy of model: {y_accuracy} \n")

Input values are:

Return value: