NaN.

discussion

Practical Machine Learning With Python [Part-1]by Savan Visalpara

In this series of posts, I will explain various Machine Learning concepts with code in Python. You can find out Python code for this part here. Check out github repository of this series.

Introduction

Understanding the classification algorithm (illustration)

Let us understand this algorithm with a classification problem. For simplicity of visualization, we'll assume that our input data is 2 dimensional. We have two possible classes - green and red.

Lets plot out training data in feature space.

There is no explicit training phase in KNN! In other words, for classifying new data points, we'll directly use our dataset (in some sense, the dataset is the model).

To classify a new data point, we find the k points in the training data closest to it, and make a prediction based on whichever class is most common among these k points (i.e. we simulate a vote). Here closest is defined by a suitable distance metric such as euclidean distance. Other distance metrics are discussed below.

For example, if we want to classify blue point as shown in following figure, we consider k nearest data points and we assign the class which has the majority.

If k = 3, we get two data points with green class and one data point with red class. Hence, we'll predict green class for the new point.

Here's another example, let us change the position of new point (blue point) as shown below.

If we take k = 5 then we get four neighbors with red class and one neighbor with green class. Hence, new point will be classified as red point.

KNN as regression algorithm

In case of regression (when target variable is a real value), we take the average of the K nearest neighbors.

Tuning the hyper-parameter K

A small value of k means that noise will have a higher influence on the result and large value will make the algorithm computationally expensive. Usually, we perform cross-validation to find out best k value (or to choose the value of k that best suits our accuracy / speed trade-off). If you don't want to try multiple values of k, a rule of thumb is to set k equal to the square root of total number of data points. For more on choosing best value of k, refer this stackoverflow thread.

Distance metrics

There are various options available for distance metric such as euclidean or manhattan distance. The most commonly used metric is euclidean distance.

Minkowski is the generalization of Euclidean and Manhattan distance.

Note that you'll want to do some pre-processing on the input data (for example, make sure each dimension has 0 mean and unit variance) so that the distance metrics above are meaningful.

Scikit-learn implementation

## load the dataset 
from sklearn.datasets import load_iris
dataset = load_iris()
X = dataset.data
y = dataset.target
 
## precessing
# standardize the data to make sure each feature contributes equally 
# to the distance
from sklearn.preprocessing import StandardScaler
ss = StandardScaler()
X_processed = ss.fit_transform(X)
 
## split the dataset into train and test set
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X_processed, y, test_size=0.3, random_state=42)
 
## fit n nearest neighbor model 
from sklearn.neighbors import KNeighborsClassifier
model = KNeighborsClassifier(n_neighbors = 5, metric="minkowski", p=2) 
# p=2 for euclidian distance
model.fit(X_train, y_train)
# output: 
# KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
#          metric_params=None, n_jobs=1, n_neighbors=5, p=2,
#          weights='uniform')
 
## evaluate 
model.score(X_test, y_test)
# output: 1.0

How to run this code on Google Colaboratory

Notebook Link

You can also play with this project directly in-browser via Google Colaboratory using the link above. Google Colab is a free tool that lets you run small Machine Learning experiments through your browser. You should read this 1 min tutorial if you're unfamiliar with Google Colaboratory.

Parametric and non-parametric models

In a parametric model, we continuously update a fixed number of parameters to learn a function which can classify new data point without requiring the training data (for example, logistic regression). In a non-parametric model, the number of parameters grows with the size of training data. This is what happens in KNN.

Dropout algorithm

Dropout is a regularization technique where during each iteration of gradient descent, we drop a set of neurons selected at random. By drop, what we mean is that we essentially act as if they do not exist.

The logistic (or sigmoid) function

The term logistic in logistic regression comes from the logistic function (also known as sigmoid function), which can be written as:

f(z)=\frac{1}{1 + e^{-z}} = =\frac{e^z}{e^z + 1}

Generalization, Overfitting and Under-fitting

It's not a good idea to test a machine learning model on a dataset which we used to train it, since it won't give any indication of how well our model performs on unseen data. The ability to perform well on unseen data is called generalization, and is the desirable characteristic we want in a model.

When a model performs well on training data (the data on which the algorithm was trained) but does not perform well on test data (new or unseen data), we say that it has overfit the training data or that the model is overfitting. This happens because the model learns the noise present in the training data as if it was a reliable pattern.

Conversely, when a model does not perform well on tr...

Supervised Learning

Currently, most of the machine learning products use supervised learning. In this, we have a set of features or inputs X (for example, an image) and our model will predict a target or output variable y (for example, caption for the image).

y = f(X)

In other words, our model learns a function that maps inputs to desired outputs. Features are independent variables and targets are the dependent variable.

Classification and Regression

Supervised learning problems can be further grouped ...

Example

Let's start with an example — suppose we have a dataset with information about the area of a house (in square feet) and its price (in thousands of dollars) and our task is to build a machine learning model which can predict the price given the area. Here is what our dataset looks like

Introduction and Overview

Gradient Descent is one of the most popular and widely used optimization algorithms. Given a machine learning model with parameters (weights and biases) and a cost function to evaluate how good a particular model is, our learning problem reduces to that of finding a good set of weights for our model which minimizes the cost function.