# Learning = Representation + Evaluation + Optimization

September 29, 2017

This tutorial describes the important components of a learning algorithm: **representation** (what the model looks like), **evaluation** (how do we differentiate good models from bad ones), and **optimization** (what is our process for finding the good models among all the possible models).

# Representation

Choosing a representation of a learner is tantamount to choosing the set of classifiers that it can possibly learn. This set is called **hypothesis space.**

**Example:** In the case of linear regression, the **representation** of our model is a **linear function**. Regardless of what weights *w* we choose, our model will be a linear function.

**Key considerations**: Is the scenario you are trying to capture well represented by the model function? Is it overly restrictive? Is it overly flexible? For example, if the data has a quadratic trend, but we are trying to fit a linear function to it, we are being overly restrictive.

# Evaluation

An evaluation or **objective function** (or **cost function**) is needed to distinguish good classifiers from bad ones.

**Example**: In the case of least-squares linear regression, the cost function was **mean-squared error** cost function.

**Key considerations**: Does your cost function capture the relative importance of different kinds of mistakes? For example, is being off by 0.3 for one data point and 0.1 for another data points better or worse than being off by 0.2 for both data points? Is a *false positive* as bad as a *false negative*?

# Optimization

The process or algorithm for finding the best model in hypothesis space.

**Example**: **Gradient descent** is an optimization algorithm that can be used for finding the optimal model for least-squares linear regression.

**Key considerations**:

- How efficient is the optimization technique in practice?
- Does it always find the optimal solution? Is it possible for it to output sub-optimal solutions? If yes, how often does it happen in practice?

# Relation between representation, evaluation and optimization

Note that the choosing any one component of the machine learning algorithm (say the optimization method) depends on our choices for the other components.

For example, gradient descent can only be applied if the cost function is differentiable, i.e. it is a smooth function (since we need to calculate the gradients).

Another more involved example is, assuming that we have a non-convex cost function, does gradient descent get stuck on local minima?

# Examples for each of the components

The table below lists a few examples of each of the components of a learning algorithm. It’s okay if you haven’t heard many of the names before, but see if you can spot ones you have heard of.

*Examples of each of the components of a learning algorithm. Source: A few useful things to know about machine learning*

# Conclusion

You will find it useful to come back to this article as you go through the journey of learning more machine learning algorithms. In particular, when you learn a new algorithm, ask yourself what **representation** does it use, what **cost function** does it use, and what is the **optimization** process.