What is a categorical feature?

In statistical modeling and machine learning, a categorical variable is a variable which can only have a fixed set of values. Some examples of categorical variables are nationality, size of clothes (small, medium, large), day of the week, genre of music, educational qualification (doctorate, graduate, diploma), etc.

Categorical Feature labels

As opposed to a continuous numerical variable such as height, age, and distance, the above variables are not intrinsically represented by continuous numbers. Instead they represented by labels. Each unique value in a categorical variable is known as a label.

For example, if the categorical variable was Size of clothing, the labels would be small, medium and large. For the categorical feature Educational qualification, some of the possible labels would be doctorate, graduate, diploma, etc.

Types of categorical features

Broadly, there are two types of categorical features: Ordinal and Nominal.

Ordinal: Ordinal variables are categorical variables whose values have a sense of order associated with them.

For example the size of a T-shirt: small, medium, large. The values have an order associated with them based on size.

Another example can be found in survey datasets, which often have responses in the Likert scale: strongly agree, agree, neutral, disagree and strongly disagree. These are ordered in terms of the respondent's level of agreement.

Nominal: Values from nominal variables do not have any order associated with them.

For example nationality, colour, sex, etc.

Strategies to deal with Categorical data

In machine learning, datasets often have categorical features.

However, many machine learning algorithms, including linear regression and logistic regression, require input features to be numerical. So we need to find a way around categorical features, transforming them into numerical features.

The strategy we use for this depends on if the categorical variable is ordinal or nominal. We use label encoding for ordinal variables, and one hot encoding for nominal variables.

Label encoding (used for Ordinal Variables)

One possible way of transforming categorical values to numerical values is to assign a numerical value to every label.

For example, let us take a look at the example dataset below. We have records of people with their ID, Education, Sex and Nationality.

Snapshot of dataset with categorical features

For the Education variable, we could assign numeric values to the labels as follows: doctorate → 3, graduate → 2 and diploma → 1. We will add another column for the numerical variable called Numerical_Ed.

Snapshot of dataset with label encoding for Education variable

Since the numerical values assigned to each label have an intrinsic order, the machine learning model will see the qualifications in an order: Diploma → Graduate → Doctorate. This means having a Doctorate is more similar to having a Graduate degree, than having a Diploma. This is desirable and works only if the categorical variable is ordinal.

However, this also illustrates why we should not use this strategy for categorical variables which are nominal. For example, suppose we assign numerical values to Nationality as follows: Canadian → 1, Indian → 2 and British → 3. Then, our model will consider being a Canadian as more similar to being an Indian than British.

Similarly, as we add more countries, the model would make more and more assumptions about which countries are more similar to which other countries, etc. This is not desirable. Hence, for nominal categorical variables, we apply a different technique called one hot encoding to avoid this problem.

One hot encoding (used for Nominal Variables)

In this method, we encode each label of the categorical feature as a one hot encoding vector. Lets see what this means with some examples.

Let us take the feature Sex. The only labels we see occurring for this feature are male and female. So we will create two columns — one for male and one for female. The values in these columns will be binary: 1 if the column name matches the value of the categorical feature, 0 otherwise.

Let us look at the male and female columns created with one hot encoding:

Columns created when performing One hot encoding for Sex feature

We can see the male column has value 1 when the value in the Sex column has the label male, and similarly for the female column.

Let us do the same for nationality column.

Columns created when performing One hot encoding for Nationality feature

Number of One Hot Encoding vector columns:

If you look at the column Sex again and the result of one hot encoding, you will notice that one of the One Hot Encoded columns is redundant.

There are only two possible labels for the Sex variable, male and female. If we just consider the male column, it has enough information to tell us the original value for the Sex variable (1 implies male and 0 implies female).

In general, if there are m possible labels for a categorical feature, then m-1 One Hot Encoded columns are enough to represent all the labels.

Summary

A categorical variable is a variable which can only have a fixed set of values.
Values of a categorical variable are represented by labels.
There are two types of categorical variables

Ordinal variables — when the values have a sense of order associated with them.
Nominal variables — when the values do not have a sense of order associated with them.

To transform categorical variables to numerical features, we use:

label encoding for ordinal variables, i.e. assign a numerical value to every label
one hot encoding for nominal variables, i.e. we create a binary column for every label, which is 1 if the column name matches the value of the categorical variable and 0 otherwise.

If there are m possible labels for a categorical feature, then m-1 One Hot Encoded columns are enough to represent all the labels.