Broadly, Natural Language Processing (or NLP for short) consists of developing a set of algorithms and tools so that machines can make sense of data available in natural (human) languages such as English, German, French, etc. Although there are traces of NLP research since a long time ago, the concept got well defined in the 1950s, with the breakthrough research work of Alan Turing and Noam Chomsky.

This tutorial introduces you to the various applications, challenges and approaches within NLP. In this course, we'll build many of these applications and encounter the challenges described.

What is Natural Language?

Natural language refers to the medium in which humans communicate with each other. This could be in the form of writings (text) for example emails, articles, news, blogs, bank documents, etc or speech for example lectures, speeches, audio calls, etc. There are trillions of web pages full of natural text, so imagine the scale of data available today.

NLP algorithms often model the hierarchical structur...

Automatic spelling correction in an important and interesting problem in NLP. This article walks you through the approaches and algorithms that can be used to understand automatic spell checking and correction.

Companies like Microsoft and Google have achieved high accuracy in spelling correction, which forms an integral part of their editors, email clients, word processors and search engines. In addition, spell checking is also often used as a pre-processing step in NLP to clean up text data before feeding it to machine learning algorithms for topic classification, search engine ranking, etc.

Spell Checking Illustration in Python

Here is a small python code that does spell checking and suggests the closest correction.

from autocorrect import spell

What is natural language identification?

Language Identification is the task of computationally determining the language of some given piece of text data. A text document could be written entirely in a single language such as English, French, German, Spanish (monolingual language identification), or each text document could have multiple languages in different parts.

Why is language identification important?

Language identification is important for most NLP applications to work accurately, since models are usually trained using data from a single language. If a model is trained on English text and is used for prediction on French text, we usually see a significant decrease in accuracy. In applications such as spam filtering, machine t...

What are word clouds?

A word cloud (also called tag cloud) is a data visualization technique which highlights the important textual data points ...

Introduction

Named Entity Recognition is one of the very useful information extraction technique to identify and classify named entities in text. These entities are pre-defined categories such a person’s names, organizations, locations, time representations, financial elements, etc.

Apart from these generic entities, there could be other specific terms that could be defined given a particular problem. These terms represent elements which have a unique context compared to the rest of the text. For example, it could be anything like operating systems, programming languages, football league team names etc. The machine learning models could be trained to categorize such custom entities which are usually denoted by proper names and therefore are mostly noun phrases in text documents.

NER Categories

Broadly NER has three top-level categorizations - en...

We got introduced to text syntax and structures and took a detailed look at part of speech tagging in part 1 of this tutorial series. In this tutorial, we will learn about phrasal structure and shallow parsing.

Phrasal Structures

A phrase can be a single word or a combination of words based on the syntax and position of the phrase in a clause or sentence. For example, in the following sentence

My dog likes his food.

there are three phrases. "My dog" is a noun phrase, "likes" is a verb phrase, and "his food" is also a noun phrase.

There are five major categories of phrases:

1. Noun phrase (NP): These are phrases where ...

We took a detailed look at part of speech tagging in part 1 and chunking in part 2 of this tutorial series. In this tutorial, we will learn about what parsing is, its different types, and techniques to automatically infer the parse tree of sentences.

Deep Parsing

Natural language parsing (also known as deep parsing) is a process of analyzing the complete syntactic structure of a sentence. This includes how different words in a sentence are related to each other, for example, which words are the subject or object of a verb. Probabilistic parsing uses language understanding such as grammatical rules or may be a supervised training set of hand-parsed sentences to try to infer the most likely syntax and structure of new sentences.

Parsing is used to solve various complex NLP problems such as conversational dialogues and text summarization. It is different from 'shallow parsing' in that it yields more expressive structur...