Recurrent Neural Networks and Long Short-Term Memory Networks
What is a Recurrent Neural Net?
Recurrent neural networks (RNNs) are neural networks designed to analyze sequences of inputs, which traditional feed-forward neural networks are not good at modeling. Text comprehension, translation, motor control and speech are all instances of problems where either the input or the output (or both) is a sequence. Depending on the problem, the sequences can be of arbitrary length and have complex time dependencies.
RNNs are built on the same computational unit as feed forward nets (neurons), but differ in the architecture of how these neurons are connected to one another. In feed forward neural networks, information flows unidirectionally from input units to output units. In particular, there are no cycles in the neural network. In RNNs, cycles are allowed. In fact, neurons are even allowed to be connected to themselves. Since the previous vector of activities is used to compute the vector of activities in each time step, RNNs are able to retain m...
Let's say you're working on a cool image processing project and your goal is to build an algorithm that analyzes faces for emotions. It takes in a 256 pixel by 256 pixel grayscale image as its input and spits out an emotion as an answer. For example, if you passed in the following image, you'd expect the algorithm to label it as "happy."
A greyscale input image of a happy man
Now this is all well and good, but before we're sati...
What is deep learning? How does it relate to machine learning?
Deep learning is a subfield of machine learning, i.e. in deep learning we are dealing with algorithms which learn from examples, similar to machine learning. The main difference between deep learning and machine learning is that deep learning models have a notion of multiple layers or multiple levels of hierarchy, which opens up the possibility of being able to learn models for more complicated tasks.
The following are some different ways of thinking ab...
Let's say you're working on a cool image processing project, and your goal is to build an algorithm that analyzes faces for emotions. It takes in a 256 pixel by 256 pixel grayscale image as its input and spits out an emotion as an answer. For example, if you passed in the following image, you'd expect the algorithm to label it as "happy."
Last time, we talked about the traditional feed-forward neural net and concepts that form the basis of deep learning. These ideas are extremely powerful! We saw how feed-forward convolutional neural networks have set records on many difficult tasks including handwritten digit recognition and object classification. And even today, feed-forward neural networks consistently outperform virtually all other approaches to solving classification tasks.
And yet, despite their well celebrated successes, most experts would agree that feed-forward neural nets are still rather limited in what they can achieve. Why? Because the task of "classification" is only one small component of the incredible computational power of the human brain. We're wired not only to recognize individual instances but to also analyze entire sequences of inputs. These sequences are ultra rich in information, have complex time dependencies, and can be of arbitrary length. For example, vision, motor control, speech, and comprehension all require us to process high-dimensional inputs as they change over time. This is something that feed-forward nets are incredibly poor at modeling.
Deep learning. Neural networks. Backpropagation. Over the past year or two, I've heard these buzz words being tossed around a lot, and it's something that has definitely seized my curiosity recently. Deep learning is an area of active research these days, and if you've kept up with the field of computer science, I'm sure you've come across at least some of these terms at least once.
Deep learning can be an indimidating concept, but it's becoming increasingly important these days. Google's already making huge strides in the space with the Google Brain project and its recent acquisition of the London-based deep learning startup DeepMind. Moreover, deep learning methods are beating out traditional machine learning approaches on virtually every single metric.
So what exactly is deep learning? How does it work? And most importantly, why should you even care?
Note to the Reader
If you're new to computer science, and you've follow...