Introduction

A novel regularization is proposed that enforces the output of a model to be isotropically smooth around each input data point irrespective of the availability of the training label. This is achieved by investigating the local anisotropy around each input data point and smoothing the model in the most anisotropic direction.

Method

The approach to investigate the local anisotropy around each data point is to find the direction of a small perturbation that the model is most sensitive to. The sensitivity of the model is measured in terms of the KL divergence between the output class probability distribution given the input and the input plus the perturbation. The desired perturbation is called the 'virtual adversarial perturbation'. The model is supposed to minimize the supervised loss (cross entropy, MSE etc) as well as the sensitivity to the virtual adversarial perturbation.

Intuition

One way to think about why this works is that VAT pu...

This is a great paper that addresses the vanishing/exploding gradient problem existing in deep neural network architecture. The solution proposed in the paper is a deep residual learning framework which allows for training extremely deep CNN models for various visual recognition tasks. The architecture consists of stacked convolutional layers, with every other layer connected with two layers below. In this way, every two layers are trained to approximate a residual function of an underlying mapping.

The claim made in the paper is that learning some underlying mapping H(x) is asymptotically approximate to learning the residual function [H(x)-x] and then adding x, but that the latter is easier to learn with several layers of a neural network. This intuition isn't very clear. Section 3.1 discusses this intuition, and I was wondering if someone could help me understand this.

Some further questions and observations:

• This framework doesn't seem to have several fully connected layers at the end, as VGG/AlexNet papers d...

Prerequisites: Standard Convolutional Neural Networks (CNNs). For Neural Machine Translation background read this reply to this post.

Motivation 1: Why character by character?

• Robust to handling spelling mistakes, typos.
• Handling long words that have sub-structure.

Motivation 2: Why multi-lingual? (one model for multiple languages)

• Language switching! i.e. using an English phrase in the middle of a French sentence (real-world scenario)
• Reduce over-fitting (when languages have similar characteristics)

Challenge = Computation

Computation is quadratic in length of source sentenc...

Training an LSTM (NTM—Neural Turing Machine—is tested too) to output an update for a steepest descent iteration at each time step. Aptly titled “Learning to learn by gradient descent by gradient descent”.

Anybody interested in trying Learning to learn to learn by gradient descent by gradient descent by gradient descent?

A clockwork RNN is a recurrent neural network (RNN) architecture designed to remember things easily over long periods of time. It is a follow up on Long Short-Term Memory architecture (also by Jurgen Schmidhuber). 

In my opinion, the paper is quite interesting and deserves more attention than it has received.