NaN.

Paper Summary: Spatial Transformer Networks

Link to paper: [1506.02025] Spatial Transformer Networks

This paper introduces a neural networks module that can learn input-dependent spatial transformations and can be inserted into any neural network. It supports transformations like scaling, cropping, rotations, and non-rigid deformations. Main contributions:

The spatial transformer network consists of the following:

Localization network that regresses to the transformation parameters given the input.
Grid generator that uses the transformation parameters to produce a grid to sample from the input.
Sampler that produces the output feature map sampled from the input at the grid points.

Differentiable sampling mechanism

The sampling is written in a way such that sub-gradients can be defined with respect to grid coordinates.
This enables gradients to be propagated through the grid generator and localization network, and for the network to jointly learn the spatial transformer along with rest of the network.

A network can have multiple STNs

at different points in the network, to model incremental transformations at different levels of abstraction.
in parallel, to learn to focus on different regions of interest. For example, on the bird classification task, they show that one STN learns to be a head detector, while the other focuses on the central part of the body.

Strengths

Their attention (and by extension transformation) mechanism is differentiable as opposed to earlier works on non-differentiable attention mechanisms that used reinforcement learning (REINFORCE). It also supports a richer variety of transformations as opposed to earlier works on learning transformations, like DRAW.
State-of-the-art classification performance on distorted MNIST, SVHN, CUB-200-2011.

Weaknesses / Notes

This is a really nice way to generalize spatial transformations in a differentiable manner so the model can be trained end-to-end. Classification performance, and more importantly, qualitative results of the kind of transformations learnt on larger datasets (like ImageNet) should be evaluated.

Copied