Introduction

A novel regularization is proposed that enforces the output of a model to be isotropically smooth around each input data point irrespective of the availability of the training label. This is achieved by investigating the local anisotropy around each input data point and smoothing the model in the most anisotropic direction.

Method

The approach to investigate the local anisotropy around each data point is to find the direction of a small perturbation that the model is most sensitive to. The sensitivity of the model is measured in terms of the KL divergence between the output class probability distribution given the input and the input plus the perturbation. The desired perturbation is called the 'virtual adversarial perturbation'. The model is supposed to minimize the supervised loss (cross entropy, MSE etc) as well as the sensitivity to the virtual adversarial perturbation.

Intuition

One way to think about why this works is that VAT pushes the decision boundary of the classifier away from regions that have high data density. Otherwise the classifier would be highly sensitive to local perturbations on data points lying close to the decision boundary. This results in improved generalization.

Results

Competitive results compared to the state of the art are reported for supervised and semi-supervised learning tasks on images. Do check out the full paper and my PyTorch implementation reproducing the results mentioned in the paper for semi-supervised learning on SVHN and CIFAR10 datasets.