Link to paper: Accelerating Learning via Knowledge Transfer
This paper presents a simple method to accelerate the training of larger neural networks by initializing them with parameters from a trained, smaller network. Networks are made wider or deeper while preserving the same output as the smaller network which maintains performance when training starts, leading to faster convergence. Main contributions:
- Initialize layers with identity weight matrices to preserve the same output.
- Only works when activation function f satisfies f(If(x)) = f(x) for example ReLU, but not sigmoid, tanh.
- Additional units in a layer are randomly sampled from existing units. Incoming weights are kept the same while outgoing weights are divided by the number of replicas of that unit so that the output at the next layer remains the same.
- Experiments on ImageNet
- Net2Deeper and Net2Wider models converge faster to the same accuracy as networks initialized randomly.
- A deeper and wider model initialized with Net2Net from the Inception model beats the validation accuracy (and converges faster).
- The Net2Net technique avoids the brief period of low performance that exists in methods that initialize some layers of a deeper network from a trained network and others randomly.
- This idea is very useful in production systems which essentially have to be lifelong learning systems. Net2Net presents an easy way to immediately shift to a model of higher capacity and reuse trained networks.
- Simple idea, clearly presented.
- The random mapping algorithm for different layers was done manually for this paper. Developing a remapping inference algorithm should be the next step in making the Net2Net technique more general.
- The final accuracy that Net2Net models achieve seems to depend only on the model capacity and not the initialization. I think this merits further investigation. In this paper, it might just be because of randomness in training (dropout) or noise added to the weights of the new units to approximately represent the same function (when not using dropout).