Deep neural networks and Deep Learning are powerful and popular algorithms. And a lot of their success lays in the careful design of the neural network architecture.

This article revisits the history of neural network design in the last few years and in the context of Deep Learning.