I've always wondered about deep learning's history, and had read in several articles that neural networks and back-propagation used to be popular in 1980s. Didn't know that there was another deep learning phase in the 1960s!
Hey, I just started with the book and I have a question on formula 2.68. Here the constraint
is stated. But this would mean that D has to be a orthogonal matrix with n=l. And this would miss the whole point of the compression. At the beginning of section 2.12 it is explained that l<n, columns of D are orthogonal and
Since shape(D) is (n,l), and transpose of D is (l,n), after left multiply D with its transpose, the output shape will be(l,l), and the constraint of all columns of D are orthonormal is essential if you wanna simplify the matrix myltiplication
Read more… (41 words)
Read more (41 words)
Chapter 3: Probability and Information Theory
If you're reading Deep Learning Book, feel free to ask questions, discuss the material or share resources.
Below is a video from a book club in San Francisco, USA discussing this chapter. Presented by Pierre Dueck.