5
$\begingroup$

What does the convolution operation do in convolutional neural network?

An interviewer told me that you can look at each convolutional filter as a vector and when it convolves with a matrix, it is essentially doing a dot product, giving you a cosine distance. Is that right? I think it is only true when both vectors are normalised to 1, but it is not the case in convolutional neural network, where we can't control the weights in the filter since they are learnt through gradient descent. Also, this doesn't seem to explain why CNN is able to learn well.

$\endgroup$
0
$\begingroup$

Dot product and cosine similarity

The dot product of two vectors $a, b$ of same length $n$ is defined as $$\sum_{i=1}^n a_i b_i$$

The cosine similarity is defined as $$\frac{ \sum\limits_{i=1}^{n}{a_i b_i} }{ \sqrt{\sum\limits_{i=1}^{n}{a_i^2}} \sqrt{\sum\limits_{i=1}^{n}{b_i^2}} }$$

So the cosine similarity is essentially a normed version of the dot product.

To compute element $(i, j)$ of the feature map resulting from a convolution with a $3 \times 3$ filter $w$ applied to a feature map $F^{(0)}$, one calculates

$$\sum_{x=-1}^1 \sum_{y=-1}^1 F^{(0)}_{x+i,y+j} \cdot w_{x, y}$$

You can, of course, find a flattened version of $F$ / $w$ so that this is a dot product. I'm not too sure how this is implemented in cuDNN. So although it is true what he says, I'm not sure how useful it is to think of convolutional layers like this.

What Conv-Layers learn

Commonly, people say that CNNs learn edge detectors. For such questions / statements, people often refer to Visualizing and Understanding Convolutional Networks

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.