I am currently a bit in doubt about the definition of a convolution...

I just been reading the paper which describes convolution as way of transforming

input image -> "convolution" -> Convolution map

But the way they describe it in the math seems a bit off in my head - They seem to describe it as it was an neural network would get an input * weight + bias.

This is how they describe it math wise.

$$q_{j,m} =\sigma\left(\sum_{c = 0}^{\text{total}~ \text{number} ~ \text{column}} \sum_{r = 0}^{\text{total}~\text{number}~\text{rows}} \text{img}(r,c) \text{W}_{c,j,r} + W_{0,j}\right)$$

And the complete convolution map $Q_j$, is computed as:

$$Q_j = \sigma \left( \sum_{c = 0}^{\text{total}~\text{number} ~\text{columns}} \text{img}(:,c) \ast W_{c,j} \right) $$

img(r,c) is the pixel unit stored in row = r and column = c. img(:,c) is a row vector for column = c of the image.

And W is a weight matrix of size $\left( (C \cdot R) \times J\right)$

\begin{bmatrix} w_{111} & w_{121} & x_{131} & \dots & x_{1J1} \\ \vdots & \vdots & \vdots & \vdots & \vdots \\ w_{C11} & w_{I21} & x_{C31} & \dots & x_{CJ1} \\ \vdots & \vdots & \vdots & \vdots & \vdots \\ w_{C12} & w_{C22} & w_{C32} & \dots & x_{CJ2} \\ \vdots & \vdots & \vdots & \vdots & \vdots \\ w_{C1R} & w_{C2R} & w_{C3R} & \dots & x_{CJR} \\ \end{bmatrix}

$W_{c,j,r}$ is one unit in the weight matrix and $W_{c,j}$ is a 1d vector of length $\left(C \cdot R \right)$

Where is the actual convolution being stated?.. where is kernel size specified, and the stride. I guess the formula is a standard convolution, but where in the formula is the stride and kernel size defined?

I guess they are using multiple convolutions, which J describes. So the weight asscociated with one convolution is with weight vector $W_{c,j}$

About weight sharing! enter image description here

Given an image - Convolution is applied which reduces the depth of the image Is the weight connecting the convoluted image to the an output neutron the same or different?

Meaning will the weight be updated as one unit, or as an individual unit,

  • $\begingroup$ All you need to know is that in convolutional neural networks, we can compute the convolution of the output of the layer $L$ with a filter $h$ to obtain $L \ast h$ that we use as the input for the neurons of the next layer. Of course, the weights of $h$ are optimized as if they were the weights of a neuron. $\endgroup$ – reuns May 9 '17 at 0:30
  • $\begingroup$ ... That isn't that helpful.. Thanks for trying though.. But i wanted to know how the weight are shared. I guess CNN reduces the depth of the image, and doing the convolution gives a reduced number of neuron which is then connected to a conv_feature map. Would that then mean that if the convolution result in one value, there would only be one connection? $\endgroup$ – ASD May 9 '17 at 0:49
  • $\begingroup$ No, the convolution doesn't give a reduced number of number of values, but you can add a decimate step. $L$ is an array of $n \times m$ values (the output of $nm$ neurons) and $L \ast h$ (decimated or not) is another array $n_2 \times m_2$ used as the input of $n_2 m_2$ neurons. That's what you need to understand. During the backpropagation, you'll need to compute the gradient with respect to the coefficients of $h$ accordingly. $\endgroup$ – reuns May 9 '17 at 1:03
  • $\begingroup$ hmm... i guess I will add a visual representation of how i understand it.. $\endgroup$ – ASD May 9 '17 at 1:32

In a general neural networks, two consecutive layers $u \in \mathbb{R}^N,v\in \mathbb{R}^M$ are related by $v = f(M u)$ (or $v = f(Mu+b)$ if you want a bias) where $M$ is a linear operator $\mathbb{R}^N\to \mathbb{R}^M$ (a matrix $n \times m$) and $f(x)_i = \tanh(x_i)$. The weights are the $M_{i,j}$.

In a convolutionnal layer, it is the same except $Mu$ is replaced by $u \ast h\ \ $ (or $d(u\ast h)$ where $d$ is a decimation operator).

So a convolutionnal layer is just a particular case of the general layer, the only difference being the number of weights, how they are shared among the layer, and how you'll compute the gradient (during the backpropagation) accordingly.

  • $\begingroup$ but how are they shared.. Is my visualisation correct? $\endgroup$ – ASD May 9 '17 at 1:51
  • $\begingroup$ @ASD Share what with what ?.. I can't be more clear than what I wrote. $\endgroup$ – reuns May 9 '17 at 1:56
  • $\begingroup$ It is the last three lines I am referring to.. How are the weight being shared, the number of weight? and whether backpropagation trains all weight or see them as the same weight? This ultimately leading me to defining the kernel size. $\endgroup$ – ASD May 9 '17 at 2:04
  • $\begingroup$ @ASD I can't help you deciphering it, you need to read your .pdf with what I wrote in mind. In convolutional neural networks there are a lot of parameters indicating the "topology of our network" ie the number of layers, the size of the filter, $n$ and $m$, ... $\endgroup$ – reuns May 9 '17 at 2:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.