How to obtain the gradients of such a complicated Deep network with batch normalization (preferably in matrix/vector notation) \begin{align} L\left(\left\{W_\ell, \gamma_\ell, \beta_\ell\right\}_{\ell=1}^3\right) := \sum_{i=1}^N \| g_3 \left(W_3 \ f_2 \left( g_2 \left(W_2 \ f_1 \left(g_1\left(W_1 x_i \right)\right) \right) \right) \right) - y_i \|_2^2 , \end{align} with respect to $\left\{W_\ell, \gamma_\ell, \beta_\ell\right\}_{\ell=1}^3$, where $g_\ell$ is parameterized by $\gamma_\ell$ and $\beta_\ell$?

The definition of $x_i \in \mathbb{R}^n$, $W_1 \in \mathbb{R}^{m \times n}$, $W_2 \in \mathbb{R}^{p \times m}$, $W_3 \in \mathbb{R}^{q \times p}$, and $y_i \in \mathbb{R}^q$, and $f_\ell(z) = \frac{1}{1 + \exp(-z)}$.

And, $$\eqalign{ g_\ell(z; \gamma_\ell, \beta_\ell) &= \gamma_\ell \frac{\left( z- \mu(z) \right)}{\left( \sigma(z) + \epsilon \right)^{1/2}} + \beta_\ell\cr \mu(z) &= \alpha \ 1^Tz \cr \sigma(z) &= \alpha \sum_{k=1}^m \left( z[k] - \mu(z) \right)^2 \equiv \alpha 1^T \left[ \left( z- \mu(z) \right) \odot \left(z - \mu(z) \right) \right]\cr }$$ where $1^T$ is a row vector with all ones, $\odot$ is an element-wise multiplication, and $\alpha$ and $\epsilon$ are known scalars.


The gradients of the cost function without $g_i(\cdot)$, batch normalization, is given in the link. But how to address it with complicated batch normalization.

Thank you so much in advance for your help


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.