3
$\begingroup$

How to obtain the gradients of such a complicated Deep network with batch normalization (preferably in matrix/vector notation) \begin{align} L\left(\left\{W_\ell, \gamma_\ell, \beta_\ell\right\}_{\ell=1}^3\right) := \sum_{i=1}^N \| g_3 \left(W_3 \ f_2 \left( g_2 \left(W_2 \ f_1 \left(g_1\left(W_1 x_i \right)\right) \right) \right) \right) - y_i \|_2^2 , \end{align} with respect to $\left\{W_\ell, \gamma_\ell, \beta_\ell\right\}_{\ell=1}^3$, where $g_\ell$ is parameterized by $\gamma_\ell$ and $\beta_\ell$?

The definition of $x_i \in \mathbb{R}^n$, $W_1 \in \mathbb{R}^{m \times n}$, $W_2 \in \mathbb{R}^{p \times m}$, $W_3 \in \mathbb{R}^{q \times p}$, and $y_i \in \mathbb{R}^q$, and $f_\ell(z) = \frac{1}{1 + \exp(-z)}$.

And, $$\eqalign{ g_\ell(z; \gamma_\ell, \beta_\ell) &= \gamma_\ell \frac{\left( z- \mu(z) \right)}{\left( \sigma(z) + \epsilon \right)^{1/2}} + \beta_\ell\cr \mu(z) &= \alpha \ 1^Tz \cr \sigma(z) &= \alpha \sum_{k=1}^m \left( z[k] - \mu(z) \right)^2 \equiv \alpha 1^T \left[ \left( z- \mu(z) \right) \odot \left(z - \mu(z) \right) \right]\cr }$$ where $1^T$ is a row vector with all ones, $\odot$ is an element-wise multiplication, and $\alpha$ and $\epsilon$ are known scalars.


Note:

The gradients of the cost function without $g_i(\cdot)$, batch normalization, is given in the link. But how to address it with complicated batch normalization.


Thank you so much in advance for your help

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.