# Gradients of Deep Neural Network With Batch Normalization

How to obtain the gradients of such a complicated Deep network with batch normalization (preferably in matrix/vector notation) \begin{align} L\left(\left\{W_\ell, \gamma_\ell, \beta_\ell\right\}_{\ell=1}^3\right) := \sum_{i=1}^N \| g_3 \left(W_3 \ f_2 \left( g_2 \left(W_2 \ f_1 \left(g_1\left(W_1 x_i \right)\right) \right) \right) \right) - y_i \|_2^2 , \end{align} with respect to $$\left\{W_\ell, \gamma_\ell, \beta_\ell\right\}_{\ell=1}^3$$, where $$g_\ell$$ is parameterized by $$\gamma_\ell$$ and $$\beta_\ell$$?

The definition of $$x_i \in \mathbb{R}^n$$, $$W_1 \in \mathbb{R}^{m \times n}$$, $$W_2 \in \mathbb{R}^{p \times m}$$, $$W_3 \in \mathbb{R}^{q \times p}$$, and $$y_i \in \mathbb{R}^q$$, and $$f_\ell(z) = \frac{1}{1 + \exp(-z)}$$.

And, \eqalign{ g_\ell(z; \gamma_\ell, \beta_\ell) &= \gamma_\ell \frac{\left( z- \mu(z) \right)}{\left( \sigma(z) + \epsilon \right)^{1/2}} + \beta_\ell\cr \mu(z) &= \alpha \ 1^Tz \cr \sigma(z) &= \alpha \sum_{k=1}^m \left( z[k] - \mu(z) \right)^2 \equiv \alpha 1^T \left[ \left( z- \mu(z) \right) \odot \left(z - \mu(z) \right) \right]\cr } where $$1^T$$ is a row vector with all ones, $$\odot$$ is an element-wise multiplication, and $$\alpha$$ and $$\epsilon$$ are known scalars.

Note:

The gradients of the cost function without $$g_i(\cdot)$$, batch normalization, is given in the link. But how to address it with complicated batch normalization.