# Gradient of $f = \gamma \ \left( x - 1\mu(x) \right) \ \left( \sigma(x) + \epsilon \right)^{-1/2} + 1\beta$ w.r.t. $x$, $\gamma$, and $\beta$

How to compute the gradient of \eqalign{ f &= \gamma \ \left( x - 1\mu(x) \right) \ \left( \sigma(x) + \epsilon \right)^{-1/2} + 1\beta \cr \mu(x) &= \alpha \ 1^Tx \cr \sigma(x) &= \alpha \sum_{k=1}^m \left( x_k - \mu(x) \right)^2 \equiv \alpha 1^T \left[ \left( x - 1 \left[\alpha 1^T x\right] \right) \odot \left(x - 1 \left[\alpha 1^T x\right] \right) \right]\cr } where $$1^T$$ is a row vector with all ones, $$\odot$$ is an element-wise multiplication, and $$\alpha$$ and $$\epsilon$$ are known scalars.

My partial attempt:

I am not sure whether I am following the matrix calculus method, but I share my attempt. So, my intention is to compute the differential firstly and then the gradient.

Following this, the matrix of scaled ones can be defined as \begin{align} J := \alpha 1 1^T \ . \end{align}

Defining another matrix $$M := I - J$$, where $$I$$ is an identity matrix.

Also, defining by following such that \begin{align} \sigma(x) := \alpha 1^T \left(M x \odot M x \right) \Longrightarrow d\sigma(x) = 2 \alpha M^2x : dx, \end{align} where double dot $$:$$ notation is for the inner product.

So, the function can be rewritten as \begin{align} f = \gamma \ M x \ \left( \sigma(x) + \epsilon \right)^{-1/2} + 1\beta \end{align}

Differential is

\begin{align} df &= d\gamma \ M x \ \left( \sigma(x) + \epsilon \right)^{-1/2} \\ & \quad + \gamma M dx \ \left( \sigma(x) + \epsilon \right)^{-1/2} \\ & \quad + \gamma \ M x \ \frac{-1}{2} \left( \sigma(x) + \epsilon \right)^{-3/2} \ d\sigma(x) \\ & \quad + 1d\beta \ , \end{align}

1. To compute $$\frac{\partial f}{\partial \gamma}$$, we set $$dx, d\sigma(x), d\beta = 0$$ such that \begin{align} \frac{\partial f}{\partial \gamma} = M x \ \left( \sigma(x) + \epsilon \right)^{-1/2} \ . \end{align}
2. And, \begin{align} \frac{\partial f}{\partial \beta} = 1 \ . \end{align}
3. To compute $$\frac{\partial f}{\partial x}$$, we set $$d\gamma, d\beta = 0$$ \begin{align} &df = \gamma M dx \ \left( \sigma(x) + \epsilon \right)^{-1/2} - \frac{1}{2} \gamma \ M x \ \left( \sigma(x) + \epsilon \right)^{-3/2} \ \left( 2 \alpha M^2x : dx \right) \\ \Longrightarrow \frac{\partial f}{\partial x} &= \gamma M \ \left( \sigma(x) + \epsilon \right)^{-1/2} - \gamma \ M x \ \left( \sigma(x) + \epsilon \right)^{-3/2} \alpha \left( M^2x \right)^T \end{align}