1
$\begingroup$

How to compute the gradient of $$\eqalign{ f &= \gamma \ \left( x - 1\mu(x) \right) \ \left( \sigma(x) + \epsilon \right)^{-1/2} + 1\beta \cr \mu(x) &= \alpha \ 1^Tx \cr \sigma(x) &= \alpha \sum_{k=1}^m \left( x_k - \mu(x) \right)^2 \equiv \alpha 1^T \left[ \left( x - 1 \left[\alpha 1^T x\right] \right) \odot \left(x - 1 \left[\alpha 1^T x\right] \right) \right]\cr }$$ where $1^T$ is a row vector with all ones, $\odot$ is an element-wise multiplication, and $\alpha$ and $\epsilon$ are known scalars.


My partial attempt:

I am not sure whether I am following the matrix calculus method, but I share my attempt. So, my intention is to compute the differential firstly and then the gradient.

Following this, the matrix of scaled ones can be defined as \begin{align} J := \alpha 1 1^T \ . \end{align}

Defining another matrix $M := I - J$, where $I$ is an identity matrix.

Also, defining by following such that \begin{align} \sigma(x) := \alpha 1^T \left(M x \odot M x \right) \Longrightarrow d\sigma(x) = 2 \alpha M^2x : dx, \end{align} where double dot $:$ notation is for the inner product.

So, the function can be rewritten as \begin{align} f = \gamma \ M x \ \left( \sigma(x) + \epsilon \right)^{-1/2} + 1\beta \end{align}

Differential is

\begin{align} df &= d\gamma \ M x \ \left( \sigma(x) + \epsilon \right)^{-1/2} \\ & \quad + \gamma M dx \ \left( \sigma(x) + \epsilon \right)^{-1/2} \\ & \quad + \gamma \ M x \ \frac{-1}{2} \left( \sigma(x) + \epsilon \right)^{-3/2} \ d\sigma(x) \\ & \quad + 1d\beta \ , \end{align}

is it correct? if not please help me to correct it.

  1. To compute $\frac{\partial f}{\partial \gamma}$, we set $dx, d\sigma(x), d\beta = 0$ such that \begin{align} \frac{\partial f}{\partial \gamma} = M x \ \left( \sigma(x) + \epsilon \right)^{-1/2} \ . \end{align}

  2. And, \begin{align} \frac{\partial f}{\partial \beta} = 1 \ . \end{align}

  3. To compute $\frac{\partial f}{\partial x}$, we set $d\gamma, d\beta = 0$ \begin{align} &df = \gamma M dx \ \left( \sigma(x) + \epsilon \right)^{-1/2} - \frac{1}{2} \gamma \ M x \ \left( \sigma(x) + \epsilon \right)^{-3/2} \ \left( 2 \alpha M^2x : dx \right) \\ \Longrightarrow \frac{\partial f}{\partial x} &= \gamma M \ \left( \sigma(x) + \epsilon \right)^{-1/2} - \gamma \ M x \ \left( \sigma(x) + \epsilon \right)^{-3/2} \alpha \left( M^2x \right)^T \end{align}

Are the above gradients correct?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.