I have a loss function of the followoing form:

$L(\mathbf{a}) = \|\mathbf{b} - \mathbf{a}\|_2^2$

Where, $\mathbf{a}$ and $\mathbf{b}$ are vectors of dimension $d\times 1$. I need to calculate $\frac{\partial L}{\partial \mathbf{a}}$

If I am correct $\frac{\partial L}{\partial \mathbf{a}} = -2(\mathbf{b} - \mathbf{a})^T$ (I am not sure)

I have two questions:

  1. What is derivation of $\frac{\partial L}{\partial \mathbf{a}}$
  2. I know that $\frac{\partial L}{\partial \mathbf{a}}$ is a vector of dimension $1\times d$ instead of $d\times 1$ ($L$ is a scalar and $\mathbf{a}$ is a vector). In that case how can I update $\mathbf{a}$ based on gradient descent (because of different dimensionality)? The update rule is: $\mathbf{a} = \mathbf{a} - \beta \frac{\partial L}{\partial \mathbf{a}}$ ? ($\beta$ is the learning rate)

1 Answer 1

  1. You need the Matrix Cookbook. But honestly, it is not that difficult to derive this particular case by looking at each element: $$\frac{\partial L}{\partial a_i} = \frac{\partial}{\partial a_i}\sum_{j=1}^n(b_j-a_j)^2 = 2(b_i-a_i)\cdot -1 = -2(b_i-a_i).$$ From here, you simply have $$\frac{\partial L}{\partial \mathbf{a}} = \begin{bmatrix} \frac{\partial L}{\partial a_1} & \frac{\partial L}{\partial a_2} & \dots & \frac{\partial L}{\partial a_d} \end{bmatrix}=-2(b-a)^T.$$So on this point you are correct!
  2. The actual update rule is $\mathbf{a} = \mathbf{a} - \beta \nabla L(\mathbf{a})$, where $\nabla L$ is the gradient of $L$. The gradient is the transpose of this quantity $\frac{\partial L}{\partial \mathbf{a}}$ above. The above quantity is sometimes called the Jacobian, but Jacobians are usually considered for multi-valued functions $f:\mathbb{R}^n\rightarrow\mathbb{R}^m$, where there is quite a material difference between an $m\times n$ matrix instead of an $n\times m$ matrix. For scalar valued functions, people are often a bit looser with the association between the gradient and the Jacobian, and often use them interchangeably, even if they shouldn't.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .