1
$\begingroup$

First, an example:

Suppose $A$ is a symmetric matrix $A = \begin{bmatrix} a & b \\ b & a \\ \end{bmatrix}$, and $\underline c = \begin{bmatrix} c_1 \\ c_2 \end{bmatrix}$

Then $A \underline c = \begin{bmatrix} ac_1 + b c_2 \\ bc_1 + ac_2 \\ \end{bmatrix}$

and hence, $\frac{\partial}{\partial \underline c}(A\underline c) = \begin{bmatrix} \frac{\partial}{\partial c_1}(ac_1 + b c_2) \\ \frac{\partial}{\partial c_2}(bc_1 + ac_2) \\ \end{bmatrix} = \begin{bmatrix} a \\a \end{bmatrix} \neq A$

However, I'm told that the derivative with respect to the vector of the product of a symmetric $(n\times n)$ matrix $A$ and an $(n\times 1)$ vector $\underline c$ is equal to the matrix $A$, such that,

$\frac{\partial}{\partial \underline c}(A\underline c) = A$

Where is the error in my understanding?

$\endgroup$
1
  • $\begingroup$ Others have answered the question, but I thought I should point out that your $A$ is not the most general $2\times 2$ symmetric matrix. The two terms on the diagonal, both $a$ in your example, can be distinct. $\endgroup$ Jul 23, 2019 at 13:58

3 Answers 3

4
$\begingroup$

You have a vector result $$ y = Ac$$

Thus, you are computing a vector-by-vector gradient which results in a second-order tensor, ie. a matrix, with the following form:

$$ \frac{\partial y}{\partial c}= \begin{bmatrix} {\frac {\partial y_{1}}{\partial c_{1}}}&{\frac {\partial y_{1}}{\partial c_{2}}}\\ {\frac {\partial y_{2}}{\partial c_{1}}}&{\frac {\partial y_{2}}{\partial c_{2}}}\\ \end{bmatrix} $$

You are just missing the off-diagonal terms.

$\endgroup$
2
$\begingroup$

Since your function is vector valued, its derivative is a matrix : $\left(\frac{\partial \mathbf{u}}{\partial\mathbf{x}}\right)_{i,j} = \frac{\partial u_i}{\partial x_j}$.

Hence, $\frac{\partial A \mathbf{c}}{\partial \mathbf{c}} = \begin{bmatrix} \frac{\partial a c_1 + b c_2}{\partial c_1} & \frac{\partial a c_1 + b c_2}{\partial c_2} \\ \frac{\partial b c_1 + a c_2}{\partial c_1} & \frac{\partial b c_1 + a c_2}{\partial c_2}\end{bmatrix} = \begin{bmatrix} a & b\\ b & a\end{bmatrix}$.

Remark : in general, $\frac{\partial A \mathbf{c}}{\partial \mathbf{c}} = A^T $, but here your matrix is symmetric (i.e. $A=A^T$).

$\endgroup$
0
$\begingroup$

Your definition of the derivative $ \frac{\partial}{\partial \overline{c}}. $

The function $$\overline{c} \mapsto A \overline{c}$$ is total differentiable if there exists a linear map $B$, with $$ \lim_{\|h\| \rightarrow 0} \frac{\|A\left(\overline{c}+\overline{h}\right) - A\overline{c} - B\overline{h}\|}{\|\overline{h}\|} = 0 $$.

Since the map is linear, it is obvious that $A = B$, so the total derivative is $A$.

$\endgroup$
2
  • $\begingroup$ I'm computing a little and getting \begin{align} ... = \lim_{||h|| \rightarrow 0} \frac{|| -A\bar{h}-B\bar{h} ||}{||\bar{h}||}=0 \end{align} can you explain how this requires $A=B$? $\endgroup$
    – djkat
    Jul 23, 2019 at 13:43
  • $\begingroup$ Yes sorry, I accidentally exchanged the $A\left(\overline{c} + \overline{h}\right)$ and the $A \overline{c}$ terms... Edited it now to bring it into the right order. $\endgroup$
    – Cadlag
    Jul 23, 2019 at 13:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .