Can someone explain the derivative of a trace of a matrix with respect to a matrix? $$\frac {d \operatorname{tr}{A}}{d A} = I$$
I just don't see how this works. For instance, if we have a matrix A that contains purely numbers with a trace of 12. Then we would take the derivative of 12 for each matrix element A, for instance A11 =1 . Then we would have $${d{12}{d 1}$$ , how would this be equal to 1, is this not equal to zero, because taking a derivative w.r.t. a constant is always zero right?
I feel like there is something wrong with my reasoning of matrix derivations...
Then I dont see how the derivative w.r.t. to A11 would differ a lot from for example A12. Both are matrix elements containing a value, so how will one be zero and the other equal to 1?
 A: We are treating the trace as a function $\operatorname{tr} \colon \mathbb{R}^{n^2} \to \mathbb{R}$, and computing partial derivatives with respect to the entries of the matrix. For example, we have
$$ \operatorname{tr} \begin{pmatrix} x_{11} & x_{12} \\ x_{21} & x_{22} \end{pmatrix} = x_{11} + x_{22},$$
and therefore the partial derivative with respect to $x_{11}$ or $x_{22}$ is $1$, and the derivative with respect to the off-diagonal entries is zero. We can then write that down using funky "matrix derivative" notation as
$$ \partial \operatorname{tr} = \begin{pmatrix}
\frac{\partial \operatorname{tr}}{\partial x_{11}} &
\frac{\partial \operatorname{tr}}{\partial x_{11}} \\
\frac{\partial \operatorname{tr}}{\partial x_{11}} &
\frac{\partial \operatorname{tr}}{\partial x_{11}} \end{pmatrix} = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} = I.$$
A: The definition of the derivative in this case is (typing out the $2\times 2$ case for simplicity):
$$
\frac{d\operatorname{tr}A}{dA} = \begin{bmatrix}\dfrac{d\operatorname{tr}A}{dA_{11}} & \dfrac{d\operatorname{tr}A}{dA_{12}}\\
\dfrac{d\operatorname{tr}A}{dA_{21}}&\dfrac{d\operatorname{tr}A}{dA_{22}}\end{bmatrix}
$$
Let's take the top left entry first. That entry asks how much the trace changes if we change $A_{11}$. And the answer is that the trace changes by exactly the same amount. Thus $\dfrac{d\operatorname{tr}A}{dA_{11}} = 1$.
On the other hand, what about $\dfrac{d\operatorname{tr}A}{dA_{12}}$? If we change $A_{12}$, that doesn't change the trace of $A$ at all. This means that $\dfrac{d\operatorname{tr}A}{dA_{12}} = 0$.
Fill in the rest of the entries of the derivative completely analogously, and you will get the identity matrix out as a result.
