# Simple matrix calculus - can't find differential

I want to find $\frac{\partial F}{\partial \beta}$ where $F(\beta) = tr((Y-X\beta ^T)(Y-X\beta ^T)^T)$

In case it matters, $Y$ is $m$ by $k$, $X$ is $m$ by $n$ and $\beta$ is $k$ by $n$, and so $(Y-X\beta ^T)(Y-X\beta ^T)^T$ is $m$ by $m$ and so we can define the trace function and everything is ok.

What I tried:

$\frac{\partial F}{\partial \beta} = \frac{\partial}{\partial \beta}tr((Y-X\beta ^T)(Y-X\beta ^T)^T) = tr(\frac{\partial}{\beta}[(Y-X\beta ^T)(Y-X\beta ^T)^T])$ which should be equal to

$tr((Y-X\beta ^T)^T\frac{\partial}{\partial \beta}(Y-X\beta ^T)+(Y-X\beta^T)\frac{\partial}{\partial \beta}(Y-X\beta ^T)^T)$ according to https://en.wikipedia.org/wiki/Matrix_calculus#Scalar-by-matrix_identities

and here I'm stuck. I don't know how to continue.

Edit - a better direction!

$F(\beta) = tr((Y-X\beta ^T)(Y-X\beta ^T)^T) = tr((Y-X\beta ^T)(Y^T-\beta X^T)) = tr(YY^T-Y\beta X^T-X\beta ^TY^T+X\beta ^T\beta X^T)$

and so $\frac{\partial F}{\partial \beta} = \frac{\partial}{\partial \beta}tr(YY^T)+\frac{\partial}{\partial \beta}tr(-Y\beta X^T)+\frac{\partial}{\partial \beta}tr(-X\beta ^TY^T)+\frac{\partial}{\partial \beta}tr(X\beta^T \beta X^T)$

and now im stuck, but this somehow seems more fruitful.

## 2 Answers

Let $M=(Y-X\beta ^T)(Y-X\beta ^T)^T$. Since $\text{tr}(M)=\sum_{i=1}^mM_{ii}$ we have $$\frac{\partial \text{tr}(M) }{\partial \beta_{st}}=\sum_{i=1}^m\frac{\partial M_{ii} }{\partial \beta_{st}}$$

Now,

$$M_{ii}=\sum_{j=1}^m[Y-X\beta ^T]_{ij}[(Y-X\beta ^T)^T]_{ji}=\sum_{j=1}^m(Y-X\beta ^T)_{ij}(Y-X\beta ^T)_{ij}.$$

By the product rule (or chain rule):

$$\frac{\partial M_{ii} }{\partial \beta_{st}}=2\sum_{j=1}^m(Y-X\beta ^T)_{ij}\frac{\partial (Y-X\beta ^T)_{ij} }{\partial \beta_{st}}=-2\sum_{j=1}^m(Y-X\beta ^T)_{ij}\frac{\partial (X\beta ^T)_{ij} }{\partial \beta_{st}}$$

Since $$(XB^T)_{ij}=\sum_{u=1}^nX_{iu}\beta^T_{uj}=\sum_{u=1}^nX_{iu}\beta_{ju}$$

we have $$\frac{\partial (X\beta ^T)_{ij} }{\partial \beta_{st}}=\begin{cases}X_{st} &j=s\\ 0 &j\neq s\end{cases}$$

Thus $$\frac{\partial M_{ii} }{\partial \beta_{st}}=-2(Y-X\beta ^T)_{is}X_{st}.$$

Finally,

$$\frac{\partial \text{tr}(M) }{\partial \beta_{st}}=-2\sum_{i=1}^m(Y-X\beta ^T)_{is}X_{st}.$$

Or with matrices starting from where you got to: $$\frac{\partial F}{\partial \beta} = \frac{\partial}{\partial \beta}\text{tr}(YY^T)+\frac{\partial}{\partial \beta}\text{tr}(-Y\beta X^T)+\frac{\partial}{\partial \beta}\text{tr}(-X\beta ^TY^T)+\frac{\partial}{\partial \beta}\text{tr}(X\beta^T \beta X^T)$$

Since $\text{tr}(A^T)=\text{tr}(A)$ and the first term is zero, we can write this as

$$\frac{\partial F}{\partial \beta} =- 2\frac{\partial}{\partial \beta}\text{tr}(Y\beta X^T)+\frac{\partial}{\partial \beta}\text{tr}(X\beta^T \beta X^T)$$

The trace is linear, so we can take the derivative inside the trace:

$$\frac{\partial F}{\partial \beta} = -2\text{tr}(\frac{\partial}{\partial \beta}Y\beta X^T)+\text{tr}(\frac{\partial}{\partial \beta}X\beta^T \beta X^T)$$

To make further progress, you might find these useful:

http://www.tc.umn.edu/~nydic001/docs/unpubs/Schonemann_Trace_Derivatives_Presentation.pdf

http://cal.cs.illinois.edu/~johannes/research/matrix%20calculus.pdf

• That's all well and good and I arrived at the same result, but I want the solution in matrix form. without summations. I thought maybe deriving with respect to a matrix is easier. – Oria Gruber Jul 29 '16 at 18:27
• also, by $\sum _{i=1}^{n}$ you mean $\sum _{i=1}^{m}$ correct? – Oria Gruber Jul 29 '16 at 18:38
• I arrived at similar results (meaning the matrix calculus). Is the correct answer overall $-2Y^TX+2I_k \beta X^T X$ ? – Oria Gruber Jul 29 '16 at 19:21

For convenience, define a new matrix $$Z=X\beta^T-Y$$ Writing the function in terms of the Frobenius (:) Inner Product and this new variable makes it almost trivial to find the differential and gradient \eqalign{ F &= Z:Z \cr\cr dF &= 2\,Z:dZ \cr &= 2\,Z:Xd\beta^T \cr &= 2\,X^TZ:d\beta^T \cr &= 2\,Z^TX:d\beta \cr\cr \frac{\partial F}{\partial\beta} &= 2\,Z^TX \cr &= 2\,(X\beta^T-Y)^TX \cr\cr }