0
$\begingroup$

I want to find $\frac{\partial F}{\partial \beta}$ where $F(\beta) = tr((Y-X\beta ^T)(Y-X\beta ^T)^T)$

In case it matters, $Y$ is $m$ by $k$, $X$ is $m$ by $n$ and $\beta$ is $k$ by $n$, and so $(Y-X\beta ^T)(Y-X\beta ^T)^T$ is $m$ by $m$ and so we can define the trace function and everything is ok.

What I tried:

$\frac{\partial F}{\partial \beta} = \frac{\partial}{\partial \beta}tr((Y-X\beta ^T)(Y-X\beta ^T)^T) = tr(\frac{\partial}{\beta}[(Y-X\beta ^T)(Y-X\beta ^T)^T])$ which should be equal to

$tr((Y-X\beta ^T)^T\frac{\partial}{\partial \beta}(Y-X\beta ^T)+(Y-X\beta^T)\frac{\partial}{\partial \beta}(Y-X\beta ^T)^T)$ according to https://en.wikipedia.org/wiki/Matrix_calculus#Scalar-by-matrix_identities

and here I'm stuck. I don't know how to continue.

Edit - a better direction!

$F(\beta) = tr((Y-X\beta ^T)(Y-X\beta ^T)^T) = tr((Y-X\beta ^T)(Y^T-\beta X^T)) = tr(YY^T-Y\beta X^T-X\beta ^TY^T+X\beta ^T\beta X^T)$

and so $\frac{\partial F}{\partial \beta} = \frac{\partial}{\partial \beta}tr(YY^T)+\frac{\partial}{\partial \beta}tr(-Y\beta X^T)+\frac{\partial}{\partial \beta}tr(-X\beta ^TY^T)+\frac{\partial}{\partial \beta}tr(X\beta^T \beta X^T)$

and now im stuck, but this somehow seems more fruitful.

$\endgroup$
1
$\begingroup$

Let $M=(Y-X\beta ^T)(Y-X\beta ^T)^T$. Since $\text{tr}(M)=\sum_{i=1}^mM_{ii}$ we have $$\frac{\partial \text{tr}(M) }{\partial \beta_{st}}=\sum_{i=1}^m\frac{\partial M_{ii} }{\partial \beta_{st}}$$

Now,

$$M_{ii}=\sum_{j=1}^m[Y-X\beta ^T]_{ij}[(Y-X\beta ^T)^T]_{ji}=\sum_{j=1}^m(Y-X\beta ^T)_{ij}(Y-X\beta ^T)_{ij}.$$

By the product rule (or chain rule):

$$\frac{\partial M_{ii} }{\partial \beta_{st}}=2\sum_{j=1}^m(Y-X\beta ^T)_{ij}\frac{\partial (Y-X\beta ^T)_{ij} }{\partial \beta_{st}}=-2\sum_{j=1}^m(Y-X\beta ^T)_{ij}\frac{\partial (X\beta ^T)_{ij} }{\partial \beta_{st}}$$

Since $$(XB^T)_{ij}=\sum_{u=1}^nX_{iu}\beta^T_{uj}=\sum_{u=1}^nX_{iu}\beta_{ju}$$

we have $$\frac{\partial (X\beta ^T)_{ij} }{\partial \beta_{st}}=\begin{cases}X_{st} &j=s\\ 0 &j\neq s\end{cases}$$

Thus $$\frac{\partial M_{ii} }{\partial \beta_{st}}=-2(Y-X\beta ^T)_{is}X_{st}.$$

Finally,

$$\frac{\partial \text{tr}(M) }{\partial \beta_{st}}=-2\sum_{i=1}^m(Y-X\beta ^T)_{is}X_{st}.$$


Or with matrices starting from where you got to: $$\frac{\partial F}{\partial \beta} = \frac{\partial}{\partial \beta}\text{tr}(YY^T)+\frac{\partial}{\partial \beta}\text{tr}(-Y\beta X^T)+\frac{\partial}{\partial \beta}\text{tr}(-X\beta ^TY^T)+\frac{\partial}{\partial \beta}\text{tr}(X\beta^T \beta X^T)$$

Since $\text{tr}(A^T)=\text{tr}(A)$ and the first term is zero, we can write this as

$$\frac{\partial F}{\partial \beta} =- 2\frac{\partial}{\partial \beta}\text{tr}(Y\beta X^T)+\frac{\partial}{\partial \beta}\text{tr}(X\beta^T \beta X^T)$$

The trace is linear, so we can take the derivative inside the trace:

$$\frac{\partial F}{\partial \beta} = -2\text{tr}(\frac{\partial}{\partial \beta}Y\beta X^T)+\text{tr}(\frac{\partial}{\partial \beta}X\beta^T \beta X^T)$$

To make further progress, you might find these useful:

http://www.tc.umn.edu/~nydic001/docs/unpubs/Schonemann_Trace_Derivatives_Presentation.pdf

http://cal.cs.illinois.edu/~johannes/research/matrix%20calculus.pdf

$\endgroup$
  • $\begingroup$ That's all well and good and I arrived at the same result, but I want the solution in matrix form. without summations. I thought maybe deriving with respect to a matrix is easier. $\endgroup$ – Oria Gruber Jul 29 '16 at 18:27
  • $\begingroup$ also, by $\sum _{i=1}^{n}$ you mean $\sum _{i=1}^{m}$ correct? $\endgroup$ – Oria Gruber Jul 29 '16 at 18:38
  • $\begingroup$ I arrived at similar results (meaning the matrix calculus). Is the correct answer overall $-2Y^TX+2I_k \beta X^T X$ ? $\endgroup$ – Oria Gruber Jul 29 '16 at 19:21
1
$\begingroup$

For convenience, define a new matrix $$Z=X\beta^T-Y$$ Writing the function in terms of the Frobenius (:) Inner Product and this new variable makes it almost trivial to find the differential and gradient $$\eqalign{ F &= Z:Z \cr\cr dF &= 2\,Z:dZ \cr &= 2\,Z:Xd\beta^T \cr &= 2\,X^TZ:d\beta^T \cr &= 2\,Z^TX:d\beta \cr\cr \frac{\partial F}{\partial\beta} &= 2\,Z^TX \cr &= 2\,(X\beta^T-Y)^TX \cr\cr }$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.