Derivative of transpose I am trying to find derivative of this :
RQ(u) = uTXTXu / uTu
I need help finding derivative : RQ/u
Optimal sol should satisfy XTXu = RQ(u)u
I am very confused, any help would be great. If you could share some references that will be very helpful too. 
 A: We have $ y = u^T X^T X u $. We compute the differential of $ y$
$$
dy = (du)^T X^T X u + u^T X^T X (du)  \\
dy = Tr ((du)^T X^T X u + u^T X^T X (du))  \\
dy = Tr ( u^T X^T X (du)) + Tr( u^T X^T X (du) ) \\
dy = 2 Tr ( u^T X^T X (du)) \\
dy = 2 (u^T X^T X)^T: du
$$
Finally $dy/du = 2 (u^T X^T X)^T = 2 X^T X u $
Additional information
Here I have used two properties of the trace:
1) $ Tr(ABC) = Tr(BCA) = Tr(CAB) $
2) $ Tr(c) = c $, where $c$ is an scalar
A: The quantity you are interested in is known as the Rayleigh Quotient. We have
$$ 
\frac{\partial}{\partial u} \frac{u^T X^T X u}{u^T u}
= \frac{1}{u^T u} \frac{\partial}{\partial u}(u^T X^T X u) + (u^T X^T X u) \frac{\partial}{\partial u}(u^T u)^{-1}
$$
Now use $\frac{\partial}{\partial u} u^T A u = 2Au$ for symmetric matrix $A$:
$$
\implies\frac{\partial}{\partial u} RQ(u)
= \frac{2X^T X u }{u^T u} - \frac{ (u^T X^T X u) 2 I u}{(u^T u)^2} = \frac{2}{u^T u }\big(X^T X - RQ(u)\cdot I\big)u
$$
In particular, if $u$ is a maximizer/minimizer of $RQ$ then
$$ 0 = \frac{\partial}{\partial u} RQ(u) \implies \big(X^T X - RQ(u)\cdot I\big)u =0 $$
If you are unsure about the matrix derivatives, I recommend this online tool: http://www.matrixcalculus.org/, and for fomulas wikipedia and The Matrix Cook Book
