Derivative of scalar multiple of matrix by scalar Let X be a n x m matrix and let $\gamma$ be a scalar. What is the derivative of $\gamma X$ with respect to $\gamma$ and X is not a function of $\gamma$? ie
$$
\frac{\partial \gamma X}{\partial \gamma} = ?
$$
I think it might be X but I always thought that the dimensions stay the same, meaning, the derivative should also be a scalar since $\gamma$ is scalar. Any hints would be appreciated!
 A: Let $$W=\gamma X-Y$$ Write a function in terms of the norm of this new variable, and find its derivative 
$$\eqalign{
 f &= \|W\|_F^2 = W:W \cr\cr
df &= 2W:dW = 2W:X\,d\gamma \cr\cr
\frac{df}{d\gamma} &= 2W:X = 2(\gamma X-Y):X \cr\cr
}$$
Set the derivative to zero and solve for $\gamma$
$$\eqalign{
 \gamma X:X &= Y:X \,\,\implies\,\, \gamma &= \frac{Y:X}{X:X} \cr\cr
}$$
The colon represents the Frobenius product, which is a convenient infix notation for the trace, i.e. $$\eqalign{A:B={\rm tr}(A^TB)\cr\cr}$$
If you actually wanted to minimize the function
$$h=\|W^TW\|_F^2$$ 
the process would be similar -- but this function seems unnatural.
$$\eqalign{
 \frac{dh}{d\gamma} &= 4WW^TW:X = 0 \cr
0 &= (\gamma X-Y)(\gamma X-Y)^T(\gamma X-Y):X \cr
}$$
This is a cubic polynomial in $\gamma$ whose coefficients are complicated scalar products of the $(X,Y)$ matrices. It seems really unlikely that this is what you were after.
A: If $X,Y$ are fixed members in the banach space $\mathbf{E} := \mathbb{R}^{n \times m}$ and $$h : \mathbb{R} \to \mathbf{E} \,\,\, ; \,\,\, t \mapsto (Y - tX)^T(Y - tX)$$
then for a fixed real number $t$ the total derivative $Dh(t) : \mathbb{R} \to \mathbf{E}$ is the continuous linear transformation that sends
$$ s \mapsto s(-Y^TX - X^TY + 2tX^TX).$$
In otherwords $Dh(t) = -Y^TX - X^TY + 2tX^TX$.
