Does $X((X^TX)^{-1}X^Ty)$ equal $X(X^{-1}y)$? Let $X$ be some matrix and $y$ be a vector of appropriate length.
I've seen someone use the expression 
$X(X^{-1}y)$
instead of
$X((X^TX)^{-1}X^Ty)$
in the context of ordinary least squares estimation.
Are these expressions equal? and if so, can someone walk me through the process of getting from the latter to the former?
Thank you
 A: Yes, they are equal, and both equal to $y$. Here are the equalities you are searching for:
$$
X((X^TX)^{-1}X^Ty) = X(X^{-1}(X^T)^{-1}X^Ty) = X(X^{-1}y) = XX^{-1}y = y.
$$
A: The two expressions are not in general equal. However, the context of least-squares is actually quite a special case, and several assumptions are typically made. Most importantly, in the linear least-squares problem
$$
Xy = b,
$$
it is often assumed that $X$ is a matrix of full column rank (that is, the map $y \mapsto Xy$ is injective). This is because this is the case in which the linear least-squares problem has a unique least-squares solution $\hat{y}$, such that
$$
\|X\hat{y} - b\|_2 \le \| Xy - b\|_2
$$
for every vector $y$.
When it is indeed the case that $X$ has full column rank, then $X^T X$ is an invertible matrix, and therefore $(X^T X)^{-1} X^T$ is a left-inverse, also called a pseudo-inverse, to $X$. This is only a true inverse if $X$ is square. Typically, to distinguish from the true inverse, we use a different symbol, such as
$$
X^{+} : = (X^T X)^{-1} X^T.
$$
With this notation, then the linear least-squares problem has the unique solution $\hat{y} = X^{+} b$. The bottom-line, then, is that $X^{+}$ gives the general solution to the problem, but when you know that $X$ is square and invertible, then you can naturally replace this with $X^{-1}$.
A: In general, they are not equal. For one thing, the design matrix $X$ would have to be square in order for $XX^{-1}$ to even make sense. Also, $X$ need not be invertible even if it is square. OTOH, in the rare instance that $X$ is square and invertible, then the identity holds, as proved by Hugo.
