$A^TA$ in summation notation $A=a_{ij}$, $A^T=a_{ji}$, $A^TA=\sum_ka_{jk}a_{kj}$ but $A^TA$ is a matrix not a vector. What's going on?
Is there anything wrong in my proof?
 A: Index notation
The index notation can be confusing from time to time, especially when one chooses unfortunate index variable names. The notation $A=a_{ij}$ is an abbreviation of
$$(*)\qquad A=(a_{ij})_{i,j=1,...,n}$$
and should only be used when it is clear from context which index-variable iterates over columns and which over rows. You implicitely made the assumption that $i$ iterates over the columns and $j$ over the rows. This gets you into trouble when applying the matrix multiplication as will be explained below.
This longer notation $(*)$ has the advantage that it clearly states that the first variable mentioned in $i,j=1,...,n$ iterates over the rows and the the second one mentioned in $i,j=1,...,n$ iterates over the columns.
Now, you introduced $A^\top=a_{ji}$ with the unwritten assumption that the long form is
$$A^\top=(a_{ji})_{i,j=1,...,n},$$
i.e. that $i$ still iterates over the rows even though it appears second in $a_{ji}$. You could also have done $A^\top=(a_{ij})_{j,i=1,...,n}$ which would be very confusing in the shorter form.

Matrix multiplication
The matrix multiplication $A^\top A$ gets applied by using the same variable (e.g. $k$) for


*

*the column-index of the left matrix $A^\top$, and 

*the row-index of the right matrix $A$.  


Then you sum over $k$. It turns out that the column-index of $A^\top=a_{ji}$ is $j$, and not $i$. Even though $j$ appears first in $a_{ji}$ this does not matter, because what is important is its position in $i,j=1,...,n$ (this was were you made the mistake in your proof). Of course the row-index of $A$ is $i$. Hence we have
$$A^\top A=\sum_k a_{ki}a_{kj}.$$

Just names
Also, index variables names are just this $-$ names. Whenever you come into conflict, you change them to be distinct. The $i$ and $j$ of $A=a_{ij}$ have little to do with the $i$ and $j$ in $A^\top=a_{ji}$ and can be independently replaced by any other pair, e.g. $x$ and $y$.
