What is transition matrix 
Every e_j har coordinates in the first base:
$$e_j = \sum_{i} s_{ij}e_i $$
so writing those coordinates as column vectors we get an important transition matrix $S = (s_{ij})$

and

Theorem: if a vector $v$ has the coordinates $X$ in the basis $e_1, \ldots, e_n$ and the coordinates $Y$ in the basis $e'_1,\ldots ,e'_n$ then
$$X=SY\iff Y=S^{-1}X$$
where $S$ is the transition matrix.

Could anyone explain short about the transition matrix and how I can prove the theorem. The book (Matrix theory) is not saying much about the transition matrix so I have no idea how it works.
 A: You can read more on such transformations in the Wikipedia article on coordinate changes. A simple example listed there is the transformation from the "usual" $x,y$ coordinate system to a $x',y'$ coordinate system obtained by rotating the standard basis counterclockwise for 45°. (The plot, and part of the explanation, is taken from here ).

In $x',y'$, there is still a vector $\overrightarrow{u'} = (1,0)$ and another vector $\overrightarrow{v'} = (0, 1)$ that form a basis (any other vector in these coordinates can be expressed as a linear combination of these two). 
But surely these are not the same vectors we had in $x,y$! If we project the new unit vectors onto the old axes, we get that they have the following coordinates in the old system:
$$ \tag{1}
\overrightarrow{u'}_{old} = \begin{bmatrix}\cos \theta \\ \sin \theta \end{bmatrix}  = \cos \theta \overrightarrow{u} + \sin \theta \overrightarrow{v} \\
\overrightarrow{v'}_{old} = \begin{bmatrix} -\sin \theta \\ \cos \theta \end{bmatrix} = -\sin \theta \overrightarrow{u} + \cos \theta \overrightarrow{v}
$$
These two equations tell us how to express the new vectors in terms of the old vectors, i.e transforming back from the new coordinates. We can put them together into a single matrix to transform any arbitrary vector expressed in terms of $u', v'$ back to $u, v$:
$$ \tag{2}
\overrightarrow{a'} = w_{1} \overrightarrow{u'} + w_{2} \overrightarrow{v'} \\
= w_{1} (\cos \theta \overrightarrow{u} + \sin \theta \overrightarrow{v}) + w_{2} (-\sin \theta \overrightarrow{u} + \cos \theta \overrightarrow{v}) \\
= (w_{1} \cos \theta - w_{2} \sin \theta) \overrightarrow{u} + (w_{1} \sin \theta + w_{2} \cos \theta) \overrightarrow{v}\\
= \begin{bmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{bmatrix}
\begin{bmatrix} w_{1} \\ w_{2}\end{bmatrix} \\
= \mathbf{S} \overrightarrow{a}
$$
Here we have identified the matrix as the transition matrix $\mathbf{S}$.
In order to go from the old coordinates to the new ones, just remember that a matrix multiplied with its inverse matrix gives the identity. Multiply both the left hand side of $(2)$ and its last right hand side term by $\mathbf{S^{-1}}$
$$ \tag{3}
\mathbf{S^{-1}} \overrightarrow{a'} = \mathbf{S^{-1}} \mathbf{S} \overrightarrow{a} \\
= \mathbb{1} \overrightarrow{a}
$$
which means, leaving out the identity and switching left hand side for right hand side:
$$ \overrightarrow{a} = \mathbf{S^{-1}} \overrightarrow{a'} $$
