Stochastic Processes: Markov Chains I would like to know how to solve question 3

 A: As said by Sandipan: If you are in states $19, 20$ or $21$, there is no way to go to any other states. For these 3 states, you have the transition matrix $$A=\begin{pmatrix}\frac12&\frac12&0\\ \frac12&0&\frac 12 \\ 0 & \frac12 & \frac12\end{pmatrix}.$$ (Formally, you can go from the original chain to this $3$-state chain and back using properties of block matrices.) The matrix $A$ is symmetric, so it is unitarily diagonalizable (cf. page 307 of [1]).
Using the standard diagonalization procedure (cf. page 236f. of [1]), we get $$A = S D S^{-1},$$ where $$S=\left(\begin{array}{ccc} 1 & -1 & 1 \\ -2 & 0 & 1 \\ 1 & 1 & 1 \\\end{array}\right)$$ and $$D=\left(\begin{array}{ccc} -\frac{1}{2} & 0 & 0 \\ 0 & \frac{1}{2} & 0 \\ 0 & 0 & 1 \\\end{array}\right)$$ and $$S^{-1}=\left(\begin{array}{ccc} \frac{1}{6} & -\frac{1}{3} & \frac{1}{6} \\ -\frac{1}{2} & 0 & \frac{1}{2} \\ \frac{1}{3} & \frac{1}{3} & \frac{1}{3} \\\end{array}\right).$$
Indeed we also notice that with respect to any norm on the space of $3\times 3$ matrices, $D^n\to\operatorname{diag}(0,0,1)$ for $n\to\infty$, so also as mentioned by Sandipan, as $n\to\infty$, we have $$A^n\to S\operatorname{diag}(0,0,1) S^{-1}=\begin{pmatrix}\frac 13&\frac 13&\frac 13\\\frac 13&\frac 13&\frac 13\\\frac 13&\frac 13&\frac 13\end{pmatrix}.$$
Multiplying this with any $$\begin{pmatrix}a\\b\\c\end{pmatrix}\in\mathbb R^3$$ gives $$\frac{a+b+c}3\begin{pmatrix}1\\1\\1\end{pmatrix}.$$ So indeed if we start with any probability measure on the states $19,20,21$ (i.e. $a+b+c=1$ and $a,b,c\in[0,1]$), we will end up asymptotically in equilibrium through this Markov chain.

Furthermore this gives an easy answer to question 3, you just need to compute $$S D^5 S^{-1} \begin{pmatrix}\frac12\\0\\\frac12\end{pmatrix}.$$
Literature
[1]: Gerd Fischer, Lineare Algebra, edition 17 (2009).
