Help on some results with Markov chains I currently have a markov chain represented by the following matrix:
$$\left[\begin{array}{ccc} 0&1&0 \\ .99&0&.01 \\ 0&0&1 \end{array}\right],$$
i.e. a row-stochastic matrix, with uniform probability on the initial state
distribution. I am trying to figure out the probability that the hitting time
for state $3$ is finite. I am not completely certain about how to go about
these calculations, but I would imagine this may have something to do with
the absorbent state qualities of state $3$. Any suggestions on how to go about
this kind of problem?
 A: Let $$\tau=\inf\{n\geqslant 0: X_n=3\} $$ and  $$\tau_i = \inf\{n\geqslant 0: X_n=3\mid X_0=i\}.$$ Conditioning on $X_1$ yields the system of equations
\begin{align}
\mathbb E[\tau_1] &= 1 + \mathbb E[\tau_2]\\
\mathbb E[\tau_2] &= 1+ \frac{99}{100}\mathbb E[\tau_1]\\
\mathbb E[\tau_3] &= 0,
\end{align}
and hence $\mathbb E[\tau_1]=200$, $\mathbb E[\tau_2]=199$. It follows that
$$\mathbb E[\tau] = \frac13(200 + 199) = 133<\infty. $$
As @ByronSchmuland pointed out, this technique requires the assumption that $\tau$ is integrable. So we may instead compute the distribution of $\tau$ directly. Clearly $\mathbb P(\tau_3=0)=1$, and further 
\begin{align}
\mathbb P(\tau_1=2)&=P_{12}P_{23} = \frac1{100}\\
\mathbb P(\tau_2=1)&=P_{23}=\frac1{100}.
\end{align} Since $P_{11}=P_{22}=0$, it follows that $$\mathbb P(\tau_1=2(n+1))= (P_{12}P_{21})^nP_{12}P_{23} = \left(\frac{99}{100}\right)^n\frac1{100}$$
and similarly
$$\mathbb P(\tau_2=2n+1) = (P_{21}P_{12})^nP_{23} = \left(\frac{99}{100}\right)^n\frac1{100} $$ for $n=1,2,\ldots$. Therefore 
\begin{align}
\mathbb E[\tau_1]&=\sum_{n=0}^\infty 2(n+1)\left(\frac{99}{100}\right)^n \frac1{100}\\
&=\frac1{50}\sum_{n=0}^\infty (n+1)\left(\frac{99}{100}\right)^n\\
&=\frac{100^2}{50}\\
&=200,
\end{align}
and similarly
$$\mathbb E[\tau_2] = \sum_{n=0}^\infty(2n+1)\left(\frac{99}{100}\right)^n \frac1{100}=199. $$
We conclude that
\begin{align}
\mathbb E[\tau] &= \sum_{i=1}^3 \mathbb E[\tau_i]\mathbb P(X_0=i)\\
&= \frac13(\mathbb E[\tau_1]+\mathbb E[\tau_2]+\mathbb E[\tau_3])\\
&= \frac13\left(200+199+0\right)\\
&= 133.
\end{align}
A: State 3 is absorbing, and from state 1 the chain always moves to state 2. So to never hit state 3, the chain must either (A) start in state 1 and bounce back and forth with state 2, or (B) start in state 2 and bounce back and forth with state 1. Both of these events have probability zero: Event $A$ implies that the chain starts at $1$ and the next $2n$ visits are $(2,1,2,\ldots,1,2,1)$; this has probability $\frac13(1)^n(.99)^n$ (after appropriate grouping of terms), so
$$
P(A) \le \frac13(.99)^n.$$
This holds for every $n$, hence $P(A)=0$. A similar argument shows $P(B)=0$. Therefore the probability is zero that the chain never hits state 3, and the prob is one that the hitting time for state 3 is finite.
