Markov chains exercise Let the transition matrix be defined as follows
The state space is defined as follow $E$={1,2,3,4}
$$
\begin{bmatrix}
0&1/2&1/2&0\\
1/4&1/2&0&1/4\\
2/3&0&0&1/3\\
0&2/3&1/3&0\\
\end{bmatrix}
$$

*

*What is the probability of getting from state 1 to state 2 in two steps?

*Assuming $X_0=1$, find the probability that the chain will be in state 2 before 4 occurs

*Assuming that $X_3 = 1$, calculate the expected value of the time to state 2.

Any help?
 A: 1/The probability of getting from state 1 to state 2 in two steps can be calculated by multiplying the transition matrix by itself. In this case, it would be:
P(1,2) = [0 1/2 1/2 0] * [1/4 1/2 0 1/4] = [1/4 1/2 0 1/4]
So, the probability of getting from state 1 to state 2 in two steps is 1/2.
2/To find the probability of being in state 2 before 4 occurs, we need to find the probability of being in state 2 first and then the probability of not being in state 4.
Let's denote the state vector at time step n as X_n. Then, the transition from X_n to X_{n+1} is given by X_{n+1} = X_n * P, where P is the transition matrix.
Starting from X_0 = [1, 0, 0, 0], the state vectors at time step 1, 2, and 3 are given by:
X_1 = X_0 * P = [0, 1/2, 1/2, 0]
X_2 = X_1 * P = [0, 1/4, 1/2, 1/4]
X_3 = X_2 * P = [0, 1/3, 1/3, 1/3]
The probability of being in state 2 before 4 occurs is the sum of the probability of being in state 2 at each time step before 4 occurs, i.e. P(X_n = 2, n < 4) = P(X_1 = 2) + P(X_2 = 2) + P(X_3 = 2) = 1/2 + 1/4 + 0 = 3/4.
3/The expected value of the time to reach state 2 can be calculated as follows:
Let's define T2 as the time to reach state 2 and X as the state of the chain at time t.
The expected value of T2 can be calculated as:
E[T2|X3=1]= ∑ t=1 to infinity (t * P(T2=t | X3=1))
where P(T2=t | X3=1) is the probability of reaching state 2 at time t given that X3=1.
To calculate the probability of reaching state 2 at time t given X3=1, we need to calculate the probability of being in state 2 at time t and having been in state 1 at time t-3.
P(T2=t | X3=1) = P(Xt=2, Xt-3=1) = P(Xt=2 | Xt-3=1) * P(Xt-3=1)
We can calculate these probabilities using the transition matrix and the initial distribution (X0=1).
This process can be repeated for each t to calculate E[T2|X3=1].
