# A question on Random Walks and Markov Chain

Suppose that $(X_n,n \in N)$ is a Markov chain whose transition probability matrix is given by

$P=\begin{bmatrix}\alpha & 1-\alpha \\1-\beta & \beta \end{bmatrix}$

(a) Deduce that $Z_n =(X_{n-1},X_n)$ is a Markov Chain with the four states $(0,0)$ $(0,1)$ $(1,0)$ $(1,1)$

(b) Determine the transition probability matrix of $(Z_n,n \in N)$

• What are the states for $(X_n)$?
– asdf
May 12 '17 at 17:29
• that is all i have been given May 12 '17 at 17:31

Ok, I'll assume that $(X_n)$ has state space $\{0,1\}$ and is time homogeneous.

What you have to prove in a) is the following: $P(Z_n=i_n|Z_{n-1}=i_{n-1},Z_{n-2}=i_{n-2}...Z_{0}=i_{0}) = P(Z_{n}=i_{n}|Z_{n-1}=i_{n-1})$, where $i_{k}$ is one of $\{(0,0), (0,1),(1,0), (1,1)\}$. This is called the Markov Property. It roughly means that the future depends only on the present, not the past.

a) This means that in your case, no matter what happened before the current $Z_n$, i.e. whatever the previous $Z_i$'s were, the transition probability depends only on $Z_{n}$. Try to formalize your argument for this particular case.

As for b) the transition probabilities $P(Z_n=i_n|Z_{n-1}=i_{n-1})$ mean what is the chance of $Z_{n}$ being equal to $i_{n}$ given $Z_{n-1}=i_{n-1}$ For example if $Z_{n-1}=(0,0)$ the transition probability $P(Z_n=(0,1)|Z_{n-1}=(0,0)) = P(X_{n-1}=0, X_{n}=1| X_{n-2}=0, X_{n-1}=0) = P(X_{n}=1|X_{n-1}=0)=1 -\alpha$

Note how we used the Markov property.

You will have to end up with a 4 by 4 matrix (1 row and 1 column for each state) and if you've done it correctly the rows will have to sum up to $1$. I can be of assistance if it's not too urgent.

• I came up with the transition matrix $\begin{bmatrix}\alpha & 1-\alpha &0&0 \\0 & 0 &1-\beta& \beta \\ \alpha & 1-\alpha & 0 & 0 \\ 0 & 0 & 1-\beta & \beta \end{bmatrix}$ May 12 '17 at 19:29
• Looks right, but it depends how you've listed the rows and columns
– asdf
May 12 '17 at 19:46