# Markov chain: Find the probability he won yesterday’s game.

Jones has been playing daily games for a very long time. If Jones wins a game, then he wins the next one with probability 0.6; if he has lost the last game but won the one preceding it, then he wins the next with probability 0.7; if he has lost the last 2 games, then he wins the next with probability 0.2.Today is Monday and Jones has just won his game. Find the probability he won yesterday’s game.

My attempt: I set up the Markov chain like below:

$$\begin{bmatrix}0&0.3&0&0.7\\0&0.8&0&0.2\\0.4&0&0.6&0\\0.4&0&0.6&0\end{bmatrix}$$

Where state 0,1,2,3 are (win,lose),(lose,lose),(win,win) and (lose,win). Then I calculated the stationary distribution which gave me- (𝜋0,𝜋1,𝜋2,𝜋3)= (0.2,0.3,0.3,0.2).

Then I checked if this Markov chain is time reversible, but it is not. I dont know what to do from here.

• The way you've stated the problem is a little confusing. Could you state what the states of your Markov chain are. Also, why do you need reversibility? Commented Aug 17, 2022 at 17:31
• The states are the result of the last two games since the win/lose in the next game depends on the result of the previous two games. About reversibility, I am not quite sure. Since the question is asking to find the probability of winning the game one day before (rather than in future) based on the present state, I thought I might wanna use reversibility. Commented Aug 17, 2022 at 17:37
• I don't understand why you have 4 states. It looks like you should have three only: the outcome of the current game, the outcome of the previous game, and the outcome of the game before the previous one. Commented Aug 17, 2022 at 17:52
• okay, supposing I set up the Markov chain with three states, what should I do to get the probability of winning the game in the past day? Commented Aug 17, 2022 at 19:40
• I think you sent up the transition matrix $P$ just fine. Well done!
– user801306
Commented Aug 18, 2022 at 11:27

$$\newcommand{\prob}{\mathrm{P}}\newcommand{\R}{{\rm I\!R}}$$Essentially you have a Markov chain, where you know $$\prob[x_{t+1} = j | x_t = i] = p_{ij},$$ but you are looking for $$\prob[x_t = i | x_{t+1} = j]$$. Using Baye's rule $$\prob[x_t = i | x_{t+1} = j] = \frac{\prob[x_{t+1} = j | x_t = i] \prob[x_t = i]}{\prob[x_{t+1} = j]} = \frac{p_{ij}\prob[x_t = i]}{\prob[x_{t+1} = j]}.$$ Now the quantities $$\prob[x_t = i]$$ and $$\prob[x_{t+1} = j]$$ depend on the initial distribution and on $$t$$. Can you take it from there?
If the initial distribution is $$v\in\R^3$$, then the unconditional probability at time $$t$$ is $$p_t = (P^t)^\top v$$ and $$\prob[x_{t} = i] = (p_t)_i$$ (the $$i$$-th element of $$p_t$$), or $$\prob[x_{t} = i] = e_i^\top (P^t)^\top v$$ so, overall, $$\prob[x_t = i | x_{t+1} = j] = p_{ij}\frac{e_i^\top (P^t)^\top v }{e_j^\top (P^{t+1})^\top v}$$