transition matrix Markov chain An individual either drives his car or walks in going from his home (h) to his office (o) in the morning, and from his office to his home in the afternoon. He uses the following strategy:
If it is raining in the morning, then he drives the car, provided it is at home to be taken. Similarly, if it is raining in the afternoon and his car is at the office, then he drives the car home.
If the car is not where he is, our man walks independently of the weather.
If it is not raining and the car is where he is, the man can decide to walk and the probability of this choice is $\alpha$ independently of anything else, $1 - \alpha$ is the probability of driving the car in this situation.
Assume that, independent of anything else, it rains during successive mornings and afternoons with constant probability p.
The suggestion is to formulate the situation as a four-state Markov chain and to denote the states in the following way: 1 = (h; h), 2 = (h; o), 3 = (o; h) and 4 = (o; o), where the first letter (home or office) corresponds to the man and the second one corresponds to this car. For instance, (o; h) means that the man is in his office and his car is at home.
I can easily determine (for example) that elements such as $P\{X_1 = 1 | X_0 = 1\}$ are all zero since they all involve the man initially being in one place and finally being in the same place. However, any consistent approach to determining the other elements of the transition matrix?
 A: If we implement the hint, then let's consider each possible transition from state $i$ to state $j$.  For example, if he is in states $1$ or $2$ at time $t$, then the only admissible states at time $t + 1$ are $3$ and $4$; conversely, if he is in states $3$ or $4$, he can only transition to states $1$ or $2$.  This is because these states are alternating:  he goes to work in the morning, and returns home in the afternoon.  Therefore, the transition matrix must have the form $$\mathcal P = \begin{bmatrix} 0 & 0 & p_{13} & p_{14} \\ 0 & 0 & p_{23} & p_{24} \\ p_{31} & p_{32} & 0 & 0 \\ p_{41} & p_{42} & 0 & 0 \end{bmatrix},$$ where rows of the matrix add to $1$.  Now we consider the probability $p_{13}$:  If he is in state $1$, the (unconditional with respect to weather) probability that he walks to work is $$p_{13} = 0 \cdot \Pr[R_t] + \alpha \cdot \Pr[\bar R_t], = \alpha (1-p),$$ where $R_t$ is the event that it rains at time $t$.  If he is in state $2$, then $$p_{23} = 0, \quad p_{24} = 1,$$ because he has no choice but to walk, and the car does not move.  If he is in state $3$, again $$p_{31} = 1, \quad p_{32} = 0,$$ since again, the car does not move.  If he is in state $4$, we calculate that he walks home with probability $$p_{42} = p_{13} = \alpha (1-p),$$ because the situation is symmetric with respect to home and office.
The key observation to make here is that the only transition between states that are stochastic are $$1 \to 3, \quad 1 \to 4, \quad 4 \to 1, \quad 4 \to 2.$$  All other transitions are not permissible, or deterministic/forced.
The only sticking point is that the way you have communicated the question, makes it seem that the probability of rain on any given afternoon is conditional on whether it rained in the morning; i.e., the statement "it rains during successive mornings and afternoons with probability $p$."  This could be interpreted to mean that if it rains in the morning, it also rains in the afternoon.  In such a case, the 4-state Markov model suggested by the hint is insufficient, because the states would necessarily have to encapsulate information about whether or not it rained in the morning if we are observing the man in the afternoon.  If, however, the statement is interpreted to mean "it rains on any given morning or afternoon with probability $p$ independent of any other morning or afternoon," then the above solution is correct.
A: Well let's see.  If $X_0=1$ then $X_1$ has to equal $3$ or $4$.  What's the probability that it's $3$?  It's the probability that he walked.  That means it didn't rain and he didn't take the car, so the probability is $(1-p)(1-\alpha)$.  Since going to $4$ is the only other possibility, going to $4$ from $1$ has probability $1-(1-p)(1-\alpha)$.  You can figure them all out like that.
A: What may be of use to others is to make use of the attached tree diagram to determine $p_{13}$. Can see from the diagram that $p_{13} = P(R) \times 0 + P(\bar{R}) \times \alpha$, the probabilities adding because they are mutually exclusive events.

