Professor has 4 umbrellas, Markov chain and Probability OK this problem is making me tear my hair out. I need someone to walk me through this in baby-steps method like 1 + 1 = 2. I am trying to figure out what I don't understand. I know this is going to be a Markov chain. 

 A: As was noted above you have to construct the transition matrix.
Say there is one umbrella in the morning at his house. How can he end up with one umbrella at his house the next morning? Lets look at the 4 cases:
Rains morning and night:
brings umbrella to office and then back hence remains with 1 at home
Rains morning but not night:
brings umbrella to office but also back because otherwise office has all 4, hence remains with 1 at home
Does not rain morning, rains night:
does not bring umbrella in the morning does bring one back at night so ends up with 2 at home
Does not rain either morning or night:
does not bring umbrella either time hence remains with 1 at home
Hence the transition probability $ 1 \to 1$ is $M_{11}=1-p(1-p)$. What is the transition probability $1 \to 2$? That only happens in the one case outlined above hence $1 \to 2$ has probability $M_{12} = p(1-p)$. The transitions $1 \to 3$ and $1 \to 4$ have zero probability.
So that's your first row of the transition matrix as you requested. Now fill in the rest. 
A: Step one is to write down the four by four transition  matrix: the probability  of moving from one state to another.  Step two is to find the limit as that matrix works on itself.
For example, with one umbrella at home, there is a $p$ chance of rain, but even if it does, and he takes the umbrella to work, he will take it back.  He can't take more than one umbrella, so he can't transition to states 3 or 4.  So the only possible transitions are to states 1 or 2.  With probability $1-p$ it doesn't rain in the morning, so with probability $(1-p)p$ it doesn't rain in the morning but rains that night, meaning he leaves one umbrella at home but brings one back from work, giving him two umbrellas at home. So the vector of transitions from state 1 into states 1,2,3,4 is $(1-p(1-p),p(1-p),0,0)$. Similar Logic should work for the other transitions, taking the state as the  number of umbrellas at home in the morning. 
