# markov chain: 2 state chain

I have a machine. It has two states, broken or working. If it is working, then it will be broken with probability $q=0.1$. If the machine is working, I will make \$1000 dollar a day. If it is broken, then repairman will charge charge me \$ $200/(1-p)$ a day to repair. He will fix the machine with probability p. Assume the transition from broken to working (and wise versa) is independent. Find p that maximize the expected profit.

Attempt:

2 state markov chain. Let state 0 be working, and let state 1 be broken. The state transition matrix is:

\begin{pmatrix} 1-q & q \\ p & 1-p \\ \end{pmatrix}

The steady state distribution is calculated by (omit showing my calculation process here since it is well-known) : $\pi_0=q/(p+q), \pi_1=p/(p+q)$

The expected profit is: $1000*\pi_0-200/(1-p)*\pi_1$. (Do you think this is correct)?

• You have the right idea, but you made a mistake in computing $\pi_0, \pi_1$ (I believe you swapped the $p$s and $q$s). The equation should be $$\pi_{working}q= \pi_{broken}p$$. – Michael Feb 11 '16 at 2:30
• To verify, you can get the same answer from renewal theory: $$\mbox{time avg profit} = \frac{1000E[working] -\frac{200}{1-p}E[broken]}{E[working] + E[broken]}$$ where $E[working]$ is the average duration of a working interval, and $E[broken]$ is the average duration of a broken interval. – Michael Feb 11 '16 at 2:34
• In that case, $E[working]/(E[working]+E[broken])$ would be the stationary distribution? – randy Feb 11 '16 at 5:57
• Yes, $\pi_{working} = \frac{E[working]}{E[working]+E[broken]}$. – Michael Feb 11 '16 at 6:39

You messed it a bit. It should be: $$\pi_0=\frac{p}{(p+q)}, \pi_1=\frac{q}{(p+q)}$$ So the profit is: $$-200\frac{0.1}{(1-p)(p+0.1)}+1000\frac{p}{p+0.1}$$ So you have to find the optimum between 0 and 1. The first derivative is: $$-20\frac{2p-0.9}{(1-p)^2(p+0.1)^2}+1000\frac{0.1}{(p+0.1)^2}$$ You put it equal to zero and you get the sollution, which is: $$p=\frac{12-\sqrt{26}}{10} \approx 0.690098049$$