A person often finds that she is up to 1 hour late for work. If she is from $1$ to $30$ minutes late, $\$4 $ is deducted from her paycheck; if she is from $31$ to $60$ minutes late for work, $\$8$ is deducted from her paycheck. If she drives to work at her normal speed (which is well under the speed limit), she can arrive in $20$ minutes. However, if she exceeds the speed limit a little here and there on her way to work, she can get there in $10$ minutes, but she runs the risk of getting a speeding ticket. With probability $\frac{1}{8}$ she will get caught speeding and will be fined $\$20$ and delayed $10$ minutes, so that it takes $20$ minutes to reach work. As she leaves home, let s be the time she has to reach work before being late; that is, $s = 10$ means she has $10$ minutes to get to work, and $s=-10$ means she is already $10$ minutes late for work. For simplicity, she considers s to be in one of four intervals: $(20, \infty), (10, 19), ( 10, 9),$ and $( 20, 11).$ The transition probabilities for s tomorrow if she does not speed today are given by

enter image description here

The transition probabilities for s tomorrow if she speeds to work today are given by

enter image description here

Note that there are no transition probabilities for $(20, \infty)$ and $(-10, 9),$ because she will get to work on time and from $1$ to $30$ minutes late, respectively, regardless of whether she speeds. Hence, speeding when in these states would not be a logical choice. Also note that the transition probabilities imply that the later she is for work and the more she has to rush to get there, the more likely she is to leave for work earlier the next day.

She wishes to determine when she should speed and when she should take her time getting to work in order to minimize her (longrun) expected average cost per day.

Formulate this problem as a Markov decision process by identifying the states and decisions and then finding the $C_{ik}$.

Attempt Let $i=0,1$ the states that represent late and on time

$\begin{array}{c|c|c} decision&action& state\\\hline 1 &speed\ given\ that (-10,9)&0\\\hline 2 &don't\ speed\ given\ that (20,\infty)\ &1\\\hline 3 &speed\ given\ that (-20,11)&0\\\hline 4 &don't\ speed\ given\ that (10,19)&1 \end{array}$

Am I correct so far?


Why there is no transition probabilities for $(-10,9)$ ?

.. $(-10, 9),$ because she will get to work on time and from $1$ to $30$ minutes late..

Why from $1$ to $30$ minutes late? should not be from $1$ to $9$ minutes late?

Can someone help me with this exercise, please?

  • $\begingroup$ There are no transition probabilities given for $(-10,9)$ if she speeds because it's illogical to speed in that situation. I don't see how you get that if she's $10$ minutes late when she leaves home, she can arrive at most $9$ minutes late. $\endgroup$ – saulspatz May 15 at 3:07
  • $\begingroup$ @saulspatz I don't understand your comment. $-9$ is in that interval which is that she is $-9$ minutes lates for work. If she speeds then she arrive in $10$ minutes thus $-1$ minute late. Hence should not this imply a probability transition? $\endgroup$ – user441848 May 15 at 3:21
  • $\begingroup$ The problem statements says, "𝑠=−10 means she is already 10 minutes late for work". $\endgroup$ – saulspatz May 15 at 13:25
  • $\begingroup$ @saulspatz yes I know, I was just trying to interprete the meaning of $-9$ $\endgroup$ – user441848 May 18 at 1:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.