Probability approach in the expected payoff of a dice game

I am trying to understand the problem of expected payoff of a dice game explained here. I can roll the dice up to three times, but after each one I decide if I want to try once again or not. The idea is to find an optimal strategy that maximizes the expected payoff (expected number of spots; they do not sum up).

Let's say I know the optimal strategy: If in the first roll I get 1,2 or 3 I roll the dice once again. If in the second I get 1,2,3 or 4 I roll the dice once again. The end.

I wanted to calculate the expected number using probability $P(X=k)$ that I end up with $k$ spots. I am probably doing something wrong because I don't get the correct answer. My reasoning is the following:

$$P(X=\{1,2,3\}) = \underbrace{\frac{3}{6} \cdot \frac{4}{6} \cdot\frac{1}{6}}_{\text{3rd roll}}$$ $$P(X=4) = \underbrace{\frac{1}{6}}_{\text{1st roll}}+\underbrace{\frac{3}{6} \cdot \frac{4}{6} \cdot\frac{1}{6}}_{\text{3rd roll}}$$
$$P(X=5) = \underbrace{\frac{1}{6}}_{\text{1st roll}}+\underbrace{\frac{3}{6}\cdot\frac{1}{6}}_{\text{2nd roll}}+\underbrace{\frac{3}{6} \cdot\frac{4}{6}\cdot\frac{1}{6}}_{\text{3rd roll}}$$ $$P(X=6) = \frac{1}{6} + \frac{3}{6}\cdot\frac{1}{6}+\frac{3}{6} \cdot \frac{4}{6} \cdot\frac{1}{6}$$

Expected number is $E[X] = \sum_{k=1}^{6}kP(X=k) \approx 4.58$, which is not correct.

I appreciate any help.

• Are you sure you don’t have that strategy backwards? You should reroll on 4 or less for the first roll, and 3 or less for the second. When you have fewer rolls you have left, you should be less willing to reroll. – Mike Earnest May 19 '18 at 15:16
• @MikeEarnest In order to end the game with e.g. $1$. You should roll 1,2 or 3 in the first try ($p=3/6$). Then you should roll 1,2,3, or 4 in the second ($p=4/6$). You get $1$ with probability $p=1/6$ in the last roll. – WoofDoggy May 19 '18 at 15:24
• Your calculations are correct in this post, I am not disputing that. However, you are basing your calculations on an optimal strategy that someone told you, and I think that that strategy you heard is incorrect, which could perhaps explain the error. – Mike Earnest May 19 '18 at 15:40
• I wrote a simple program in C++ in order to check if probabilities are correct. Yes, they are. Now I am puzzled. The recurrence formula given by @Ben gives $4.6333$ instead of $4.58$. Moreover, in my approach I do not see a recurrence. – WoofDoggy May 19 '18 at 16:44
• Ok, I start to understand what is wrong with my reasoning after reading this. What I calculated is correct, but as mentioned by @MikeEarnest the optimal strategy is different. My strategy is fixed and that is the problem. – WoofDoggy May 19 '18 at 16:56

$1$-roll game: $$E(X)=\sum_{k=1}^6 kP(X=k)=1\cdot \frac{1}{6}+2\cdot \frac16+3\cdot \frac16+4\cdot \frac16+5\cdot \frac16+6\cdot \frac16=3.5.$$ $2$-roll game:
$$E(X)=\sum_{k=1}^6 kP(X=k)=1\cdot \frac{3}{36}+2\cdot \frac{3}{36}+3\cdot \frac{3}{36}+\\ 4\cdot \left(\frac16+\frac{3}{36}\right)+5\cdot \left(\frac16+\frac{3}{36}\right)+6\cdot \left(\frac16+\frac{3}{36}\right)=4.25.$$ Note: The player rerolls if the first roll was $1,2,3$.
$3$-roll game: $$E(X)=\sum_{k=1}^6 kP(X=k)=1\cdot \frac{12}{216}+2\cdot \frac{12}{216}+3\cdot \frac{12}{216}+\\ 4\cdot \left(\frac{4}{36}+\frac{12}{216}\right)+5\cdot \left(\frac16+\frac{4}{36}+\frac{12}{216}\right)+6\cdot \left(\frac16+\frac{4}{36}+\frac{12}{216}\right)=4\frac23.$$ Note: The player rerolls if the first roll is $1,2,3,4$ and the second roll is $1,2,3$. You did the other way around!
I think you can approach this type of question inductively. Suppose you use an optimal stragedy for n rolls the expected value is $e_n$. Then for $n+1$ rolls, the expected value $e_{n+1}=\sum_{i: i\le e_n} \frac{1}{6} e_n+ \sum_{i: i>e_n} \frac{1}{6}(i)$. If your first roll is greater than $e_n$ then you dont roll anymore because you already get a value bigger than the expected optimal value for n rolls and if the value of first roll is smaller than $e_n$, then you are in the situation of n rolls and you keep performing the stragedy because the expected value for n rolls is higher than the firet roll.