Analysis of “Tiny Dice Dungeon” (I) Consider the following one-player game:  Initially, there is a pot of $\$0$. On each turn, the player may opt to end the game and collect the contents of the pot, or roll a single die.  In the latter case, if the roll is $1$, the game is over and the player collects nothing.  Otherwise, the amount of the die roll (from $\$2–\$6$) is added to the pot, and the player gets another move.  The player continues rolling until they either get a $1$, and collect nothing, or opt to collect the pot.  We would like to find the optimal strategy for this game, and the expected payoff under the optimal strategy.
A simple (but not quite correct) analysis goes like this:  Any rational strategy must take the form “Stop if and only if the pot has value at least $\$S$.”  If the pot contains $\$P$, then the expected gain from the next die roll is $\frac16(-P+2+3+4+5+6 )$ dollars.  This is positive whenever $P\ge 20$, so the player should roll until the pot reaches $\$20$ or more, and then stop and collect it.
This is in fact an optimal strategy, but the analysis is not exactly correct,  because it assumes, incorrectly, that the expected gain from a successful die roll is exactly $\$4$.  But actually the expectation is a little bit more than this, because a successful die roll not only adds $\$4$ to the pot, it enables further die rolls that may add more.  In fact strategies $S=20$ and $S=21$ yield identical expected payoffs of $\$8.141794893727$.
A more careful analysis follows:  Let $X_S(P)$ be the expected amount won by following the stop-at-$S$ strategy when $\$P$ is already in the pot.  Then $X_S$ satisfies the following unusual recurrence:
$$X_S(P) = \begin{cases}
P & \text{if $P\ge S$} \\
\frac16\sum_{i=2}^6 X_S(P+i) & \text{otherwise}
\end{cases}
$$
We are interested in the behavior of $X_S(0)$, which we can calculate exactly using this recurrence.  This is tedious, so I had the computer do it.  $X_S(0)$ is maximized, as already mentioned, for $S=20$ and $S=21$.
My questions are:

  
*
  
*Is there an easier method to calculate $X_S(0)$ without the help of the computer?
  
*Is there an argument that shows that $X_{20}(0) = X_{21}(0)$?
  

 A: Let $X(P)$ denote the expected value of the game in its entirety, with \$$P$ in the pot.    Thus $$X(P)=\max(P,\frac{1}{6}\left(0+X(P+2)+X(P+3)+X(P+4)+X(P+5)+X(P+6)\right)$$
Because $X(P)\ge P$, we have $X(P+2)\ge P+2$ (etc) and plug in and have $$X(P)\ge \frac{5P+20}{6}$$
Now for $P< 20$, we have $P< \frac{5P+20}{6}$ so we will not cash out until at least $P=20$.  On the other hand,  for all $k\in\mathbb{N}$, we have $X(P+k)\le X(P)+k$.  Proof: Play two games in parallel, one with an extra \$k in the pot.  If you bust, the extra money doesn't help, and if you cash out you've got an extra \$k.  
Now, suppose $X(P)>P$ but $X(P-1)=P-1$.  We have $$6(P-1)\ge X(P+1)+X(P+2)+X(P+3)+X(P+4)+X(P+5)$$
$$X(P+2)+X(P+3)+X(P+4)+X(P+5)+X(P+6)>6P$$
$$6P-6-X(P+1)>6P-X(P+6)$$
$$X(P+1)+5\ge X(P+6)>X(P+1)+6$$
This contradiction proves that once $X(P)=P$ then $X(P+k)=P+k$ for all $k\in \mathbb{N}$.
Further, for some $P$ sufficiently large we must have $X(P)=P$ since otherwise the right strategy would be to play forever, which yields an expected value of \$0 almost surely.  Set $P^\star$ to be the maximal value such that $X(P^\star)>P^\star$; from the above we have $P^\star\ge 19$.
We have $P^\star<20$ since $$6P^\star<X(P^\star+2)+X(P^\star+3)+X(P^\star+4)+X(P^\star+5)+X(P^\star+6)=5P^\star+20$$
Thus $P^\star=19$, and the optimal strategy is to keep rolling while we have 19 or less.  To find the expected value of the original game, i.e. $X(0)$, we need to work backwards.
$X(20)=20, X(21)=21$, etc.
$X(19)=\frac{1}{6}(21+22+23+24+25)=\frac{115}{6}$
$X(18)=\frac{1}{6}(20+21+22+23+24)=\frac{55}{3}$
$X(17)=\frac{1}{6}(\frac{115}{6}+20+21+22+23)=\frac{631}{36}$
This is too many to calculate by hand, but I'm sure someone can write a few lines of code to figure it out.
