# Two-Armed Poisson Bandit in Continuous Time: Interpreting the Value Function

Consider a continuous-time version of the two-armed Poisson bandit, as in Presman (1991).

A perfectly-divisible unit resource is allocated to a risky arm $R$ or a safe arm $S$. $R$ generates lump-sum payoffs $h>0$ according to an inhomogeneous Poisson process $N_t$, with instantaneous rate $\lambda_\theta k_t$, where $\theta\in \{0,1 \}$ is the unknown type of the risky arm, and $k_t$ is the proportion of the resource allocated to $R$. We are given that $\Pr (\theta =1)=p_0$. $S$ gives flow payoffs of $(1-k_t)s$. That is, given an allocation process $\{k_t\}_{t\ge0}$ and a realisation of the process $\{N_t\}$, realised payoffs are given by $$\int_0^\infty re^{-rt} h\,\mathrm{d}N_t +\int_0^\infty re^{-rt}(1-k_t)s\,\mathrm{d}t .$$

$r>0$ is the discount rate. (The payoff is multiplied by $r$ for normalisation.) The objective is to choose the (appropriately measurable) process $\{ k_t \}_{t \ge 0}$ to maximise the expectation of the above expression. Hence, this is a problem of controlling the rate of the process $N_t$. To make the problem non-trivial, assume that $\lambda_1 h > s > \lambda_0 h \ge 0$.

One way to solve this problem is to derive the associated Hamilton-Jacobi-Bellman equation, which determines an ordinary differential-difference equation which can be solved using standard methods, as in Keller and Rady (2010). Let $p_t$ denote the posterior probability that $\theta = 1$ given observations of the process $N_t$. The dynamics of $p_t$, following Bayes' rule, are given by

$$\dot{p}_t = -(\lambda_1 - \lambda_0)p_t (1-p_t)k_t ,$$

whenever no lump-sum arrives. At the time of the arrival of a lump-sum, we have that

$$p_t = \frac{\lambda_1 p_{t-}}{\lambda(p_{t-})},$$

where $p_{t-} = \lim_{t\downarrow 0} p_t$, and

$$\lambda(p) = p \lambda_1 + (1-p )\lambda_0.$$

This gives the value function

$$V (p) = \lambda (p)h + (s -\lambda(p)h) \frac{1-p}{1-p^*} \left[ \frac{\Omega(p)}{\Omega(p^*)} \right]^{\mu},$$

for $p > p^*$, and $V(p)=s$ otherwise. In the above expression,

$$\Omega(p)=\frac{1-p}{p},$$

$\mu$ is the solution to

$$r+\lambda_0 = \mu(\lambda_1 -\lambda_0) + \lambda_0 \left(\frac{\lambda_0}{\lambda_1}\right)^\mu,$$

and,

$$p^* = \frac{\mu (s-\lambda_0 h)}{(\mu + 1)(\lambda_1 h -s )+\mu(s-\lambda_0h)}.$$

Intuitively, this can be thought of as a stopping problem where you set $k=1$ as long as $p>p^*$, and switch to $k=0$ forever after once your posterior reaches the level $p^*$.

Now, my question. Is there an intuitive interpretation of the second term of $V(p)$?

I know that in the case where $\lambda_0 = 0$ (Keller, et al. (2005)), the value function admits a simple interpretation. In particular, the term $$s - \lambda(p)h$$ captures the change in payoffs upon reaching the posterior $p^*$, $$\frac{1-p}{1-p^*}$$ is the probability that no lump-sum payoffs arrive before the posterior p reaches the level $p^*$, while $$\left[ \frac{\Omega(p)}{\Omega(p^*)} \right]^{\mu}$$ discounts this term to the time the switch to the safe arm occurs. In other words, the first term of the value function is the payoff from using $R$ forever, while the second term is an adjustment factor for the event where you switch to $S$ after becoming sufficiently pessimistic about $R$.

Does this interpretation still apply when $\lambda_0 > 0$? I've tried a few calculations and it doesn't seem to be the case, but perhaps I've made an error somewhere.

• Hi there, downvoter. If you had more concrete feedback, that'd be great. Apr 13, 2018 at 17:31