# Stochastic Dynamic Programming: Deriving the Steady-State for a Lottery

This question was originally posted here, but as the Math.SE community is more active I provide an extended version of the post here.

I am working through the basic examples of the stochastic optimization models in the book by McCandless (2008): The ABCs of RBCs, pp. 71 - 75 [RBC stands for real business cycle' model in the economists' jargon]

### A Standard Stochastic Dynamic Programming Problem

In what follows next, I assume that the domain of the variables and the range of the functions all belong to $\mathcal{R}_0^+$ and I assume there are no corner solutions. Here is a formulation of a basic stochastic dynamic programming model: $$y_t = A^t f(k_t)$$

$$A^t = \cases{A_1 \text{ with probability } p \\ A_2 \text{ with probability } (1 - p) }$$

$$k_{t+1} = A^tf(k_t) + (1 - \delta)k_t - c_t$$

With an agent maximizing the expected utility function $u(c_t)$, which is concave, monotone strictly increasing in $c_t$: $$E_t \sum_{t}^\infty \beta^t u(c_t)$$

Substituting consumption from the previous equation and using the recursive formulation of the problem gives the following problem: $$V(k_t, A^t) = \max_{k_{t+1}} \left[u(A^tf(k_t) + (1 - \delta)k_t - k_{t+1}) + \beta E_t V(k_{t+1},A^{t+1})\right]$$

Then McCandless proceeds saying that the algorithm to solve to the problem is almost identical to the deterministic case. One finds the first-order conditions (a derivative of the value function with respect to $k_{t+1}$) for the control variables, then does the same for $k_t$ and applies the Envelope theorem to get an analytical solution.

(1) Find the first order conditions for the control variables (period $t+1$): $$\frac{\partial V_t}{\partial k_{t+1}} = - u_{k_{t+1}}(A^tf(k_t) + (1 - \delta)k_t - k_{t+1}) + \beta E_t V_{k_{t+1}}(k_{t+1},A^{t+1})$$

To go around $V_{k_{t+1}}(k_{t+1},A^{t+1})$ he applies the results from Benveniste and Scheinkman

(2) Find the derivatives for the value function for the control variables of the previous time $t$ (which are the state variables in the case):

$$\partial V_t / \partial k_{t} = A^t (f_{k_t}(k_t) - \delta) u_{k_t}(A^tf(k_t) + (1 - \delta)k_t - k_{t+1})$$

And obtain the value of $\partial V_t / \partial k_{t}$ by simply updating' it by one period (refering to Benveniste and Scheinkman):

$$\partial V_t / \partial k_{t+1} = A^t (f_{k_{t+1}}(k_{t+1}) - \delta) u_{k_{t+1}}(A^tf(k_{t+1}) + (1 - \delta)k_{t+1} - k_{t+2})$$

Then you can impute the above equation into the first-order conditions and derive a steady-state solution with the Euler equation. Model written, implications discussed, paper submitted. Profit.

### A Lottery Augmented Version

Now I want to investigate a little different case. Take the very same model but introduce another variable. Now allow agent to save some of his/her income in every period (call it $s_t$ for a security) to accumulate in a variable $l$: $$l_{t+1} = l_t + s_t$$

And I assume here that l_{t+1} is the control variable because in terms of accounting it does not matter, which one to pick $l_{t+1}$ or $s_t$ but it helps to derive the optimality conditions later on.

And the $l_t$ enters the problem through the variable $A^t$: $$A^t = \cases{A_1 \text{ with probability } p \\ A_2l_{t} \text{ with probability }(1 - p)}$$

The main difference is easy to see if one writes the equation for income in a period $t$ explicitly opening up the expectation sign: $$y_t = pA_1f(k_t) + (1-p)A_2l_tf(k_t)$$

In this case we have a deterministic control variable $l_t$ "turning on" when a certain event happens (as if you would win in a lottery, which increases your income by the factor of how much you invested in it; yes the example makes little sense but I am interested in the principle itself).

The Question: Would the previous algorithm described by McCandless give the right solution as well?

P.S. If someone could point me a paper with an example model, which is very close to the one I described, that would be helpful.