In reinforcement learning theory, from Sutton and Barto, page 46-47 the Bellman equation for a state-value function is:

\begin{equation} \begin{aligned} v_\pi (s) :&= \mathbb{E}\left[G_t | S_t=s\right]\\ &= \mathbb{E}_\pi\left[R_{t+1} + \gamma G_{t+1} | S_t=s\right]\\ &= \sum_{a}\pi(s|a)\sum_{s', r}p(s', r|s, a)\left[r+\gamma\mathbb{E}_\pi[G_{t+1}|S_t=s\right]]\\ &= \sum_{a}\pi(s|a)\sum_{s', r}p(s', r|s, a)\left[r + \gamma v_\pi(s')\right] \end{aligned} \end{equation}


\begin{equation} \begin{aligned} v_\pi(s) &= \text{A state value function}\\ r &= \text{A reward}\\ G_t &= \text{Discounted reward or return from time t}\\ S_t, s, s' &= \text{State at time t, a state and the next state respectively}\\ \gamma &= \text{A discount factor, constant}\\ p(s', r|s, a) &= \text{probability of transition to state } s' \text{with reward } r \text{, from state } s \text{and action } a\\ \pi(s, a) &= \text{probability of taking action } a \text{ in state } s \text{ under stochastic policy } \pi\\ \end{aligned} \end{equation}

I understand the relationship between lines 1 and 2 (from a previous equation in Sutton and Barto, eqn. 3.9 if your interested) and I also understand the final substitution, i.e. the recursive bit. However I don't fully understand how to you get from line 2 to 3. I think you probably need to refer to the definition of conditional expectation, i.e.

\begin{equation} \mathbb{E}(X|Y) = \sum_{x}x P(X=x|Y=y) \end{equation}

but I'm having a hard time understanding the precise logic. Does anybody have any further insight that could help me understand both mathematically and intuitively why this is so?


Perhaps you need a bit more of details and notational enrichment of the equations in order to make the derivation step. These are spelled out in fair amount of detail at the link below.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.