# Understanding the derivation of the Bellman equation for state value function

In reinforcement learning theory, from Sutton and Barto, page 46-47 the Bellman equation for a state-value function is:

\begin{equation} \begin{aligned} v_\pi (s) :&= \mathbb{E}\left[G_t | S_t=s\right]\\ &= \mathbb{E}_\pi\left[R_{t+1} + \gamma G_{t+1} | S_t=s\right]\\ &= \sum_{a}\pi(s|a)\sum_{s', r}p(s', r|s, a)\left[r+\gamma\mathbb{E}_\pi[G_{t+1}|S_t=s\right]]\\ &= \sum_{a}\pi(s|a)\sum_{s', r}p(s', r|s, a)\left[r + \gamma v_\pi(s')\right] \end{aligned} \end{equation}

Where

\begin{equation} \begin{aligned} v_\pi(s) &= \text{A state value function}\\ r &= \text{A reward}\\ G_t &= \text{Discounted reward or return from time t}\\ S_t, s, s' &= \text{State at time t, a state and the next state respectively}\\ \gamma &= \text{A discount factor, constant}\\ p(s', r|s, a) &= \text{probability of transition to state } s' \text{with reward } r \text{, from state } s \text{and action } a\\ \pi(s, a) &= \text{probability of taking action } a \text{ in state } s \text{ under stochastic policy } \pi\\ \end{aligned} \end{equation}

I understand the relationship between lines 1 and 2 (from a previous equation in Sutton and Barto, eqn. 3.9 if your interested) and I also understand the final substitution, i.e. the recursive bit. However I don't fully understand how to you get from line 2 to 3. I think you probably need to refer to the definition of conditional expectation, i.e.

$$\begin{equation} \mathbb{E}(X|Y) = \sum_{x}x P(X=x|Y=y) \end{equation}$$

but I'm having a hard time understanding the precise logic. Does anybody have any further insight that could help me understand both mathematically and intuitively why this is so?