# Can the action-value function be "expanded"?

I know the basic language of probability, but not fluent in it, and the text books on reinforcement learning usually assumes one is fluent in it, so they don't define everything rigorously, here is my attempt to define every symbol appears in the action value function:

$$q_\pi{(s, a)}=E_\pi[G_t|S_t=s,A_t=a]$$

where:

$s$ and $a$ are concrete state and action (no problems here)

$S_t$ is a random variable represents the state of the system at time step $t$ (a little philosophical problem, but let's not get into it, so let's say no problem here)

$A_t$ is a random variable represents the action the agent take at time step $t$ (more on this later)

$G_t$ is the return at time step $t$, defined to be $\sum_{i>=t+1}^T{\gamma^{i-t-1}R_i}$, where $T$ is the terminal step, and $\gamma \in [0,1]$ is the discount factor (just a sum of $R_\_$s, so if I understand $R_\_$, I understand this, so no problem here)

$R_{i+1}$ is a random variable represents the reward the environment gives to the agent due to its action $A_i$ in state $S_i$ (more on this later)

$\pi(a|s)=P[A_t=a|S_t=s]$ is the probability of taking action $a$ at time $t$ given state $s$ (just a conditional probability, if I can understand $A_\_$, I can understand this)

$E_\pi[\_]$ expectation of some kind, more on this later

Questions:

1. It seems to me that, $R_{t+1}$ depends on the current state and action, so, is it technically correct that if I write $R_{t+1}=r(S_t,A_t)$, if it's correct, what is $r$? is it a function whose domain is $\{S_i\}_{i=1}^T \times \{A_i\}_{i=1}^T$? (no, I don't mean $StateSpace \times ActionSpace$, that would be $r(s, a)$), is this (r.v. valued function of r.v.s) studied in math?

2. About the notation $E_\pi[\_]$, there is where everything really starts to fall apart, (I undersdand expectation and conditional expectation), the text book says, and I quote: we define the value of taking action $a$ in state $s$ under a policy $\pi$, as the expected return starting from $s$, taking the action $a$, and thereafter following policy $\pi$. Especially on the "thereafter following policy $\pi$" part, how can one just put a $\pi$ at the subscript of the expectation willy-nilly? I know this is how mathematician do things, they define things for a living ^_^ but still, at least symbols in the definition should related to each other more obviously, not just by words-behind-the-scenes.

(2 continued) When you don't understand something, it sometimes can be hard to know that it is what exactly that you don't understand. I guess that the main source confusion is that, there are too many "free random variables"/"free moving parts" and they are "verbally defined" (contrast to formally defined, where you can follow things more or less mechanically), and all held together by the sentence in 2. For example, there seems to be a relation between $A_t$ and $\pi$, can this relation be used to eliminate $A_t$? (probably cannot, since this is a MDP, but still), Can the definition be expanded to something like $E_{sth \sim \pi(sth)}[sth(sth)|S_t=s,sth]$ where $sth$ meanings "something", and one $sth$ don't have to be equal to another $sth$, if it could, then I think I will have a easier time understanding it, since all "moving parts" are somewhat "bounded"/"qualified".

(this 2 is too long) I guess that at this point, the best I can say is that, is there a formulation of this expectation, that reflects the sentence: we define the value of taking action $a$ in state $s$ under a policy $\pi$, as the expected return starting from $s$, taking the action $a$, and thereafter following policy $\pi$ in a more obvious way? and not like "Oh, this is the state of art, just think this sentence when you look at the definition equation, sometime later, you will grew used to it" - I want to understand it, not treat it "axiomatically" - that's what "grew used to it" means.

I know that this is a really really long question, and a badly-stated one, you probably don't know what I'm talking about, that's because I really don't know it is what exactly I don't understand. So I put it here, Preying to Aristotle and all the souls of the great mathematicians, hoping they would send me an experienced teacher/mathematician with a kind heart to spot my problem and save me from frustration. QED.

Thank you for even considering reading this question.

• Not much clear to me what you don't understand. Seems to me that you mostly got it. However, think about this. You choose a policy. Then, you play the game many times and each time receive a total reward which might be different from other runs, both because you don't choose your actions deterministically and because the rewards are random. The expectation you are talking about is just the average of your reward playing the game and following that policy. You see that different policies might produce different expected total rewards. Feb 2, 2018 at 11:36
• Thanks for reading the whole thing :) I tried to convince myself like you said before I ask the question, yeah, it seems everything follows, but there always seems to be some thing hidden that I don't get. Thanks anyway. Feb 2, 2018 at 11:59

It's a long question, so sorry if I misunderstand something.

You have the action-value function: $$Q_\pi(s,a) = \mathbb{E}_\pi\left[G_t|S_t = s,A_t=a\right]$$

Notice how this relates to the value function: $$V_\pi(s) = \mathbb{E}_\pi\left[ G_t| S_t=s\right] = \mathbb{E}_{a\sim\pi(a|s)}\left[ Q_\pi(s,a) \right]$$

In other words, if $V$ tells the agent the value of a given state $s$ (under $\pi$), then $Q$ tells them the value of taking a particular action when in $s$. Thus, $V$ is given by "integrating out" the actions in $Q$, i.e. it is the value of $s$ averaged over the possible actions we could take in $s$.

I thought that might give some intuition. Anyway, for your questions:

1. It depends on the situation. Sometimes the reward depends on both the state and the action; other times, it depends only on the state. It can also be stochastic, i.e. $r_t\sim P_R(s_t,a_t)$ is a random variable that depends on the current state and action, or a deterministic function. Usually, we consider some kind of Markov decision process, so we don't need to consider the older states and actions.

2. I think it is written that way because it is hard to fully write out. In many cases, we have $a_t\sim \pi(a|s_t)$. Thus, the expected reward for a single step is given by $\mathbb{E}_{a\sim\pi}[r_t]$, where $r_t$ depends (stochastically) on $s_t$ and $a_t$. Also remember that state transitions can be stochastic too, i.e. $s_{t+1}\sim P_E(s_{t},a_t)$, which can be more important for model-based RL. So then the expected reward in the next step depends on which action was (randomly) chosen from the policy, and so on. The notation just encapsulates the idea that the expectation is over all time steps, all distributions over the environmental dynamics, and over the stochasticity of the policy itself. Only $\pi$ is really under the agent's control, so the subscript only includes it. Overall you seem to get this though, so I'm not sure what your question is beyond clarifying notation.