All the books on Reinforcement Learning that I've seen seem to do an argument approximately as follows to define the Bellman equations. Assume that we have a Markov Decision Process (MDP) and a policy $\pi$ with an action-value function $Q^\pi$. We first create a new deterministic policy, at least as good as the original, nondeterministic policy by setting

$$\tilde\pi(s)=\arg \max_a Q^\pi(s,a).$$

We then proceed to define the Bellman operator for a deterministic policy. The obvious problem from a mathematical perspective is then that even if we set some assumption that guarantees that the supremum over the action is finite (e.g. bounded rewards with either a discount factor or bounded episode lengths), it still doesn't guarantee that there is an action that obtains a maximum value. Of course one assumption that does always guarantees it is a finite action space, which from the point of view of a practitioner is probably completely fine.

Now as a mathematician this bugs me a little and I was wondering if someone could point me to resources that don't sweep things like this under the rug? I'm also interested in general how to handle the case of uncountable state and action spaces. In other words, see how they are handled mathematically for no other reason than curiosity.

ps. It looks like there's no tag for MDPs...



You must log in to answer this question.