Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Reward R(S) in a Markov Decision Process is a mapping from a State S -> Bounded number. I want to know how a Utility Function is defined for an MDP.

I think it has to be a mapping from a sequence of states to a bounded number - where the utility function is just the Summation of the rewards R(S) of each of the states visited by an agent. Would that be correct?

share|cite|improve this question
up vote 2 down vote accepted

Your notation is not completely clear to me (especially with regards to bounded number).

Markov Decision Process (MDP) is a Markov process (MP) where (probabilistic) control is allowed, that name usually refers to discrete-time processes. Probabilistic control means that at each step you choose just a distribution of the next value from the class of admissible distributions. Again, $$\text{MDP} = \text{MP} + \text{probabilistic control}.$$

Talking about the reward $R:S\to\mathbb R$ the name for the model I met in the literature (Computer Science as well) is Markov Reward Model (MRM): that is MP + reward function, not necessary with a control in general. The underlying process for MRM can be just MP or may be MDP.

Utility function can be defined e.g. as $U=\sum\limits_{i=0}^nR(X_i)$ given that $X_0,X_1,...,X_n$ is a realization of the process. In the case when the underlying process is uncontrolled, one studies distribution of this function. In the case when control is allowed, optimal control problems are solved. On the other hand, nothing will prevent you from defining $U$ is another form and studying its properties.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.