# Real-world example for utility theory

I'm stuck with Utility Theory. As I understand it, we have

• vector $\vec{x}$ representing evidence about the world
• $n$ possible states of the world $S = \{S_{1}, S_{2}, ..., S_{n}\}$
• $m$ possible actions of the agent $\alpha = \{\alpha_{1}, \alpha_{2}, ..., \alpha_{m}\}$
• utility function $U_{ik}$ which returns a number representing the benefit of taking action $\alpha_{i}$ when the state is $S_{k}$
• the expected utility of any action defined as $EU(\alpha_{i}|x) = \sum_{k}U_{ik} \cdot P(S_{k}|x)$

And, say, our agent would always choose the action which maximizes $EU$. That's pretty much it about the theory basics.

Now, I want to get a better understanding of all those abstractions by imagining a real world scenario and putting some meaningful values/properties into $\vec{x}$, $S_{i}$ etc, so that I can see how to write a program for choosing the best action.

I've seen examples in literature, but when tried to come up with my own one, the questions started to bubble up!

So, my example: an agent has some money and can invest a portion of that into a risky asset. Here's how I think about approaching the solution (with my questions inline):

1. Evidence about the world $\vec{x}$ - it may be, say, asset worst case and best case returns, and asset price difference over the last week. (Is that ok? Any other good candidates?)
2. States of the world $S_{k}$ - it may be, say, asset price difference between today and yesterday. (Is that ok? Any other good candidates?)
3. Actions $\alpha_{i}$ - it may be, say, "invest", "do nothing" and "let a human to decide" (Is that ok to have less actions than states? How to incorporate the idea of "invest a portion" here?)
4. Utility function $U_{ik}$ - it may be just a function of agent's current wealth and asset's returns (say, $\sqrt{w}$ and its compare certainty equivalent to the expected gain) but I believe it should be more intelligent. What's the best way to define it (proper function? matrix?), and who should define it (human expert? learned by an algorithm?). How to emphasize the penalty of deferring the decision to a human, by just having a smaller utility?
5. Probabilities $P(S_{k}|x)$ - with the $S_{k}$ and $\vec{x}$ I've chosen, I think the probabilities could and should be learned by the algorithm, but I'm not sure. What's the best way to defined them? should they be learned by an algorithm or provided by a human expert? should they be updated over time?

Hopefully I'm not asking too much - thanks!

-

Expected utility starts from the work of von Neumann and Morgenstern. In their framework, there exists a set of outcomes $Z$ (finite, for simplicity) and an agent has preferences over lotteries over the outcomes. The outcomes can be amounts of money the agent gets and the lotteries can actually be random assignments of money. In this framework, probabilities are taken to be objective. They might be coming from a roulette wheel. So $\Delta(Z)$ is the set of probability distributions over $Z$ and the agent has a preference relation $\succ$ on $\Delta(Z)$, where we interpret $p\succ q$ as "the agent would prefer the outcome to be decided by $p$ compared to $q$". Under certain conditions on $\succ$, interpreted as rationality requirements, there exists a function $u:Z\to\mathbb{R}$ such that $p\succ q$ if and only if $\sum_{z\in Z}p(z)u(z)>\sum_{z\in Z}q(z)u(z)$, that is if the expectation of $u$ is higher under $p$ than under $q$.

A different, more complex approach to utility is based on subjective probability. This approach was developed by, among others, De Finetti, Ramsey, and Savage. In the framework of Savage, one starts with a set of states $S$, a set of outcomes $Z$, and and a decision maker has preferences over acts, which are functions from states to consequences. The states can be $S=\{$ rain, sunshine $\}$, the outcomes $Z=\{$ being dry, being wet, enjoying summer $\}$. An act can be taking an ubrella, so that one is dry, no matter what state. Another act might be leaving the umbrella at home. Then one enjoys summer when the state is sunshine and gets wet when the state is rain. Now the decisionmaker has preferences over the acts. Under certain conditions, much more complicated than for the von Neumann-Morgenstern approach, there exists a function $u:Z\to\mathbb{R}$ and a probability distribution $p$ over $S$ (actually, a finitely additive probability...) such that for two acts $a$ and $b$, one has $a\succ b$ if and only if $\int_S u(a(s)) p(s)>\int_S u(b(s)) p(s)$. It is also possible to weaken that for a finite state space such that $a\succ b$ if and only if $\sum_S u(a(s))p(s)>\sum_Su(b(s))$, but in that case, the probability distributions are not uniquely determined.

One can write much more about this and decision theory is a booming area. But the point is that utilities and, for subjective utility theory, also the probabilities, are constructed from the preferences of a decison maker in situations of risk and uncertainty. A good introduction to the essentials of the theory is Notes On The Theory Of Choice by David Kreps. A more advanced book with lots of philosophical discussions and insights is Theory of Decision under Uncertainty by Itzhak Gilboa.

-
Thanks for the answer and book references. Looks like what I called "evidence vector x" you call "outcomes Z", but in general your definition seems to be the same as mine. And still all my questions are not answered - (1) how to program the utility function, is it a proper function or some kind of a mapper? (2) is it possible (or, is it better) to write an algorithm which would learn the utility function on the go rather than hardcoding it? (3) who defines the probabilities and what's the recommended way to do that, should they be updated over time? –  andreister Feb 7 '12 at 6:47
The point of my answer is that these questions don't have a real answer. If utility and probability are an expression of subjective references, there is no point to ask some expert what it is supposed to be. And you don't have to learn your subjective preferences. –  Michael Greinecker Feb 7 '12 at 9:22