# Understanding large deviation theory/principle

From WIkipedia

Given a Polish space $\mathcal{X}$, let $\{ \mathbb{P}_N\}$ be a sequence of Borel probability measures on $\mathcal{X}$, let $\{a_N\}$ be a sequence of positive real numbers such that $\lim_N a_N=+\infty$, and finally let $I:\mathcal{X}\to [0,+\infty]$ be a lower semicontinuous functional on $\mathcal{X}$. The sequence $\{ \mathbb{P}_N\}$ is said to satisfy a large deviation principle with speed $\{a_n\}$ and rate $I$ if, and only if, for each Borel measurable set $E \subset \mathcal{X}$, $$-\inf_{x \in E^\circ} I(x) \le \varliminf_N a_N^{-1} \log\big(\mathbb{P}_N(E)\big) \le \varlimsup_N a_N^{-1} \log\big(\mathbb{P}_N(E)\big) \le -\inf_{x \in \bar{E}} I(x) ,$$ where $\bar{E}$ and $E^\circ$ denote respectively the closure and interior of $E$.

I was wondering how the above formal definition corresponds to the following informal interpretation:

the theory of large deviations concerns the asymptotic behaviour of remote tails of sequences of probability distributions. ... Roughly speaking, large deviations theory concerns itself with the exponential decay of the probability measures of certain kinds of extreme or tail events, as the number of observations grows arbitrarily large.

In particular,

1. what are the certain kinds of extreme or tail events in the formal definition? $E \subset \mathcal{X}$ is any Borel measurable set, not necessarily tail or extreme events.
2. How is the "exponential decay" represented in the formal definition? Is it represented as $a_n$ being a exponential function of $n$?
3. What are the interpretations for $a_n$ and $I$ in the formal definition?

Thanks!

Think of the inequalities as follows: for every Borel $E$ essentially

$P_N(E) = \exp[-a_N \ (\inf_{x\in E}I(x) + o(1) )]$ (*)

Think of $a_N= N$ and $I(x)=\frac 12 x^2$ as simple example, which is the LDP for gaussian $P_N$ with variance $\frac 1N$.

The rare events are those such that $\inf_{x\in E}I(x)>0$. They are rare because their probability is exponentialy small as $N\to \infty$.

EDIT after comments below: (1) you can see the LDP as a refinement of the LLN (Law of large numbers). The latter tells you what is typical, i.e. only that I(x)=0 for some x. So strictly speaking the LDP is about rare events AND typical events. But usually one stresses that it is about rare events, because it is what it distinguishes it from a LLN.

(2) The equivalence between $(*)$ your inequalities should be clear (but check it!) for $E$ such that $\inf_{x\in \mathring{E}} I(x) = \inf_{x\in \bar{E}}I(x)$. For general $E$ $(*)$ is stronger. ($*$) is better to understand the idea, but it's bit too strong in general for having good theorems. You can compare this with weak convergence of measures: the idea is $P_N(E) \to P(E)$, but the formal definition is bit more complicated.

Maybe this is a better point to explain why one does not use $(*)$ in general: in many interesting situations $\{x\}$ is measurable and $P_n(\{x\})=0$ for every $x\in \mathcal X$ (this is so already for the Gaussian case). Then $(*)$ can hold only for $I\equiv \infty$.

• Thanks! (1) Since $I$ map into $[0, \infty)$, $\inf_{x\in E} I(x)$ can only be $0$ or $>0$. If rare events are those defined to be the latter case, then does the formal definition of large deviation principle not use rare events? (2) How did you get $P_N(E) = \exp[-a_N \ (\inf_{x\in E}I(x) + o(1) )]$? In particular $o(1)$? – Tim Oct 19 '12 at 14:45
• Thanks! About ($*$), (1) when $\inf_{x\in E} I(x) = 0$, how do you know $P_N(E) \to 0$, given that there is $o(1)$? (2) In your last sentence, why ($*$) canhold only for $I \equiv \infty$? How about when $I$ is constant over $E$? – Tim Oct 19 '12 at 18:22
• (1) I say that if $\inf_{x\in E}I(x)>0$ then $P_N(E)\to 0$ (2) I mean $P_n(\{x\})=0$ for every $x\in\mathcal X$ AND for every $n\in\mathbb N$. Then only $I\equiv 0$ is possible. Does this answer your questions? – Hans Oct 20 '12 at 11:27