3
$\begingroup$

The background of the problem is this:

Assume that we have a parameter vector $\Theta$ which satisfies $\Theta^\prime\Theta=1$. If we let this vector have the uniform prior, the density of the prior is $$ \pi(\Theta)=\frac{\Gamma(q/2)}{(2\pi)^{q/2}} $$ where $q$ is the dimension of $\Theta$. Now, we have a set of possible models in the set $\mathcal{M}=\{2, \dots, k\}$, where each (integer) $i\in\mathcal{M}$ corresponds to the dimension of $\Theta_i$.

What I am supposed to answer is how this prior behaves when comparing dimensions $q+1$ to $q$. A hint I got was to look at what happens to the ratio of a priori probabilities when $q$ increases from for instance 2 to 10 (actually, he said when $k$ increases, but I assume he means $q$). Doing this yields: $$ \frac{\pi(\Theta_{10})}{\pi(\Theta_{2})}=\frac{\frac{\Gamma(5)}{(2\pi)^5}}{\frac{1}{2\pi}}=\frac{\Gamma(5)}{(2\pi)^4} $$

but what on earth does this tell me?

Also, I think I found something interesting here on page 3. More specifically, this:

This becomes more problematic in higher dimensions: the uniform prior in large dimension does not integrate anymore. [...] it tells that most of the probability mass lies at $+\infty$.

This sounds very interesting, but is not expanded on as far as I can tell. How do I show this? I'm guessing that is what I need to show (even though my vector of uniform distributions is restricted to length 1). Furthermore, the next sentence is very interesting too as I in another exercise am to do the same thing but with standard normal priors;

If instead one considers a high-dimensional Gaussian distribution $X\sim N (0,1)$, most of the mass is concentrated in a (high dimensional) unit sphere centered at the origin.

So, can anyone shed some light on this issue? I'm a bit lost, and would appreciate help!

$\endgroup$
  • $\begingroup$ There is a typo here; in your first equation, the denominator is supposed to be $2\pi^{q/2}$, not $(2\pi)^{q/2}$. $\endgroup$ – Ferdinand.kraft Jun 11 '13 at 2:27
1
$\begingroup$

I've had trouble working out what the question is, but I think it might be getting a the great circle problem, or, as it is more generally known the Borel-Kolmogorov paradox.

Basically, if you naively "cut through" a uniform density on an $n$-sphere to get an $n-1$ sphere and renormalise, you end up getting different results depending on your parameters. The problem stems from the $n$-sphere being a measure zero subset of the $n+1$-sphere. The solution, as presented by Jaynes, is making sure that you take a limit, and are clear about how to do it.

Note:

Something related to the measurability: If you gave units to your densities, would comparing them make so much sense?

Answer 2

I was going to suggest this before, but thought the other more likely. It's a classic problem for physicists (I don't know why) - I know it as the "infinite orange", but other names are the "infinite onion" and hyper-dimensional Christmas present.

The point it makes is the infinite orange is made only of skin, somewhat unintuitive (only=almost everywhere). In this case, it's saying that as you increase the number of dimensions the chance of drawing a point that is close (by some measure) to the surface increases. It's easy to show a that if you have an orange of radius $r$, and skin of thickness $\epsilon$, that the eddible inside is:

$$V_\text{inside} = \frac{\pi^{n/2}}{\Gamma(1+n/2)}(r-\epsilon)^n$$

and the volume which is skin is

$$V_\text{skin} = \frac{\pi^{n/2}}{\Gamma(1+n/2)}[r^n - (r-\epsilon)^n]$$

so the ratio of delicious, juicy, inside to yucky skin is:

$$\frac{V_\text{skin}}{V_\text{inside}} = \frac{r^n - (r-\epsilon)^n}{(r-\epsilon)^n} = \left(\frac{r}{r-\epsilon}\right)^n - 1 = \frac{1}{f}$$

(with $f$ from the book). For any $\epsilon$ in $(0,1)$ the limit as $n \rightarrow \infty$ is $\infty$, i.e. it is all skin.

The way of thinking about this probabilistically is simpler if we consider a $n$-cube (for which the limit above also holds). For uniform distributions, a point sampled from the unit $n$-cube can be thought of as $n$ samples from the unit interval. i.e.

$$[0,1]^n = [0,1]\times[0,1]\times ... [0,1] \;\;\;\text{$n$ times}$$

Then, lying within $\epsilon$ from the edge of the cube means that at least one of the samples from the $n$ samples from unit interval is not in $[\epsilon, 1-\epsilon]$. Clearly, as $n$ goes to infinity, at least one sample will be outside of $[\epsilon, 1-\epsilon]$, almost surely.

More probabilistically, the chance of being away from the edge is, $p_\epsilon$, is $1-2\epsilon$, so the probability of at least one lying on the edge after $n$ samples is:

$$Pr(\text{on edge}) = 1-p_\epsilon^n$$

which is $1$ in the limit of $n \rightarrow \infty$.

$\endgroup$
  • $\begingroup$ Great edit. Thanks for explaining it so meticulously :) $\endgroup$ – hejseb Jul 3 '13 at 12:31
2
$\begingroup$

The solution apparently goes something like this.

If you divide two of these uniform priors, you get a ratio that depends on $q$:

$$ \frac{\pi(\Theta_{q+1})}{\pi(\Theta_q)}=\frac{\frac{\Gamma((q+1)/2)}{(2\pi)^{\frac{q+1}{2}}}}{\frac{\Gamma(q/2)}{(2\pi)^{q/2}}}=\frac{\Gamma((q+1)/2)}{\sqrt{2\pi}\Gamma(q/2)} $$

This is, of course, fairly reasonable since we have the restriction $\Theta^\prime\Theta=1$; thus, the individual elements in the vector are not independent. This is basically what we're supposed to arrive at -- the comparison does not only depend on the difference of the dimensions in the comparison, but also on the levels of the dimensions.

This can then be compared to the case where we do not have any restrictions on $\Theta^\prime\Theta$ and instead let each element in the vector be iid $N(0,1)$. In this case, we get

$$ \frac{\pi(\Theta_{q+1})}{\pi(\Theta_q)}=\frac{\prod_{i=1}^{q+1}p(\theta_i)}{\prod_{i=1}^{q}p(\theta_i)}=p(\theta_{q+1})=(2\pi)^{-1/2}e^{-\theta_i^2/2} $$ which clearly does not depend on the level of the dimensions, but simply the difference.

$\endgroup$
  • $\begingroup$ This seems to have everything to do with geometry and very little to do with probability. I'm intrigued as to where your teacher is going with this. Do let me know when you find out. $\endgroup$ – Lucas Jun 17 '13 at 15:08
  • $\begingroup$ @Lucas: I think what this exercise is meant to showcase is that priors can behave weirdly. Have a look at this figure. As you can see, when comparing q+1 and q and q>14, the ratio exceeds 1. This doesn't make much sense, does it? It basically means the numerator is more likely, while it at the same time has its mass uniformly spread out on a greater sphere, doesn't it? I sent my answer to my teacher yesterday, and I'll let you know if I get any comments. My answer was basically what I posted here (but a little nicer if I may say so myself). $\endgroup$ – hejseb Jun 17 '13 at 15:25
  • $\begingroup$ It doesn't mean that, as they are densities, and densities with respect to different measures. Try this excercise, imagine the the $n$-sphere has a radius 1 meter, what are the units of $\pi(\Theta_{q})$? what are the units of your ratio? $\endgroup$ – Lucas Jun 17 '13 at 15:35
  • $\begingroup$ @Lucas - if you are still interested, have a look at Exercise 2.20 (p. 37) here: inference.phy.cam.ac.uk/mackay/itprnn/ps/22.40.pdf and then the answer on page 42 here: inference.phy.cam.ac.uk/mackay/itprnn/ps/41.46.pdf. I can't say I fully understand it, but perhaps you might... $\endgroup$ – hejseb Jul 3 '13 at 6:40
  • $\begingroup$ I am. I have seen this problem a number of times, have explained it below... $\endgroup$ – Lucas Jul 3 '13 at 12:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.