I have a zero-mean Gaussian random variable $Y\sim\mathcal{N}(0,\sigma^2_X)$ with known variance $\sigma_X^2$. I also have a zero-mean random variable $X$, which may be dependent on $Y$ (though, I can tolerate independence assumption if it doesn't hurt the final bound too much). Besides having mean zero, I know two sets of facts about $X$:

  1. Its variance, while unknown, is greater than $\sigma_X^2$;
  2. One (or more, but one is sufficient) of the following facts hold about the distance between distributions of $X$ and $Y$: $$\begin{align} D(p_Y\|p_X)&=\int_{-\infty}^{\infty}\frac{\exp(-x^{2}/2\sigma_X^2)}{\sqrt{2\pi}\sigma_X}\log\frac{\exp(-x^{2}/2\sigma_X^2)/\sqrt{2\pi}\sigma_X}{p_X(x)}dx\leq\epsilon_{KL}\\ H^2(Y,X)&=1-\int_{-\infty}^{\infty}\sqrt{\frac{\exp(-x^{2}/2\sigma_X^2)}{\sqrt{2\pi}\sigma_X}p_X(x)}dx\leq\epsilon_{H}\\ TV(Y,X)&=\int_{-\infty}^{\infty}\left|\frac{\exp(-x^{2}/2\sigma_X^2)}{\sqrt{2\pi}\sigma_X}-p_X(x)\right|dx\leq \epsilon_{TV} \end{align}$$

$D(p_Y\|p_X)$, $H^2(Y,X)$, and $TV(Y,X)$ are Kullback-Leibler divergence, Hellinger distance, and Total variation distance, respectively. These three quantities are commonly used to characterise distance between distributions.

I am trying to find the maximum variance of $X$ such that the distribution for $X$ satisfies any of the above distance requirements. I don't really care about the distribution, just its second moment.

Does anyone have any ideas?

This is a related to a question I asked earlier, but here I've generalised it here quite a bit.


First, I think that dependence and independence are moot here. Your question is just about two probability distributions, and their joint distribution never enters.

I'm not sure you can bound the variance of $X$ at all. For instance, suppose $X$ is a mixture of distributions which with probability $1-\epsilon$ is $N(0, \sigma_X^2)$, and with probability $\epsilon$ is some distribution with some very large variance $a$. By choosing $\epsilon$ very small, you can at least make the total variation distance small, independent of $a$. Then you can choose $a$ very large so that $X$ has large variance. For the other distances it's not quite as clear what happens, but I invite you to try it.

  • $\begingroup$ Hmmm... you are right about $X$ having an unbounded variance and still satisfying the TV constraint if it's a $(1-\epsilon,\epsilon)$ mixture of $Y$ and some high-variance distribution $Z$. But suppose we had Fact 3 about $X$: $X=Y+Z$ where $Z$ is arbitrary? Can we bound the variance of $X$ (which I guess boils down to bounding variance of $Z$) in that case, given Fact 2? Your answer tells me that I should've included that fact in the original question... $\endgroup$ – M.B.M. Oct 13 '11 at 12:53
  • $\begingroup$ I would like to ask the community: should I edit my question to include Fact 3 (which is important)? Or should I accept @Nate's answer and start a different question? I am new here, and would like to do the right thing and not be rude by "moving the target." $\endgroup$ – M.B.M. Oct 13 '11 at 13:01
  • 1
    $\begingroup$ @Bullmoose: Fact 3 doesn't help. Choose $Z$ to be a mixture which is $0$ with probability $1-\epsilon$ and something with large variance otherwise, and independent of $Y$. Then $Y+Z$ is effectively a mixture of $Y$ and something with large variance. $\endgroup$ – Nate Eldredge Oct 13 '11 at 13:51
  • $\begingroup$ @Bullmoose: In general I think it's fine to make updates and clarifications to a question. It's nice to add a comment to any answer which is affected. This is better than having two very similar questions, which can lead to unnecessary duplication of efforts. $\endgroup$ – Nate Eldredge Oct 13 '11 at 13:55
  • $\begingroup$ Great point re: Fact 3. I've accepted your answer. This actually adds some insight to the underlying problem that I am working on. $\endgroup$ – M.B.M. Oct 13 '11 at 14:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.