12
$\begingroup$

Let me start with the excerpt out of Casella & Berger's Statistical Inference (2nd edition, pg. 470) that inspired this question.


Definition 10.1.7 For an estimator $T_n$, if $\lim_{n\to\infty}k_n\mathrm{Var}T_n=\tau^2<\infty$, where $\{k_n\}$ is a sequence of constants, then $\tau^2$ is called the limiting variance or limit of the variances.

Example 10.1.8 (Limiting variances) For the mean $\bar X_n$ of $n$ iid normal observations with $\mathrm EX=\mu$ and $\mathrm{Var}\,X=\sigma^2$, if we take $T_n=\bar X_n$, then $\lim n\mathrm{Var}\bar X_n=\sigma^2$ is the limiting variance of $T_n$.

But a troubling thing happens if, for example, we are instead interested in estimating $1/\mu$ using $1/\bar X_n$. If we now take $T_n=1/\bar X_n$, we find that the variance is $\mathrm{Var}\,T_n=\infty$, so the limit of the variances is infinity. But recall Example 5.5.23, where we said that the "approximate" mean and variance of $1/\bar X_n$ are $$ \mathrm E\left(\frac{1}{\bar X_n}\right)\approx\frac{1}{\mu}, $$ $$ \mathrm{Var}\left(\frac{1}{\bar X_n}\right)\approx\left(\frac{1}{\mu}\right)^4\mathrm{Var}\bar X_n, $$ and thus by this second calculation the variance is $\mathrm{Var}\,T_n\approx\frac{\sigma^2}{n\mu^4}<\infty$.

This example points out the problems of using the limit of the variances as a large sample measure. Of course the exact finite sample variance of $1/\bar X$ is $\infty$. However, if $\mu\neq 0$, the region where $1/\bar X$ gets very large has probability going to $0$. So the second approximation in Example 10.1.8 is more realistic (as well as being much more useful). It is this second approach to calculating large sample variances that we adopt.


Now, I disagree that $\mathrm{Var}\left(1/\bar X_n\right)=\infty$. While I do agree that $\mathrm E\left(1/\bar X_n^2\right)=\infty$, the first negative moment $\mathrm E\left(1/\bar X_n\right)$ is clearly undefined and thus so is the variance. Given that both $\mathrm E\left(1/\bar X_n\right)$ and $\mathrm{Var}\left(1/\bar X_n\right)$ are strictly undefined, I have the following question.

Question: What exactly do the delta method approximations for $\mathrm E\left(1/\bar X_n\right)$ and $\mathrm{Var}\left(1/\bar X_n\right)$ approximate if the moments of $1/\bar X_n$ are undefined?

My thoughts:

Let $\bar X_n\sim\mathcal N(\mu,\sigma^2/n)$ and define for $t\in\Bbb R$ $$ \mathcal H[f_{\bar X_n}](t)=\lim_{\epsilon\to 0^+}\int_{\Bbb R\setminus(t-\epsilon,t+\epsilon)}\frac{f_{\bar X_n}(x)}{x-t}\,\mathrm dx, $$ which is the Hilbert transform of the density function for $\bar X_n$. Since the Hilbert transform commutes with derivatives, i.e. $$ \mathcal H[\partial_t^k u]=\partial_t^k\mathcal H[u], $$ the expression $\mathcal H[f_{\bar X_n}](t)$ represents a sort of generating function for the negative moments of $\bar X_n$, which do not exist in the traditional sense. We define $$ \mathrm E\bar X_n^{-k}:=\frac{1}{(k-1)!}\partial_t^{k-1}\mathcal H[f_{\bar X_n}](t)\Big|_{t=0}. $$ For our particular example $$ \mathcal H[f_{\bar X_n}](t)=\frac{\sqrt 2}{\sigma/\sqrt n}\mathcal D\left(\frac{\mu-t}{\sqrt 2\,\sigma/\sqrt n}\right), $$ with $\mathcal{D}(z)=e^{-z^{2}}\int_{0}^{z}e^{t^{2}}\,\mathrm{d}t$ being Dawson's integral; thus $$ \mathrm E\bar X_n^{-1}=\frac{\sqrt 2}{\sigma/\sqrt n}\mathcal D\left(\frac{\mu}{\sqrt 2\,\sigma/\sqrt n}\right) $$ and $$ \mathrm{Var}\bar X_n^{-1}:=\mathrm E\bar X_n^{-2}-(\mathrm E\bar X_n^{-1})^2 $$ with $$ \mathrm E\bar X_n^{-2}=\frac{\sqrt 2\,\mu}{(\sigma/\sqrt n)^3}\mathcal D\left(\frac{\mu}{\sqrt 2\,\sigma/\sqrt n}\right)-\frac{1}{(\sigma/\sqrt n)^2}. $$ Now, the Dawson integral admits the following asymptotic expansion for $x\to\infty$ $$ \mathcal D(x/\sqrt 2)\sim\frac{1}{\sqrt 2\,x}\sum_{k=0}^\infty (2k-1)!!\frac{1}{x^{2k}}, $$ so letting $n\to\infty$ we have for the first moment $$ \begin{align} \mathrm E\bar X_n^{-1} &\sim\frac{1}{\mu}\sum_{k=0}^\infty (2k-1)!!\left(\frac{\sigma/\sqrt n}{\mu}\right)^{2 k}\\ &=\frac{1}{\mu}+\mathcal O\left(\frac{1}{n}\right), \end{align} $$ which is the first-order delta method approximation for the mean of $1/\bar X_n$. Likewise, for the second moment we find $$ \mathrm E\bar X_n^{-2} \sim\frac{1}{(\sigma/\sqrt n)^2}\sum_{k=1}^\infty (2k-1)!!\left(\frac{\sigma/\sqrt n}{\mu}\right)^{2 k}, $$ which upon combining with the expansion for $\mathrm E\bar X_n^{-1}$ gives the asymptotic approximation for $n\to\infty$ $$ \begin{align} \mathrm{Var}\bar X_n^{-1} &=\frac{1}{\mu^2}+3\frac{(\sigma/\sqrt n)^2}{\mu^4}+\mathcal O\left(\frac{1}{n^2}\right)-\left(\frac{1}{\mu^2}+2\frac{(\sigma/\sqrt n)^2}{\mu^4}+\mathcal O\left(\frac{1}{n^2}\right)\right)\\ &=\frac{(\sigma/\sqrt n)^2}{\mu^4}+\mathcal O\left(\frac{1}{n^2}\right), \end{align} $$ which is the first-order delta method approximation for the variance of $1/\bar X_n$.

So in this context, the delta method moment "approximations" are equal to the first term in the asymptotic expansions for our exact generalized moments in the case $n\to\infty$.

To ask my question from a different perspective: If we are to assign any value to the moments of $1/\bar X_n$, should we instead assign these generalized values for $E\bar X_n^{-k}$ with the delta method moment estimates simply being an approximation of these "exact" moments?

Indeed, the generalized moments seem to provide a more accurate picture of the moments for $1/\bar X_n$ whenever $|\mu/(\sigma/\sqrt n)|\to\infty$ ($n$ does not necessarily have to be large), which implies $f_{\bar X_n}(0)$ is vanishingly small.

Consider the case $\mu=6$, $\sigma=1$, and $n=1$ so that $\bar X_n=X\sim\mathcal N(\mu,\sigma^2)$. I simulated $X$ in MATLAB with the following code:

mu = 6;
sigma = 1;
Y = 1./normrnd(mu,sigma,1e8,1);

EY = sqrt(2)/sigma*dawson(mu/(sqrt(2)*sigma));
[EY 1/mu mean(Y)]

EY2 = sqrt(2)*mu/sigma^3*dawson(mu/(sqrt(2)*sigma))-1/sigma^2;
[EY2 1/mu^2 mean(Y.^2)]

EY3 = (sqrt(2)*(mu^2-sigma^2)*dawson(mu/(sqrt(2)*sigma))-mu*sigma)/(2*sigma^5);
[EY3 1/mu^3 mean(Y.^3)]

VarY = EY2-(EY)^2;
[VarY sigma^2/mu^4 var(Y)]

fY = @(y) normpdf(1./y,mu,sigma)./y.^2;
ax = linspace(0.05,0.35,512);

figure
hold on;
histogram(Y,linspace(0.05,0.35,40),'Normalization','pdf')
plot(ax,fY(ax),'Color',[0 0 0],'LineWidth',2)

Here are the results showing that the generalized moments more accurately reflect the sample statistics.

$$ \begin{array}{cccc} &\text{generalized} &\text{delta} &\text{sample}\\ \mathrm EX^{-1} &0.1718 &0.1667 &0.1718 \\ \mathrm EX^{-2} &0.0305 &0.0278 &0.0305 \\ \mathrm EX^{-3} &0.0056 &0.0046 &0.0056 \\ \mathrm{Var}X^{-1} &0.0010 &0.0008 &0.0010 \end{array} $$

enter image description here

$\endgroup$
5
  • 1
    $\begingroup$ I like this question but it's not self contained enough (+1 anyway) $\endgroup$
    – Snoop
    Commented Mar 2, 2022 at 15:39
  • $\begingroup$ The "delta method" is nothing more than Taylor of first or second order. Under suitable conditions on the function, the error terms converge to zero in probability. I don't think there is much more here. So, you have $T_n = o_P(1) + Z_n$ where $Z_n$ is normal. The convergence in distribution of $T_n$ follows. If anything can be extracted is that the function $f:\mathsf{P} \to \bar{\mathbf{R}}$ given by $f(\mu) = \int x^2 d\mu(x)$ is not continuous, where $\mathsf{P}$ is the space of probability measures. $\endgroup$
    – William M.
    Commented Mar 2, 2022 at 15:47
  • 2
    $\begingroup$ I have another comment for you: you said that when $E(X)$ does not exists, then you disagree with $V(X) = \infty.$ In a sense, you are correct that the formula for $V(X) := E(((X - E(X))^2)$ is not defined. However, it is convention that when $E(X)$ does not exists or is infinite, then $V(X) = \infty$ as well. The reason is that $V(X)$ measures the variation of $X.$ It should be intuitively obvious that $|X|$ varies less than $X$ for the variation is the same among same-sign values while the variation decreases sharply for opposite sign values... $\endgroup$
    – William M.
    Commented Mar 18, 2022 at 18:09
  • 1
    $\begingroup$ Thus, for a sensible measure of variation, $V(|X|) \leq V(X).$ When $E(X)$ is not defined, then $E(|X|) = \infty$ and then $V(|X|) = E((|X| - E(|X|))^2) = \infty$ since $|X| - E(|X|) = \infty$ almost surely. Therefore, we ought to have $V(X) = \infty$ as well. Hence, that is why we have the convention that whenever a random variable has no finite first moment, then its variance is infinite. $\endgroup$
    – William M.
    Commented Mar 18, 2022 at 18:10
  • $\begingroup$ @WilliamM. This is good feedback. Thank you for sharing. $\endgroup$ Commented Mar 18, 2022 at 18:46

1 Answer 1

5
$\begingroup$

Too long for comment.

The asymptotic variances are more useful than the variances because some estimators do not have variance for any finite sample and yet the do have asymptotic variance. Consider $Y_1$ and $Y_2$ two independent binomial distributions. The odds ratio is $\theta = \dfrac{\frac{P(Y_2=1)}{P(Y_2=0)}}{\frac{P(Y_1=1)}{P(Y_1=0)}}.$ Write now a (2,2) table of observations as $(n_{ij})$ with $i=1,2,$ denoting $Y_1$ or $Y_2,$ and $j = 1$ denotes failure, while $j = 2$ denotes success. The most natural estimate is $\hat\theta = \dfrac{n_{11}n_{22}}{n_{12}n_{21}}$ which takes the values $\infty$ if $n_{12}n_{21} = 0,$ so its variance does not exists. However, for very large samples, $n_{12}n_{21} = 0$ has an exponentially small probability and this event will not occur in practice and so for all intents and purposes we should be able to calculate the variance. In this sense, the sample variance (not as defined by Casella and Berger but by the "delta method") is the one that we use in practice. The odds ratio can also be written as $\theta = \dfrac{P(Y_2 = 1)}{P(Y_1 = 1)} \dfrac{P(Y_1 = 0)}{P(Y_2 = 0)}.$ When $Y_1$ and $Y_2$ model something rare (so that "success" is a rare event, as in effectiveness of new treatment and we are comparing two treatments, or number of people who use drugs and we are comparing two social factors or number of car accidents than result in death comparing seat-bealt use vs not) an estimator of interest is the relative risk $r = \dfrac{P(Y_2 = 1)}{P(Y_1 = 1)}$ which cannot be estimated usually due to ethical or cost contraints (it can be estimated in clinical trials but not in the social-factor/drug-use scenario or the fatal-accident/seat-belt use), since the probabilty of "failure" is rare for both $Y_1$ and $Y_2$ the odds ratio is very close to the relative risk since $\dfrac{P(Y_1 = 0)}{P(Y_2 = 0)} \approx 1$ and in practice it is not rare that this fraction is $\in [0.97, 1.3].$ (By the way, it can be shown that the odds ratio is an estimator that is invariant under changes of scales, swaping of columns and or of rows, so it is really invariant under retrospective or prospective analyses.)

$\endgroup$
6
  • $\begingroup$ "However, for very large samples, $n_{12}n_{21} = 0$ has an exponentially small probability and this event will not occur in practice and so for all intents and purposes we should be able to calculate the variance." This is the type of explanation I have heard before and it bothers me. Essentially what were saying is because the probability $n_{12}n_{21} = 0$ is very small our sample will likely not contain any "bad observations" and so our sample moments will agree with the delta method approximations. This strikes me as being so hand-wavy. Perhaps I'm just thinking about this too hard. $\endgroup$ Commented Mar 2, 2022 at 16:23
  • $\begingroup$ We already know that moments aren't continuous functions. Taylor theorem will give $T_n = o_P(1) + Z_n$ and in my particular case, $o_P(1)$ is the set where $n_{12}n_{21} = 0,$ since we didn't observe these, the sample correspond to $Z_n.$ Also, mathematics and real-world, two different domains. Mathematics is a pure branch of knowledge that study itself in a consistent way. It is only popular because when we assume mathematical models or representations, these work very well (and in social domains, human or animal, the mathematical framework is at best a crude approximation to reality). $\endgroup$
    – William M.
    Commented Mar 2, 2022 at 16:30
  • $\begingroup$ (+1) to your question too, I really like the theoretical approach to applied mathematics. $\endgroup$
    – William M.
    Commented Mar 2, 2022 at 17:40
  • $\begingroup$ Hi William. In my example what would $o_P(1)$ be? The set $1/\bar X_n=0$? $\endgroup$ Commented May 9, 2022 at 14:26
  • 1
    $\begingroup$ It would be sets of the form $S(\varepsilon) = \left\{ \left| \bar X_n \right| < \varepsilon \right\};$ note that your Hilbert transform ignores these sets and then you do have a limit. In other words, the variance of $1/\bar X_n$ seems to fundamentally blow up on the small-probability sets $S(\varepsilon).$ $\endgroup$
    – William M.
    Commented May 9, 2022 at 15:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .