What does big $O$ notation mean in statistics? I encountered the following proposition about the variance of the sample mean in my text book.
$$\text{Var}(\bar{X}) = \frac{\text{Var}(X_1)}{n} = O(n^{-1}).$$
From this I understand big $O$ notation in statistics to mean the same as it does in $\text{CS}$ (namely $\exists C$ such that $\frac{\text{Var}(X_1)}{n} \leq Cn^{-1}$ for $n \to \infty$) which makes sense because $\text{Var}(X_1)$ is a constant.
However, later on I encounter the following.
$$\text{Var}(S^2) = \mathbb{E}[(S^2-\sigma^2)^2] = \frac{1}{n}(\mathbb{E}[(X_1-\mu)^4]-\frac{n-3}{n-1}\sigma^4)= O(n^{-1}).$$
Which doesn't seem to make sense since ($\mathbb{E}[(X_1-\mu)^4]-\frac{n-3}{n-1}\sigma^4)$ obviously varies with $n.$ Am I missing something?
 A: 
Which doesn't seem to make sense since ($\mathbb{E}[(X_1-\mu)^4]-\frac{n-3}{n-1}\sigma^4)$ obviously varies with n. Am I missing something?

I think you misunderstand what the big-O notation means. For two functions $f,g$ over naturals we say that $f\in O(g)$ if there exists a constant $C$ such that $f(n)\leq Cg(n)$ eventually.
So if $f$ is bounded by say $M$, then by the definition $f(n)\leq M$ for any $n$ and thus $f\in O(1)$. In particular $\sin(n)$ is $O(1)$ even though it varies with $n$.
And so your $f(n)=\mathbb{E}[(X_1-\mu)^4]-\frac{n-3}{n-1}\sigma^4$ component is bounded, thus it is $O(1)$. Therefore $\frac{1}{n}f(n)$ is $O(\frac{1}{n})$ since the big-O preserves multiplication by nonnegative functions.
A: You can rewrite $\frac{n-3}{n-1}=\frac{\frac{n}{n}-\frac3n}{\frac nn-\frac1n}=\frac{1-\frac3n}{1-\frac 1n}$ which clearly tends to $1$ as $n$ goes to infinity. Therefore, your term still goes to zero as fast as $\frac 1n$ as the second factor becomes constant in the limit.
More formally, $f\in O(g)$ means that $\frac fg \leq C$ is bounded for all but finitely many values and one fixed value $C$. In your example, where $f(n)=\frac{1}{n}(E(X_1-\mu)^4-\frac{n-3}{n-1}\sigma^4)$ and $g(n)=\frac 1n$, we can compute $$\frac{f(n)}{g(n)}=\frac{\frac{1}{n}(E(X_1-\mu)^4-\frac{n-3}{n-1}\sigma^4)}{\frac 1n}=E(X_1-\mu)^4-\frac{n-3}{n-1}\sigma^4\to E(X_1-\mu)^4-\sigma^4=:C. $$
You can find the formal definition on wikipedia.
