6
$\begingroup$

I'm measuring a characteristic of a device that has a normal distribution ($0$ mean and std dev of $\sigma_M$).

There is, however, noise in the measurement process, which also has a normal distribution ($0$ mean and std dev of $\sigma_N$). I can measure this noise independently.

I can estimate the device's true characteristic (without noise) as $\sigma_D = \sqrt{\sigma_M^2 - \sigma_N^2}$. To be compliant with a specific spec, $\sigma_D$ must be less than $L$.

If the noise is small compared to the measured value, I have high confidence in my data. But my confidence drops as the noise approaches the measured value. In the extreme case, if $\sigma_M = \sigma_N$, my estimation returns $\sigma_D = 0$, indicating that I've reached the noise floor of my equipment (I think that's the correct interpretation, but let me know if not).

confidence in data, scenarios

My question is, how close can $\sigma_N$ be to $\sigma_M$ such that $\sigma_D$ is still "accurate"?

I don't want to report a value of $\sigma_D$ that contains too much error. Rather, I'd like to report some lower bound for $\sigma_D$ once $\sigma_N$ becomes too close to $\sigma_M$. Any light you can shed to help me define that lower bound would be much appreciated.

UPDATE 1

  • To clarify, I measure $(1)$ the device with noise and, separately at a later time, $(2)$ the noise (without the device).
  • The distribution of noise measured directly (without the device) can be assumed to also exist when the device is measured with noise, and that it is the ONLY noise present when the device is measured with noise.

UPDATE 2

Is the following statistically meaningful as a condition where $\sigma_D$ is inaccurate: $$ \sigma_M - \sigma_N < \dfrac{1.96}{\sqrt{n}}(\sigma_M + \sigma_N) \;, $$ where $n$ is the number of samples used to compute the sigmas?

$\endgroup$
  • $\begingroup$ Are you saying you can measure what the contribution of the noise was in a given measurement of the device, or that you can only separately measure the device with noise and the noise? The former problem is very easy, the latter problem seems subtle. $\endgroup$ – Ian Jul 1 '17 at 15:40
  • $\begingroup$ It's an issue of when the measurements happen. You can measure the noise, but presumably you can't measure the actual contribution of the noise at the same time as you measure the device. Instead you can measure the distribution of the noise but not how it actually contributed to your device experiments. Is that correct? Again I ask because the other interpretation is a trivial problem (since you can simply subtract off the noise measurements). $\endgroup$ – Ian Jul 1 '17 at 15:54
  • $\begingroup$ In symbols, I'm asking whether you have $\{ X_i \}_{i=1}^n$ and $\{ X_i + Y_i \}_{i=1}^n$ vs. $\{ X_i \}_{i=1}^n$ and $\{ X_i + Y_i \}_{i=n+1}^{2n}$, where $X$ is the noise and $Y$ is the signal. I think you have the second one. $\endgroup$ – Ian Jul 1 '17 at 15:59
  • $\begingroup$ The second one. See update 1 above. $\endgroup$ – user46688 Jul 1 '17 at 16:27
  • $\begingroup$ What you can really find is actually a probability and a confidence interval. The notation in Update 2 should be changed, as you do not know the true values of the variances, but only their estimates. Please, denote estimated values with a hat, e.g. $\hat{\sigma}_D$. The bound you are looking for is the Cramer-Rao Lower Bound. If you knew the true value of $\sigma_D$, the problem would be much, much easier. But you don't and you have to take that into account. $\endgroup$ – PseudoRandom Jul 2 '17 at 9:02
2
$\begingroup$

The question can be reduced to evaluation of the probability $\mathbb{P}(\hat{\sigma}_D < L)$, where $L>0$ is known. It is sufficient to calculate the CDF of $\hat{\sigma}_D$. Unfortunately, it appears no closed-form exists.

Part 1. We call secondary data the noise-only measures. They are a sequence $\{ y_1, \dots, y_n \}$ such that $y_i = n_i$, for $i=1,\dots,n$, where $n_i \sim \mathcal{N}(0, \sigma^2_N)$ are i.i.d. gaussian random variables (RVs), aka, noise. From standard estimation theory, (see note) $$ \hat{\sigma}^2_N = \frac{1}{n} \sum_{i=1}^n y_i^2 $$ is the MLE (Maximum Likelihood Estimator) of $\sigma^2_N$ and it follows a $\chi^2$ distribution with $n$ degrees of freedom. More precisely: $$ n \frac{\hat{\sigma}^2_N}{\sigma^2_N} \sim \chi^2_{n} $$ Let $z_i$ denote primary data (signal+noise). We have that $z_i = x_i + w_i$, for $i=1,\dots,n$, where: $x_i \sim \mathcal{N}(0, \sigma^2_D)$ represents the device, $w_i \sim \mathcal{N}(0, \sigma^2_N)$ is noise, independent of $x_i$ and $n_i$. Again, $$ \hat{\sigma}^2_M = \frac{1}{n} \sum_{i=1}^n z_i^2 $$ from which $$ n \frac{\hat{\sigma}^2_M}{\sigma^2_M} \sim \chi^2_n $$ Note: There is no need to subtract the sample mean $\bar{y}$ since we know that $\mathbb{E}[y_i] = 0$. If you actually use $$ \tilde{\sigma}^2_N = \frac{1}{n} \sum_{i=1}^n (y_i - \bar{y})^2 $$ this is not ML anymore. It still is $\chi^2$ distribution, but with $(n-1)$ degrees of freedom. More precisely, $(n-1)s^2 / \sigma^2 \sim \chi^2_{n-1}$, where $s^2$ is the unbiased sample variance, defined as: $s^2 = \frac{1}{n-1} \sum_{i=1}^n (x_i - \bar{x})^2$. The precise mathematical statement follows from Cochran's theorem.


Part 2. We know that $\rm Var[z_i] = Var[x_i] + Var[w_i]$, so we can compute $$ \hat{\sigma}^2_D = \hat{\sigma}^2_M - \hat{\sigma}^2_N $$ Essentially, we now need to compute the CDF of the difference between two independent $\chi^2$ RVs, which is not trivial. This is complicated by the fact that some coefficients are needed to make things right. We need to use the following result.

Lemma. Let $X,Y$ be two independent $\chi^2_n$. The PDF of $Z=X-Y$ is given by $$ f_Z(z) = \frac{1}{\sqrt{\pi} 2^{n/2}} \frac{1}{\Gamma \Big( \frac{n}{2} \Big)} |z|^{(n-1)/2} K_{\frac{n-1}{2}}\Big( |z| \Big) $$ where $K(\cdot)$ is the modified Bessel function of the second kind and $\Gamma(\cdot)$ is the Gamma function. Proof. See here.

Denoting the PDF of $\hat{\sigma}^2_D$ with $f_Z(z)$, the CDF is given by $$ \mathbb{P}(\hat{\sigma}^2_D \leq t) = F_Z(t) = \int_{-\infty}^t f_Z(z) dz $$ Since $\hat{\sigma}_D = \sqrt{\hat{\sigma}^2_D}$, your solution is $$ \mathbb{P}(\sqrt{\hat{\sigma}^2_D} < L) = \mathbb{P}(\hat{\sigma}^2_D < L^2) = F_Z(L^2) = \int_{-\infty}^{L^2} f_Z(z) dz $$ which is the probability that the device is compliant.


ADDENDUM. To answer the accuracy question, define the Signal-to-Noise Ratio (SNR) as follows $$ SNR = \frac{\sigma^2_D}{\sigma^2_N} $$ which you can compute using estimated values (use big values of $n$, since, ideally, you would like to have $n \rightarrow +\infty$). SNR is a useful measure. First, $SNR \geq 0$ always. Second, in the limit $\sigma^2_N \rightarrow +\infty$ (infinitely powerful noise), we have $SNR=0$, while $\sigma^2_D \rightarrow +\infty$ (infinitely powerful signal) implies $SNR=+\infty$. In other words, the bigger the SNR, the better.

SNR is a quantitative metric tied to the accuracy of your measurements. Sometimes, you will see a threshold-based approach to define "accuracy": if $SNR \geq \gamma$, where $\gamma>0$ is arbitrarily decided (e.g. $\gamma = 10^3$), then you label the results as ``accurate'', inaccurate otherwise. But this approach is flawed, since accuracy is treated as a binary value, which is too simplistic.

A better approach is to compute $$ \eta = 1 - \frac{1}{SNR +1} $$ Why and how does this work? For $SNR=0$ (infinitely powerful noise or zero signal), $\eta=0$. For $SNR=+\infty$ (zero noise or infinitely powerful signal), $\eta=1$. So, clearly, $\eta \in [0,1]$, with extreme values taken only under limiting conditions. If you now use $a_{[\%]} = 100\eta$, you can interpret $a_{[\%]}$ directly as accuracy itself expressed in percentage. So, for example, $\eta=0.9$ implies 90% accurate measures, while $\eta=0.1$ implies rather inaccurate measures. This gives us a quantitative measure of the accuracy of our measures, which is also simple to calculate and intuitively appealing.

$\endgroup$
  • $\begingroup$ Thanks @PseudoRandom. What is $f_z(z)$ (besides the PDF of $\hat{\sigma_d}^2$)? Does it relate to $\hat{\sigma_M}$ or $\hat{\sigma_N}$? Can I, for example, create a model for $f_z(z)$ empirically from the population of $z_i$ measurements, then integrate it to L to solve for the probability that $\sqrt{\hat{\sigma_d}^2}<L$? Not sure how to proceed. $\endgroup$ – user46688 Jul 5 '17 at 18:56
  • $\begingroup$ I edited my answer, adding an answer to the accuracy question you asked, which you may find particularly useful since it is easy to calculate. Regarding $f_Z(z)$, it is not easy to find its expression, since we don't have a "simple" case of $Z=X-Y$, where $X,Y$ are $\chi^2_n$ independent, but $Z = c_1X - c_2 Y$, with $c_1 = n / \sigma^2_M$ and $c_2 = n/ \sigma^2_N $. For $c_1 = c_2 =1$, $f_Z(z)$ reduces to the PDF in the given link, which is still quite complicated, as it involves modified Bessel functions of the second kind. $\endgroup$ – PseudoRandom Jul 5 '17 at 19:27
  • $\begingroup$ After you find $f_Z(z)$, you just integrate it from $-\infty$ to $L^2$. That's it. Most probably, you will need to approximate it numerically. It will give you the desired $\mathbb{P}(\hat{\sigma}_D < L)$. The parameters $\sigma^2_M$ and $\sigma^2_N$ will be inside $f_Z(z)$. Since you don't actually know the true values, just use estimates. At the end of the day, you will have a big integral, from $-\infty$ to $L^2$, of a function which depends on $n, \hat{\sigma}^2_M, \hat{\sigma}^2_N$ in complicated ways. At this point, I think SNR is better. $\endgroup$ – PseudoRandom Jul 5 '17 at 19:41
  • $\begingroup$ In fact, I strongly suggest you to: (1) use SNR to evaluate accuracy (really, SNR is a very good idea!); (2) if you really want that probability, look for a paper along the lines of "Linear Combination of two independent chi-squared random variables". Or just use the $f_Z(z)$ given in the link as an approximation, it will still work. $\endgroup$ – PseudoRandom Jul 5 '17 at 19:49
  • $\begingroup$ Thanks @PseudoRandom, really appreciate all your work here. The SNR is a nice way to formulate some measure of accuracy. $\endgroup$ – user46688 Jul 5 '17 at 21:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.