0
$\begingroup$

If I have a list of 99 values, say $x_1$ to $x_{99}$, and I have a value for the standard deviation of an $x_{1}$ to $x_{100}$ data set, containing these 99 values + 1 other, can I calculate the potential values for the extra data point? Is there a simple algebraic solution to this that I am missing?

$\endgroup$
7
  • $\begingroup$ Hint: see the definitions of mean, variance and standard deviaton. Then write them as applied to this case. $\endgroup$
    – Matti P.
    Nov 29, 2019 at 11:36
  • $\begingroup$ looking... haha $\endgroup$
    – user197686
    Nov 29, 2019 at 11:36
  • $\begingroup$ so let's say you have: (N-1)std^2 = sum(xi-sum(xi)/N)^2 (i=1 to 3) $\endgroup$
    – user197686
    Nov 29, 2019 at 11:40
  • $\begingroup$ I'll help you get started. So let's first look at the $99$ data points. I'll assume that you have calculated their mean $\mu_{99}$ and variance $\sigma_{99}^2$. Now we assume that $\sigma^2_{100}$ is also given. The formula for $\sigma_{100}^2$ is $$ \sigma_{100}^2 = \frac{1}{100} \sum_{i=1}^{100} (x_i - \mu_{100})^2 = \frac{1}{100} \sum_{i=1}^{99} (x_i - \mu_{100})^2 + \frac{1}{100} (x_{100}- \mu_{100})^2 $$ On the other hand, we know that $\mu_{100} = \frac{99\mu_{99} + x_{100}}{100}$ ... $\endgroup$
    – Matti P.
    Nov 29, 2019 at 11:43
  • $\begingroup$ never used LaTex before, can I post a picture on here to show my working? see if you can point me in the right direction? my assumption was to put mu 100 into the first equation for sigma squared 100 and then expand the squared brackets and re-arrange so that you have two sums, one can be reduced down simply to sigma squared 99 $\endgroup$
    – user197686
    Nov 29, 2019 at 15:48

1 Answer 1

0
$\begingroup$

We are basically comparing two data sets, one with $99$ entries and another with $100$ entries. In the latter, the first $99$ are the same as in the first set and the last value is "unknown". Additionally, we know

  • the mean of the first $99$ values, $\mu_{99}$
  • the variance of smaller set, $\sigma^2_{99}$
  • the variance of larger set, $\sigma^2_{100}$

What we want to know, is the value of the last entry on the larger set, the unknown value $x_{100}$. We can start by considering how to calculate means and variances. It's easy to see that the mean of the larger set is just $$\tag{1} \mu_{100} = \frac{99 \mu_{99}+x_{100}}{100} $$ The variances are $$\tag{2} \sigma^2_{99} = \frac{1}{99}\sum_{i=1}^{99} \left(x_i - \mu_{99} \right)^2 $$ $$\tag{3} \sigma^2_{100} = \frac{1}{100}\sum_{i=1}^{100} \left(x_i - \mu_{100} \right)^2 $$ Let's try to write the latter in terms of the former. Splitting the sum and using Equation ($1$), we get $$ \begin{split} \sigma^2_{100} =& \frac{1}{100}\sum_{i=1}^{99} \left(x_i - \mu_{100} \right)^2 + \frac{1}{100} \left(x_{100} - \mu_{100} \right)^2\\ =& \frac{1}{100}\sum_{i=1}^{99} \left(x_i - \frac{99 \mu_{99}+x_{100}}{100} \right)^2 + \frac{1}{100} \left(x_{100} - \frac{99 \mu_{99}+x_{100}}{100} \right)^2\\ =& \frac{1}{100}\sum_{i=1}^{99} \left(\frac{100x_i -99 \mu_{99}-x_{100}}{100} \right)^2 + \frac{1}{100} \left(\frac{100x_{100}-99 \mu_{99}-x_{100}}{100} \right)^2\\ =& \frac{1}{100}\sum_{i=1}^{99} \left(\frac{100x_i -100 \mu_{99} + \mu_{99}-x_{100}}{100} \right)^2 + \frac{1}{100} \left(\frac{99x_{100}-99 \mu_{99}}{100} \right)^2\\ =& \frac{1}{100}\sum_{i=1}^{99} \left([x_i -\mu_{99}]+\left[\frac{\mu_{99}-x_{100}}{100}\right] \right)^2 + \frac{99^2}{100} \left(\frac{x_{100}- \mu_{99}}{100} \right)^2\\ \end{split} $$ Let's look at the first term. We have $$ \begin{split} \sum_{i=1}^{99} \left([x_i -\mu_{99}]+\left[\frac{\mu_{99}-x_{100}}{100}\right] \right)^2 =& \sum_{i=1}^{99} \left\{ [x_i -\mu_{99}]^2 + 2[x_i -\mu_{99}]\left[\frac{\mu_{99}-x_{100}}{100}\right] + \left[\frac{\mu_{99}-x_{100}}{100}\right]^2 \right\} \\ =& \underbrace{\sum_{i=1}^{99} [x_i -\mu_{99}]^2}_{=99\cdot \sigma^2_{99}} + 2\underbrace{\sum_{i=1}^{99}[x_i -\mu_{99}]\left[\frac{\mu_{99}-x_{100}}{100}\right]}_{=0} + \underbrace{\sum_{i=1}^{99}\left[\frac{\mu_{99}-x_{100}}{100}\right]^2}_{=99\cdot \left[\frac{\mu_{99}-x_{100}}{100}\right]^2} \\ =& 99 \sigma^2_{99} + 99\cdot \left[\frac{\mu_{99}-x_{100}}{100}\right]^2 \end{split} $$ Plugging this back in, we have the equation $$ \sigma^2_{100} = \frac{99}{100} \sigma^2_{99} + \frac{99}{100}\cdot \left(\frac{x_{100}- \mu_{99}}{100}\right)^2 + \frac{99^2}{100} \left(\frac{x_{100}- \mu_{99}}{100} \right)^2 $$ Phew. Can you continue from here?

$\endgroup$
2
  • $\begingroup$ That helped massively! Thanks. One question, why do you use N instead of N-1 when calculating standard deviation? so 1/N instead of 1/(N-1)? $\endgroup$
    – user197686
    Dec 2, 2019 at 14:35
  • $\begingroup$ I assume that it's the population standard deviation. A sample standard deviation would use $\frac{1}{N-1}$. $\endgroup$
    – Matti P.
    Dec 3, 2019 at 5:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .