2
$\begingroup$

NB: I was gonna post on physics stack exchange, not really sure where this fits in. But I'm only a lowly Engineer so please go easy on the notation if you can

Using Student's t-distribution I can infer the parameters ($\mu,\sigma^2$) of a probability distribution based on $n$ samples of data that I assume will fit a gaussian prior. However in all the examples I've seen, the $n$ samples are all simple values. How can I infer a probability distribution based on samples of data with uncertainty; if my $n$ samples are not simple values but probability distributions themselves? What is the effect of measurement uncertainty on the shape of the inferred distribution?

Context

I'm trying to measure how long some code takes to run on a computer. The timer is low resolution - similar order of magnitude to the duration I'm trying to measure - and so the true timestamps are quantized into 100 ms bins. Assuming a uniform rectangular probability distribution within these bins, then the time differences have a triangular probability distribution.

i.e. A task starting at $142ms$ and ending at $331 ms$ when quantised will appear to start at $100\pm50ms$ and end at $300\pm50ms$. Then the difference will be a triangular probability distribution, centered on $200ms$ and with a width of $\pm 100ms$.

I have several of these triangular timespan measurements, and I'd like to use them to determine the parameters of a distribution. As I say, I could just ignore the quantisation errors in my samples, and plug the modal (centre) values into the t-distribution, but surely those errors will increase the uncertainty ($\sigma$) of my inferred gaussian?

$\endgroup$
4
  • 1
    $\begingroup$ What if you just record the time required for 100 back-to-back tasks? Then the resolution of the timer is less significant in the overall measurement, and you would be getting a good measure of $\sum_{i=1}^{100} X_i$. Or make it 1000 back-to-back tasks if you like. $\endgroup$
    – Michael
    Jul 18, 2020 at 18:48
  • 1
    $\begingroup$ @Michael true, I'll be doing that next. But this is more a question of theoretical understanding than practical - I want to try and understand all sources of uncertainty, then work out how to mitigate them $\endgroup$
    – Greedo
    Jul 18, 2020 at 18:50
  • $\begingroup$ Very nice question. (+1) If we suppose $\{X_i\}$ are i.i.d. then a drawback of measuring $\sum_{i=1}^{n} X_i$ is that the measurement will appear "more Gaussian" (by the central limit theorem). By aggregating $n$ tasks we mitigate the resolution issue and get more accuracy about the mean, but less accurate information about actual distribution. So what is the fundamental tradeoff with $n$ here, given we have a fixed resolution? $\endgroup$
    – Michael
    Jul 18, 2020 at 18:53
  • $\begingroup$ @Michael TBH I'm not sure - how can these to effects be quantified? I think ultimately it'll be down to a user whether they want to know average running time with small uncertainty, or the nature of the distribution of running times - where each individual measurement has larger uncertainty. But I'm not sure how you might go about combining the results of these 2 approaches to decide how many tasks you want in each batch to give good insight on average running time and variance? $\endgroup$
    – Greedo
    Aug 23, 2020 at 11:12

1 Answer 1

2
$\begingroup$

All you have to do is incorporate this uncertainty into the statistics that you are measuring. I'll give you examples using $\bar{x}$ and $s^2$, but it's not difficult to see how this generalizes.

Suppose we have some function of $n$ variables: $$f:\mathbb{R}^n\to \mathbb{R} ~; f:(x_1,...,x_n)\mapsto f(x_1,...,x_n)$$ and we have some error $\delta x_i$ on each argument, the error in $f$ will be roughly $$\delta f=\sqrt{\sum_{i=1}^{n} \left(\frac{\partial f}{\partial x_i}\delta x_i\right)^2}$$

Let's apply this, for example, to the sample mean. For a sample of data $x_1,...x_n$ the sample mean is $$\bar{x}=\frac{1}{n}\sum_{i=1}^n x_i$$ Therefore, $\forall i\in [1,n]$, $$\frac{\partial \bar{x}}{\partial x_i}=\frac{1}{n}.$$ Therefore if each of our measurements has an associated error $\delta x_i$, the maximum error when measuring the sample mean is $$\delta \bar{x}=\frac{1}{n}\sqrt{\sum_{i=1}^n {\delta x_i}^2}$$ Thus, when estimating the population mean, you have to incorporate both the standard error, as usual, but also add on whatever error you get from the above. Let's also do the sample standard deviation as well. Recall that $$s^2=\frac{1}{n-1}\sum_{i=1}^n (x_i-\bar{x})^2$$ And so $$\frac{\partial s^2}{\partial x_i}=\frac{1}{n-1}\frac{\partial}{\partial x_i}(x_i-\bar{x})^2$$ Using what we determined above, this is $$\frac{\partial s^2}{\partial x_i}=\frac{2}{n-1}(x_i-\bar{x})\left(1-\frac{1}{n}\right)=\frac{2}{n}(x_i-\bar{x})$$ Thus $$\delta s^2=\frac{2}{n}\sqrt{\sum_{i=1}^n (x_i-\bar{x})^2{\delta x_i}^2}$$ Note: Don't confuse $\delta s^2$ with $(\delta s)^2$ !

So to expand this answer using some of the topics mentioned in the comments: suppose we have some real data points $x_1,...,x_n=\mathbf{x}$ which are intended to measure the position in $\Bbb{R}^n$ of some desired point $\mathbf{x}_0.$ Now suppose that each measurement has corresponding errors $\delta x_i$ which I.I.D random variables, each following a PDF $p$ on $\Bbb{R}$. We assume that the desired point $\mathbf{x}_0$ is within the vector range $(x_1\pm\delta x_1,...,x_n\pm\delta x_n)$. The question is: How large do we expect the net error, $\Vert \mathbf{x}-\mathbf{x}_0\Vert$, to be? Basically what we need to do is define the random vector $\delta\mathbf{x}=(\delta x_1,...,\delta x_n)$ and take the expected value of $\Vert\delta\mathbf{x}\Vert.$ So in principle this should look like $$\mathrm{E}\left(\Vert\delta\mathbf{x}\Vert\right)=\int_0^\infty \epsilon\cdot \mathrm{P}(\Vert\delta\mathbf{x}\Vert=\epsilon)\mathrm{d}\epsilon$$ The $\mathrm{P}(\Vert\delta\mathbf{x}\Vert=\epsilon)$ bit is an integral in and of itself, and is is quite tricky: it has to do with the volume of an $n$ dimensional spherical shell bound by the radii $\epsilon$ ; $\epsilon + \mathrm{d}\epsilon$ and weighted by the value of the PDF $p$ for each $\delta x_i$ at that point. This is much easier if we assume the $\delta x_i$ are all I.I.D, since then the integral is symmetric and can be reduced to one dimension. I'll need a bit more time if I'm to iron out all the details, though.

$\endgroup$
6
  • $\begingroup$ Hi, sorry I'm replying so late, it took a while to let this sink in! I'm trying to think about how I take what you've told me and use it for uncertainty distributions. I guess what I do is imagine $\delta x_i$ taking on a distribution of values - substituting this into your formula for $\delta f$ would turn the sum into a convolution of all the distributions scaled by $\frac{\partial f}{\partial x_i}$ I think, which would make the $\delta f$ distribution tend to a gaussian for large $n$. However I'm not sure is this value for $\delta f$ valid for large $\delta x_i$? $\endgroup$
    – Greedo
    Aug 23, 2020 at 10:57
  • $\begingroup$ ... for example, if $x_i$ follows a gaussian distribution, then it is most likely that $\delta x_i$ takes on a value that is small relative to $x_i$ - meaning using the partial differential evaluated at $x_i$ to "step along" the function $f$ is a good first order approximation. But because $x_i$ has uncertainty distributed all the way to infinity, surely the resultant $\delta f$ distribution will be inaccurate at the fringes of what $x_i$ could possibly take on? Also why do we use Root Sum Squared for getting the error on $f$? $\endgroup$
    – Greedo
    Aug 23, 2020 at 11:05
  • $\begingroup$ Basically, would you be able to expand a little on how this all applies to uncertainty distributions rather than maximum errors as in your examples? PS, thanks so much for all the help so far:) $\endgroup$
    – Greedo
    Aug 23, 2020 at 11:07
  • $\begingroup$ @Greedo Perhaps "maximum" was a bad way of putting it. Yes, it is not possible for $\delta f$ to be larger than the value described above. But, it is usually true that it will be of the same order of magnitude as the quantity described above. But, the above is not a probabilistic argument, simply a geometric one. A simple example: Let's say there is a particle in 3 dimensional space. Now, let's say someone has told you the $x,y,z$ coordinates of the particle, but each figure is actually a $\pm 0.1$ meter range.... $\endgroup$
    – K.defaoite
    Aug 23, 2020 at 12:30
  • $\begingroup$ What is the volume of the region that contains all possible values for the position of the particle? Well, it is simply a sphere in 3D space (with unequal errors, it would be an ellipsoid in general) and its radius is simply the square root of the sum of squares of the uncertainties: In this case $\sqrt{3\cdot 0.1^2}.$ It is then a very different question to ask: What is the expected amount of error when we guess the particles position to simply be the center of the sphere? To ask this question we need to know about the distributions of the uncertainties. $\endgroup$
    – K.defaoite
    Aug 23, 2020 at 12:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.