0
$\begingroup$

In wikipedia the following is said: "We model a set of observations as a random sample from an unknown joint probability distribution which is expressed in terms of a set of parameters."

I don't understand this entire statement.

What is meant with modeling the set of observations ?

what is this joint probability, why is it joint, should it be joint exclusively, where does it comes from?

What I know is that if you have a sample of Data from a population, with assumed probability distribution,then you can have an estimator for this sample. The estimators of an arbitrary parameter can be different. In this case the estimator is called the Maximum likelihood estimator. So while I know this much, I don't understand this joint probability. What is this? The sample can be that of a single random variable, so joint probability doesn't exist here.

EDIT: The same goes for the likelihood function. It's definition in wikipedia is confusing: "The likelihood function (often simply called the likelihood) is the joint probability of the observed data viewed as a function of the parameters of the chosen statistical model."

joint of what with what? I don't understand

$\endgroup$
2
  • $\begingroup$ If the sample size is at least 2, then it is a joint distribution because you are looking at say the distribution of $x_1,x_2$ $\endgroup$
    – Andrew
    Nov 24, 2022 at 21:33
  • $\begingroup$ the sample size which contains actual values is somehow considered a joint distribution, which in the simple case of a discrete case, it is a probability distribution. How can we make this assumption ? $\endgroup$
    – imbAF
    Nov 24, 2022 at 22:02

1 Answer 1

1
$\begingroup$

Take a series of $n$ events or observations of any kind (for example, dice rolls) and call them $x_1, x_2, ..., x_n$. We generally understand $x_1, x_2, ..., x_3$ as randomly coming out of some well-defined, although possibly unknown, probability space.

This is simply an assumption, but is a reasonable one to make. Most series of events do not occur in a completely entropic or senseless way. On the contrary, for the occurence of every $x_i$ we assume there is a definite probability $P(x_i)$, and that probability comes from some probability distribution that governs the variable.

Furthermore, if you observed $x_1, x_2, ..., x_n$ then not only each $x_i$ has a probability, but the fact that they all occurred together has some probability $P(x_1 \land x_2 \land...\land x_n)$ (this reads "the probability that $x_1$ and $x_2$ and ... $x_n$ all occur"). This is the joint probability of $x_1,...x,_n$.

This is what the quote

"We model a set of observations as a random sample from an unknown joint probability distribution which is expressed in terms of a set of parameters."

means. That we assume any set of observations as coming (or being samples of) some joint (simultaneous occurrence) probability distribution. Probability distributions are governed by parameters (for example, the mean and standard deviation for the normal distribution), which is what's stated in the last phrase of the quote.


The same goes for the likelihood function. It's definition in wikipedia is confusing: "The likelihood function (often simply called the likelihood) is the joint probability of the observed data viewed as a function of the parameters of the chosen statistical model."

A probability function tells the probability of an event $P(x_i)$ or some joint probability for a series of events $P(x_1 \land x_2 \land .... \land x_n)$ given some distribution parameters. For example, $P(1.25|\mu=0, \sigma=1)$ would mean "the probability of sampling $1.25$ from a normal distribution with mean $0$ and standard deviation $1$". This extends to joint probabilities: $P(1.25 \land 0.92 \land ... |\mu=0, \sigma=1)$ would be the probability of sampling $1.25$ and $0.92$ and a bunch of other values when the parameters of the distribution are those.

A likelihood function tells you the opposite: the "probability" that a distribution has certain parameters given some sampled data. For example, $P(\mu=0, \sigma=1|1.25 \land 0.92 \land ...$ tells what is the "probability" that the parameters of the distribution are $\mu=0, \sigma=1$ when we have sampled $1.25, 0.92,...$ etc.

I put quotation marks around "probability" because the likelihood function does not actually return a probability (it's codomain is not $[0, 1]$), but a likelihood (higher values means greater likelihood, lower values lower likelihood). That is why the symbol $\mathcal{L}$ and not $P$ is actually used.

In other words, likelihood estimation estimates how well some distribution parameters fit given a series of observations. Again, here the word "join" comes from the fact that probabilites (or likelihoods) are taken not on a single event but on a series of events occuring together.

$\endgroup$
1
  • $\begingroup$ The answer is correct, however the probability of a single value applies only to discrete random variables. The given example shows non integer values suggesting that variables are continuous. In this second case, it will be more appropiate to use probability density functions instead. $\endgroup$ May 14 at 0:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .