Mixture Distribution/ Mixture Model

Question

I have a question concerning mixture Distributions and mixture models. Suppose I am in the following situation: I get a data set with K groups, each group is normally distributed. Now I know, that the complete data is distributed with a mixture distribution, which is a weighted sum of the K normal distributions, that is, the density function is $$f(y)=\sum_{i=1}^K w_i f(y|i) $$ and each $f(y|i)$ is a density of a normal distribution. Now, the common algorithms (for example the EM algorithm) estimate the distribution parameters and the weights $w_i$. But in my data set it is known, to which group the observation belongs, thus we have more information. Now I am not sure, wheter this model makes sense in this case. Are the $w_i$ then simply given by the proportions of the different groups? Suppose I look for the weight of male and female, thus K=2, 100 observations, and I have 20 male, 80 female. Do I get $w_1=0.2$ and $w_2=0.8$? Are these values the REAL ones or are they only estimates? Because if I look for a good estimator for the common distribution for the weight of men and women it doesn't make sense, to use these weights, as the reality is rather about 0.5/0.5.

In other words: I'm not sure, what the $w_i$ really mean...and whether they come from the data or have to be specified in advance.

Thanks!

Answer 1

There are two random variables of interest:

$Y$ - the continuous outcome (e.g. weight)

$I$ - which group the observation belongs to (e.g. male or female)

The $\textit{marginal}$ density of $Y$ is given by the formula \begin{align} f(y) = \sum_{i = 1}^I f(y|i)\Pr(I = i) \tag{1} \end{align} where $f(y|i)$ is the density of $Y$ conditional on $I = i$. Thus $w_i = \Pr(I = i)$ is the true population proportion in group $I$.

Now suppose you have an i.i.d. sample from the population. If you observe both $Y$ and $I$, you can estimate $f(y|i)$ directly. You can also estimate $w_i$ using the sample proportion in group $i$. However this estimate of the distribution of $I$ is not in general equal to the true value $Pr(I = i)$. Weighting your estimates of $f(y|g)$ by estimates of $w_i$ will give you a consistent estimator of the marginal density of $Y$. However if you know the true values of $w_i$, then using these instead of the sample proportions will improve the performance of your estimator (e.g. lower mean squared error).