I have a question concerning mixture Distributions and mixture models. Suppose I am in the following situation: I get a data set with K groups, each group is normally distributed. Now I know, that the complete data is distributed with a mixture distribution, which is a weighted sum of the K normal distributions, that is, the density function is $$f(y)=\sum_{i=1}^K w_i f(y|i) $$ and each $f(y|i)$ is a density of a normal distribution. Now, the common algorithms (for example the EM algorithm) estimate the distribution parameters and the weights $w_i$. But in my data set it is known, to which group the observation belongs, thus we have more information. Now I am not sure, wheter this model makes sense in this case. Are the $w_i$ then simply given by the proportions of the different groups? Suppose I look for the weight of male and female, thus K=2, 100 observations, and I have 20 male, 80 female. Do I get $w_1=0.2$ and $w_2=0.8$? Are these values the REAL ones or are they only estimates? Because if I look for a good estimator for the common distribution for the weight of men and women it doesn't make sense, to use these weights, as the reality is rather about 0.5/0.5.

In other words: I'm not sure, what the $w_i$ really mean...and whether they come from the data or have to be specified in advance.



There are two random variables of interest:

$Y$ - the continuous outcome (e.g. weight)

$I$ - which group the observation belongs to (e.g. male or female)

The $\textit{marginal}$ density of $Y$ is given by the formula \begin{align} f(y) = \sum_{i = 1}^I f(y|i)\Pr(I = i) \tag{1} \end{align} where $f(y|i)$ is the density of $Y$ conditional on $I = i$. Thus $w_i = \Pr(I = i)$ is the true population proportion in group $I$.

Now suppose you have an i.i.d. sample from the population. If you observe both $Y$ and $I$, you can estimate $f(y|i)$ directly. You can also estimate $w_i$ using the sample proportion in group $i$. However this estimate of the distribution of $I$ is not in general equal to the true value $Pr(I = i)$. Weighting your estimates of $f(y|g)$ by estimates of $w_i$ will give you a consistent estimator of the marginal density of $Y$. However if you know the true values of $w_i$, then using these instead of the sample proportions will improve the performance of your estimator (e.g. lower mean squared error).

  • $\begingroup$ Thank you very much. I really understand it better now. Now I am wondering, whether it is possible to fit a regression model to the whole dataset. I can do this for each group (e.g. one model for men and one for women), but if I consider all my observations, their common distribution is given by this sum and therefore difficult to handle. I think ordinary methods as least squares cannot work then. $\endgroup$ – Kathrin Sep 22 '15 at 8:44
  • $\begingroup$ One approach would be to include a dummy variable for gender in the regression and correct for heteroskedasticity. $\endgroup$ – David Sep 22 '15 at 14:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.