What does it mean for a probability model to be "well-specified" or "misspecified"? I read in several articles that a model $\{p_\theta , \theta \in \Theta\}$ is well-specified if there exists a $\theta^* \in \Theta$ such that $p_{\theta^*} = p^*$.
Similarly, a model $\{p_\theta , \theta \in \Theta\}$ is misspecified if there does not exists a $\theta^* \in \Theta$ such that $p_{\theta^*} = p^*$.
Here, the definition of $p^*$ is that it is the true "data-generating distribution".
I can understand the terminology but what is the underlying logic here? I feel that I am missing the deeper story here. What is an example of a misspecified model? thanks!
 A: Well-specified means that the class of distribution $\mathcal{C}$ you are assuming for your modeling actually contains the unknown probability distribution $p$ from where the sample is drawn.
Misspecified means, on the other hand,  that $\mathcal{C}$ does not contain $p$. You made a modeling assumption, and it is not perfect: for instance, you assume your sample is Gaussian, but (maybe due to noise, or just inherently) it is not actually originating from any Gaussian distribution. It may be close to some Gaussian, if you're lucky, but not exactly.
For instance, many papers dealing with learning the structure of data (think of social sciences, biology, physics) may assume some underlying structure: "the data is coming from some multidimensional Gaussian," or "the different coordinates are independent (product distribution)." This is a modelling assumption which simplifies a lot and typically avoids intractability or impossibility to reasonably analyze the data; but it's most likely only an approximation of the real, actual data. This is what a misspecified model is: a convenient (or even necessary) assumption on the distribution underlying the data, which may only be a reasonable approximation.
