generative vs discriminative machine learning

Could someone provide differences between these two techniques? Also, I would very much appreciate if you could give an example of a ML method that falls into discriminative model or gnerative one. For instance, is Perceptron discriminative? K-Means?

-

2 Answers

Let $X$ be your observed data and let $Y$ be their unobserved/hidden properties. In a ML setting, $Y$ usually holds the categories of $X$.

A generative model models their joint distribution, $P(X,Y)$.

A discriminative model models the posterior probability of the categories, $P(Y|X)$.

Depending on what you want to do, you choose between generative versus discriminative modeling. For example, if you are interested in doing classification, a discriminative model would be your choice because you are interested in $P(Y|X)$. However, you can use a generative model, $P(X,Y)$, for classification, too.

A generative model allows you to generate synthetic data ($X$) using the joint but you cannot do this with a discriminative model.

-

Consider a classification problem at the simplest approach one can get to its goal by designing a discriminant function. for example: a perceptron $y=sign(\theta^Tx)$ you can learn the parameters by a algorithm. in this case you should use Perceptron Learning Rule.

the above approach is not probabilistic and is called 'discriminant function' approach.

a better goal is trying to predict the labels probabilistically. the best goal is to have P(y|X) that is called the posterior.

Example of a discriminative approach: logistic regression

$P(y=1|X)=g(\theta^Tx)$,where g is the sigmoid function.

Example of a generative model:

$P(y|X)=P(X|y)P(y)$

you can model $P(X|y=1)=N(\mu_1,\Sigma_1)$ and $P(X|y=0)=N(\mu_0,\Sigma_0)$ and estimates its parameters from data. (the simple approach is Maximum likelihood).

-