1
$\begingroup$

In a multinomial logistic regression, the predicted probability $\pi$ of each outcome $j$ (in a total of $J$ possible outcomes) is given by:

$ \pi_j = \frac{e^{A_j}}{1+\sum_{g \neq j}^Je^{A_j}} $

where the value $A_j$ is predicted by a series of predictor variables. For instance, here it is predicted by two covariates ($x_1$ and $x_2$), with their associated regression slopes $\beta_1$ and $\beta_2$, and the interaction between the two covariates (with associated regression slope $\beta_{12}$):

$ A_j = e^{(\alpha_j+\beta_{1,j}x_1+\beta_{2,j}x_2+\beta_{12,j}x_1x_2)} $

The model needs to be fitted to real data, and we will want to know how well it fits. For instance, perhaps out of a draw of 10 balls from a sack, 5 were red, 2 were green, and 3 were yellow. We are interested in whether the variables $x_1$ and $x_2$ associated with the sack allow us to predict this result if we plug in the values for $\beta_1$, $\beta_2$, and $\beta_{12}$ from the fitted model.

To obtain a measure of the goodness-of-fit of the model, we need to calculate the log-likelihood formula for a multinomial logistic regression. I am unsure how to go about this. What is the formula for log-likelihood in a multinomial logistic regression of the kind described above?

$\endgroup$
2
$\begingroup$

Let us assume that the probability of observing $\boldsymbol{x}$ under the condition that we are in class $\mathcal{C}_i$ is given by

$$p(\boldsymbol{x}|\mathcal{C}_i)=\dfrac{\exp(-\boldsymbol{w}^T_i\boldsymbol{x})}{\sum_{l=1}^{J}\exp(-\boldsymbol{w}^T_l\boldsymbol{x})}.$$

If we get data $\mathcal{D}=\{(\boldsymbol{x}_1,\boldsymbol{t}_1),\ldots,(\boldsymbol{x}_N,\boldsymbol{t}_N) \}$, in which $\boldsymbol{t}$ is encoding the class as one-hot-encoding vector. For three classes $\boldsymbol{t}=[0,0,1]^T$ signifies that the corresponding observation $\boldsymbol{x}$ is from the third class.

We are assuming that all observations are drawn independently (this allows us to multiply all probability distributions) from the same distribution (this allows using the same probability distribution for all observations). We rewrite the previous probability by using $\boldsymbol{t}=[t_1,\ldots,t_J]^T$. Note, that these are scalars (not bold face).

$$p(\boldsymbol{x}|\boldsymbol{t})=\prod_{j=1}^{J}\left[\dfrac{\exp(-\boldsymbol{w}^T_i\boldsymbol{x})}{\sum_{l=1}^{J}\exp(-\boldsymbol{w}^T_l\boldsymbol{x})}\right]^{t_j}.$$

The likelihood to observe the data $\mathcal{D}$ is given by

$$p(\boldsymbol{x}_1,\ldots,\boldsymbol{x}_N|\boldsymbol{t}_1,\ldots,\boldsymbol{t}_N)=\prod_{n=1}^{N}\prod_{j=1}^{J}\left[\dfrac{\exp(-\boldsymbol{w}^T_i\boldsymbol{x}_n)}{\sum_{l=1}^{J}\exp(-\boldsymbol{w}^T_l\boldsymbol{x}_n)}\right]^{t_{nj}}.$$

Hence, the log-likelihood is given by

$$\log p(\boldsymbol{x}_1,\ldots,\boldsymbol{x}_N|\boldsymbol{t}_1,\ldots,\boldsymbol{t}_N)=\sum_{n=1}^{N}\sum_{j=1}^{J} t_{nj} \log\left[\dfrac{\exp(-\boldsymbol{w}^T_i\boldsymbol{x}_n)}{\sum_{l=1}^{J}\exp(-\boldsymbol{w}^T_l\boldsymbol{x}_n)}\right],$$

in which $t_{nj}$ is the $j^{\text{th}}$ component of the class vector $\boldsymbol{t}_n$ for the $n^\text{th}$ observation $\boldsymbol{x}_n$.

$\endgroup$
  • $\begingroup$ Thanks very much - very helpful! Makes sense that the log-likelihood for the whole model is the sum of the individual log-likelihoods over $N$. Is this still true if the model we are fitting has random effects? In my example, say some observations (coloured balls picked out of sacks) are drawn from sacks by the same person, so 'person' is a random effect. So the model used to predict the weights $w$ would include random effects. In this case, in a sense, the observations are not independent. Can we still sum the individual log-likelihoods to get the overall model's log-likelihood? $\endgroup$ – Sprog Feb 27 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.