I was wondering if you could provide some clarifications regarding the derivation of the negative log likelihood function.

Let $\ell := \frac{1}{N}\sum_{n=1}^{N}\left[-log(\sum_{n=1}^{N} \alpha(x)\;\mathcal{N}\left(\mu(x),\;\sigma(x))\right)\right]$ be our mean negative log likelihood function where:

$ \begin{align*} x, y &= \mathbb{R}^{N} \\ \alpha(x) &= \frac{e^{f(x_{i})}}{\sum_{j=1}^{N}e^{f(x_{j})}} \\ \mu(x) &= f(x) \\ \sigma(x) &= e^{f(x)} \\ f(x) &= W_{\alpha,\mu,\sigma}x + b_{\alpha,\mu,\sigma}\;\mbox{some affine transformation} \end{align*} $

Now the derivatives are accordingly:

$ \begin{align*} \gamma &= -\frac{1}{\sum_{n=1}^{N}\mathcal{N}(\mu(x),\;\sigma(x))}\alpha(x)\;\mathcal{N}(\mu(x),\;\sigma(x)) \\ \frac{\partial\ell}{\partial\mu} &= \gamma\frac{\mu-y}{N\sigma^{2}} \\ \frac{\partial\ell}{\partial\sigma} &= \gamma(\frac{1}{N} - \frac{(y-\mu)^{2}}{N\sigma^{2}}) \\ \frac{\partial\ell}{\alpha} &= \frac{\alpha - \gamma}{N} \end{align*} $

Now my confusion starts with $\gamma$, I believe that the numerator should have been $N\cdot\alpha(x)\cdot(1-\alpha(x))\cdot\mathcal{N}(\mu, \sigma)$ ?

Am I missing something here?

According to my understanding $\frac{1}{N}\sum_{n=1}^{N}-log(\phi(x_{n})) = -log(\phi(x_{n}))$ and $\sum_{n=1}^{N}\alpha(x)\mathcal{N}(\mu(x),\sigma(x)) = N\cdot\alpha(x)\cdot\mathcal{N}(\mu(x),\sigma(x))$.

Then the derivative of $\frac{\partial-log(\phi(x_{n}))}{\partial\phi} = \frac{1}{\phi(x_{n})}\phi\prime(x_{n})$ if we let $\phi(x_{n}) = \sum_{n=1}^{N}\alpha(x)\mathcal{N}(\mu(x),\sigma(x))$.

Did I miss something in the derivation? I can't see clearly where does $\gamma$ come from if $log(f(x)g(x)) = log(f(x)) + log(g(x))$ then the derivative should be $\frac{1}{f(x)}f\prime(x) + \frac{1}{g(x)}g\prime(x)$

Thanks in advance for your help!

  • $\begingroup$ I would strongly suggest you to ask this question at stats.stackoverflow.com as it's a quite complex Statistics-related question $\endgroup$ – David Jul 1 at 16:21
  • $\begingroup$ I'll do thanks for the suggestion. $\endgroup$ – Jane Dane Jul 1 at 18:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.