1
$\begingroup$

Context

The maximum likelihood estimators for a Normal distribution with unknown mean and unknown variance are $$ \widehat{\mu} = \frac{1}{n}\sum_{i=1}^n x_i \qquad \text{and} \qquad \widehat{\sigma}^2 = \frac{1}{n}\sum_{i=1}^n (x_i - \mu)^2 $$ These can be found (for example) by taking derivatives of the average log-likelihood $$ \frac{1}{n}\sum_{i=1}^n \log p(x_i) = -\frac{1}{2}\log(2\pi) - \frac{1}{2n\sigma^2}\sum^n_{i=1} (x^{(i)} - \mu)^2 - \log \sigma $$

Question: What if I want to use a gradient-based method?

Yes, I know I can just use the estimators found above. However, I want to find such estimators using a gradient-based method such as coordinate descent or gradient descent. These are the gradients with respect to $\mu$ and with respect to $\sigma$ (which you can set equal to zero to find the estimators above)

$$ \begin{align} \frac{\partial}{\partial \mu} \frac{1}{n} \sum^n_{i=1} \log p(x^{(i)}) &= \frac{\overline{x}}{\sigma^2} - \frac{\mu}{\sigma^2} \\ \frac{\partial}{\partial \sigma} \frac{1}{n}\sum^n_{i=1} \log p(x^{(i)}) &= \frac{1}{n\sigma^3}\sum^n_{i=1}(x^{(i)} - \mu)^2 - \frac{1}{\sigma} \end{align} $$ I tried using them in gradient descent $$ \begin{align} \mu_{t+1} &\longleftarrow \mu_t + \gamma \left(\frac{\overline{x}}{\sigma^2_t} - \frac{\mu_t}{\sigma^2_t}\right) \\ \sigma_{t+1} &\longleftarrow \sigma_t + \gamma\left(\frac{1}{n\sigma^3_t}\sum^n_{i=1}(x^{(i)} - \mu_{t+1})^2 - \frac{1}{\sigma_t}\right) \end{align} $$ or in coordinate ascent (where I would keep, say $\sigma_t$ fixed and optimize $\mu_t$ for $n_{\text{inner}}$ times and then switch: keep $\mu_t$ fixed and optimize $\sigma_t$ for $n_{\text{inner}}$ times. All this for $n_{\text{outer}}$ times. However it seems to blow up for some reason and not give me the obvious answer. You can run the code here.

maximum likelihood fails

What am I doing wrong?

$\endgroup$

1 Answer 1

2
$\begingroup$

The gradient with respect to $\sigma$ is very large when $\sigma$ is less than than the maximizer, and decreases gradually as $\sigma$ grows when it is larger than the maximizer. (See this plot.) The behavior you are seeing is likely due to a large step size; gradient descent creeps down to the maximizer, but if it overshoots and ends below the maximizer, it will use the huge gradient to take a giant step upward, erasing all the work you have done.

You have to be careful with gradient methods when the gradient is unbounded or varies sharply like this.

$\endgroup$
7
  • $\begingroup$ Would you have any advice then? Should I use a second order method, or simply make the step size smaller? $\endgroup$ Nov 6, 2020 at 23:37
  • $\begingroup$ @Euler_Salter Both ideas might be good, although I am not sure if the second order method would suffer from a similar issue. $\endgroup$
    – angryavian
    Nov 6, 2020 at 23:42
  • $\begingroup$ I tried changing the step size many times but doesn't seem to be leading to any improvement.. $\endgroup$ Nov 6, 2020 at 23:47
  • $\begingroup$ @Euler_Salter You have a typo; the last term of the gradient is $1/\sigma_t$ without a square. $\endgroup$
    – angryavian
    Nov 6, 2020 at 23:54
  • 1
    $\begingroup$ @Euler_Salter I meant in the code you had sigma**2 mistakenly, but I think you fixed it. $\endgroup$
    – angryavian
    Nov 7, 2020 at 0:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .