0
$\begingroup$

In machine learning, when using L2 Regularization we add a penalty of $\lambda \lVert w \rVert^2$ to our error function to get:

$ E = \sum{(y_n - \hat{y}_n)^2 } + \lambda\lVert w \rVert^2$ as our new error function.

This is equivalent to the negative log-likelihood, which we want to minimize. In order to maximize the likelihood we multiply both sides by $-1$ and get:

$ -E = -\sum{(y_n - \hat{y}_n)^2 } - \lambda\lVert w \rVert^2$

which is equivalent to the log-likelihood. Then, exponentiating the function, we obtain:

$\exp(-E) = [\prod\exp{\{-(y_n - w^Tx_n)^2}\}]\exp{\{-\lambda\lVert w \rVert^2}\}$.

Now I am told this yields two gaussians; the one of interest to me is the second one:

$\exp{\{-\lambda\lVert w \rVert^2}\}$.

This is a gaussian with $\mu = 0$, and $\sigma^2 = \frac{1}{\lambda}$.

I can intuitively get this, since the gaussian take the form $\exp\{{-\frac{1}{2}\frac{(x-\mu)^2}{\sigma^2}\}} $, which is then multiplied by some constant as a normalization factor.

But when I tried to convince myself, I came to the point that $\int\exp{\{-\lambda\lVert w \rVert^2}\} = \sqrt{\frac{\lambda}{2\pi}}$. I can't get any farther than this. Is there a way I can finish proving this equality to be true?

$\endgroup$
3
$\begingroup$

From the PDF of the $N(0, \sigma^2)$ distribution, we have $$\frac{1}{\sqrt{2\pi \sigma^2}} \int \exp\{-\frac{z^2}{2 \sigma^2}\} \, dz = 1.$$

Taking $\sigma^2 = \frac{1}{2 \lambda}$ yields $$\int_{\mathbb{R}} \exp\{-\lambda z^2\} \, dz = \sqrt{\pi / \lambda}.$$

Writing $\|w\|^2 = w_1^2 + \cdots + w_d^2$, we have $$\int_{\mathbb{R}^d} \exp\{-\lambda \|w\|^2\} \, dw = \left(\int_{\mathbb{R}^d} \exp\{-\lambda w_1^2\} \, dw_1 \right)^d.$$

$\endgroup$
  • $\begingroup$ Thank you; but I’m curious. I thought the $\sigma^2 = \frac{1}{\lambda}$. Just curious why did you take $\sigma = \frac{1}{2\lambda}$? $\endgroup$ – Hanzy Jul 9 '18 at 22:55
  • 1
    $\begingroup$ @Hanzy $\exp(-\lambda z) = \exp(-\frac{1}{2\sigma^2}z) \Rightarrow \lambda = \frac{1}{2\sigma^2} \Rightarrow \sigma^2 = \frac{1}{2\lambda}$ $\endgroup$ – AlexanderJ93 Jul 9 '18 at 23:14
  • $\begingroup$ @AlexanderJ93 Maybe I typed my MathJax wrong or did some step wrong because the instructor mentions that $\sigma^2 = \frac{1}{\lambda}$. I also see that this post similarly imposes a gaussian with $\sigma^2 = \frac{1}{\lambda}$. So I have messed up somewhere. Your answer makes sense from my original question, but I can't see what I've done wrong yet. $\endgroup$ – Hanzy Jul 9 '18 at 23:52
  • $\begingroup$ @Hanzy In the linked post, they are being loose with constants. Note that the difference in the two versions of $\lambda$ is ultimately just a factor of two in the log-likelihood. $\endgroup$ – angryavian Jul 9 '18 at 23:55
  • 1
    $\begingroup$ @angryavian, ok I think I see my confusion now. I'm trying to prove the wrong distribution. I have a follow up question but I will put it in a new post so as not to follow up. I understand why this answer is correct. $\endgroup$ – Hanzy Jul 10 '18 at 0:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.