0
$\begingroup$

In machine learning, when using L2 Regularization we add a penalty of $\lambda \lVert w \rVert^2$ to our error function to get:

$ E = \sum{(y_n - \hat{y}_n)^2 } + \lambda\lVert w \rVert^2$ as our new error function.

This is equivalent to the negative log-likelihood, which we want to minimize. In order to maximize the likelihood we multiply both sides by $-1$ and get:

$ -E = -\sum{(y_n - \hat{y}_n)^2 } - \lambda\lVert w \rVert^2$

which is equivalent to the log-likelihood. Then, exponentiating the function, we obtain:

$\exp(-E) = [\prod\exp{\{-(y_n - w^Tx_n)^2}\}]\exp{\{-\lambda\lVert w \rVert^2}\}$.

Now I am told this yields two gaussians; the one of interest to me is the second one:

$\exp{\{-\lambda\lVert w \rVert^2}\}$.

This is a gaussian with $\mu = 0$, and $\sigma^2 = \frac{1}{\lambda}$.

I can intuitively get this, since the gaussian take the form $\exp\{{-\frac{1}{2}\frac{(x-\mu)^2}{\sigma^2}\}} $, which is then multiplied by some constant as a normalization factor.

But when I tried to convince myself, I came to the point that $\int\exp{\{-\lambda\lVert w \rVert^2}\} = \sqrt{\frac{\lambda}{2\pi}}$. I can't get any farther than this. Is there a way I can finish proving this equality to be true?

$\endgroup$

1 Answer 1

3
$\begingroup$

From the PDF of the $N(0, \sigma^2)$ distribution, we have $$\frac{1}{\sqrt{2\pi \sigma^2}} \int \exp\{-\frac{z^2}{2 \sigma^2}\} \, dz = 1.$$

Taking $\sigma^2 = \frac{1}{2 \lambda}$ yields $$\int_{\mathbb{R}} \exp\{-\lambda z^2\} \, dz = \sqrt{\pi / \lambda}.$$

Writing $\|w\|^2 = w_1^2 + \cdots + w_d^2$, we have $$\int_{\mathbb{R}^d} \exp\{-\lambda \|w\|^2\} \, dw = \left(\int_{\mathbb{R}^d} \exp\{-\lambda w_1^2\} \, dw_1 \right)^d.$$

$\endgroup$
6
  • $\begingroup$ Thank you; but I’m curious. I thought the $\sigma^2 = \frac{1}{\lambda}$. Just curious why did you take $\sigma = \frac{1}{2\lambda}$? $\endgroup$
    – Hanzy
    Jul 9, 2018 at 22:55
  • 1
    $\begingroup$ @Hanzy $\exp(-\lambda z) = \exp(-\frac{1}{2\sigma^2}z) \Rightarrow \lambda = \frac{1}{2\sigma^2} \Rightarrow \sigma^2 = \frac{1}{2\lambda}$ $\endgroup$
    – Alex Jones
    Jul 9, 2018 at 23:14
  • $\begingroup$ @AlexanderJ93 Maybe I typed my MathJax wrong or did some step wrong because the instructor mentions that $\sigma^2 = \frac{1}{\lambda}$. I also see that this post similarly imposes a gaussian with $\sigma^2 = \frac{1}{\lambda}$. So I have messed up somewhere. Your answer makes sense from my original question, but I can't see what I've done wrong yet. $\endgroup$
    – Hanzy
    Jul 9, 2018 at 23:52
  • $\begingroup$ @Hanzy In the linked post, they are being loose with constants. Note that the difference in the two versions of $\lambda$ is ultimately just a factor of two in the log-likelihood. $\endgroup$
    – angryavian
    Jul 9, 2018 at 23:55
  • 1
    $\begingroup$ @angryavian, ok I think I see my confusion now. I'm trying to prove the wrong distribution. I have a follow up question but I will put it in a new post so as not to follow up. I understand why this answer is correct. $\endgroup$
    – Hanzy
    Jul 10, 2018 at 0:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .