3
$\begingroup$

I am new to machine learning. I am currently studying various kinds of regressions using the scikit-learn Python library. Here's an example from the documentation: https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression There are a lot of mathematical formulas here and I don't understand many of the notation.

For example : There is some function :

y(w,x) = w0 + w1*x1 + ... + wp * xp
and 
vector w = (w1,....wp)

In that case linear regression looks like:

enter image description here

What does it mean "||" and two "2" in bottom and top? I guess "2" on top means square.

Could anyone explain on the fingers or give a link to the source, where the symbols are explained. Thanks.

$\endgroup$
4
  • $\begingroup$ What do you understand and don't you understand? You have to be more specific; what's your current knowledge? Are these the only examples? $\endgroup$ Aug 17, 2021 at 8:37
  • $\begingroup$ I edited my question $\endgroup$
    – harp1814
    Aug 17, 2021 at 9:08
  • 1
    $\begingroup$ "2" on top indeed means square. "2" at the bottom means the 2-norm, the case $p=2$ of the p-norm: en.wikipedia.org/wiki/Lp_space#Definition $\endgroup$ Aug 17, 2021 at 9:16
  • 1
    $\begingroup$ You can think of $\Vert \cdot \Vert_2$ as the length of a vector. Then $\Vert Xw-y \Vert_2$ is the distance between $Xw$ and $y$. Minimizing the square of this term is a linear regression problem (en.wikipedia.org/wiki/Linear_least_squares). By adding $\alpha \Vert w \Vert_2^2$ to the minimization problem, you are trying to "keep $w$ small" (en.wikipedia.org/wiki/…) $\endgroup$
    – user592521
    Aug 17, 2021 at 9:25

1 Answer 1

6
$\begingroup$
  • The $\min$ means you want to minimize the function with respect to a variable, in this case $w$. So you want to find out which value does $w$ has to have so that your function is minimized. In this case, $w$ is a vector and you want to find the vector which minimizes the function.

  • The ||$.$||$_2$ is the norm and the subscript, in your case $2$, defines which norm. Your function calculates the L2-norm, which can be calculated as: $\sqrt{x_1^2+x_2^2+...+x_n^2}$.

  • $X$ is a matrix of size $n \times m$. $Xw$ represents the matrix-vector product between the matrix $X$ and the vector $w$. https://mathinsight.org/matrix_vector_multiplication. This gives a vector back.

  • This resulting vector is subtracted with the vector $y$, which is your target vector (the thing you want to predict).

  • The $2$ at the upper right side of ||.|| is simply the square. That means that the equation in point two becomes $x_1^2+x_2^2+...+x_n^2$. So the error you got from the previous point (which is again a vector) is then put into this calculation.

  • Basically you have your input matrix $X$ which you want to adjust with a fixed vector of coefficients $w$ of length $n$. To find this vector, you execute the non-linear least squares problem, described by the steps above and represented mathematically by the function in your question.

  • Then you have something similar with $\alpha||w||_2^2$. This is the regularisation part of the function and prevents a variable to massively dominate the NNLS solution.

  • So you chose a value for $\alpha$. This is done outside the model, so we usually call this a "hyperparameter". Now, $||w||_2^2$ is calculated again as: $w_1^2+w_2^2+...+w_n^2$. This means that when a value in the vector $w$ would becomes too large when minimizing the first part, it is "regularized" by this part, because a too large value in the vector $w$ would let $\alpha||w||_2^2$ become very high.

In Machine learning, you will be dealing with numerical optimization and linear algebra all the time, you I suggest you become familiar with these fields asap. It will take some time, but once you got the hang of it, you can read functions like this and variations of it quite easy.

Good luck!

$\endgroup$
2
  • 1
    $\begingroup$ Just one more thing to Steven's already excellent answer. When mathematicians explain their notation by saying that "oh, by $||\cdot||_2$ I mean the L2-norm", they are using meta-notation where the dot is a placeholder for the argument. Similarly, they might use $\sqrt{\cdot}$ to refer to "the square root function". $\endgroup$ Aug 17, 2021 at 12:19
  • 1
    $\begingroup$ Great answer , Steven! Thanks! $\endgroup$
    – harp1814
    Aug 17, 2021 at 16:11

Not the answer you're looking for? Browse other questions tagged .