2
$\begingroup$

How would I go about calculating the total training loss, and the optimal parameter values of $w$ (given as an optimal weight vector)?

Given an expression such as:

$$\sum_{n=1}^N (w^Tx_n-t_n)^2$$

How would I engage such a problem? Dealing with total training loss, I usually see expressions of the form: $f(x;w) = w_o+w_1x$ (using linear regression) for computing the optimal model from: $$\frac{1}{N} \sum_{n=1}^N ((w_o + w_1x) - t_n)^2$$

But I cannot fit the expression i provided above, into this expression, or should i consider the $w^Tx_n$-term as $w_0x_0 + w_1x_1$? It confuses me a lot.

I would "normally" solve an expression by finding its partial derivatives, with respect to each value of $w$ (given the optimal model expression from above), and then evaluate these at 0 to find the optimal values. Then I would apply the second derivative test to find a minimum.

Can anyone explain how a similar approach would look like given an expression as the one above?

$\endgroup$

1 Answer 1

2
$\begingroup$

"or should i consider the $w^Tx_n$-term as $w_0x_0 + w_1x_1$?"

Almost. $w$ is a vector and $w^T$ is the transpose of it. So, what you want is to find the optimal values of $w$ that minimizes the following function

$$\sum_{n=1}^N (w^Tx+w_o-t_n)^2$$, where $x$ is also a vector and $w_0$ is a scalar. Which can be written as:

$$\sum_{n=1}^N error_n^2$$, which we can call the cost function.

Via the chain rule, you can take the derivative of this cost function with respect to $w$:

$$\frac{dcostfunction}{dw} = \frac{dcostfunction}{derror}*\frac{derror}{dw} $$

, which gives us:

$$2* error * x$$ for $w$

and for $w_o$:

$$2*error$$

Now, you can solve it via what we call the 'normal equations', but a popular approach is to use gradient descent, where the vector $w$ just moves iteratively via a learning rate towards the negative gradient descent. So in our case, we should iteratively update our vector $w$ via:

$$w = w + error * learning rate * x $$ and $$w_0 = w_0+ error * learning rate $$

So, what we do is: we make a guess in the form of $$w^Tx+w_o$$ and after every guess, we calculate the error and update our weight vector $w$ and $w_0$ as I showed, which will converge to the optimal values of $w$ and $w_0$. This is pre-programmed in many libraries and easy to use in e.g. Python.

$\endgroup$
3
  • $\begingroup$ Hmm this is actually a very well explained case. Is it not possible to calculate the values directly without using Python etc. In the case of the simplest linear regression evaluating $w_0$ and $w$, where $w_0$ would be solved for $w$ and this substituted into the expression of $w$ and solve backwards. Eventually you would have two expressions in terms of each parameter value of $w_0,w$ and just calculate these with respect to 0, to find their values. Then just add the second derivative test to determine if these cases are in fact a minimum. Hopes this makes sense. $\endgroup$
    – NewDev90
    Nov 23, 2019 at 16:41
  • $\begingroup$ yes, that's also possible if I understand your question correctly. Maybe this helps? youtube.com/watch?v=JvS2triCgOY $\endgroup$ Nov 23, 2019 at 19:49
  • 1
    $\begingroup$ Alright thanks :) I will have a look. Thanks a lot of taking you time to explain! $\endgroup$
    – NewDev90
    Nov 24, 2019 at 15:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .