# Total Training Loss

How would I go about calculating the total training loss, and the optimal parameter values of $$w$$ (given as an optimal weight vector)?

Given an expression such as:

$$\sum_{n=1}^N (w^Tx_n-t_n)^2$$

How would I engage such a problem? Dealing with total training loss, I usually see expressions of the form: $$f(x;w) = w_o+w_1x$$ (using linear regression) for computing the optimal model from: $$\frac{1}{N} \sum_{n=1}^N ((w_o + w_1x) - t_n)^2$$

But I cannot fit the expression i provided above, into this expression, or should i consider the $$w^Tx_n$$-term as $$w_0x_0 + w_1x_1$$? It confuses me a lot.

I would "normally" solve an expression by finding its partial derivatives, with respect to each value of $$w$$ (given the optimal model expression from above), and then evaluate these at 0 to find the optimal values. Then I would apply the second derivative test to find a minimum.

Can anyone explain how a similar approach would look like given an expression as the one above?

"or should i consider the $$w^Tx_n$$-term as $$w_0x_0 + w_1x_1$$?"

Almost. $$w$$ is a vector and $$w^T$$ is the transpose of it. So, what you want is to find the optimal values of $$w$$ that minimizes the following function

$$\sum_{n=1}^N (w^Tx+w_o-t_n)^2$$, where $$x$$ is also a vector and $$w_0$$ is a scalar. Which can be written as:

$$\sum_{n=1}^N error_n^2$$, which we can call the cost function.

Via the chain rule, you can take the derivative of this cost function with respect to $$w$$:

$$\frac{dcostfunction}{dw} = \frac{dcostfunction}{derror}*\frac{derror}{dw}$$

, which gives us:

$$2* error * x$$ for $$w$$

and for $$w_o$$:

$$2*error$$

Now, you can solve it via what we call the 'normal equations', but a popular approach is to use gradient descent, where the vector $$w$$ just moves iteratively via a learning rate towards the negative gradient descent. So in our case, we should iteratively update our vector $$w$$ via:

$$w = w + error * learning rate * x$$ and $$w_0 = w_0+ error * learning rate$$

So, what we do is: we make a guess in the form of $$w^Tx+w_o$$ and after every guess, we calculate the error and update our weight vector $$w$$ and $$w_0$$ as I showed, which will converge to the optimal values of $$w$$ and $$w_0$$. This is pre-programmed in many libraries and easy to use in e.g. Python.

• Hmm this is actually a very well explained case. Is it not possible to calculate the values directly without using Python etc. In the case of the simplest linear regression evaluating $w_0$ and $w$, where $w_0$ would be solved for $w$ and this substituted into the expression of $w$ and solve backwards. Eventually you would have two expressions in terms of each parameter value of $w_0,w$ and just calculate these with respect to 0, to find their values. Then just add the second derivative test to determine if these cases are in fact a minimum. Hopes this makes sense. Nov 23, 2019 at 16:41
• yes, that's also possible if I understand your question correctly. Maybe this helps? youtube.com/watch?v=JvS2triCgOY Nov 23, 2019 at 19:49
• Alright thanks :) I will have a look. Thanks a lot of taking you time to explain! Nov 24, 2019 at 15:49