I'm trying to figure the derivation of the Moore-Penrose pseudoinverse for linear regression. The starting expression is the standard error function. I'm not quite sure how to expand on this since all the terms are vectors (my linear algebra is a bit rusty):

$\frac12 (\sum_{n=1}^N(w^Tx^{(n)} - t^{(n)})^2)$

where x is data vector, w is weight vector and t is target vector. So this is the error function taken over the N sample datapoints. Each datapoint consists of a vector of parameters, which is the vector $x$

I need to take the gradient of this w.r.t to $w$ to get:

$X^TXw - X^Tt$

I should clarify that $X$ here is the matrix of all $x^{(n)}$. So its an NxM matrix where N is the total datapoints and M is the number of features in each datapoint.

The Moore-Penrose pseudoinverse could then be used to minimize this function by setting the gradient at 0:

$X^TXw - X^Tt = 0$

$w = (X^TX)^{-1}X^Tt$

Can someone please explain how to do this?

  • 1
    $\begingroup$ Sorry, both $x$ and $w$ are vectors? Then isn't $x^T w$ a number? $\endgroup$ – Muphrid Jan 25 '15 at 5:31
  • $\begingroup$ Interesting... maybe I reached the expression incorrectly. I've updated my original post with the starting error function. I need to get the final expression by taking the derivative of this function w.r.t $w$. I'm not sure exactly how to go about doing this $\endgroup$ – user3425451 Jan 25 '15 at 5:44
  • $\begingroup$ Okay, $X$ is a matrix, $X^T X$ is $m\times m$, so $w$ has $m$ elements, and $t$ has $n$ elements? $\endgroup$ – Muphrid Jan 25 '15 at 6:22
  • $\begingroup$ Yes. That seems to be correct. $\endgroup$ – user3425451 Jan 25 '15 at 6:43
  • $\begingroup$ You have an expression for the pseudoinverse solution. What is the question now? $\endgroup$ – Algebraic Pavel Jan 25 '15 at 11:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.