Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

In machine learning a very common technique to use as a training algorithm (in NN) is the gradient descent rule. I understand that it is an iterative process of increasing each of the weights based on the partial derivative. Why could we not simply take partial derivative of all weights, set up set of lieanr equations, and solve them? Is it the computational cost?

share|improve this question
    
my bad... spelling was not an issue, but funny comment indeed :) –  Bober02 May 6 '12 at 10:53
add comment

1 Answer

up vote 2 down vote accepted

If you do that you'll get a non-linear rather than a linear equation.

This is a common strategy for solving some optimization problems, but then that leads to finding a root of a nonlinear system of equations. This can be done using Newton's method (and generalizations), but this will generally involve dense matrix computations.

The dense matrix computations are the issue. Just setting up and solving the Newton's equations is costly (making a matrix will be O(n^2) without including the cost of computing the entries, and solving a matrix equation is O(n^3)).

Another issue in the NN context is online algorithms vs. batch algorithms. In that context it's much more common to use sequential gradient descent (SGD) than the standard gradient descent. (The

share|improve this answer
    
well, for instance, in perceptron, you have w*x as your output. as you differentiate with respect to w_i you get set of Linear Equations. I guess maybe you meant full NN with hidden layers? –  Bober02 May 6 '12 at 10:55
    
Hidden layers and/or nonlinear functions connecting the layers. Linear functions aren't used very much in the NN setting. It tends to reduce to well-known linear algebra models and results. –  cjordan1 May 6 '12 at 11:04
    
but perceptron is linear, isn't it? –  Bober02 May 6 '12 at 14:23
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.