In machine learning a very common technique to use as a training algorithm (in NN) is the gradient descent rule. I understand that it is an iterative process of increasing each of the weights based on the partial derivative. Why could we not simply take partial derivative of all weights, set up set of linear equations, and solve them? Is it the computational cost?

• my bad... spelling was not an issue, but funny comment indeed :) – Bober02 May 6 '12 at 10:53
• I'm even not sure that usage of gradient descend method is legal for such non-linear-system (cascading of linear and steps functions) from math point of view math.stackexchange.com/questions/1580425/… – bruziuz Dec 17 '15 at 23:07

If you do that you'll get a non-linear rather than a linear equation.

This is a common strategy for solving some optimization problems, but then that leads to finding a root of a nonlinear system of equations. This can be done using Newton's method (and generalizations), but this will generally involve dense matrix computations.

The dense matrix computations are the issue. Just setting up and solving the Newton's equations is costly (making a matrix will be O(n^2) without including the cost of computing the entries, and solving a matrix equation is O(n^3)).

Another issue in the NN context is online algorithms vs. batch algorithms. In that context it's much more common to use sequential gradient descent (SGD) than the standard gradient descent. (The

• well, for instance, in perceptron, you have w*x as your output. as you differentiate with respect to w_i you get set of Linear Equations. I guess maybe you meant full NN with hidden layers? – Bober02 May 6 '12 at 10:55
• Hidden layers and/or nonlinear functions connecting the layers. Linear functions aren't used very much in the NN setting. It tends to reduce to well-known linear algebra models and results. – cjordan1 May 6 '12 at 11:04
• but perceptron is linear, isn't it? – Bober02 May 6 '12 at 14:23