# Neural Network Gradient Descent of Cost Function Misunderstanding

I've taken an interest in neural networks recently and have been progressing rather well but came to a standstill while learning about gradient descent (I've done multivariable calculus previously). Specifically, I struggle with this: Say our neural network is designed to recognise digits 0-9, and we have the MSE Cost function which, given a certain vector of weights and biases, after a large number of training examples, will spit out the average 'cost' as a scalar. I have read that now one must compute the gradient -$\nabla{C}$ in order to find the vector of greatest descent, but I am confused on this:

How does one graph the cost function/find an explicit formula for the cost function/compute $\nabla{C}$ - these would all require one to try an infinite number of weights and biases surely...??

• You do not graph the function. You need to have a formula for the function $C$, to which you apply the partial differentiation rules from multivariable calculus to obtain a formula for the gradient $\nabla C$. – Rahul Jun 23 '18 at 14:17
• @Rahul So, for example, if $C(w,b)={1/2n}\sum_{x=1}^n (y(x)-a)^2$, where y(x) is the desired vector (e.g. (0,0,0,0,1,0,0,0,0,0)) and a is the vector you get (absolute value bars around y(x)-a)), how would one compute $\nabla{C}$ – Daniele1234 Jun 23 '18 at 14:22

Automatic differentiation doesn't require or produce an explicit expression for the derivative, nor does it approximate the derivative. The basic idea (for forward mode AD at least) is straightforward. Write $Df$ for the derivative of $f$ with respect to its argument. Let's say we wanted to know what the derivative of $f+g$ is at $x$, i.e. $D(f+g)(x)$. We know from basic calculus that this is $(Df+Dg)(x)=Df(x)+Dg(x)$. So if the program implementing $f$ when evaluated at $x$ produced not only the value $f(x)$ but also the (single numerical value!) $Df(x)$ and similarly for $g$, we could calculate $D(f+g)(x)$ by simply adding those two extra outputs. Similarly, for $D(fg)(x)=f(x)Dg(x)+g(x)Df(x)$ we use both the "normal" outputs of $f$ and $g$ and the "extra" derivative outputs and easily calculate the the "extra" derivative output of the product of the functions. We can do similar things for various other combinations of functions (scaling, composition, exponentiation) and easily implement primitive operations (like $\sin$ and $\cos$) to produce these "extra" derivative values. With operator overloading, type classes, or program rewriting, you can just work in terms of the "normal" values and automatically, in parallel, the derivatives will also be calculated. Reverse mode AD is a little more complicated but the end experience is much the same.