# How to Find the Maximum of a Function Represented by a Back-Propagation Neural Network?

First, I train a standard feed-forward neural network over a training set of data points. I get an approximate function, say $F(x)$, represented implicitly by that neural network.

1. How do I find its (global) maximum?

Note that the function $F(x)$ is not given explicitly; it is only implicitly realized by the neural network.

$F(x)$ should be differentiable, and gradient descent may be applied to it. But I'm not sure how this could be done given only the neural network.

2. Could the structure of the neural network be exploited so that I could find the maxima faster?

PS: the question is not the same as finding the maximum value of a set of data points -- I need the neural network for other purposes, so the neural network must be trained first, and the maximum be found afterwards.

• This is a fascinating question. I really hope someone finds an answer Aug 9, 2016 at 4:31
• You know those NN demos where you feed in an input and it returns a morphed version where the net has 'found' faces or buildings etc in the input that weren't there to start with? This is what they're doing. They take some late stage activation (e.g. one which indicates classification as a cat) and perform gradient descent on the input to maximise that activation. It's just a matter of 'training' the input representation rather than the weights of the network. Dec 16, 2016 at 9:09

A traditional FFNN has L layers where each layer will have a set of input responses $a^{(l)} = \{a^{(l)}_1, a^{(l)}_2, \dots, a^{(l)}_n\}$ and a matrix of weights $W^{(l)}$ where $W^{(l)}_{jk}$ represents the weight connecting the j-th node from the previous layer to the i-th node of the current layer. There are also activation functions $f^{(l)}(z^{(l)}_k)$ where $z^{(l)}_k = [W^{(l)}_{1k}a^{(l)}_1, W^{(l)}_{2k}a^{(l)}_2, \dots, W^{(l)}_{nk}a^{(l)}_n]$ that are applied to every node in every layer. What you are looking for is the gradient of the activation function of the output layer $\nabla{f}^{(L)}$ (in this case just one node) with respect to the input of the first layer. Using the chain rule, $$\nabla (f^{(l)}(z^{(l)}_k)) = \nabla f^{(l)}(z^{(l)}_k)\nabla z^{(l)}_k \\ = \nabla f^{(l)}(z^{(l)}_k) \cdot \sum_j W_{jk}\nabla(f^{(l - 1)}(z^{(l - 1)}_j))$$
$\nabla f^{(l)}$ can be calculated analytically from the activation function. $z^{(l)}_k$ should already be calculated during the forward pass. By propagating backwards you will eventually calculate $\nabla_{a^{(1)}} f^{(L)}$