# how to find the input for this optimization problem?

Suppose I have a neural network, with input variables $a,b,c,d,f,g$ and output variables $m,n,o,p,q$.

Given different input values, the neural network will output corresponding $m,n,o,p,q$.

Now I want find out the best input values which can maxmize m,n, while minimize $o,p,q$ with different weights as well. So how can I find the best $a,b,c,d,f,g$?

Currently I use a simple way, which calculate $x=w_1m+w_2n+\frac{w_3}{o}+\frac{w_4}{p}+\frac{w_5}{q}$, then simply search all the possible inputs to find the input to get maxmization of x. However this simple method is not efficient and it also assume m,n,o,p,q are independent, which is not the case.

Meanwhile, normally Neural network will not give you a formula how the output related to the input, all the optimization approach I known all need a specific function how the output related to input. Even for genetic algorithm, I need a fitness function like $x=w_1m+w_2n+\frac{w_3}{o}+\frac{w_4}{p}+\frac{w_5}{q}$, but in this case, x has to related to inputs $a,b,c,d,f,g$ instead of outputs.

Any suggestions? Many thanks

-
How is your specific problem different from the usual, everyday setting? Have you consulted any introductory machine learning text? –  Yuval Filmus Feb 7 '11 at 13:26
Also, normally in neural networks there are hidden layers. And you haven't specified how the input is related to the output - what weights are you trying to adjust? –  Yuval Filmus Feb 7 '11 at 13:27
The problem is this, I can not define the relation between the input and output as a function. but I can represent it as a neural network. I have tried normal regression on the input data, but the result is not good. a neural network can fit the data fine, but normally it is very difficult to represent the network as formula, because the hidden layer, active function etc.... –  user6743 Feb 7 '11 at 13:58

@zhang: Your goal is optimizing some objective function that has some unknown, black box relationship with the inputs. Either you have some samples of that black box or are able to sample the function yourself. You've chosen to use some very general regression method (artificial neural networks) to model the black box function--but with this approach your objective function will be smooth but non-convex, meaning you might not be to use convex techniques for the optimization.

@Yuval: He wants to generalize his data to a smooth function with the neural network, and then design or optimize the input for some objective function. The standard case only covers the generalization/prediction part.

1. Use your neural network and plug it in to the objective function of some general nonlinear programming technique such as Matlab's fmincon. To do this, you won't even need to necessarily know the form of the relation.
2. Another option is using a more restricted regression (such as linear or convex quadratic) for either the relation or the objective function itself. In this case, once you have the regression, you can solve a convex problem to get the optimum.
3. Use a trust region method (which #1 may be using anyway) directly on your samples of the relation/objective. This will build local convex approximations of the relation/objective, which obviates the need for training a global approximation (the neural net) as in #1.

About the structure of the relation from the neural net: you'll need to know exactly what kind of net you are using and any parameters. Below is an explanation of the structure of a common neural net variety.

For example, if your network is a multilayer perceptron (one of the most common types), the output for a single output or hidden node will follow this form: $\phi(x) = f( w^Tx+b )$

$x$ is a vector of inputs for that node (in the input layer, these will be your inputs $a,b,c,d,f,g$)
$w$ is the learned weights for for that node
$b$ is a bias term for the node, like another weight
$f$ is some nonlinear function, which for a multilayer perceptron can be $\tanh$, or $(1 + e^{-x})^{-1}$ (there are other choices)

So, if you know the structure of the network, and the weights at each node (you should be able to get this from your neural network package), you can compose the functions to get expressions for the output nodes. Then you can plug in the output expressions into your formula for x, your objective function.

-