# How can I optimize this neural network? - Basic Deep learning

I'm trying to understand deep learning and I think all has to do with least squares for non-linear algebra.

Let's say that we have inputs $$X$$ and outputs $$Y$$ in a 2-layer neural network, e.g deep learning. My goal is to find $$a(i,j), b(i,j), c(i,j)$$ where $$j$$ stands for which neural we are focusing on e.g $$2$$ or $$5$$ and $$i$$ stands for which line for the neural we are focusing on e.g line number $$1$$ or line number $$3$$

On other words $$a(i,j), b(i,j), c(i,j)$$ are matrices.

If we want to display this in beautiful linear algebra. We write:

$$Z1 = X^T a(i, j)$$ $$Z2 = Z1 b(i, j)$$ $$Y = Z2 c(i, j)$$

Or we can put all together: $$Y = X^T a(i, j) b(i, j) c(i, j)$$

Notice that $$X$$ and $$Y$$ changes for each data set.

Question:

How can I find $$a(i, j) b(i, j) c(i, j)$$ if I know $$Y$$ and $$X$$?

I'm assuming that I will get lots of solutions depending on which $$Y$$ and $$X$$ I'm using. What solution should I use then? The best fit? How?

• I think you will find that at each node some non-linear function, traditionally a sigmoid, is applied. Then the learning process is indeed similar to a non-linear least squares problem using gradient descent or some convergence enhancing modification thereof, conjugate gradient, quasi-Newton like BFGS, on all the parameters at once or block-wise by layers or nodes. Jul 15, 2019 at 12:53
• Just wondering: why do you need 2 hidden layers for a linear regression model? Usually multiple layers are used in non-linear models. Jul 15, 2019 at 12:53
• @denklo Beacuase the model is not linear :) Jul 15, 2019 at 12:54
• @LutzL so.....what should I do and learn? Jul 15, 2019 at 12:56
• Numerical methods with a focus on optimization/non-linear programming, large-scale numerical linear algebra, that is, iterative methods and sparse systems, perhaps also an overview of automatic/algorithmic differentiation, as that is how the gradients are computed in back-propagation. Jul 15, 2019 at 13:08

You will use the gradient descent method to backpropagate the error of your prediction to lower layers. Your goal is to adjust the $$a(i,j), b(i,j)$$, and $$c(i,j)$$ values to minimize the error. Minimizing the error is usually represented a minimizing a loss. A loss function usually used is $$L= y\log(y_0)$$ where $$y_0$$ will be the prediction you get from the model and $$y$$ is the actual values $$Y$$ should have. By gradient descent you will calculate $$\frac{dL}{da(i,j)}, \frac{dL}{db(i,j)}$$, and $$\frac{dL}{dc(i,j)}$$. Then you will update the parameters as follows;
$$a(i,j)= a(i,j)-\alpha dL/da(i,j) \\ b(i,j)= b(i,j)-\alpha dL/db(i,j) \\ c(i,j)= c(i,j)-\alpha dL/dc(i,j)\$$
where $$\alpha$$ is the learning rate. You will keep doing this for multiple data samples for multiple iterations until you get a model that shows a low loss.