I'm trying to understand deep learning and I think all has to do with least squares for non-linear algebra.

Let's say that we have inputs $X$ and outputs $Y$ in a 2-layer neural network, e.g deep learning. My goal is to find $a(i,j), b(i,j), c(i,j)$ where $j$ stands for which neural we are focusing on e.g $2$ or $5$ and $i$ stands for which line for the neural we are focusing on e.g line number $1$ or line number $3$

On other words $a(i,j), b(i,j), c(i,j)$ are matrices.

enter image description here

If we want to display this in beautiful linear algebra. We write:

$$Z1 = X^T a(i, j)$$ $$Z2 = Z1 b(i, j)$$ $$Y = Z2 c(i, j)$$

Or we can put all together: $$Y = X^T a(i, j) b(i, j) c(i, j)$$

Notice that $X$ and $Y$ changes for each data set.


How can I find $a(i, j) b(i, j) c(i, j)$ if I know $Y$ and $X$?

I'm assuming that I will get lots of solutions depending on which $Y$ and $X$ I'm using. What solution should I use then? The best fit? How?

  • 1
    $\begingroup$ I think you will find that at each node some non-linear function, traditionally a sigmoid, is applied. Then the learning process is indeed similar to a non-linear least squares problem using gradient descent or some convergence enhancing modification thereof, conjugate gradient, quasi-Newton like BFGS, on all the parameters at once or block-wise by layers or nodes. $\endgroup$ Jul 15, 2019 at 12:53
  • $\begingroup$ Just wondering: why do you need 2 hidden layers for a linear regression model? Usually multiple layers are used in non-linear models. $\endgroup$
    – denklo
    Jul 15, 2019 at 12:53
  • $\begingroup$ @denklo Beacuase the model is not linear :) $\endgroup$
    – euraad
    Jul 15, 2019 at 12:54
  • $\begingroup$ @LutzL so.....what should I do and learn? $\endgroup$
    – euraad
    Jul 15, 2019 at 12:56
  • 1
    $\begingroup$ Numerical methods with a focus on optimization/non-linear programming, large-scale numerical linear algebra, that is, iterative methods and sparse systems, perhaps also an overview of automatic/algorithmic differentiation, as that is how the gradients are computed in back-propagation. $\endgroup$ Jul 15, 2019 at 13:08

1 Answer 1


You will use the gradient descent method to backpropagate the error of your prediction to lower layers. Your goal is to adjust the $a(i,j), b(i,j)$, and $c(i,j)$ values to minimize the error. Minimizing the error is usually represented a minimizing a loss. A loss function usually used is $L= y\log(y_0)$ where $y_0$ will be the prediction you get from the model and $y$ is the actual values $Y$ should have. By gradient descent you will calculate $\frac{dL}{da(i,j)}, \frac{dL}{db(i,j)}$, and $\frac{dL}{dc(i,j)}$. Then you will update the parameters as follows;

$$ a(i,j)= a(i,j)-\alpha dL/da(i,j) \\ b(i,j)= b(i,j)-\alpha dL/db(i,j) \\ c(i,j)= c(i,j)-\alpha dL/dc(i,j)\ $$

where $\alpha$ is the learning rate. You will keep doing this for multiple data samples for multiple iterations until you get a model that shows a low loss.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .