0
$\begingroup$

The full question is: Can you modify a shallow (1 hidden layer) fully connected feed-forward neural network in some way (either adding or removing some aspect of the neural network) to produce a linear regression equation for the same input variables?

I'm still working on getting information on what it means to modify and what the extent of those modifications can be. I'm also finding out if there are any restrictions on the number of inputs, hidden layer nodes, or outputs. I'll update this post soon when I get that information. Edit So any modifications to the neural net are okay such as weights, biases, number of hidden nodes, etc. as long as it still is a fully connected feedforward network with 1 hidden layer.

I know that the main difference between a neural network (NN) and a linear regression (LR) is that an NN has an activation function on each node which allows for non-linear functions to be applied so that the input data can better fit output data that is not linear (or classification data that can't be divided with a linear boundary). The set of multiple linear regression equations for multiple outputs looks like this: multiple linear regression equations

The NN would look something like this with multiple inputs and outputs and only 1 hidden layer. NN with multiple inputs and outputs

I'm thinking that a NN can achieve that same set of equations by forcing some of the weights and biases to be equal to 1 or 0 such that value that ultimately passes through to the final output layer is the same as the multiple linear regression equations. The activation function can simply be the linear activation function (which is akin to just not having one). The biases on all nodes in the hidden layer can be 0 while the biases on the final output layer could simply be retained to whatever their true value is.

Any thoughts on how to approach the problem? Thanks.

$\endgroup$
2
  • 1
    $\begingroup$ Why would you want to do this (as opposed to just using linear regression directly, or just using neural networks)? $\endgroup$ May 30, 2022 at 2:15
  • $\begingroup$ It was just a hypothetical conceptual question. In practicality, there are situations where the straightforward multiple linear regression is best and some where the non-linearity of the neural network is best. $\endgroup$
    – DuperSoup
    May 30, 2022 at 15:22

1 Answer 1

1
$\begingroup$

I think you mean "multivariate linear regression" (multiple outputs) rather than "multiple linear regression" (multiple inputs but one output) [see here].


Multivariate linear regression is equivalent to a network without hidden layers and without activation functions. If you must have a hidden layer, you can always have each node of the hidden layer pass its output directly to a corresponding node of the output layer, as you have suggested.

$\endgroup$
2
  • $\begingroup$ When you say "pass its output directly to a corresponding node of the output layer" do you mean just have each hidden node pass its output only once to 1 output node? For a fully connected network, each hidden node passes its output to all other output nodes which is the part that confuses me. $\endgroup$
    – DuperSoup
    May 30, 2022 at 16:15
  • $\begingroup$ @DuperSoup Sure but you can set certain weights to zero so that each output node ignores all nodes from the previous layer except one. Again, I am not really sure why you want to do this though. $\endgroup$
    – angryavian
    May 30, 2022 at 16:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .