2
$\begingroup$

I am trying to remember how a neural network is used for classification (Been a while since I took the course).

Let the training data have output labels $1,2,3$, consider a single layer perceptron and let there be only one training sample for simplicity. Let the activation function be denoted by $f$. I remember that for a training sample $x\in \mathbb{R}^n$, the output of the network is $f(w_0+w_1x_1+\ldots+w_nx_n)$ where $f$ is a Linear function and the network learns weights iteratively by minimizing an objective function like, for example, the square of difference of predicted and observed output values.

Theoretically can I use any activation function for a classification problem? For example, if I use the activation function $f(y)=y$ for $y\in \mathbb{R}$, then the output of the neural network, after the iterative process of learning weights, is a real number. But the desired output is an integer in $\{1,2,3\}$. So that is unsettling for me?

$\endgroup$

1 Answer 1

1
$\begingroup$

For classification problems, we generally use different output nodes for different labels. So we will have output $y_1, y_2, y_3$, and each output will go through that linear weighted sum and activation function. Then the probabilities of it being a specific label, given the outputs is calculated using another function, maybe something like $p_1 = \frac{y_1}{y_1+y_2+y_3}$ just for an example. Given enough data points it can learn these probabilities for all types of $\textbf{linear}$ problems.

For more common use problems which are non-linear, you have to use a non linear activation function like ReLu or sigmoid, and the network also has to have more than one layer.

$\endgroup$
5
  • $\begingroup$ Sorry, I did not understand the answer. I like the idea of the output being a probability of belonging to a class. But I don't know what you mean by $y_1,y_2,y_3$? The output that I get from my single layer network is a real number, how can I use that to find the probabilities? Thanks. $\endgroup$
    – Garfield
    Sep 29, 2022 at 18:38
  • $\begingroup$ @Garfield You make three output nodes instead of one, so instead of having $(n+1)$ biases $w_0, w_1 \dots w_n$, you will have $3*(n+1)$ biases $w_{0,0}, w_{0,1} \dots w_{0,n}, w_{1,0}, w_{1,1}, \dots w_{1,n}, w_{2,0}, w_{2,1} , \dots w_{2,n}$. Also you will apply the activation function $3$ different times, $y_0 = f(w_{0,0}+w_{0,1}x_1+\ldots+w_{0,n}x_n), y_1 = f(w_{1,0}+w_{1,1}x_1+\ldots+w_{1,n}x_n), y_2 = f(w_{2,0}+w_{2,1}x_1+\ldots+w_{2,n}x_n)$ $\endgroup$
    – EnEm
    Sep 29, 2022 at 18:43
  • $\begingroup$ Makes sense now. $\endgroup$
    – Garfield
    Sep 29, 2022 at 18:48
  • $\begingroup$ @Garfield Also just to clarify, $p_1 = \frac{y_1}{y_1+y_2+y_3}$ is a terrible practical probability function. Generally people use something like softmax function $\endgroup$
    – EnEm
    Sep 29, 2022 at 18:49
  • $\begingroup$ True, I understand that the example was for purposes of demonstration. $\endgroup$
    – Garfield
    Sep 29, 2022 at 18:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .