About classification using a neural network

I am trying to remember how a neural network is used for classification (Been a while since I took the course).

Let the training data have output labels $$1,2,3$$, consider a single layer perceptron and let there be only one training sample for simplicity. Let the activation function be denoted by $$f$$. I remember that for a training sample $$x\in \mathbb{R}^n$$, the output of the network is $$f(w_0+w_1x_1+\ldots+w_nx_n)$$ where $$f$$ is a Linear function and the network learns weights iteratively by minimizing an objective function like, for example, the square of difference of predicted and observed output values.

Theoretically can I use any activation function for a classification problem? For example, if I use the activation function $$f(y)=y$$ for $$y\in \mathbb{R}$$, then the output of the neural network, after the iterative process of learning weights, is a real number. But the desired output is an integer in $$\{1,2,3\}$$. So that is unsettling for me?

For classification problems, we generally use different output nodes for different labels. So we will have output $$y_1, y_2, y_3$$, and each output will go through that linear weighted sum and activation function. Then the probabilities of it being a specific label, given the outputs is calculated using another function, maybe something like $$p_1 = \frac{y_1}{y_1+y_2+y_3}$$ just for an example. Given enough data points it can learn these probabilities for all types of $$\textbf{linear}$$ problems.

For more common use problems which are non-linear, you have to use a non linear activation function like ReLu or sigmoid, and the network also has to have more than one layer.

• Sorry, I did not understand the answer. I like the idea of the output being a probability of belonging to a class. But I don't know what you mean by $y_1,y_2,y_3$? The output that I get from my single layer network is a real number, how can I use that to find the probabilities? Thanks. Sep 29, 2022 at 18:38
• @Garfield You make three output nodes instead of one, so instead of having $(n+1)$ biases $w_0, w_1 \dots w_n$, you will have $3*(n+1)$ biases $w_{0,0}, w_{0,1} \dots w_{0,n}, w_{1,0}, w_{1,1}, \dots w_{1,n}, w_{2,0}, w_{2,1} , \dots w_{2,n}$. Also you will apply the activation function $3$ different times, $y_0 = f(w_{0,0}+w_{0,1}x_1+\ldots+w_{0,n}x_n), y_1 = f(w_{1,0}+w_{1,1}x_1+\ldots+w_{1,n}x_n), y_2 = f(w_{2,0}+w_{2,1}x_1+\ldots+w_{2,n}x_n)$
– EnEm
Sep 29, 2022 at 18:43
• Makes sense now. Sep 29, 2022 at 18:48
• @Garfield Also just to clarify, $p_1 = \frac{y_1}{y_1+y_2+y_3}$ is a terrible practical probability function. Generally people use something like softmax function
– EnEm
Sep 29, 2022 at 18:49
• True, I understand that the example was for purposes of demonstration. Sep 29, 2022 at 18:50