# Activation function and its slope in Neural network

I have the following question about the Neural network. In the following paper https://arxiv.org/pdf/2210.05189.pdf The first layer output for each neuron is given by the following equation $$o_i = \sum_{i} w_i x_i +b$$ where $$w_i$$ is the weight and $$b$$ the bias. Now we compose it with the activation function. I see the activation function as scaling the output to make it in a bound. The activation function we use is $$\textit{Relu, tanh}$$ etc. Now my question is how eq 2 in the paper represents it as Hadamard multiplication. That is multiplication by $$\bf{a}_i$$ by $$o_i$$. I don't understand what are this $$a_i$$ how it's calculated. How it is determined.

Eq 1 is representing a feed-forward neural network $$N(\bf{x_0})= ....\sigma(W_{1}^{T}(\sigma(\bf{W_0^{T}}\bf{x_{0}})))$$

Eq 2 is represented as follows

$$\sigma(\bf{W_0^{T}}\bf{x_{0}}) = a_i \odot W_0^{T}x_0$$ I am interpreting $$\odot$$ as multipication of two vector like hadamard. That is $$(a_1 , a_2, \ldots, a_n)\odot (x_1, x_2, \ldots, x_n) = (a_1 x_1 , a_2 x_2, \ldots ,a_n x_n)$$

I don't understand the following paragraph

• You should include Equation 2 at least in this question.
– user934527
Commented Feb 19, 2023 at 23:46
• Thanks for the comment, I have just updated it.
– GGT
Commented Feb 19, 2023 at 23:59

Each layer in a neural network is a composition of the form $$\sigma\circ L$$ where $$L:\mathbb{R}^n\to\mathbb{R}^m$$ is a linear function (matrix multiplication by $$W_i^T$$ in the author's notation) and $$\sigma:\mathbb{R}^m\to\mathbb{R}^m$$ is a nonlinear function. But more specifically, it seems traditional in machine learning to specify the function $$\sigma$$ by indicating a single function $$g:\mathbb{R}\to\mathbb{R}$$ with the understanding that $$\sigma$$ is the function that applies $$g$$ to each coordinate of its inputs, i.e., $$\sigma = (g)_{i=1}^m$$ in symbols.
The author indicates that $$\sigma$$ should be a piecewise linear activation function'', which is to say that the function $$g:\mathbb{R}\to\mathbb{R}$$ defining the components of $$\sigma$$ is a piece-wise linear function, link if useful. So in particular, $$g(x) = a_x x$$ for any $$x\in\mathbb{R}$$, where $$a_x$$ is determined by which piece of the function $$g$$ the number $$x$$ is contained in.
The vector $$\mathbf{a}_{i-1}$$ that the author indicates collects the slopes of the individual linear pieces of $$g$$ that the components of the vector $$L(\mathbf{x}_{i-1})$$ falls in.
More precisely, let $$L(\mathbf{x}_{i-1})$$ be $$(x_1,\dots,x_m)^T$$ in $$\mathbb{R}^m$$. Define $$a_1 = a_{x_1}, a_2 = a_{x_2}$$, etc. and $$\mathbf{a}_{i-1}$$ by $$(a_1,\dots,a_m)^T$$. Then $$\sigma(\mathbf{x}) = \mathbf{a}_{i-1} \odot L(\mathbf{x}_{i-1})$$.