0
$\begingroup$

I have the following question about the Neural network. In the following paper https://arxiv.org/pdf/2210.05189.pdf The first layer output for each neuron is given by the following equation $$ o_i = \sum_{i} w_i x_i +b $$ where $w_i$ is the weight and $b$ the bias. Now we compose it with the activation function. I see the activation function as scaling the output to make it in a bound. The activation function we use is $\textit{Relu, tanh} $ etc. Now my question is how eq 2 in the paper represents it as Hadamard multiplication. That is multiplication by $\bf{a}_i$ by $o_i$. I don't understand what are this $ a_i$ how it's calculated. How it is determined.

Eq 1 is representing a feed-forward neural network $$N(\bf{x_0})= ....\sigma(W_{1}^{T}(\sigma(\bf{W_0^{T}}\bf{x_{0}}))) $$

Eq 2 is represented as follows

$$\sigma(\bf{W_0^{T}}\bf{x_{0}}) = a_i \odot W_0^{T}x_0 $$ I am interpreting $\odot$ as multipication of two vector like hadamard. That is $(a_1 , a_2, \ldots, a_n)\odot (x_1, x_2, \ldots, x_n) = (a_1 x_1 , a_2 x_2, \ldots ,a_n x_n)$

I don't understand the following paragraph enter image description here

Any helpful explanation is appreciated.

$\endgroup$
2
  • $\begingroup$ You should include Equation 2 at least in this question. $\endgroup$
    – user934527
    Feb 19, 2023 at 23:46
  • 1
    $\begingroup$ Thanks for the comment, I have just updated it. $\endgroup$
    – GGT
    Feb 19, 2023 at 23:59

1 Answer 1

0
$\begingroup$

I think you're misunderstanding the domains and codomains of the functions in Equation 2. Which is reasonable, this is tersely written.

Each layer in a neural network is a composition of the form $\sigma\circ L$ where $L:\mathbb{R}^n\to\mathbb{R}^m$ is a linear function (matrix multiplication by $W_i^T$ in the author's notation) and $\sigma:\mathbb{R}^m\to\mathbb{R}^m$ is a nonlinear function. But more specifically, it seems traditional in machine learning to specify the function $\sigma$ by indicating a single function $g:\mathbb{R}\to\mathbb{R}$ with the understanding that $\sigma$ is the function that applies $g$ to each coordinate of its inputs, i.e., $\sigma = (g)_{i=1}^m$ in symbols.

The author indicates that $\sigma$ should be a ``piecewise linear activation function'', which is to say that the function $g:\mathbb{R}\to\mathbb{R}$ defining the components of $\sigma$ is a piece-wise linear function, link if useful. So in particular, $g(x) = a_x x$ for any $x\in\mathbb{R}$, where $a_x$ is determined by which piece of the function $g$ the number $x$ is contained in.

The vector $\mathbf{a}_{i-1}$ that the author indicates collects the slopes of the individual linear pieces of $g$ that the components of the vector $L(\mathbf{x}_{i-1})$ falls in.

More precisely, let $L(\mathbf{x}_{i-1})$ be $(x_1,\dots,x_m)^T$ in $\mathbb{R}^m$. Define $a_1 = a_{x_1}, a_2 = a_{x_2}$, etc. and $\mathbf{a}_{i-1}$ by $(a_1,\dots,a_m)^T$. Then $\sigma(\mathbf{x}) = \mathbf{a}_{i-1} \odot L(\mathbf{x}_{i-1})$.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .