Notation for element-wise function application Is there some kind of specific notation I could use to specify that function $f$ is applied to each element of matrix $W$ and not to a matrix as a whole. Specifically, I am writing about applying activation function for a layer of neural network. Or just writing this is clear enough: $f(W)$?
 A: If $W_{ij}$ represents the elements of the matrix, then $f(W_{ij})$
A: An elegant choice of notation would be $f \circ W$ where $\circ$ denotes function composition, because
\begin{align*}
(f \circ W)_{i,j}
&= (f \circ W)(i,j) \\
&= f(W(i,j)) \\
&= f(W_{i,j})
\end{align*}
A: I think there are no common notation to describe elementwise function application, but for some binary operators like $+, \cdot$ it is common to use $\oplus, \odot$ in neural networks. Some programming languages like Julia adopt dot operator f.(W), X .* Y to denote elementwise application. $f \circ W$ is not appropriate because
$$
\exists W, \exists i, j, \exp(\text{get}_{ij}(W)) = \exp (W_{ij}) \ne (\exp (W))_{ij} = \text{get}_{ij}(\exp(W)),
$$
this means $\exp \circ \text{get}_{ij} \ne \text{get}_{ij} \circ \exp$.
I personally like to use adding a symbol $\circ$ or others to the bottom or top of the function like $\overset{\circ}{f}(\{ a_i \}_{i \in I}) := \{ f(a_i) \}_{i \in I}, \overset{\circ}{f}(W) = ( f(W_{ij}) )_{ij}$ to mean an elementwise function.
