1
$\begingroup$

For a linear transformation $W$, one can easily find an orthogonal projection matrix $P$ such that for all $x$, $WPx=0$. $P$ projects $x$ to the null space of $W$.

Assume a nonlinear transformation $f(Wx)$, where $f$ is some analytic nonlinear function that is almost everywhere differentiable (e.g., tanh or relu in the context of neural networks). Is there exists such a $f$ with a closed, analytic operation that "projects" each vector to its zero set?

Thanks!

$\endgroup$
4
  • $\begingroup$ Do you mean to find $P$ such that $f(Px)=0$ for all $x$? $\endgroup$
    – obareey
    Jan 12, 2020 at 8:26
  • $\begingroup$ Yes, although $P$ can also be a nonlinear function. But crucially, I want $P$ to act analogously to orthogonal projection in the sense it projects to the closest zero - to prevent trivial solutions of e.g. mapping all x's to a negative number for relu. $\endgroup$
    – user1767774
    Jan 12, 2020 at 9:21
  • $\begingroup$ So you want $p(x) \in f^{-1}(0), \forall x$ and $\lVert p(x) - x \rVert$ is minimized. I doubt you can find a $p$ for a generic $f$. In linear case, you need to find a basis for $W$ and similarly first you need to characterize $f^{-1}(0)$. For example, for single variable relu function $p(x)=\begin{cases}0 & x > 0 \\ x & x \leq 0\end{cases}$. $\endgroup$
    – obareey
    Jan 14, 2020 at 13:10
  • $\begingroup$ Thanks for the comment. I don't want to find a $P$ for every $f$; finding such $P$ for a specific "well behaved" (differentiable, analytic) nonlinear $f$ would be enough. $\endgroup$
    – user1767774
    Jan 14, 2020 at 13:29

0

Reset to default

You must log in to answer this question.