# Minimizing a neural network containing relu

I have a question regarding the minimization of a neural network containing $$Relu(x):=\max\{0,x\}$$.

Since Relu is convex, the subgradient method can be used to minimize the Relu function.

However, Relu is part of a neural network $$f$$. Training a neural network means to minimize a function $$L(W)= \sum_{i = 1}^{N} d(f(x_{i},W),g_{i})$$, where we have $$N$$ training sample $$x_{i}$$ with corresponding label $$g_{i}$$ and $$d$$ is a metric.

How do we minimize $$L$$ with respect to $$W$$ if relu is non-differentiable, but $$f$$ non-convex?

In more detail, there are two subsequent questions:

1. How is it optimized in practice ?
2. What are the theoretical aspects to this question? In which case is convergence to a local optimum guaranteed (may be only almost sure).

• For a proper convex function $f:\mathbb{E}\to(-\infty,\infty]$, a subgradient at $\mathbf{x}$ is a vector $\mathbf{g}\in\mathbb{E}^*$ (the dual space) for which $f(\mathbf{y})\geq f(\mathbf{x})+\mathbf{g}^T(\mathbf{y}-\mathbf{x}), \forall \mathbf{y}\in\mathbb{E}$. The group of all subgradient at $\mathbf{x}$ is call the subdifferential set. If this set is a singleton (single element), then the subgradient and gradient (derivative) coincide. When using the subgradient method, then you can choose any element from the set, if more than a single subgradient exist (think of $|x|$ at $x=0$). Sep 8 '20 at 9:10