I want to study the convexity of the Mean Squared Error with regularization loss function. I am using an artificial neural network to compute the output.
$$E(w) = MSE(w) = \frac{1}{\mid D \mid}\sum_{d \in D} E_d(w)$$
$E_d$ is the error for a pattern $d$, defined as
$$E_{d}(w)=\frac{1}{2}\sum_{j \in outputLayer}^{} (t_{j}-o_{j}(w))^2 + \frac{1}{2}\lambda||w||^2$$
where $t_j$ and $o_j$ are respectively the target and the output of the $j^{th}$ output neuron of the network. $w$ are the parameters. $\lambda$ is the regularization term.
convexity study
Intuitively I can say that the function is neither convex nor concave, since there are several local minima. In detail...
- $E_d(w)$ is a nonnegative weighted sum of two terms. This operation preserves convexity if the two terms are convex.
- The second term, $\frac{1}{2}\lambda||w||^2$, is convex since the squared euclidean norm is convex and $\lambda$ is positive.
- The first term, $\frac{1}{2}\sum_{j \in outputLayer}^{} (t_{j}-o_{j}(w))^2$, is neither convex nor concave because of the neural network...
for example, at the first layer of the neural network it is computed
$$g(x) = f(W \cdot x + b)$$
where $W \in R^{m \times n}$ is the first weight matrix, $x \in R^n$ is the network input, $b \in R^m$ is the bias and $f: R^m \rightarrow R^m$ is the element-wise activation function.
Here is when I break my analysis saying that the whole function is neither convex nor concave, depending on the activation function:
- if $f := identity$ then $g$ is convex on $R$
- if $f := ReLU$, then $g$ is convex on $R$
- if $f := Sigmoid$, the $g$ is convex in $R_-$ but concave in $R_+$
- if $f := tanh$, the $g$ is convex in $R_-$ but concave in $R_+$
I don't believe this analysis is sufficient, how can I improve it?