Considering the cross entropy cost function of a neural network: $$Cost = -\frac{1}{N} \sum_{N}^{n=1} y_n \log(\hat{y}_n) + (1 - y_n) \log(1-\hat{y}_n)$$ where $y_n$ is the label (either 0 or 1) and $\hat{y}_n$ is the output of the network (a number between 0 and 1). For simplicity consider a network with one hidden layer: $$ \hat{y}_n = sigmoid(W_2^T \cdot sigmoid(W_1^T \cdot X + b_1) +b_2),$$ where $X$ is the input o the network and $W_1$, $W_2$, $b_1$, $b_2$ are the wieghts and biases of the two layers.

We know this is not convex, when there is at least one hidden layer.

From Wikipedia:

In convex analysis and the calculus of variations, branches of mathematics, a pseudoconvex function is a function that behaves like a convex function with respect to finding its local minima, but need not actually be convex. Informally, a differentiable function is pseudoconvex if it is increasing in any direction where it has a positive directional derivative.

Intuitively, this seems to be the case.

My questions are:

  • Is it pseudo-convex?

  • How can we prove/disprove it?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.