# Derivative of a summation function in order to minimize the function

I'm asked to minimize this function $$f\left(x\right)= \sum_{k=1}^K \left(g\left(w\left(k\right)+\alpha\right)-t\left(k\right)\right)^2$$ with respect only to $\alpha$. Function $g\left(w\left(k\right)+\alpha\right)$ is a sigmoid function, defined as $$g\left(w\left(k\right)+\alpha\right) = \frac{1}{1+exp^{-\left(w\left(k\right)+\alpha\right)}}$$ Using Newton's iteration method, I've to take the first and second derivative of the function with respect to $\alpha$.

Derivative of sigmoid function is $$g'\left(w\left(k\right)+\alpha\right) = g\left(w\left(k\right)+\alpha\right)\left(1-g\left(w\left(k\right)+\alpha\right)\right)$$

In order to find the $\alpha$ which minimize the function $f\left(\alpha\right)$, I would like to use Newton's iteration method, so I need to determine first and second derivative of function $f\left(\alpha\right)$.

\begin{align*} f\left(x\right)= \sum_{k=1}^K \left(g\left(w\left(k\right)+\alpha\right)-t\left(k\right)\right)^2 \\f'\left(x\right)= \sum_{k=1}^K 2\left(g\left(w\left(k\right)+\alpha\right)-t\left(k\right)\right)\left(g'\left(w\left(k\right)+\alpha\right)\right) \\f'\left(x\right)= \sum_{k=1}^K 2\left(g\left(w\left(k\right)+\alpha\right)-t\left(k\right)\right)\left(g\left(w\left(k\right)+\alpha\right)\left(1-g\left(w\left(k\right)+\alpha\right)\right)\right) \end{align*}

Is it correct? How about the second derivative, should it be determined by in the same way?

## 1 Answer

Your approach is correct.

What you're looking for is an $\alpha^*$ that minimizes the function $f(x,\alpha)$.

A sufficient ($2^{\text{nd}}$-order) condition for a local minimum in the vicinity of $\alpha^*$ is

$$(\partial_{\alpha}f)(\alpha^*) = 0 \wedge (\partial_{\alpha\alpha}f)(\alpha^*) > 0.$$

In a first step you need to solve the nonlinear algebraic equation

$$\left.(\partial_{\alpha}f)(\alpha^*) = 2\sum\limits_{k=1}^K (g(w(k)+\alpha)-t(k))\frac{\partial g}{\partial(w(k) + \alpha)}\right|_{\alpha^*} = 0,$$

and then check any solution $\alpha^*$ if it satisfies $(\partial_{\alpha\alpha}f)(\alpha^*) > 0$.