I'm asked to minimize this function $$f\left(x\right)= \sum_{k=1}^K \left(g\left(w\left(k\right)+\alpha\right)-t\left(k\right)\right)^2$$ with respect only to $\alpha$. Function $g\left(w\left(k\right)+\alpha\right)$ is a sigmoid function, defined as $$g\left(w\left(k\right)+\alpha\right) = \frac{1}{1+exp^{-\left(w\left(k\right)+\alpha\right)}}$$ Using Newton's iteration method, I've to take the first and second derivative of the function with respect to $\alpha$.

Derivative of sigmoid function is $$g'\left(w\left(k\right)+\alpha\right) = g\left(w\left(k\right)+\alpha\right)\left(1-g\left(w\left(k\right)+\alpha\right)\right)$$

In order to find the $\alpha$ which minimize the function $f\left(\alpha\right)$, I would like to use Newton's iteration method, so I need to determine first and second derivative of function $f\left(\alpha\right)$.

$$\begin{align*} f\left(x\right)= \sum_{k=1}^K \left(g\left(w\left(k\right)+\alpha\right)-t\left(k\right)\right)^2 \\f'\left(x\right)= \sum_{k=1}^K 2\left(g\left(w\left(k\right)+\alpha\right)-t\left(k\right)\right)\left(g'\left(w\left(k\right)+\alpha\right)\right) \\f'\left(x\right)= \sum_{k=1}^K 2\left(g\left(w\left(k\right)+\alpha\right)-t\left(k\right)\right)\left(g\left(w\left(k\right)+\alpha\right)\left(1-g\left(w\left(k\right)+\alpha\right)\right)\right) \end{align*}$$

Is it correct? How about the second derivative, should it be determined by in the same way?


Your approach is correct.

What you're looking for is an $\alpha^*$ that minimizes the function $f(x,\alpha)$.

A sufficient ($2^{\text{nd}}$-order) condition for a local minimum in the vicinity of $\alpha^*$ is

$$ (\partial_{\alpha}f)(\alpha^*) = 0 \wedge (\partial_{\alpha\alpha}f)(\alpha^*) > 0. $$

In a first step you need to solve the nonlinear algebraic equation

$$ \left.(\partial_{\alpha}f)(\alpha^*) = 2\sum\limits_{k=1}^K (g(w(k)+\alpha)-t(k))\frac{\partial g}{\partial(w(k) + \alpha)}\right|_{\alpha^*} = 0,$$

and then check any solution $\alpha^*$ if it satisfies $(\partial_{\alpha\alpha}f)(\alpha^*) > 0$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.