0
$\begingroup$

If Cost function is L , $$ L=−(\frac{1}{m})(y(log(h(x))+(1−y)( log(1−h(x) ) ) $$ $$ h(x)=\frac{1}{1+e^{-(w^{T}x+b)}} $$ First order partial deriavative of L with respect to w is , $$\frac{\partial L}{\partial w} = - ( \frac{1}{m} ) ( h(w) - y )x $$
Question :
how do i find the second order partial derivative of L with respect to w ?, that is $$ \frac{\partial ^{2}L}{\partial w^{2}}$$
So that i can compute the error gradient by using Newton's method and update Weights $ w $, like this $$ w_{new} = w_{old} - (\frac{\partial ^{2}L}{\partial w^{2}})^{-1} \ ( \frac{\partial L}{\partial w}) $$
Am just trying to figure out how Newton's method works with logistic regression.

$\endgroup$

3 Answers 3

2
$\begingroup$

It is not so clear that you get these concepts.

You should clarify inputs and outputs. What you seem to have done is calculated second derivative of a scalar valued function of one variable. In other words : $$\mathbb R^{1} \to \mathbb R^{1}$$ function. Jacobians take all different partial differentials with respect to all different input variables. For a function $$\cases{x \in \mathbb R^n\\f(x) \in \mathbb R^m}$$

you get an output that is a $n\times m$ matrix.

For a Hessian to be a matrix we would need for a function $f(x)$ to be $$\mathbb R^{n} \to \mathbb R^{1}$$

the more general case

$$\mathbb R^{n} \to \mathbb R^{m}$$

it will be a 3 indexed tensor.

$\endgroup$
7
  • $\begingroup$ Since I am a beginner i got an idea about the gradients , tensor which is a n-dimensional vector representation. As far second order optimization i am just starting off so Please enlighten me ! $\endgroup$
    – guru_007
    Commented Aug 5, 2019 at 18:21
  • $\begingroup$ I think your derivation of second order derivative is correct, I just wanted to point out that we usually use multivariable functions when talkning of Jacobians and Hessians. $\endgroup$ Commented Aug 5, 2019 at 18:32
  • $\begingroup$ i just dont how these things add up to minimizing the loss function as this$ L=−(\frac{1}{m})(y(log(h(x))+(1−y)( log(1−h(x) ) ) $, where $h(x)=\frac{1}{1+e^-{wx+b}}$ , m = len of vector x and how did L became $\frac{dL}{dw} = - ( \frac{1}{m} ) ( h(x) - y )x $ ? $\endgroup$
    – guru_007
    Commented Aug 6, 2019 at 10:10
  • $\begingroup$ Keep reading. Probably something you missed. $\endgroup$ Commented Aug 6, 2019 at 12:31
  • $\begingroup$ you saying that the function has to be positive semi-definite for newton's method to work ? can you brief me on that please $\endgroup$
    – guru_007
    Commented Aug 7, 2019 at 4:51
1
$\begingroup$

For the second derivative, you could do is faster $$\sigma' (x)= \sigma(x)(1-\sigma(x))=\sigma(x)-\sigma^2(x)$$ $$\sigma'' (x)=\sigma' (x)-2 \sigma (x)\sigma' (x)=\sigma' (x)(1-2\sigma (x))=\sigma(x)(1-\sigma(x))(1-2\sigma (x))$$ which is not what you obtain.

Edit

In order to check the result, let us use the second-order central derivative $$f''(x) = \frac{f(x+h) - 2 f(x) + f(x-h)}{h^{2}}$$ at $x=\frac 12$ and $h=\frac 1 {200}$.

This would give $-0.0575566$ while the formula I wrote gives $-0.0575568$; your formula leads to $0.292561$.

At least, we do not agree.

$\endgroup$
9
  • $\begingroup$ But mine is so simpler than yours ! $\endgroup$
    – guru_007
    Commented Aug 6, 2019 at 9:56
  • $\begingroup$ @guru_007. Simpler, not sure ! Is your correct ? $\endgroup$ Commented Aug 6, 2019 at 10:05
  • $\begingroup$ Leiboivici am preety sure about it but am having trouble how does it add to deriving $\frac{d^{2}L}{dx^{2}} $, where $ L=−(\frac{1}{m})(y(log(h(x))+(1−y)( log(1−h(x) ) ) $, where $ h(x)=\frac{1}{1+e^-{wx+b}} $ and $\frac{dL}{dw} = - ( \frac{1}{m} ) ( h(w) - y )x $ $\endgroup$
    – guru_007
    Commented Aug 6, 2019 at 10:15
  • $\begingroup$ @guru_007. Have a look at my edit. $\endgroup$ Commented Aug 6, 2019 at 10:47
  • 1
    $\begingroup$ Changing problems after answer has been gotten is not nice. It will give bad responses and probably fewer answers in the future. $\endgroup$ Commented Aug 7, 2019 at 8:30
0
$\begingroup$

First of all $f(x)$ has to satisfy the condition where its hessian has to be $$\mathbb R^{n} \to \mathbb R^{1}$$ Meaning that $f(x)$ has to be twice differentiable and it is positive semi-definite.

we already know from here that ,

$$ \frac{\partial L} { \partial w} = (h_\theta(x) - y)x $$ $$ \sigma(x) = \frac{1}{1+e^{-(w^Tx+b)}}$$ Refer here for proof on first deriavative of $ \sigma(x)$ , $$ \sigma^{'}(x) = \sigma(x)(1-\sigma(x)) $$

Please note that here $ h_\theta(x) $ and $ \sigma(x) $ are one and the same , i just used $ \sigma(x)$ for representation sake.

Now we need to find $ \frac{\partial^2 L}{ \partial w^2} $ , $$ \begin{align*} \frac{\partial^2 L}{ \partial w^2} &= \frac{\partial L}{\partial w}(xh_\theta(x) - xy) \\ \\ &= x^2 \frac{\partial L}{\partial w} (h_\theta(x)) \ \ \ \ \ \ \ \ \ [ \ h_\theta^{'}(x) = \sigma^{'}(x) \ ] \\ \\ &= x^2 ( \sigma(x) - \sigma(x)^2) \\ \\ & (or) \\ \\ &= x .( \sigma(x) - \sigma(x)^2).x^{T} \end{align*} $$ Am hoping that am correct !

$\endgroup$
3
  • 1
    $\begingroup$ How could you have answered the question if you haven't even formulated one? $\endgroup$ Commented Aug 13, 2019 at 5:40
  • $\begingroup$ I have ignored the unwanted parts & framed the question clearly now ! @mathreadler Is my implementation of answer is correct if not can you provide me an answer ? Thank you $\endgroup$
    – guru_007
    Commented Aug 17, 2019 at 7:42
  • $\begingroup$ Here's the proof for my answer @mathreadler $\endgroup$
    – guru_007
    Commented Aug 25, 2019 at 16:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .