# Why are terms flipped in partial derivative of logistic regression cost function?

When calculating the partial derivative: $$\frac{\partial}{\partial\theta_{j}}J(\theta)$$ from: $$J(\theta)=-\frac{1}{m}\sum_{i=1}^{m}(y^{i}\log(h_\theta(x^{i}))+(1-y^{i})\log(1-h_\theta(x^{i})))$$ with: $$h_{\theta}(x)=g(\theta^{T}x)$$ and $$g(z)=\frac{1}{1+e^{-z}}$$ as documented in the question here: Derivative of cost function for logistic regression (Note: Some of my content is taken from that post.)

So, how are we deriving:

$$\frac{\partial}{\partial\theta_{j}}J(\theta) =\sum_{i=1}^{m}(h_\theta(x^{i})-y^i)x_j^i$$ rather than: $$\frac{\partial}{\partial\theta_{j}}J(\theta) =\sum_{i=1}^{m}(y^i - h_\theta(x^{i}))x_j^i$$ ? When I work it out, I get the latter, rather than the former.

Here's my work (attempted proof):

Let $$a = \log(h_\theta(x^i))$$ and $$b=\log(1-h_\theta(x^i))$$ Then,

$$J(\theta)=-\frac{1}{m}\sum_{i=1}^{m}(y^{i} a+(1-y^{i}) b)$$

We can then simplify a: $$a = \log(h_\theta(x^i))$$ $$= \log(\frac{1}{1+e^{-\theta x^i}})$$ $$= \log(1) - \log(1+e^{-\theta x^i})$$ $$= 0 - \log(1+e^{-\theta x^i})$$ $$= -\log(1+e^{-\theta x^i})$$

and since $$1 = \frac{(1+e^{-\theta x^i})}{(1+e^{-\theta x^i})}$$

we can simplify b:

$$b = \log(1-h_\theta(x^i))$$ $$= \log(1 - \frac{1}{(1+e^{-\theta x^i})})$$ $$= \log(\frac{(1+e^{-\theta x^i})}{(1+e^{-\theta x^i})} - \frac{1}{(1+e^{-\theta x^i})})$$ $$= \log(\frac{(1+e^{-\theta x^i}) - 1}{(1+e^{-\theta x^i})})$$ $$= \log(\frac{1-1+e^{-\theta x^i}}{(1+e^{-\theta x^i})})$$ $$= \log(\frac{0+e^{-\theta x^i}}{(1+e^{-\theta x^i})})$$ $$= \log(\frac{e^{-\theta x^i}}{(1+e^{-\theta x^i})})$$ $$= \log(e^{-\theta x^i}) - \log(1+e^{-\theta x^i})$$ $$= -\theta x^i - \log(1 + e^{-\theta x^i})$$

by associativity, simplifying by common denominators, evaluating the logarithm on e, and the logarithm property that $$\log_b(M/N) = \log_b(M) - \log_b(N)$$

Thus, we simplified to: $$a = -\log(1+e^{-\theta x^i})$$ and $$b = -\theta x^i - \log(1 + e^{-\theta x^i})$$

Let us then set: $$t = \log(1 + e^{-\theta x^i})$$ so that: $$a = -t$$ and $$b = -\theta x^i - t$$

We may then substitute: $$J(\theta) =-\frac{1}{m}\sum_{i=1}^{m}(y^{i}\log(h_\theta(x^{i}))+(1-y^{i})\log(1-h_\theta(x^{i})))$$ $$= -\frac{1}{m}\sum_{i=1}^{m}(y^i a + (1 - y^i) b))$$ Expanding to t, we then get: $$J(\theta) =-\frac{1}{m}\sum_{i=1}^{m}(y^i (-t) + (1 - y^i) (-\theta x^i - t)) = -\frac{1}{m}\sum_{i=1}^{m}(-t y^i + (1 - y^i) (-\theta x^i - t))$$ This allows us to expand via FOIL since: $$(1 - y^i) (-\theta x^i - t) = (-\theta x^i + \theta x^iy^i - t + ty^i)$$

So, $$J(\theta) = -\frac{1}{m}\sum_{i=1}^{m}(-ty^i + (-\theta x^i + \theta x^iy^i -t + ty^i))$$ and again by associativity: $$J(\theta) = -\frac{1}{m}\sum_{i=1}^{m}(-ty^i + ty^i + -\theta x^i + \theta x^iy^i -t )$$ $$= -\frac{1}{m}\sum_{i=1}^{m}(0 + -\theta x^i + \theta x^iy^i -t )$$ $$= -\frac{1}{m}\sum_{i=1}^{m}(-\theta x^i + \theta x^iy^i -t )$$

Now we can expand t back out to reduce further because

$$-\theta x^i = \log(e^{-\theta x^i})$$ and $$-\theta x^i = -\log(e^{\theta x^i})$$

So, $$J(\theta) = -\frac{1}{m}\sum_{i=1}^{m}(-\theta x^i + \theta x^iy^i - \log(1 + e^{-\theta x^i}) )$$

$$= -\frac{1}{m}\sum_{i=1}^{m}(-\log(e^{\theta x^i}) + \theta x^iy^i - \log(1 + e^{-\theta x^i}))$$

$$= -\frac{1}{m}\sum_{i=1}^{m}(\theta x^iy^i + (-\log(e^{\theta x^i})) - \log(1 + e^{-\theta x^i}))$$

$$= -\frac{1}{m}\sum_{i=1}^{m}(\theta x^iy^i + (-[\log(e^{\theta x^i}) + \log(1 + e^{-\theta x^i})]))$$

and since: $$\log(M) + \log(N) = \log(MN)$$

we can apply the distributive property and simplify the bracketed term: $$[\log(e^{\theta x^i}) + \log(1 + e^{-\theta x^i})]$$ $$= \log(e^{\theta x^i} * (1 + e^{-\theta x^i})$$ $$= \log(e^{\theta x^i} + (e^{\theta x^i} * e^{-\theta x^i} ))$$ $$= \log(e^{\theta x^i} + (e^{(\theta x^i) + (-\theta x^i)} ))$$ $$= \log(e^{\theta x^i} + (e^{0} ))$$ $$= \log(e^{\theta x^i} + (1) )$$ $$= \log(e^{\theta x^i} + 1 )$$

Substituting the simplified bracketed term back in: $$J(\theta) = -\frac{1}{m}\sum_{i=1}^{m}(\theta x^iy^i + (-[\log(e^{\theta x^i} + 1 )]))$$

and dropping the extra brackets and parentheses:

$$J(\theta) = -\frac{1}{m}\sum_{i=1}^{m}(\theta x^iy^i -\log(e^{\theta x^i} + 1 ))$$

Next, calculating the partial derivatives:

$$\frac{\partial}{\partial \theta_j} (\theta x^i y^i)= x^i_j y^i$$ because the corresponding theta component becomes 1 and drops out. Then, let:

$$s = \theta x^i$$ $$r = 1 + e^{s}$$

$$\frac{\partial}{\partial \theta_j} (\log(r)) = \frac{\partial}{\partial r} \frac{\partial r}{\partial s} \frac{\partial s}{\partial \theta_j} (\log(r))$$ So, $$\frac{\partial}{\partial r}(\log(r)) = \frac{1}{r}$$ (since we're really referring to the natural log, ln) and $$\frac{\partial r}{\partial s} = e^s$$ and $$\frac{\partial s}{\partial \theta_j} = x_j^i$$ because the theta component again becomes 1 and drops out. So, by the chain rule and substitution: $$\frac{\partial}{\partial \theta_j}(\log(r)) = \frac{\partial}{\partial r} \frac{\partial r}{\partial s} \frac{\partial s}{\partial \theta_j}$$ $$= \frac{1}{r} e^s x_j^i$$ $$= \frac{1}{1 + e^{s}} e^s x_j^i$$ $$= \frac{1}{1 + e^{\theta x^i}} e^{\theta x^i} x_j^i$$ $$= \frac{e^{\theta x^i} x_j^i}{1 + e^{\theta x^i}}$$ We can then move the e term from the top to the bottom and apply the distributive property again to simplify: $$\frac{\partial}{\partial \theta_j} (\log(r)) = \frac{x_j^i}{e^{-\theta x^i} (1 + e^{\theta x^i})}$$ $$= \frac{x_j^i}{(e^{-\theta x^i} + e^{-\theta x^i} * e^{\theta x^i})}$$ and then using exponent multiplication rules: $$\frac{\partial}{\partial \theta_j} (\log(r)) = \frac{x_j^i}{(e^{-\theta x^i} + e^{(-\theta x^i) + (\theta x^i)})}$$ $$= \frac{x_j^i}{(e^{-\theta x^i} + e^{(0)})}$$ $$= \frac{x_j^i}{(e^{-\theta x^i} + 1)}$$ $$= \frac{x_j^i}{(1 + e^{-\theta x^i})}$$

By using the notation from the link above: $$\theta x^i:=\theta_0+\theta_1 x^i_1+\dots+\theta_p x^i_p.$$ Resolving: $$h_\theta(x^i) = g(\theta^T x^i) = \frac{1}{1 + e^{-\theta x^i}}$$ and then factoring out:

$$\frac{\partial}{\partial \theta_j}(\log(r)) = \frac{x_j^i}{(1 + e^{-\theta x^i})} = x_j^i \frac{1}{(1 + e^{-\theta x^i})} = x_j^i h_\theta(x^i)$$

So, our inner partial derivatives are:

$$\frac{\partial}{\partial \theta_j} (\theta x^i y^i)= x^i_j y^i$$ and $$\frac{\partial}{\partial \theta_j}(\log(e^{\theta x^i} + 1)) = x_j^i h_\theta(x^i)$$

So, for J: $$J(\theta) = -\frac{1}{m}\sum_{i=1}^{m}(\theta x^iy^i -\log(e^{\theta x^i} + 1 ))$$ computing the partial derivative:

$$\frac{\partial}{\partial \theta_j}(J(\theta)) = -\frac{1}{m}\sum_{i=1}^{m}(\frac{\partial}{\partial \theta_j} (\theta x^i y^i) - \frac{\partial}{\partial \theta_j}(\log(e^{\theta x^i} + 1)))$$ $$= -\frac{1}{m}\sum_{i=1}^{m}(x^i_j y^i - x_j^i h_\theta(x^i))$$ which we can factor: $$= -\frac{1}{m}\sum_{i=1}^{m}((y^i - h_\theta(x^i))x^i_j)$$

and yet the course instructor obtained: $$\frac{\partial}{\partial\theta_{j}}J(\theta) =\sum_{i=1}^{m}(h_\theta(x^{i})-y^i)x_j^i$$

How is this possible?

• In your final result bring the minus sign inside the parentheses? Mar 23, 2017 at 21:13
• @xidgel Oh my gosh.... I can't believe I didn't see that. Thank you!!! Mar 23, 2017 at 21:18

So, the answer is that the course instructor brought the minus sign inside the parentheses. Thanks to @xidgel for noticing it! So, $$-\frac{1}{m}\sum_{i=1}^{m}((y^i - h_\theta(x^i))x^i_j)$$
$$= \frac{1}{m}\sum_{i=1}^{m}(-(y^i - h_\theta(x^i))x^i_j)$$ $$= \frac{1}{m}\sum_{i=1}^{m}((-y^i + h_\theta(x^i))x^i_j)$$ $$= \frac{1}{m}\sum_{i=1}^{m}((h_\theta(x^i)-y^i)x^i_j)$$
Also, when I took a closer look at the instructor's derivative, contrary to what was posted in Derivative of cost function for Logistic Regression, the instructor's term did still have the $$\frac{1}{m}$$ in front of the sum. But it was positive, not negative.