I'm reading a paper that shows the following:

Takes 2 vectors, one containing data, one containing weights and takes the dot product, and then divides by the norms, obtaining the cosine of the angle between them. Calling this "net_norm"

$$net_{norm} =\cos{\theta} = \frac{\vec{x}\cdot \vec{w}}{|\vec{x}||\vec{w}|}$$

Then it rewrites this in a different form

$$net_{norm} = \frac{\sum_{i=1}^{N}w_ix_i}{\sqrt{\sum_{i=1}^{N}w_i^2}\sqrt{\sum_{i=1}^{N}x_i^2}}$$

And finally says that the derivatives wrt $x_i$ and $w_i$ are

$$\frac{\partial net_{norm}}{\partial w_i} = \frac{x_i}{|\vec{x}||\vec{w}|} - \frac{w_i(\vec{w} \cdot \vec{x})}{|\vec{w}|^3|\vec{x}|}$$

$$\frac{\partial net_{norm}}{\partial x_i} = \frac{w_i}{|\vec{x}||\vec{w}|} - \frac{x_i(\vec{w} \cdot \vec{x})}{|\vec{x}|^3|\vec{w}|}$$ My try

$$\frac{\partial net_{norm}}{\partial w_i} = \frac{x_i}{\left(\frac{1}{2\sqrt{\sum{w_i^2}}}2w_i\sqrt{\sum{x_i^2}} + \frac{1}{2\sqrt{\sum{x_i^2}}}2x_i\sqrt{\sum{w_i^2}}\right)} = \frac{x_i}{\left(\frac{w_i|\vec{x}|}{|\vec{w}|} + \frac{x_i|\vec{w_i}|}{|\vec{x_i}|}\right)}$$ which becomes $$\frac{x_i|\vec{w}||\vec{x}|}{(w_i|\vec{x}|^2 + x_i|\vec{w}|^2)}$$

But I don't know how to continue


For the notational purposes of this answer, I'm going to assume that $N=2$. This changes nothing, except that it allows me to use some convenient explicit notation.

Let's look at $$ \frac{\partial}{\partial w_1} \frac{\vec{x} \cdot \vec{w}}{\|\vec x \| \| \vec w \|} = \frac{\partial}{\partial w_1} \frac{x_1 w_1 + x_2 w_2}{\sqrt{x_1^2 + x_2^2}\sqrt{w_1^2 + w_2^2}}.$$ I will apply the "product rule" of differentiation, and think of the derivative as (derivative of numerator) $\times$ (1/denominator) + (numerator) $\times$(derivative of 1/denominator). Then this is exactly $$ x_1\frac{1}{\sqrt{x_1^2 + x_2^2}\sqrt{w_1^2 + w_2^2}} + (x_1w_1 + x_2w_2) \frac{-\frac{1}{2}}{\sqrt{x_1^2 + x_2^2} \; (w_1^2 + w_2^2)^{3/2}} \underbrace{(2w_1)}_{\text{chain rule}},$$ in which I've written the order in the same order in which I would write this derivative in an introductory calculus course, showing the product rule and chain rule steps implicitly. This can be simplified as $$ \frac{x_1}{\|\vec x \| \|\vec w\|} - \frac{w_1 \; (\vec x \cdot \vec w)}{\| \vec x \| \| \vec w \|^3}$$ as desired.

The general partial derivatives for any $N$ and for $\partial_{x_i}$ are done similarly.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.