1
$\begingroup$

I'm reading a paper that shows the following:

Takes 2 vectors, one containing data, one containing weights and takes the dot product, and then divides by the norms, obtaining the cosine of the angle between them. Calling this "net_norm"

$$net_{norm} =\cos{\theta} = \frac{\vec{x}\cdot \vec{w}}{|\vec{x}||\vec{w}|}$$

Then it rewrites this in a different form

$$net_{norm} = \frac{\sum_{i=1}^{N}w_ix_i}{\sqrt{\sum_{i=1}^{N}w_i^2}\sqrt{\sum_{i=1}^{N}x_i^2}}$$

And finally says that the derivatives wrt $x_i$ and $w_i$ are

$$\frac{\partial net_{norm}}{\partial w_i} = \frac{x_i}{|\vec{x}||\vec{w}|} - \frac{w_i(\vec{w} \cdot \vec{x})}{|\vec{w}|^3|\vec{x}|}$$

$$\frac{\partial net_{norm}}{\partial x_i} = \frac{w_i}{|\vec{x}||\vec{w}|} - \frac{x_i(\vec{w} \cdot \vec{x})}{|\vec{x}|^3|\vec{w}|}$$ My try

$$\frac{\partial net_{norm}}{\partial w_i} = \frac{x_i}{\left(\frac{1}{2\sqrt{\sum{w_i^2}}}2w_i\sqrt{\sum{x_i^2}} + \frac{1}{2\sqrt{\sum{x_i^2}}}2x_i\sqrt{\sum{w_i^2}}\right)} = \frac{x_i}{\left(\frac{w_i|\vec{x}|}{|\vec{w}|} + \frac{x_i|\vec{w_i}|}{|\vec{x_i}|}\right)}$$ which becomes $$\frac{x_i|\vec{w}||\vec{x}|}{(w_i|\vec{x}|^2 + x_i|\vec{w}|^2)}$$

But I don't know how to continue

$\endgroup$

1 Answer 1

2
$\begingroup$

For the notational purposes of this answer, I'm going to assume that $N=2$. This changes nothing, except that it allows me to use some convenient explicit notation.

Let's look at $$ \frac{\partial}{\partial w_1} \frac{\vec{x} \cdot \vec{w}}{\|\vec x \| \| \vec w \|} = \frac{\partial}{\partial w_1} \frac{x_1 w_1 + x_2 w_2}{\sqrt{x_1^2 + x_2^2}\sqrt{w_1^2 + w_2^2}}.$$ I will apply the "product rule" of differentiation, and think of the derivative as (derivative of numerator) $\times$ (1/denominator) + (numerator) $\times$(derivative of 1/denominator). Then this is exactly $$ x_1\frac{1}{\sqrt{x_1^2 + x_2^2}\sqrt{w_1^2 + w_2^2}} + (x_1w_1 + x_2w_2) \frac{-\frac{1}{2}}{\sqrt{x_1^2 + x_2^2} \; (w_1^2 + w_2^2)^{3/2}} \underbrace{(2w_1)}_{\text{chain rule}},$$ in which I've written the order in the same order in which I would write this derivative in an introductory calculus course, showing the product rule and chain rule steps implicitly. This can be simplified as $$ \frac{x_1}{\|\vec x \| \|\vec w\|} - \frac{w_1 \; (\vec x \cdot \vec w)}{\| \vec x \| \| \vec w \|^3}$$ as desired.

The general partial derivatives for any $N$ and for $\partial_{x_i}$ are done similarly.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .