So, I am trying to take the derivative of the following equation, because it is needed in an optimization problem. I want to make sure I am on the right track. The equation is:

$$ -3 \mathbb E[(w^Tz)^2]^2 $$

So my question is, what is:

$$ \frac{\delta (-3 \mathbb E[(w^Tz)^2]^2)}{\delta w} = ? $$

Please assume here that $w$ is a 2-dimensional column vector, just like $z$. $z$ is also a zero mean, unit variance (joint) random variable. ($w$ is a deterministic vector).

I would like a break down of the steps for evaluating the derivative here - I half syspect the chain rule is involved, however I am getting thrown off by the presence of the expectation operator.



Notice that $(w^T z)^2 = w^T z z^T w = w^T (z z^T) w$. Thus $f(w) = E_z (w^T z)^2 = w^T E_z (z z^T) w$. The derivative is given by $\frac{\partial f(w)}{\partial w} = 2 w^T E_z (z z^T)$. Just as a reminder, $\frac{\partial (w^TAw)}{\partial w} = (A+A^T)w$. This becomes $2Aw$ when $A$ is symmetric.

You wished to compute the derivative of $\phi(w) = -3 f(w)^2$. This can be computed using the usual calculus rules as $\frac{\partial \phi(w)}{\partial w} = -6 f(w) \frac{\partial f(w)}{\partial w} = -12 \, (w^T E_z (z z^T) w) \, w^T E_z (z z^T)$.

Now $E_z(zz^T) = I$ because $z$ is zero mean with covariance $I$. Hence, the final answer is: $-12 \, (w^T w) \, w^T = -12||w||_2^2 w^T$, where $||w||_2$ is the $L2$-norm of $w$.

  • $\begingroup$ +1. And $\phi(w)$ is simply $\phi(w)=-3(w^Tw)^2=-3\|w\|_2^4$. $\endgroup$
    – Did
    Aug 11 '12 at 8:18
  • $\begingroup$ Thanks @Kartik Audhkhasi for the additional contribution. I missed the unit variance statement in the question. $\endgroup$
    – copper.hat
    Aug 11 '12 at 8:37
  • $\begingroup$ Thanks you copperhat and @Kartik Audhkhasi for that, it is in fact the correct answer. I see that what you did here is 'open up' $f(w)$ a little before taking the derivative. What I am not clear about is, how do we generally treat the expectation operator in relation to derivatives? Would it have been wrong to move the $\frac{\delta }{\delta w}$ inside the expectation and go from there? When might it be ok or not ok to do so? Thank you. $\endgroup$
    – Spacey
    Aug 11 '12 at 13:27
  • $\begingroup$ For some more general conditions under which you can exchange differentiation and expectation, look at Qiaochu Yuan's answer here math.stackexchange.com/questions/12909/… $\endgroup$
    – copper.hat
    Aug 11 '12 at 15:23
  • $\begingroup$ @copper.hat I took a look but did not quite get it. Let me phrase it another way. Why didn't you take the derivative operator inside the expectation in this example? Is it because the outside squaring makes it non linear? $\endgroup$
    – Spacey
    Aug 12 '12 at 0:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.