Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

What is the formula for the derivative of the product of two matrices when they are different sizes? Further, what is the formula for the derivative of a Hadamard product when the derivatives of the inputs aren't the same size as the inputs?

Here's what I'm thinking: assume an $n\times 1$ vector $x$, a $n\times n$ symmetric matrix $S$, $f(x)=Sw$, $g(x)=w'Sw$, $h(x)=\frac{f(x)}{g(x)}$, and $j(x)=\operatorname{had}(x, h(x))$ where $\operatorname{had}(a, b)$ is the Hadamard product between the two vectors.

For $f(x)$ and $g(x)$ the derivatives are $S$ and $2\cdot Sw$, respectively. One would think that the derivative of $h(x)$ would be $\dfrac{g(x)f'(x)-g'(x)f(x)}{g(x)^2}$ from the quotient rule. However, the numerator is $n\times n$ and the second part is multiplying a $n\times 1$ vector by an $n\times 1$ vector, which doesn't work.

Similarly, I found a formula for the derivative of Hadamard product and applying it to this would give $\operatorname{had}(x, h'(x)) + \operatorname{had}(I, h(x)) =\operatorname{had}(x, h'(x)) + h(x)$. Since $h(x)$ is $n\times 1$, $h'(x)$ must be $n\times n$. However, the Hadamard product of an $n\times 1$ and $n\times n$ matrix doesn't work. Similarly, the opposite is the case in the second term. So far as I can tell, the terms that are $n\times 1$ need to be copied over to become $n\times n$. This would indicate that the proper formula is something like $\operatorname{had}(\operatorname{kron}(x,\operatorname{ones}),h'(x))+\operatorname{had}(I,\operatorname{kron}(h(x),\operatorname{ones}))$, where $I$ is an $n\times n$ identity matrix and ones is a $1\times n$ vector of ones.

I think that's on the right track since I found a matrix calculus guide that connects the problem to Kronecker products. However, I still had the problem of conformability when using their product rule formula.

share|cite|improve this question
I think that you mean Hadamard... Capital H. – Mariano Suárez-Alvarez Feb 17 '11 at 6:46

Someone has provided the answers to me.

He noted that $g'(x)$ is $1 \times n$ not $n \times 1$ so there is no conformability problem with $h'(x)$. Second, the derivative of the Hadamard product $f(x) \circ g(x)$ is $\operatorname{diag}(g(x)) f'(x) + \operatorname{diag}(f(x)) g'(x)$.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.