# How to derive and compute second derivative of matrix quadratic form?

I want to take the first and second derivative of the following matrix quadratic form,

$$Q = {\phi}(\beta)'W{\phi}(\beta),$$

where, $$\phi(\beta)$$ is a $$q \times 1$$ vector, $$\beta$$ is a $$k \times 1$$ vector, and W is a $$q \times q$$ symmetric matrix, then Q is a scalar. So, I would expect the first derivative Q w.r.t to $$\beta$$ to be a $$k \times 1$$ vector $$\frac{\partial{Q}}{\partial{\beta}}$$, and the second derivative to be a $$k \times k$$ matrix $$\frac{\partial^2{Q}}{\partial{\beta}\partial\beta'}$$.

Then, taking the first derivative of Q w.r.t $$\beta$$ yields,

$$\frac{\partial{Q}}{\partial{\beta}} = 2\frac{\partial{\phi}'}{\partial{\beta}}W\phi(\beta),$$

which is $$k \times 1$$ as expected. Then, the second derivative yields me,

$$\frac{\partial^2{Q}}{\partial{\beta}\partial\beta'} = 2\left(\frac{\partial^2{\phi(\beta)'}}{\partial{\beta}\partial\beta'}W\phi(\beta) + \frac{\partial{\phi(\beta)'}}{\partial\beta}W\frac{\partial{\phi(\beta)}}{\partial\beta'}\right),$$

which should be $$k \times k$$. The second term inside the parenthesis is $$k \times k$$, but the first one isn't? It doesn't quite make sense to me, but it can be of the right size because $$W$$ and $$\phi$$ give together $$q \times 1$$, while $$\frac{\partial^2{\phi(\beta)'}}{\partial{\beta}\partial\beta'}$$ I don't know what shape it might have ($$\frac{\partial{\phi(\beta)'}}{\partial{\beta}}$$, is $$k \times q$$, but differentiating such a matrix w.r.t. a $$1 \times k$$ vector..., how does that work? I don't see how would it be compatible with the terms that follow it).

In any case, I know it is wrong, because I have a formula outlining what the correct form should be, but I don't understand how to arrive to it or how does it work. According to this formula, the second derivative should be,

$$\frac{\partial^2{Q}}{\partial{\beta}\partial\beta'} = 2\left(({\phi}(\beta)'W\otimes I_k)\frac{\partial vec(\partial\phi(\beta)'/\partial\beta)}{\partial{\beta'}}+\frac{\partial{\phi(\beta)'}}{\partial\beta}W\frac{\partial{\phi(\beta)}}{\partial\beta'}\right).$$ Here, $$vec(\partial\phi(\beta)'/\partial\beta)$$ is the vectorisation of a $$k \times q$$ matrix, which would be, to my understanding, a $$(k \times q)\times 1$$ vector(?). Differentiating this vector w.r.t. a $$1 \times k$$ vector $$\beta'$$ would yield a $$(k \times q)\times k$$ matrix. Furthermore, following my understanding of the Kronecker product, $$W \otimes I_k$$ would yield a $$(k \times q)\times (k \times q)$$ matrix. And all this would be alright except the term $$\phi(\beta)'$$ is still a $$1 \times q$$ vector and thus incompatible with the other terms following -- unless we do $$\phi(\beta)W$$ and then follow with $$I_k$$ yielding a $$1 \times q$$ vector $$\otimes$$ a $$k \times k$$ matrix, but I can't anywhere where it says the Kronecker product works between a vector and a matrix (and if so, what shape would the result then have? How would I compute it?)

I don't know where I am going wrong, and I would appreciate it very much if someone could guide me through this conundrum.

• I haven't read all what you wrote but I just would like to remark that you can treat your matrix $W$ to be diagonal because $W = U^TDU$ with a diagonal $D$ and an orthogonal $U$. Hence, $Q = \phi^TW\phi = (U\phi)^TD(U\phi) = \psi^T D\psi = \sum d_i\phi_i^2$, where $\psi = U\phi$. I have $Q'' = 2\sum d_i[(\nabla\psi_i)(\nabla\psi_i)^T + \psi_i\cdot H_i]$, where $H_i$ is the Hessian of $\psi_i$. – amsmath Aug 28 '19 at 17:11
• Your question is similar to this: "What is the second derivative of $f(x)$?" Without specifying the function, the best answer you will get is the blindingly obvious: $f''(x)$. So if you expect a useful answer to this question, you'll need to specify the functional dependence of the vector $\phi$ upon the vector $\beta$. – greg Aug 29 '19 at 19:50
• @greg $\phi(\beta)$ = $Z'(y-\exp(X\beta))$, where $Z$ is a $n \times q$ matrix, $y$ is a $n \times 1$ vector, $X$ is a $n \times k$ matrix, and $\beta$ is a $k \times 1$ vector. Then, $$Q = (Z'(y - e^{X\beta})'W(Z'(y - e^{X\beta}))$$ – porcos Aug 29 '19 at 21:57

For typing convenience, define the following variables \eqalign{ &A = ZWZ^T, \quad &b = \beta, \quad &e = \exp(Xb), \quad &E = {\rm Diag}(e) \\ & &p = (e-y), \quad &r = Ap, \quad &R = {\rm Diag}(r) \\ } And the differentials \eqalign{ dp &= de = e\odot(X\,db) = EX\,db \\ dr &= A\,dp = AEX\,db \\ } where the elementwise/Hadamard product $$(\odot)$$ of vectors was replaced by ordinary matrix multiplication by a diagonal matrix. This is a standard trick, e.g. $$Er = e\odot r = r\odot e = Re$$
Write the quadratic form in terms of these new variables. Then find its differential and gradient. \eqalign{ Q &= p^TZWZ^Tp &= p^Tr \\ dQ &= p^Tdr + r^Tdp &= 2r^TEX\,db \\&= (2X^TEr)^Tdb \\ g=\;\frac{\partial Q}{\partial b} &= 2X^TEr &= 2X^TRe \\ } Now find the differential of the gradient, and then the hessian. \eqalign{ dg &= 2X^TE\,dr + 2X^TR\,de \\ &= 2X^TEAEX\,db + 2X^TREX\,db \\ &= 2X^T(EA + R)EX\,db \\ H =\; \frac{\partial g}{\partial b} &= 2X^T(EA + R)EX \\ }
The hessian is symmetric since the matrices $$(A,E,R,W)$$ are all symmetric, and the matrices $$(E,R)$$ are diagonal (and therefore commute with each other).
• Thanks so much! That was really helpful!. Just one more thing, for $p = (e - y)$, shouldn't it be $(y - e)$ as I originally wrote? Do you change the order for some reason? I think that writing it $(y-e)$ shouldn't change anything (except maybe the sign?), right? – porcos Sep 6 '19 at 11:28
• It avoids carrying around a bunch of minus signs in the derivation, but makes no difference to the quadratic form since \eqalign{ Q&=p^TZWZ^Tp \\&= (-p)^TZWZ^T(-p) } – greg Sep 6 '19 at 13:50