Calculate the differential of the function $f: \Bbb R^n \to \Bbb R$ given by $$f(x) = x^T A x$$ with $A$ symmetric. Also, differentiate this function with respect to $x^T$.

How exactly does this work in the case of vectors and matrices? Could anyone please help me out?

  • $\begingroup$ And yes, I will soon try to learn to use Latex :). $\endgroup$ – dreamer Feb 23 '13 at 16:03
  • 2
    $\begingroup$ I've edited your math formatting, could you look through it and see that it is still correct? $\endgroup$ – Arthur Feb 23 '13 at 16:06
  • 1
    $\begingroup$ Write math between \$...\$, you can find symbols etc. here: codecogs.com/latex/eqneditor.php $\endgroup$ – Kasper Feb 23 '13 at 16:06
  • 1
    $\begingroup$ try a $2 \times 2$ case explicitly and see if you can guess the general form of answer. $\endgroup$ – Maesumi Feb 23 '13 at 16:09
  • 1
    $\begingroup$ This is the composition of the linear map $x\longmapsto (x,x)$ and the bilinear map $(x,y)\longmapsto x^tAy$. You can use the chain rule. $\endgroup$ – Julien Feb 23 '13 at 16:10

As a start, things work "as usual": You calculate the difference between $f(x+h)$ and $f(x)$ and check how it depends on $h$, looking for a dominant linear part as $h\to 0$. Here, $f(x+h)=(x+h)^TA(x+h)=x^TAx+ h^TAx+x^TAh+h^TAh=f(x)+2x^TAh+h^TAh$, so $f(x+h)-f(x)=2x^TA\cdot h + h^TAh$. The first summand is linear in $h$ with a factor $2x^TA$, the second summand is quadratic in $h$, i.e. goes to $0$ faster than the first / is negligible against the first for small $h$. So the row vector $2x^TA$ is our derivative (or transposed: $2Ax$ is the derivative with respect to $x^T$).

  • $\begingroup$ Thank you. This is also what I tried. However, what confused me is that the question mentions that you should differentiate with respect to xT. From your answer, I see that you took the transpose of the 'ordinary' derivative. Does that imply that the ordinary derive is always taken with respect to x so that you can just take the transpose when you differentiate with respect to xT? $\endgroup$ – dreamer Feb 23 '13 at 16:21
  • 2
    $\begingroup$ Are you sure that $x^TAh = h^TAx$? Why? $\endgroup$ – mavavilj Nov 17 '16 at 13:24
  • $\begingroup$ @mavavij it's not. The dimensions don't necessarily check out. $\endgroup$ – Michael Chav Sep 28 '17 at 23:28
  • 3
    $\begingroup$ Actually, it's because A is symmetric. $\endgroup$ – Michael Chav Sep 28 '17 at 23:43

There is another way to solve the problem:

Let $\mathbf{x}^{n\times 1}=(x_1,\dots ,x_n)'$ be a vector, the derivative of $\mathbf y=f(\mathbf x)$ with respect to the vector $\mathbf{x}$ is defined by $$\frac{\partial f}{\partial \mathbf x}=\begin{pmatrix} \frac{\partial f}{\partial x_1} \\ \vdots\\ \frac{\partial f}{\partial x_n} \end{pmatrix}$$ Let \begin{align} \mathbf y&=f(\mathbf x)\\&=\mathbf x'A\mathbf x \\&=\sum_{i=1}^n\sum_{j=1}^n a_{ij}x_ix_j\\&=\sum_{i=1}^na_{i1}x_ix_1+\sum_{j=1}^na_{1j}x_1x_j+\sum_{i=2}^n\sum_{j=2}^n a_{ij}x_ix_j \\\frac{\partial f}{\partial x_1} &=\sum_{i=1}^na_{i1}x_i+\sum_{j=1}^na_{1j}x_j\\&=\sum_{i=1}^na_{1i}x_i+\sum_{i=1}^na_{1i}x_i \,[\text{since}\,\, a_{1i}=a_{1i}]\\ &=2 \sum_{i=1}^na_{1i}x_i \\ \frac{\partial f}{\partial \mathbf x}&=\begin{pmatrix} 2 \sum_{i=1}^na_{1i}x_i \\ \vdots\\ 2 \sum_{i=1}^na_{ni}x_i \end{pmatrix} \\&=2\begin{pmatrix} a_{11} & a_{12} & \dots & a_{1n}\\ \vdots & \vdots &\ddots & \vdots \\ a_{n1} & a_{n2} & \dots & a_{nn} \end{pmatrix}\begin{pmatrix}x_1 \\ \vdots \\ x_n \end{pmatrix}\\ &= 2A\mathbf x \end{align}

  • $\begingroup$ Thanks for showing me this way as well :). Gives me more options :) $\endgroup$ – dreamer Feb 24 '13 at 10:01
  • 1
    $\begingroup$ The only thing that is slightly unclear to me is how x'Ax becomes the double summation (aijxixj). Why is the order reversed here? I mean, why arent the a's in the middle anymore? @Argha $\endgroup$ – dreamer Feb 24 '13 at 10:04
  • $\begingroup$ Note that $\mathbf x'A\mathbf x=(x_1,\dots ,x_n)\begin{pmatrix} a_{11} & a_{12} & \dots & a_{1n}\\ \vdots & \vdots &\ddots & \vdots \\ a_{11} & a_{12} & \dots & a_{1n} \end{pmatrix}\begin{pmatrix}x_1 \\ \vdots \\ x_n \end{pmatrix}$ and simply multipling we get required result. Also note order of $\mathbf x'$ is $1 \times n$ and order of $A$ is $n \times n$. So order of $\mathbf x'A\mathbf x$ is $1 \times 1$. $\endgroup$ – Argha Feb 24 '13 at 11:24
  • $\begingroup$ Note that $a_{ij}\,x_i\,x_j \equiv x_i\,a_{ij}\,x_j$. So there is no problem at all. $\endgroup$ – Argha Feb 24 '13 at 11:36
  • 1
    $\begingroup$ On the first summation of the line that says [since a_1i = a_1i, how did you swap the indices from the previous step? $\endgroup$ – Michael Chav Sep 28 '17 at 23:40

@Hagen von Eitzen's answer is certainly the fastest route here, but since you asked, here is a chain rule.

Here are two useful facts about linear and bilinear bounded maps from normed vectors spaces to normed vector spaces.

If $f$ is linear and bounded, then trivially: $$ df_x(h)=f(h). $$

And if $g$ is bilinear and bounded ($\|g(h,k)\|\leq C\|h\|\|k\|$), we have $$ dg_{(x,y)}(h,k)=g(x,k)+g(h,y). $$

Now take $f(x)=(x,x)$ and $g(x,y)=x^tAy$. The former is linear and bounded, the latter is bilinear and bounded.

So, by the chain rule, $g\circ f(x)=x^tAx$ is differentiable and $$ d(g\circ f)_x(h)=dg_{f(x)}\circ df_x(h)=dg_{(x,x)} (h,h)=x^tAh+h^tAx. $$

This is true for any matrix $A$. Now if $A$ is symmetric, this can be simplified since $$ x^tAh+h^tAx=x^tAh+h^tA^tx=x^tAh+(Ah)^tx=2x^tAh. $$

Removing $h$, this gives $$ d(g\circ f)_x=2x^tA. $$

  • $\begingroup$ Thank you. The other answer is indeed quicker but I am glad that I know now how to do it in this way as well. Much appreciated :). $\endgroup$ – dreamer Feb 23 '13 at 17:03
  • 2
    $\begingroup$ @user48288 You're welcome. And I am sure these general facts about bounded linear and bilinear maps will prove useful sooner or later. $\endgroup$ – Julien Feb 23 '13 at 17:04
  • $\begingroup$ Can I know in detail? I want to know this, but it can be hard to understand. $\endgroup$ – jakeoung Apr 15 '15 at 23:37

Here is relationship between directional derivative whenever f is differentiable.

$f'(p: v)$ denotes the derivative of $f$ at $p$ in the direction of $v$.

Let $f:U \subset \mathbb{R}^n \rightarrow \mathbb{R}$ and $p\in U$, $v \in \mathbb{R}^n$. Suppose that $f$ is differentiable at $p$. Then we have \begin{equation*} df_p(v)=f(p;v)= \lim_{t \rightarrow 0}\frac{f(\sigma(t))-f(p)}{t} \end{equation*} for any differentiable curve $\sigma:(-\epsilon, \epsilon)\rightarrow U$ such that $\sigma(0)=p$ and $\sigma '(0)=v$.

In our case $f(x)=x^TAx$ and $\sigma (t) = x+th$,

$$f'(x; h) = \lim_{t\rightarrow 0} \frac{(x+th)^TA(x+th)-x^TAx}{t}$$ $$f'(x; h) = x^TAh+h^TAx$$

Since A is symmetric and we have the follwing: $$f'(x; h) = x^TAh+h^TAx=x^TAh+x^TA^Th$$ $$f'(x; h) = x^T(A+A^T)h$$

So the differential/gradient is simply $2x^TA$. $$f'(x; h) = 2x^TAh $$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.