Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Calculate the differential of the function $f:\Bbb R^n\rightarrow\Bbb R$ given by $f(x)=x^TAx$, with $A$ symmetric. Also differentiate this function to $x^T$.


How exactly does this work in the case of vectors and matrices? Could anyone please help me out?

share|improve this question
    
And yes, I will soon try to learn to use Latex :). –  dreamer Feb 23 '13 at 16:03
2  
I've edited your math formatting, could you look through it and see that it is still correct? –  Arthur Feb 23 '13 at 16:06
1  
Write math between \$...\$, you can find symbols etc. here: codecogs.com/latex/eqneditor.php –  Kasper Feb 23 '13 at 16:06
1  
try a $2 \times 2$ case explicitly and see if you can guess the general form of answer. –  Maesumi Feb 23 '13 at 16:09
1  
This is the composition of the linear map $x\longmapsto (x,x)$ and the bilinear map $(x,y)\longmapsto x^tAy$. You can use the chain rule. –  1015 Feb 23 '13 at 16:10

3 Answers 3

up vote 5 down vote accepted

As a start, things work "as usual": You calculate the difference between $f(x+h)$ and $f(x)$ and check how it depends on $h$, looking for a dominant linear part as $h\to 0$. Here, $f(x+h)=(x+h)^TA(x+h)=x^TAx+ h^TAx+x^TAh+h^TAh=f(x)+2x^TAh+h^TAh$, so $f(x+h)-f(x)=2x^TA\cdot h + h^TAh$. The first summand is linear in $h$ with a factor $2x^TA$, the second summand is quadratic in $h$, i.e. goes to $0$ faster than the first / is negligible against the first for small $h$. So the row vector $2x^TA$ is our derivative (or transposed: $2Ax$ is the derivative with respect to $x^T$).

share|improve this answer
    
Thank you. This is also what I tried. However, what confused me is that the question mentions that you should differentiate with respect to xT. From your answer, I see that you took the transpose of the 'ordinary' derivative. Does that imply that the ordinary derive is always taken with respect to x so that you can just take the transpose when you differentiate with respect to xT? –  dreamer Feb 23 '13 at 16:21

@Hagen von Eitzen's answer is certainly the fastest route here, but since you asked, here is a chain rule.

Here are two useful facts about linear and bilinear bounded maps from normed vectors spaces to normed vector spaces.

If $f$ is linear and bounded, then trivially: $$ df_x(h)=f(h). $$

And if $g$ is bilinear and bounded ($\|g(h,k)\|\leq C\|h\|\|k\|$), we have $$ dg_{(x,y)}(h,k)=g(x,k)+g(h,y). $$

Now take $f(x)=(x,x)$ and $g(x,y)=x^tAy$. The former is linear and bounded, the latter is bilinear and bounded.

So, by the chain rule, $g\circ f(x)=x^tAx$ is differentiable and $$ d(g\circ f)_x(h)=dg_{f(x)}\circ df_x(h)=dg_{(x,x)} (h,h)=x^tAh+h^tAx. $$

This is true for any matrix $A$. Now if $A$ is symmetric, this can be simplified since $$ x^tAh+h^tAx=x^tAh+h^tA^tx=x^tAh+(Ah)^tx=2x^tAh. $$

Removing $h$, this gives $$ d(g\circ f)_x=2x^tA. $$

share|improve this answer
    
Thank you. The other answer is indeed quicker but I am glad that I know now how to do it in this way as well. Much appreciated :). –  dreamer Feb 23 '13 at 17:03
1  
@user48288 You're welcome. And I am sure these general facts about bounded linear and bilinear maps will prove useful sooner or later. –  1015 Feb 23 '13 at 17:04

There is another way to solve the problem:

Let $\mathbf{x}^{n\times 1}=(x_1,\dots ,x_n)'$ be a vector, the derivative of $\mathbf y=f(\mathbf x)$ with respect to the vector $\mathbf{x}$ is defined by $$\frac{\partial f}{\partial \mathbf x}=\begin{pmatrix} \frac{\partial f}{\partial x_1} \\ \vdots\\ \frac{\partial f}{\partial x_n} \end{pmatrix}$$ Let \begin{align} \mathbf y&=f(\mathbf x)\\&=\mathbf x'A\mathbf x \\&=\sum_{i=1}^n\sum_{j=1}^n a_{ij}x_ix_j\\&=\sum_{i=1}^na_{i1}x_ix_1+\sum_{j=1}^na_{1j}x_1x_j+\sum_{i=2}^n\sum_{j=2}^n a_{ij}x_ix_j \\\frac{\partial f}{\partial x_1} &=\sum_{i=1}^na_{i1}x_i+\sum_{j=1}^na_{1j}x_j\\&=\sum_{i=1}^na_{1i}x_i+\sum_{i=1}^na_{1i}x_i \,[\text{since}\,\, a_{1i}=a_{1i}]\\ &=2 \sum_{i=1}^na_{1i}x_i \\ \frac{\partial f}{\partial \mathbf x}&=\begin{pmatrix} 2 \sum_{i=1}^na_{1i}x_i \\ \vdots\\ 2 \sum_{i=1}^na_{ni}x_i \end{pmatrix} \\&=2\begin{pmatrix} a_{11} & a_{12} & \dots & a_{1n}\\ \vdots & \vdots &\ddots & \vdots \\ a_{11} & a_{12} & \dots & a_{1n} \end{pmatrix}\begin{pmatrix}x_1 \\ \vdots \\ x_n \end{pmatrix}\\ &= 2A\mathbf x \end{align}

share|improve this answer
    
Thanks for showing me this way as well :). Gives me more options :) –  dreamer Feb 24 '13 at 10:01
    
The only thing that is slightly unclear to me is how x'Ax becomes the double summation (aijxixj). Why is the order reversed here? I mean, why arent the a's in the middle anymore? @Argha –  dreamer Feb 24 '13 at 10:04
    
Note that $\mathbf x'A\mathbf x=(x_1,\dots ,x_n)\begin{pmatrix} a_{11} & a_{12} & \dots & a_{1n}\\ \vdots & \vdots &\ddots & \vdots \\ a_{11} & a_{12} & \dots & a_{1n} \end{pmatrix}\begin{pmatrix}x_1 \\ \vdots \\ x_n \end{pmatrix}$ and simply multipling we get required result. Also note order of $\mathbf x'$ is $1 \times n$ and order of $A$ is $n \times n$. So order of $\mathbf x'A\mathbf x$ is $1 \times 1$. –  Argha Feb 24 '13 at 11:24
    
Note that $a_{ij}\,x_i\,x_j \equiv x_i\,a_{ij}\,x_j$. So there is no problem at all. –  Argha Feb 24 '13 at 11:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.