7
$\begingroup$

If the inner product of some vector $\mathbf{x}$ can be expressed as

$$\langle \mathbf{x}, \mathbf{x}\rangle_G = \mathbf{x}^T G\mathbf{x}$$

where $G$ is some symmetric matrix, if I want the derivative of this inner product with respect to $\mathbf{x}$, I should get a vector as a result since this is the derivative of a scalar function by a vector (https://en.wikipedia.org/wiki/Matrix_calculus#Scalar-by-vector).

Nevertheless, this formula tells me that I should get a row-vector, and not a normal vector.

$$\frac{\mathrm{d}}{\mathrm{d} \mathbf{x}} (\mathbf{x}^TG\mathbf{x}) = 2\mathbf{x}^T G$$

(http://www.cs.huji.ac.il/~csip/tirgul3_derivatives.pdf) which is a row-vector.

Why do I get this contradiction?

$\endgroup$

4 Answers 4

13
$\begingroup$

For a smooth $f:\mathbb{R}^n\to\mathbb{R}^m$, you have $df:\mathbb{R}^n\to\mathcal{L}(\mathbb{R}^n,\mathbb{R}^m)$

Being differentiable is equivalent to: $$ f(x+h)=f(x)+df(x)\cdot h+o(\|h\|) $$

In your case, $f(x)=\langle x,x \rangle_G$ and $m=1$, hence differential at $x$, $df(x)$ is in $\mathcal{L}(\mathbb{R}^n,\mathbb{R})$. It's a linear form.

Let's be more explicit: \begin{align*} f(x+h)=& \langle x+h,x+h \rangle_G \\ =& \underbrace{\langle x,x \rangle_G}_{f(x)} + \underbrace{2\langle x,h \rangle_G }_{df(x)\cdot h}+ \underbrace{\langle h,h \rangle_G}_{\in o(\|h\|)}\\ \end{align*}

Hence your differential is defined by $$ df(x)\cdot h = 2\langle x,h \rangle_G = (2x^tG)h $$ where $2x^tG=\left(\partial_{x_1} f,\dots,\partial_{x_n} f\right)$ is your "row" vector.

Note that, because $m=1$, you can also use a vector $\nabla f(x)$ to represent $df(x)$ using the canonical scalar product. This vector is by definition the gradient of $f$:

$$ df(x)\cdot h = \langle \nabla f(x),h \rangle = \langle 2Gx,h \rangle $$ where $\nabla f(x)=2Gx=\left(\begin{array}{c}\partial_{x_1} f \\ ... \\\partial_{x_n} f\end{array}\right)$. This is your "column" vector.

$\endgroup$
5
$\begingroup$

The difference is in the fact the author in the second reference prefers to arrange the components of the gradient. In the first paragraph they state

Let $x\in \mathbb{R}^n$ (a column vector) and let $f : \mathbb{R}^n \to R$. The derivative of $f$ with respect to $x$ is a row vector: $$ \frac{\partial f}{\partial x} = \left(\frac{\partial f}{\partial x_1}, \cdots , \frac{\partial f}{\partial x_n} \right) $$

You can argue this is a better option than the first one (e.g. this answer), but at the end of the day is just a matter of notation. Pick the one you prefer and stick with it to avoid problems down the line

$\endgroup$
4
$\begingroup$

More generally, suppose we differentiate any scalar-valued function $f$ of a vector $\mathbf{x}$ with respect to $\mathbf{x}$. By the chain rule, $$df=\sum_i\frac{\partial f}{\partial x_i}dx_i=\boldsymbol{\nabla}f\cdot d\mathbf{x}=\boldsymbol{\nabla}f^T d\mathbf{x}.$$(Technically, I should write $df=(\boldsymbol{\nabla}f^T d\mathbf{x})_{11}$ to take the unique entry of a $1\times 1$ matrix.)

If you want to define the derivative of $f$ with respect to $\mathbf{x}$ as the $d\mathbf{x}$ coefficient in $df$, you use the last expression, obtaining the row vector $\boldsymbol{\nabla}f^T$. Defining it instead as the left-hand argument of the dot product, giving the column vector $\boldsymbol{\nabla}f$, is an alternative convention.

$\endgroup$
1
$\begingroup$

Why not use the Leibniz-rule? We have, where $\langle .,.\rangle$ denotes the standard inner product $$D_p(\langle x,x\rangle_G)=2\langle p,x\rangle_G=2p^TGx=2\langle p,Gx\rangle.$$

Note that the derivative of $f\colon\mathbb R^n\to\mathbb R$ is not a vector, but a linear form instead. The gradient $\nabla^{\langle .,.\rangle_G}f$ in respect to the inner product $\langle .,.\rangle_G$ is the unique vector which represents this linear form in presence of the specified inner product. In our case we have $$\nabla^{\langle .,.\rangle_G}f(x)=2x,\quad\text{that is}\quad D_p(\langle x,x\rangle_G)=\langle p,2x\rangle_G$$ whereas $$\nabla^{\langle .,.\rangle}f(x)=2Gx,\quad\text{and that is}\quad D_p(\langle x,x\rangle_G)=\langle p,2Gx\rangle$$

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .