Derivative of inner product If the inner product of some vector $\mathbf{x}$ can be expressed as 
$$\langle \mathbf{x}, \mathbf{x}\rangle_G = \mathbf{x}^T G\mathbf{x}$$
where $G$ is some symmetric matrix, if I want the derivative of this inner product with respect to $\mathbf{x}$, I should get a vector as a result since this is the derivative of a scalar function by a vector (https://en.wikipedia.org/wiki/Matrix_calculus#Scalar-by-vector).
Nevertheless, this formula tells me that I should get a row-vector, and not a normal vector. 
$$\frac{\mathrm{d}}{\mathrm{d} \mathbf{x}} (\mathbf{x}^TG\mathbf{x}) = 2\mathbf{x}^T G$$
(http://www.cs.huji.ac.il/~csip/tirgul3_derivatives.pdf)
which is a row-vector.
Why do I get this contradiction?
 A: The difference is in the fact the author in the second reference prefers to arrange the components of the gradient. In the first paragraph they state

Let $x\in \mathbb{R}^n$ (a column vector) and let $f : \mathbb{R}^n \to R$. The derivative of $f$ with respect to $x$ is a row vector:
  $$
\frac{\partial f}{\partial x} = \left(\frac{\partial f}{\partial x_1}, \cdots , \frac{\partial f}{\partial x_n} \right)
$$

You can argue this is a better option than the first one (e.g. this answer), but at the end of the day is just a matter of notation. Pick the one you prefer and stick with it to avoid problems down the line
A: More generally, suppose we differentiate any scalar-valued function $f$ of a vector $\mathbf{x}$ with respect to $\mathbf{x}$. By the chain rule, $$df=\sum_i\frac{\partial f}{\partial x_i}dx_i=\boldsymbol{\nabla}f\cdot d\mathbf{x}=\boldsymbol{\nabla}f^T d\mathbf{x}.$$(Technically, I should write $df=(\boldsymbol{\nabla}f^T d\mathbf{x})_{11}$ to take the unique entry of a $1\times 1$ matrix.)
If you want to define the derivative of $f$ with respect to $\mathbf{x}$ as the $d\mathbf{x}$ coefficient in $df$, you use the last expression, obtaining the row vector $\boldsymbol{\nabla}f^T$. Defining it instead as the left-hand argument of the dot product, giving the column vector $\boldsymbol{\nabla}f$, is an alternative convention.
A: For a smooth $f:\mathbb{R}^n\to\mathbb{R}^m$, you have $df:\mathbb{R}^n\to\mathcal{L}(\mathbb{R}^n,\mathbb{R}^m)$
Being differentiable is equivalent to:
$$
f(x+h)=f(x)+df(x)\cdot h+o(\|h\|)
$$
In your case, $f(x)=\langle x,x \rangle_G$ and $m=1$, hence differential at $x$, $df(x)$ is in $\mathcal{L}(\mathbb{R}^n,\mathbb{R})$. It's a linear form.
Let's be more explicit:
\begin{align*}
f(x+h)=& \langle x+h,x+h \rangle_G \\
      =& \underbrace{\langle x,x \rangle_G}_{f(x)} + \underbrace{2\langle x,h \rangle_G }_{df(x)\cdot h}+ \underbrace{\langle h,h \rangle_G}_{\in o(\|h\|)}\\
\end{align*}
Hence your differential is defined by 
$$
df(x)\cdot h = 2\langle x,h \rangle_G = (2x^tG)h
$$
where $2x^tG=\left(\partial_{x_1} f,\dots,\partial_{x_n} f\right)$ is your "row" vector.
Note that, because $m=1$, you can also use a vector $\nabla f(x)$ to represent $df(x)$ using the canonical scalar product. This vector is by definition the gradient of $f$:
$$
df(x)\cdot h = \langle \nabla f(x),h \rangle = \langle 2Gx,h \rangle 
$$
where $\nabla f(x)=2Gx=\left(\begin{array}{c}\partial_{x_1} f \\ ... \\\partial_{x_n} f\end{array}\right)$. This is your "column" vector.
A: Why not use the Leibniz-rule?  We have, where $\langle .,.\rangle$ denotes the standard inner product
$$D_p(\langle x,x\rangle_G)=2\langle p,x\rangle_G=2p^TGx=2\langle p,Gx\rangle.$$
Note that the derivative  of $f\colon\mathbb R^n\to\mathbb R$ is not a vector, but a linear form instead.  The gradient $\nabla^{\langle .,.\rangle_G}f$ in respect to the inner product $\langle .,.\rangle_G$  is the unique vector which represents this linear form in presence of the specified inner product.  In our case we have 
$$\nabla^{\langle .,.\rangle_G}f(x)=2x,\quad\text{that is}\quad
D_p(\langle x,x\rangle_G)=\langle p,2x\rangle_G$$
whereas 
$$\nabla^{\langle .,.\rangle}f(x)=2Gx,\quad\text{and that is}\quad
D_p(\langle x,x\rangle_G)=\langle p,2Gx\rangle$$
