Correct method to differentiate a first order vector/matrix

After asking a few questions earlier, I think I've been able to describe my main issue. Could someone show me why this identity is true:

$$\frac{\partial \textbf{x}^{T} B \textbf{x}}{\partial \textbf{x}} = (B + B^{T})\textbf{x}$$

I've gone through things like Matrixcookbook which seem to just show this identity without any derivation of it. I have tried to unsuccessfully derive the formula below:

Via product rule:

$\frac{\partial \textbf{x}^{T} B \textbf{x}}{\partial \textbf{x}} = \frac{\partial \textbf{x}^{T}}{\partial \textbf{x}}B\textbf{x} + \textbf{x}^{T}\frac{\partial B \textbf{x}}{\partial \textbf{x}}$

I realize product rule for scalars is not the same for matrices, but it still holds as long as order is preserved.

At this point, I'm stuck because I'm not sure if $\frac{\partial \textbf{x}^{T}}{\partial \textbf{x}}$ is identity $I$.

I assume this definition of the Jacobian:

$\begin{bmatrix} \dfrac{\partial y_1}{\partial x_1} &; \cdots &; \dfrac{\partial y_1}{\partial x_n} \\ \vdots &; \ddots &; \vdots \\ \dfrac{\partial y_m}{\partial x_1} &; \cdots &; \dfrac{\partial y_m}{\partial x_n} \end{bmatrix}$

And if I write $\frac{\partial B \textbf{x}}{\partial \textbf{x}}$ component by component with the definition above, it becomes simply $B$.

Thus,

$\frac{\partial \textbf{x}^{T} B \textbf{x}}{\partial \textbf{x}} = B\textbf{x} + \textbf{x}^{T}B$

which is not correct, the left side is a nx1 vector and the right side is a nx1 vector.

I'm not sure where my faulty assumption is. I've gone through various sources which don't explain the situation I showed above. Am I missing some very simple point that those books assume?

-

Here is a somewhat simplified view of things to get you started (one can avoid all of the un-necessary elementwise notation down there, but it seems that to help you start, it might still prove to be useful):

Suppose $x \in \mathbb{R}^n$. Further assume the convention that $x$ is a column vector so that

$$x = \begin{bmatrix} x_1\\\\ x_2\\\\ \vdots\\\\ x_n \end{bmatrix}$$

Now suppose that we have some function $f: \mathbb{R}^n \to \mathbb{R}$ (i.e., a scalar valued function of the vector $x$). We define the partial derivatives of $f(x)$ as the vector $$\frac{\partial f(x)}{\partial x} = \begin{bmatrix} \frac{\partial f(x)}{\partial x_1}\\\\ \frac{\partial f(x)}{\partial x_2}\\\\ \vdots\\\\ \frac{\partial f(x)}{\partial x_n} \end{bmatrix}$$

Now let us look at your function $f(x) = x^TBx$. To simplify things for our brute-force attack, we rewrite this is: $$f(x) = \sum_{ij} x_iB_{ij}x_j$$

Now let us try to compute the partial derivative wrt the $p$-th coordinate of $x$, i.e., $\partial f(x) / \partial x_p$. This is given by $$\frac{\partial f(x)}{\partial x_p} = \frac{\partial}{\partial x_p}\sum_{ij} x_i B_{ij} x_j$$

$$= \sum_{j} B_{pj}x_j + \sum_i x_iB_{ip}$$ using the product rule for differentiation. But recognize that $$\sum_i B_{ip}x_i = \sum_i B^T_{pi}x_i = (B^Tx)_p$$ while $$\sum_j B_{pj}x_j = (Bx)_p$$

Thus, we conclude that $$\frac{\partial f(x)}{\partial x_p} = (Bx)_p + (B^Tx)_p$$

Collecting derivatives for $p=1,\ldots,n$ yields the desired identity.

Now you can follow this approach to obtain derivatives for other functions.

Another point that might help you is this: If $f$ is a function as defined above, i.e., $f$ is a function of a column-vector $x$, then $\partial f / \partial x$ is a column-vector, while $\partial f / \partial x^T$ is a row-vector

Please have a look at the matrix-calculus book by Magnus that I have cited in this answer:here

-
Thank you very much, this was the explanation I needed. I guess my biggest misstep was assuming a static definition of differentiation with the Jacobian. – Christopher Dorian Nov 17 '10 at 22:07

$\frac{\partial}{\partial x}$ is just the gradient.

$xBx$ is scalar, namely $\sum_i x_i \sum_j B_{ij} x_j = \sum_i \sum_j B_{ij} x_i x_j$. This can be simplified to

$sum_i B_{ii} x_i^2 + \sum_{i<j} x_i x_j ( B_{ij} + B_{ji})$.

For any index $i$, the derivative with respect to $i$ becomes.

$2 B_{ii} x_i + \sum_{j \neq i} = x_j ( B_{ij} + B_{ji})$.

But for any $i$, the above term is just the i-th component of the vector

$(B + B^T)x$

-

You're right about suspecting that $\frac{ \partial \mathbf x^T }{\partial \mathbf x} \neq I$ because the dimensions no longer match up. I think you might be wading into tensor territory with such operations.

-