How do I find the minimum of this function? This might seem trivial to some of you, but I can't for the life of me figure out how to solve this.
$$\underset{x}\arg \min  (x - b)^T Ax$$
$$x \in \mathbb{R^n}$$
We may assume A to be invertable, but it is not symmetric.
My idea was to calculate the first and second derivative.
I know that $\frac{dx^T}{dx} = (\frac{dx}{dx})^T$, but when I try to apply the chain rule, I get 
$$\frac{d}{dx} = Ax + (x-b)^Tx$$
which doesn't make sense, as it's a vector plus a scalar.
Even if there is another way to find the x for which the function is minimal, I am now more interested in how to derive this kind of formula.
 A: $$f(x) = (x-b)^TAx = x^TAx - b^TAx$$
$$\frac{\partial}{\partial x}f(x) = (A+A^T)x - A^Tb = 0$$
that is the minimizer should satisfy 
$$(A+A^T)x^* = A^Tb $$
If $A$ is invertible then
$$x^* = (A+A^T)^{-1}A^Tb = \big(A^{-T}(A+A^T)\big)^{-1}b = (I + A^{-T}A)^{-1} b$$
A: As mentioned in the comments, you need to take the gradient, not the derivative. 
I usually get confused when trying to take the gradient of a function written in vector/matrix form (as opposed to coordinate form), so I use the following method. Let $f(x)=(x-b)^\top Ax$,  and consider $f(x+\epsilon)-f(x)$ for a small vector $\epsilon$. The result is
$$
\epsilon^\top Ax + x^\top A\epsilon+\require{cancel}\cancel{\epsilon^\top A\epsilon} - \epsilon^\top Ab=\epsilon^\top (A(x-b)+A^\top x)
$$
Note that the $\epsilon^\top A\epsilon$ is canceled because it goes to zero quadratically quickly as $|\epsilon|\to 0$, whereas the other terms converge to $0$ linearly. 
Since $f(x+\epsilon)-f(x)\approx \epsilon^T(A(x-b)+A^\top x)$, the gradient is $A(x-b)+A^\top x$. Setting this equal to zero, you get $$x=(A+A^\top)^{-1}Ab.$$
To determine if this is indeed a minimum, you need to look at the second derivative. This works out to be $A+A^\top$, which will be a minimum as long as this is positive definite.
A: You can rewrite it to a standard quadratic program and use corresponding methods as follows:
$(x-b)^T A x = x^T A x - b^T A x = x^T A x - c^T x$
for $c := A^T b$.
Your method can work too but your derivative calculation was wrong, it would be:
$ \frac{d}{dx} (x-b)^T A x = (x-b)^T A + x^T A = 0 \Leftrightarrow 2A^T x = A^Tb $
A: To differentiate this sort of functions, it is easier to develop them as sums:
Let
$$
f(x) = Ax + (x - b)^T x = \sum_{i=1}^{n} \sum_{j=1}^{n} a_{ij} \, (x_i - b_i) \, x_j,
$$
where $a_{ij}, 1 \leq i, j \leq n$ are the elements of $A$ 
and $b_i$, $1 \leq i \leq n$ are the elements of $b$.
Differentiating w.r.t. $x_k$,
\begin{align}
    \frac{\partial f}{\partial x_k} &= \sum_{i=1}^{n} \sum_{j=1}^{n} a_{ij} \, \left( \delta_{ik} \, x_j  + (x_i - b_i) \, \delta_{jk} \right), \\
                                    &= \sum_{j=1}^{n} \left( a_{kj}\, x_j \right) + \sum_{i=1}^{n} \left( a_{ik} \, (x_i - b_i) \right),
\end{align}
so if you gather the derivatives in a vector $(\nabla f)_k = \partial_{x_k} f$, 
\begin{align}
    \nabla f = Ax + A^T(x - b)
\end{align}
The gradient is 0 when
$$
(A + A^T) \, x = A^T \, b
$$
