derivative of $f(x)=(x^⊤x)^a$ What book shall I read to be able to solve this problem? I don't need to solve this particular problem, I want to know where I can learn to solve such problems. It seems like usual multivariate calculus, but in Stewart book which I studied for multivariate calculus I have never seen such notation. Please recommend some book, thank you in advance for any help.
 A: 
Recommended books
You would need some background on Linear Algebra, which i'd recommend: "Introduction to Linear Algebra, by Gilbert Strang". Furthermore, you'd need to get familiar with matrix calculus, in which i'd recommend "Generalized Vectorization, Cross-Products, and Matrix Calculus, by Darrell A. Turkington".

Solution of your title
You know that $$g(x) = x^T x = \sum_{i=1}^n x_i^2$$
So the derivative of 
$f(x) = g^a(x) $
wrt $x_k$ is 
$$\frac{\partial}{\partial x_k} f(x) = \frac{1}{a} g^{a-1}(x) \frac{\partial}{\partial x_k} g(x)= \frac{1}{a} g^{a-1}(x) (0 +\ldots + 0+2x_k + 0 + \ldots + 0), \quad 1 \leq k \leq n$$
so
$$\frac{\partial}{\partial x_k} f(x) = \frac{2x_k}{a} (x^T x)^{a-1} , \quad 1 \leq k \leq n$$
A: If you know multivariate calculus, then the only thing you might possibly be confused about is the inner product $x^\top x$, sometimes written as $\langle x, x \rangle$ or $x \cdot x$. This ought to be in Stewart somewhere (check the index for "inner product" or "dot product"), but if not, you can find it in linear algebra texts.
For vectors $x$ and $y$, the $x^\top y$ is defined as $x_1 y_1 + x_2 y_2 + \cdots + x_n y_n$. So $f(x) = \left(\sum_{i=1}^n x_i^2\right)^a$ and you can use your standard multivariable calculus techniques to answer the question.
