Disclaimer: I don't know the book you are reading, but I know something about differentiation on $\Bbb{R}^n$ (or more generally on Banach spaces).
Here, it is important to say that the derivative of $f$ at some fixed point $x \in A$ is a linear map, written $Df(x) : \Bbb{R}^p \to \Bbb{R}^q$, so that the value of the linear map $Df(x)$ at $v \in \Bbb{R}^p$ is $Df(x)(v) \in \Bbb{R}^q$. This also answers part (2) of your question. Personally, I prefer to write this as $Df(x)\langle v \rangle$.
Here, I have written $\Vert \cdot \Vert$ and $|\cdot|$ to distinguish the different norms (one on $\Bbb{R}^q$, one on $\Bbb{R}^p$).
For the exact definition of this map, $Df(x)$ is the unique linear map so that
$$
\lim_{y \to x} \frac{\Vert f(y) - f(x) - Df(x) \langle y-x \rangle \Vert}{|x-y|} = 0,
$$
if such a map exists (then it is unique). This means that the map $y \mapsto f(y) + Df(x)\langle y -x \rangle$ is the best (affine-) linear approximation to $f$ at $x$.
- See (1)
- This is normally called the directional derivative and not the partial derivative w.r.t. an arbitrary vector. The directional derivative $D_v f (x)$ of $f$ in direction $v$ at $x$ is usually defined as
$$
D_v f(x) := \lim_{t \to 0} \frac{f(x + tv) - f(x)}{t}.
$$
You should check that $D_v f(x) = Df(x)\langle v \rangle$ holds, if $Df(x)$ exists, i.e. if $f$ is differentiable at $x$.
Let us plug in the definition and calculate. \begin{eqnarray*}
D_{\left(a,b\right)}f\left(0,0\right) & = & \lim_{t\to0}\frac{f\left(\left(0,0\right)+t\left(a,b\right)\right)-f\left(\left(0,0\right)\right)}{t}\\
& = & \lim_{t\to0}\frac{\left(ta\right)\left(tb\right)^{2}}{\left(ta\right)^{2}+\left(tb\right)^{2}}\\
& = & \lim_{t\to0}\frac{t^{3}}{t^{2}}\frac{ab^{2}}{a^{2}+b^{2}}=0\neq\frac{ab^{2}}{a^{2}+b^{2}}
\end{eqnarray*}
in general, so I think that you (or the book) miswrote the definition of $f$ (check it).
The point of an example like this (replace e.g. the $b^2$ in the nominator by $b$ and repeat the calculation) is to show that it can happen that all partial derivatives exist without the function being differentiable at $x$ (at $0$ in this case). You can even deduce that $f$ is not differentiable, because otherwise, part (3) would show that the directional-derivative map
$$
v \mapsto D_v f(x) = Df(x)\langle v \rangle
$$
is linear, which is not the case in your example.
EDIT: Maybe the following can also help you. In the setting of $\Bbb{R}^p$ and $\Bbb{R}^q$, any linear map $T : \Bbb{R}^p \to \Bbb{R}^q$ can always be identified with its matrix (w.r.t. the standard basis), i.e. we identify the matrix $A \in \Bbb{R}^{q \times p}$ with the linear map
$$
x \mapsto Ax.
$$
With this notation, the linear map $Df(x)$ is identified with the Jacobian matrix containing the partial derivatives, i.e.
$$
Df(x) \leftrightarrow \left(\frac{\partial f_i (x)}{\partial x_j} \right)_{i,j},
$$
where I used the standard notation
$$
\frac{\partial f_i (x)}{\partial x_j} := D_{e_j} f_i (x),
$$
where $e_j$ is the $j$-th vector of the standard basis and $f_i$ is the $i$-th component of $f$.
In fact, sometimes the derivative in this setting is defined as a matrix instead of a linear map, so that $\widetilde{D} f(x)$ (I just invented that notation) is defined as the unique matrix such that
$$
\lim_{y \to x} \frac{\Vert f(y) - f(x) - \widetilde{D}f(x) \cdot (y-x) \Vert}{|y - x|} = 0.
$$
So, why do we bother with the (seemingly) more complicated definition using linear maps instead of plain old matrices? The answer is that this becomes much more convenient in the setting of spaces that are finite-dimensional, but do not look like $\Bbb{R}^n$ in a natural way. For example, consider the determinant map
$$
\det : \Bbb{R}^{n\times n} \to \Bbb{R}.
$$
You could of course identify $\Bbb{R}^{n \times n}$ with $\Bbb{R}^{n^2}$ by introducing some arbitrary enumeration of the coordinates (or even some other basis), but this will make it complicated (not painless) to compute the matrix-derivative.
Instead, it is not so hard to compute the linear-map version of the derivative (at least at the identity matrix $I$). First, note that the determinant is certainly differentiable (because it is a polynomial map), so that it suffices to calculate the directional derivative
$$
D_A \det(I) = \lim_{t \to 0} \frac{\det (I + tA) - \det(I)}{t} = \rm{tr}(A),
$$
where the last equality comes from the fact that the Leibniz formula yields
$$
\det(I + tA) = \sum_\sigma \rm{sgn}(\sigma) \cdot \prod_i (I + tA)_{i, \sigma(i)} = 1 + t \cdot \rm{trace}(A) + t^2 \cdot \ast,
$$
where $\ast$ is a polynomial in $t$, as you can check.
One can also calculate the derivative at other points than at the identity, but I willl skip that here.
