Quadratic variation is the right variation for continuous martingales of unbounded variation Notation: $M_2^c$ are continuous square integrables martingales, $\langle X\rangle_t$ is the quadratic variation and $V_t^{(2)}(\Pi)$ the sum of square of increments of $X$ over the partition $\Pi$.
In the book Brownian motion and stochastic calculus by Karatzas and Shreves, 3 results are used to motivate the study of quadratic variation.

Theorem 5.8. Let $X$ be in $M_2^c$. For partition $\Pi$ of $[0,t]$, we have $\lim_{\|\Pi\|\to0} V_t^{(2)}(\Pi)=\langle X\rangle_t$ (in probability); i.e., for every $\varepsilon>0$,$\eta>0$
there exists $\delta>0$ such that $\|\Pi\|<\delta$ implies
$$P[|V_t^{(2)}(\Pi)-\langle X\rangle_t|>\varepsilon]<\eta.$$
5.11 Problem. Let $\{X_t,\mathcal{F}_t;0\le t<\infty\}$ be a continuous process with the property that for each fixed $t>0$ and for
some $p>0$, $$\lim\limits_{\|\Pi\|\to0}V_t^{(p)}(\Pi)=L_t\;\;\;\text{(in probability),}$$ where $L_t$ is a random variable taking
values in $[0,\infty)$ a.s. Show that for $q>p$, $\lim_{\|\Pi\|\to0}V_t^{(q)}(\Pi)=0$ (in probability), and for $0<q<p,
\lim_{\|\Pi\|\to0}V_t^{(p)}(\Pi)=\infty$ (in probability) on the event
$\{L_t>0\}$.
5.12 Problem. Let $X$ be in $M_2^c$, and $T$ be a stopping time of $\{\mathcal{F}_t\}$. If $\langle X\rangle_T=0$, a.s. $P$, then we have
$P[X_{T\wedge t}=0;\forall 0\le t<\infty]=1$.
The conclusion to be drawn from Theorem 5.8 and Problems 5.11 and 5.12
is that for continuous, square-integrable martingales, quadratic
variation is the "right" variation to study. All variations of higher
order are zero, and, except in trivial cases where the martingale is
a.s. constant on an initial interval, all variations of lower order
are infinite with positive probability.
Thus, the sample paths of continuous, square-integrable martingales are quite
different from "ordinary" continuous functions. Being of unbounded first
variation, they cannot be differentiable, nor is it possible to define integrals of
the form Ito $\int_0^t Y_s(\omega) dX_s(\omega)$ with respect to $X\in M_2^c$, in a pathwise (i.e., for every
or P-almost every $\omega\in\Omega$), Lebesgue-Stieltjes sense.

First, I don't understand how problem 5.12 comes into play and how it is used to argue that the quadratic variation is "right" variation to study (see last quoted paragraph).
Second, I don't understand how they can deduce from this discussion that we cannot define a pathwise integral with respect to martingales of unbounded variations.
 A: 
First, I don't understand how problem 5.12 comes into play and how it is used to argue that the quadratic variation is "right" variation to study.

What this is saying is that a martingale is either constant or it has non-vanishing (and locally finite) quadratic variation. This highlights the fact that the quadratic order is the right order at which to study the variation of martingales.

Second, I don't understand how they can deduce from this discussion that we cannot define a pathwise integral with respect to martingales of unbounded variations.

The Lebesgue-Stieltjes theory allows you to define integrals where your integrators have bounded variation, as functions of bounded variation are the difference of two increasing functions (which in turn can be associated with measures). If you wish to define a stochastic integral with a sufficiently rich class of integrands, a pathwise integral will fail for the following reason:
Theorem: Suppose $G:[0,T] \to \mathbb{R}$ is a function for which the Riemann-Stieltjes integral $\int_0^T f dG$ exists for all continuous functions integrands $f$. Then $G$ is of bounded variation.
Proof. This is a consequence of the Banach-Steinhaus theorem. $\blacksquare$
A pathwise stochastic calculus does exist (e.g. see here for some references); it certainly is more elementary than Itô calculus, but the class of integrands is too restrictive for some applications.
