how to prove that the formula for the volume center(centroid) is incorrect? Why the following derivation is incorrect?
Because
$$
\frac{1}{2}\nabla\left(\vec{x}\cdot\vec{x}\right)=\vec{x}\cdot\nabla\vec{x}=\vec{x},
$$
the centroid/center $\vec{X^c}$ of the mass of a volume $V$ is by Gauss' Theorem:
$$
 \frac{1}{V}\iiint_V\vec{x}\,dV=\frac{1}{2V}\iiint_V\nabla\left(\vec{x}\cdot\vec{x}\right)dV=\frac{1}{2V}\iint_S\left(\vec{x}\cdot\vec{x}\right)\cdot\vec{n}\,dS
$$
However obviously the correct $\vec{X^c}$ is
$$
X^c_i=\frac{1}{2V}\iint_Sx_i^2n_j\delta_{ij}dS
$$
which is not equal to $$
\frac{1}{2V}\iint_S\left(x_1^2+x_2^2+x_3^2\right)n_idS
$$
EDIT: When revert back, still bewildered by why do we have to apply Gauss' Theorem component-wisely here?
 A: One should notice that the integrand in the following equation contains dirac delta function,
$$X^c_i=\frac{1}{2V}\iint_Sx_i^2n_j\delta_{ij}dS$$
which suggests that the explicit form of the integral will look like:
$$X^c_i=\frac{1}{2V}\iint_S\left(
\begin{array}{ccc}
 x_1^2 & 0 & 0 \\
 0 & x_2^2 & 0 \\
 0 & 0 & x_3^2 \\
\end{array}
\right) \left(
\begin{array}{c}
 n_1 \\
 n_2 \\
 n_3 \\
\end{array}
\right)dS$$
$$X^c_i=\left(
\begin{array}{c}
 X_1^c \\
 X_2^c \\
 X_3^c \\
\end{array}
\right)=\frac{1}{2V}\iint_S\left(
\begin{array}{c}
 n_1 x_1^2 \\
 n_2 x_2^2 \\
 n_3 x_3^2 \\
\end{array}
\right)dS$$
A: Hint:
As noted by @Vlad you cannot use the divergence theorem because in the integral
$$
{2V}\iiint_V\nabla\left(\vec{x}\cdot\vec{x}\right)dV \qquad (1)
$$
you have not a divergence (a scalar) but a gradient (a vector). 
So the integral in $(1)$ is really a vector:
$$
\iiint_V\nabla\left(\vec{x}\cdot\vec{x}\right)dV=\hat x_1 \iiint_V2x_1dV +\hat x_2 \iiint_V2x_2dV+\hat x_3 \iiint_V2x_3dV
$$
Now you can apply the divergence theorem to the three integrals at the right, where ve have a scalar function such that $2x_i=\frac{d}{dx_i}x_i^2$ .... And you find the correct result.(and this is essentially the generalization suggested by @achillehui).
