Find sum of squares of numbers that are smaller than $n$ and coprime to it Find sum of squares of numbers that are smaller than $n$ and coprime to it.The book wrote the answer:
$\frac{\phi (n)}{6}(2n^2+(-1)^m p_1p_2 \dots p_m)$
Where $p_1,p_2,\dots p_m$ are only primes that divide it.It was in the part inclusion-exclusion principle part but I don't understand how inclusion-exclusion principle can solve this?
 A: $$f(n) =\sum_{m \le n} m^2 = \frac{n(n+1)(2n+1)}{6}= \sum_{l=1}^3 c_l n^l, \qquad c_1= \frac16,c_2=\frac12,c_3 = \frac13$$ 
Look at the multiplicative functions $$h_l(n) = \sum_{d | n} \mu(d) d^2 (n/d)^l=\prod_{p^k \| n} h(p^k)= \prod_{p^k \| n} (p^{lk}-p^{2+l(k-1)})$$
$$h_1(n) = n \frac{\phi(n)}{n}(-1)^{\omega(n)} \prod_{p | n} p,\qquad h_2(n) = 0 (n \ne 1), \qquad  h_3(n) = n^3 \frac{\phi(n)}{n}$$
 Therefore
$$\sum_{m \le n, (m,n)=1} m^2 =\sum_{d | n} \sum_{m \le n, d | m} m^2= \sum_{d | n} \mu(d) d^2 f(n/d)= \sum_{l=1}^3 c_l h_l(n)\\= \frac{\phi(n)}{6}(2n^2+(-1)^{\omega(n)} \prod_{p | n} p)$$
A: We suppose  that OP understands the  inclusion-exclusion argument used
to evaluate $\varphi(n).$ Using the very same procedure and
introducing
$$a(n) = \sum_{k=1}^n k^2 =
\frac{1}{3} n^3 + \frac{1}{2} n^2 + \frac{1}{6} n$$
and with $\mathcal{P}$ the set of primes that divide $n$
we thus have for the desired sum
$$\sum_{\mathcal{Q}\subseteq\mathcal{P}}
(-1)^{|\mathcal{Q}|}
a\left(\frac{n}{\prod_{p\in \mathcal{Q}} p}\right)
\prod_{p\in \mathcal{Q}} p^2.$$
Now do  the three terms in  turn starting with the  lowest order term,
which yields
$$\frac{1}{6} n \sum_{\mathcal{Q}\subseteq\mathcal{P}}
(-1)^{|\mathcal{Q}|}
\prod_{p\in \mathcal{Q}} p
= \frac{1}{6} n \prod_{p\in \mathcal{P}} (1-p)
= \frac{1}{6} n \prod_{p\in \mathcal{P}} \left(\frac{1}{p}-1\right)
\prod_{p\in \mathcal{P}} p
\\ = \frac{1}{6}\varphi(n) (-1)^{|\mathcal{P}|}
\prod_{p\in \mathcal{P}} p.$$
The next term is
$$\frac{1}{2} n^2 \sum_{\mathcal{Q}\subseteq\mathcal{P}}
(-1)^{|\mathcal{Q}|} = \frac{1}{2} \prod_{p\in \mathcal{P}} (1-1) = 0.$$
We get for the third and last term
$$\frac{1}{3} n^3 \sum_{\mathcal{Q}\subseteq\mathcal{P}}
(-1)^{|\mathcal{Q}|}
\prod_{p\in \mathcal{Q}} \frac{1}{p}
= \frac{1}{3} n^3 \prod_{p\in \mathcal{P}} \left(1-\frac{1}{p}\right)
= \frac{1}{3} n^2 \varphi(n).$$
Collecting everything we find
$$\bbox[5px,border:2px solid #00A000]{
\frac{1}{6} \varphi(n)
\left(2n^2 + (-1)^{|\mathcal{P}|}
\prod_{p\in \mathcal{P}} p\right).}$$
The following Maple code may be used to explore these statistics.

with(numtheory);
with(combinat);

a := unapply(expand(sum(k^2, k=1..n)), n);

S :=
proc(n)
option remember;
local k, res;
    res := 0;

    for k to n do
        if gcd(n, k) = 1 then
            res := res + k^2;
        fi;
    od;

    res;
end;

S1 :=
proc(n)
local res, P, S, Q;
    res := 0;

    P := factorset(n);
    S := subsets(P);

    while not S[finished] do
        Q := S[nextvalue]();

        res := res +
        (-1)^nops(Q)*a(n/mul(p, p in Q))
        *mul(p^2, p in Q);
    end do;

    res;
end;

S2 :=
proc(n)
local P;
    P := factorset(n);
    1/6*phi(n)*(2*n^2 + (-1)^nops(P)*mul(p, p in P));
end;

