Measurable subset of unit circle invariant under translation by infinitely many points.

I'm working through Katznelson's An Introduction to Harmonic Analysis. Currently, I'm looking at an exercise in the first chapter: Prove that if $E$ is a measurable set on $\mathbb{T}=\mathbb{R}/2\pi\mathbb{Z}$ and $E$ is invariant under translation by infinitely many $\tau\in\mathbb{T}$, then either $E$ or its complement has measure zero. A similar question, but doesn't really clear my confusion. My attempt:

Denote by $G$ the subgroup of elements of $\mathbb{T}$ under which $E$ is invariant, that is, such that $E+\tau=E$. By a previous exercise, $G$ must be dense in $\mathbb{T}$ as it is infinite. Now if $\mu(E)=0$, then there is nothing to show, so assume $\mu(E)>0$. The Lebesgue density theorem implies that $E$ has a point of density, say $t$. I believe it suffices to show that every point of $\mathbb{T}$ is a point of density of $E$, as then $E^c$ would have no points of density and hence have measure zero.

Fixing $\tau\in\mathbb{T}$, we want to show $$\lim_{r\rightarrow0}\frac{\mu(E\cap(\tau-r,\tau+r))}{2r}=1.$$ At this point, I want to use the density of $G$ to translate $t$ to $\tau$. As $G$ is dense, there exists a sequence $\{t_n\}$ in $G$ approaching $\tau-t$. For each $t_n$ and $r>0$, we know that $$\mu(E\cap(t+r,t-r))=\mu(E\cap(t+t_n-r,t+t_n+r))$$ since $E \pm t_n=E$. This shifts the point of density of $E$ arbitrarily close to $\tau$, but I am not sure how to conclude that $\tau$ itself has metric density 1. Thanks!

-
Is it obvious that a point cannot be a density point for $E$ and $E^c$ simultaneously ? – Epsilon Aug 19 '15 at 23:11

Hint: the difference between $(t+t_n - r, t + t_n+r)$ and $(\tau - r, \tau + r)$ has measure at most $2|\tau - t - t_n|$. In particular this can be made less than $\epsilon r$.
Oh, by difference you mean symmetric difference? Then, the difference between $E\cap(t+t_n-r,t+t_n+r)$ and $E\cap(\tau-r,\tau+r)$ also has measure less than $\epsilon r$. Thus the desired limit holds. – dls Jun 29 '12 at 17:54