$(n^2 \alpha \bmod 1)$ is equidistributed in $\mathbb{T}^2$ if $\alpha \in \mathbb{R} \setminus \mathbb{Q}$ I did the following homework question, can you tell me if I have it right?
We want to show that the sequence $(n^2 \alpha \bmod 1)$ is equidistributed if $\alpha \in \mathbb{R} \setminus \mathbb{Q}$. To that end we consider the transformation $T: (x,y) \mapsto (x + \alpha, y + 2x + \alpha)$ on the $2$-dimensional torus $\mathbb{T}^2$ endowed with the Lebesgue measure $\lambda \times \lambda$.

a) Show that the action of $T$ on the torus is ergodic, i.e., if a measurable set $A \subset \mathbb{T}^2$ is invariant under $T$, then $(\lambda \times \lambda)(A) \in \{0, 1\}$. Show this by checking the following equivalent definition of ergodicity:
$\forall f \in L^2( \mathbb{T}^2)$, we have: if $f$ is $T$-invariant, then $f$ has to be constant almost everywhere.
Hint: Use Fourier series.
My answer: 
$$ 
\begin{align}
f(x,y) &= \sum_{j,k \in \mathbb{Z}} c_{jk} e^{ijx} e^{iky}
\\ &\stackrel{f = f\circ T}{=} \sum_{j,k \in \mathbb{Z}} c_{jk} e^{ij(x + \alpha)} e^{ik(y + 2x + \alpha)} 
\\ &=
\sum_{j,k \in \mathbb{Z}} c_{jk} e^{ij\alpha + ik\alpha} e^{ijx + ik2x}e^{iky}
\\ &\stackrel{j \rightarrow j-2k}{=}
\sum_{j,k \in \mathbb{Z}} c_{(j-2k)k} e^{i(j-k)\alpha} e^{ijx}e^{iky} .
\end{align}$$
Now we want $c_{jk} = c_{(j-2k)k} e^{i(j-k)\alpha}$, and we have $|c_{jk}| = |c_{(j-2k)k}| = |c_{(j-4k)k}| = \cdots$, and so on.
The series only converges if $|c_{(j-4k)k}| \; \xrightarrow{k \rightarrow \infty} \; 0$ and so $c_{jk}$ has to be $0$, too.

b) For $x \in \mathbb{T}$ show that $$ \frac{1}{m} \sum_{n=1}^m T^n_\ast (\delta_x \times \lambda) \rightarrow \lambda \times \lambda$$ using the equidistribution of $(n \alpha \bmod 1)$.
My answer:
$$ 
\begin{align}
\frac{1}{m} \sum_{n=1}^m T_\ast^n(\delta_x \times \lambda (A \times B)) 
&= 
\frac{1}{m} \sum_{n=1}^m \delta_x \times \lambda ((T^{-1})^n(A \times B)) 
\\ &= 
\frac{1}{m} \sum_{n=1}^m \delta_x \times \lambda (T^n(A \times B)) ,
\end{align}
$$
where I have the last equality because it doesn't matter which way the points are shifted. Then writing out what $T^n$ does I get:
$$ 
\begin{align}
\frac{1}{m} \sum_{n=1}^m \delta_x \times \lambda (T^n(A \times B)) 
&= 
\frac{1}{m} \sum_{n=1}^m \delta_x \times \lambda ( (A + \alpha n) \times (B + \alpha n) ) \\ &=
\frac{1}{m} \sum_{n=1}^m \delta_x (A + \alpha n) \lambda(B + \alpha n)
\\ &=
\frac{1}{m} \sum_{n=1}^m \chi_{A + \alpha n}(x) \lambda(B + \alpha n) .
\end{align}
$$
And then using that the Lebesgue measure $\lambda$ is translation invariant I get:
$$
\begin{align}
\frac{1}{m} \sum_{n=1}^m \chi_{A + \alpha n}(x) \lambda(B + \alpha n) = 
\frac{1}{m} \sum_{n=1}^m \chi_{A + \alpha n}(x) \lambda(B) .
\end{align}
$$
And finally, by using the ergodic theorem:
$$
\begin{align}
\frac{1}{m} \sum_{n=1}^m \chi_{A + \alpha n}(x) \lambda(B)=
\lambda(B) \int_{\mathbb{T}} \chi_{A + \alpha n} (x) d \lambda(x) = \lambda(B)\lambda(A) = \lambda \times \lambda (A \times B) .
\end{align}
$$

c) For $\eta \in (0,1)$ and $x,y \in \mathbb{T}$ define the two sequences
$$ 
\begin{align*}
\mu_m &= 
\frac{1}{m} \sum_{n=1}^m T^n_\ast \left(\delta_x \times \left(\frac{1}{2 \eta} \left. \lambda \right \vert_{[y-\eta, y + \eta]} \right) \right)
\\
\nu_m &= 
\frac{1}{m} \sum_{n=1}^m T^n_\ast \left(\delta_x \times \left( \frac{1}{1 - 2 \eta} \left. \lambda \right\vert_{\mathbb{T} \smallsetminus [y-\eta, y + \eta]} \right) \right)
\end{align*}
$$
Using exercise 3 of assignment 10 and weak$^\ast$-compactness of the unit ball we know that there exists a subsequence in $\mathbb{N}$ such that both sequences converge along these subsequences. Call the limit points $\mu$ and $\nu$ respectively. Show that $2 \eta \mu + (1 - 2 \eta) \nu = \lambda \times \lambda$.
My answer:
Exercies 3 of assignment 10 was to show that any weak$^\ast$ limit point $\mu$ of the sequence $\mu_n = \frac{1}{n}\sum_{j=0}^{n-1} T_\ast^j \nu_n$ is a Borel probability measure with $\mu = T_\ast \mu$.
$$
2 \eta \lim_{m \to \infty} \frac{1}{m} \sum_{n=1}^m T_\ast^n \left(\delta_x \times \left( \frac{1}{2 \eta} \left. \lambda \right\vert_{[y- \eta, y + \eta]} \right) \right) +

(1 - 2 \eta) \lim_{m \to \infty} \frac{1}{m} \sum_{n=1}^m T_\ast^n \left( \delta_x \times \left( \frac{1}{1- 2 \eta} \left.  \lambda \right\vert_{\mathbb{T} \smallsetminus [y- \eta, y + \eta]} \right) \right) ,
$$
which, by part (b), is equal to 
$$
\left. \lambda \times \lambda \right\vert_{[y - \eta , y + \eta]} +  \left. \lambda \times \lambda \right \vert_{\mathbb{T} \smallsetminus [y - \eta , y + \eta]} = \lambda \times \lambda.
$$ 

d) Using the following proposition, show that $\mu = \lambda \times \lambda$.
Proposition: A $T$-invariant probability measure is extremal if and only if its action is ergodic. 
My answer:
Distinguish the cases $\eta \geq \frac{1}{2}$ and $\eta < \frac{1}{2}$.
If  $\eta < \frac{1}{2}$ then by c) $\lambda \times \lambda = 2 \eta \mu + (1 - 2 \eta) \nu$ and since $2 \eta < 1$, by extremality, $\lambda \times \lambda = \mu = \nu$.
If $\eta \geq \frac{1}{2}$ then $[y - \eta , y + \eta] = \mathbb{T}$ and so
$$ \begin{align}
\mu(A \times B) &= \lim_{m \to \infty} \mu_m (A \times B) 
\\ &= \lim_{m \to \infty} \frac{1}{m} \frac{1}{2 \eta} \sum_{n=1}^m \chi_A(x + \alpha n) \lambda \mid_{[y - \eta , y + \eta]} (B) \\
&\stackrel{b)}{=} \frac{1}{ 2 \eta} \lambda \times \lambda
\end{align}$$
So I think I got the sums wrong here.

e) Show that $\mu_m \to \lambda \times \lambda$. To that end prove and apply the following:
Lemma: Let $X$ be a metric space and $x \in X$, $x_n$ a sequence in $X$. Assume that each subsequence of $x_n$ has a subsequence converging to $x$. Then $x_n$ itself converges to $x$.
My answer:
Assume that $x_n$ converges to $y \neq x$. Then there is a (sub)sequence (the sequence itself is a subsequence) not converging to $x$. Contradiction. Same argument for $x_n$ diverges.
Now with c) and d), $\mu_m \to \mu = \lambda \times \lambda$.

I'm stuck on:
f) Show that for all $f \in C(\mathbb{T}^2)$ and for all $\varepsilon > 0$, there exists $\eta > 0$ such that we have 
$$  \left\vert \int f d \mu_m - \int f d \omega_m  \right\vert < \varepsilon ,$$
where $$ \omega_m = \frac{1}{m} \sum_{n=1}^m T^n_\ast (\delta_x \times \delta_y) .$$
Thanks for your help!
 A: Obviously this answer is very late, but the argument sketched in this exercise is so pretty it really deserves an answer. One thing that we will use on a couple of occasions is that $T^n(x,y) = (x + n\alpha, y + 2nx + n^2\alpha)$, which is easily verifiable by induction.
(a) Your answer for this part is close to right. Since you are working mod 1, you should use exponentials like $e^{2\pi ijx}$ instead of $e^{ijx}$ when doing your Fourier analysis. Nonetheless, you correctly derive that if $f\circ T = f$, then $c_{j,k} = c_{j-2k,k}e^{2\pi i(j-k)\alpha}$. If $k\neq 0$, then you also correctly note that $|c_{j,k}| = |c_{j-2nk,k}|$ for all $n$, and hence in order to get convergence of the Fourier series you need that $c_{j,k} = 0$. This argument fails when $k = 0$, however. If $k = 0$, we have derived that $c_{j,0} = c_{j,0}e^{2\pi ij\alpha}$. If $j\neq 0$, then $e^{2\pi ij\alpha}\neq 1$, and thus one must have $c_{j,0} = 0$. This proves that the only possible nonzero Fourier coefficient for $f$ is $c_{0,0}$, and hence that $f$ is constant.
(b) Your argument doesn't make use of the equidistribution of $n\alpha$ mod 1, so I think it has some problems. A simpler argument is as follows. Let $f\in C(\mathbb{T}^2)$, and let $g\in C(\mathbb{T})$ be the function $$g(x) = \int_{\mathbb{T}} f(x,y)\,d\lambda(y).$$ Then $$\frac{1}{m}\sum_{n=1}^m\int_{\mathbb{T}^2}f\circ T^n\,d(\delta_x\times \lambda) = \frac{1}{m}\sum_{n=1}^m\int_\mathbb{T}f(x+n\alpha,y + 2nx + n^2\alpha)\,d\lambda(y) = \frac{1}{m}\sum_{n=1}^mg(x + n\alpha).$$ Here the last equality uses the fact that $\lambda$ is translation invariant on $\mathbb{T}$. Now by equidistribution of $n\alpha$ mod 1, we get that $$\frac{1}{m}\sum_{n=1}^m\int_{\mathbb{T}^2} f\circ T^n\,d(\delta_x\times \lambda) = \frac{1}{m}\sum_{n=1}^mg(x + n\alpha)\to \int_\mathbb{T}g\,d\lambda = \int_{\mathbb{T}^2} f\,d(\lambda\times\lambda).$$ This limit is precisely what it means for $m^{-1}\sum_{n=1}^mT^n_*(\delta_x\times\lambda)\to \lambda\times\lambda$.
(c) Pretty sure you need $\eta\in (0,1/2)$ here, since otherwise $\mu_m$ and $\nu_m$ aren't probability measures. From now on I assume $\eta\in (0,1/2)$. By weak* compactness, there is a subsequence $m_k$ such that $\mu_{m_k}$ converges to some $\mu$. Again by compactness, we can pass to a subsequence $m_{k_l}$ of $m_k$ to get that $\nu_{m_{k_l}}$ converges to some $\nu$. From (b), we know that $2\eta\mu_{m_{k_l}} + (1 - 2\eta)\nu_{m_{k_l}}\to \lambda\times\lambda$. On the other hand, $2\eta\mu_{m_{k_l}} + (1 - 2\eta)\nu_{m_{k_l}}\to 2\eta\mu + (1-2\eta)\nu$, so $\lambda\times\lambda = 2\eta\mu + (1-2\eta)\nu$.
(d) $\lambda\times \lambda = 2\eta\mu + (1-2\eta)\nu$ is a decomposition of $\lambda\times\lambda$ into a convex combination of two measures $\mu$ and $\nu$. Since $\lambda\times\lambda$ is ergodic (part a), it is extremal, and hence this can only happen if $\mu = \nu = \lambda\times\lambda$.
(e) Note that $\mu$ was any arbitrary limit point of the sequence $\mu_m$. Thus $\lambda\times\lambda$ is the only limit point of $\mu_m$. The lemma then says $\mu_m\to \lambda\times \lambda$.
(f) Equip $\mathbb{T}$ with the flat metric (the metric inherited from $\mathbb{R}$), and $\mathbb{T}^2$ with the product metric $d$. You are given $f$ and $\epsilon$. Since $\mathbb{T}^2$ is compact, $f$ is uniformly continuous, and hence there is an $\eta>0$ (with $\eta<1/2$) such that if $(x,y),(x',y')\in \mathbb{T}^2$ are such that $d((x,y),(x',y'))\leq\eta$, then $|f(x,y) - f(x',y')|<\epsilon$. The triangle inequality (and unravelling definitions) give $$\left|\int_{\mathbb{T}^2} f\,d\mu_m - \int_{\mathbb{T}^2}f\,d\omega_m\right|\leq \frac{1}{m}\sum_{n=1}^m\frac{1}{2\eta}\left|\int_{\mathbb{T}} f\circ T^n(x,z) - f\circ T^n(x,y)\,d\lambda|_{[y-\eta,y+\eta]}(z)\right|.$$ Because $T^n(x,z) = (x + n\alpha, z + 2nx + n^2\alpha)$ and $T^n(x,y) = (x + n\alpha, y + 2nx + n^2\alpha)$, we see that if $z\in [y-\eta,y + \eta]$, then $d(T^n(x,z), T^n(x,y))\leq\eta$. In particular, if $z\in [y-\eta,y + \eta]$, then $|f\circ T^n(x,z) - f\circ T^n(x,y)|<\epsilon$. This shows that each of the terms in the above some is bounded above by $\epsilon$, and hence that the entire right hand side is bounded above by $\epsilon$. This finishes (f).
To complete the argument, we note that (f) shows that $\omega_m\to \lambda\times\lambda$ for any fixed $x,y\in \mathbb{T}$. Applying this to $x = y = 0$ will give the desired equidistribution. Indeed, let $g\in C(\mathbb{T})$, and set $f\in C(\mathbb{T}^2)$ to be $f(x,y) = g(y)$. The fact that $\omega_m\to \lambda\times\lambda$ says precisely that $$\frac{1}{m}\sum_{n=1}^m g(n^2\alpha) = \frac{1}{m}\sum_{n=1}^m f(n\alpha, n^2\alpha)\to \int_{\mathbb{T}^2}f\,d(\lambda\times\lambda) = \int_\mathbb{T}g\,d\lambda.$$
