Gâteaux derivate of the Tikhonov functional 
Let $X,Y$ be Hilbert spaces, and let $A\colon X\to Y$ be a compact operator. The Tikhonov functional is given by
  $$
F(x)=\lVert Ax-y\rVert_X^2+\alpha\lVert x\rVert_X^2.
$$
  Calculate the Gâteaux derivative of the Tikhonov functional.

Tip: Use $\lVert a+b\rVert=\lVert a\rVert^2+2(a,b)+\lVert b\rVert^2$.

First question: Is it really $\lVert Ax-y\rVert_X^2$ and not the Y-norm?
Second question: What I have to do here is to my opinion calculate
$$
\lim\limits_{t\to 0}\frac{F(x+th)-F(x)}{t}.
$$
So I would start with calculating
$F(x+th)=\lVert A(x+th)-y)\rVert_X^2+\alpha\lVert x+th\rVert_X^2$.
I start with calculating $\lVert A(x+th)-y)_X^2$. I can not read from the text if the operator $A$ is linear, too. Is it right to calculate with the given tip
$$\lVert A(x+th)-y\rVert_X^2=\lVert A(th)+Ax-y\rVert_X^2=\lVert A(th)\rVert_X^2+2(A(th),Ax-y)+\lVert Ax-y\rVert_X^2?$$
 A: Since $Ax$ and $y$ are both in $Y$, the first norm should indeed be the norm in $Y$, not in $X$.
Compact operator is (often, at least) implicitly assumed to be linear and the notation $Ax$ instead of $A(x)$ also suggests linearity.
Let us calculate the derivative of $F$ at $x\in X$.
I will assume that your Hilbert spaces are over the reals; the complex version is similar.
For any $h\in X$ we have
$$
\begin{split}
F(x+h)
&=
\|A(x+h)-y\|_Y^2+\alpha\|x+h\|_X^2
\\&=
\|Ax-y\|_Y^2+\|Ah\|_Y^2-2\langle Ax-y,Ah\rangle_Y+\alpha\|x\|_X^2+\alpha\|h\|_X^2+2\alpha\langle x,h\rangle_X
\\&=
F(x)+2\alpha\langle x,h\rangle_X-2\langle Ax-y,Ah\rangle_Y+\|Ah\|_Y^2+\alpha\|h\|_X^2
\\&=
F(x)+2\alpha\langle x,h\rangle_X-2\langle A^*(Ax-y),h\rangle_X+\|Ah\|_Y^2+\alpha\|h\|_X^2
\\&=
F(x)+2\langle\alpha x-A^*Ax+A^*y,h\rangle_X+\|Ah\|_Y^2+\alpha\|h\|_X^2.
\end{split}
$$
The Gâteaux derivative of $F$ at $x$ in direction $h$ is
$$
\begin{split}
D_GF(x)h
&=
\lim_{t\to0}\frac{F(x+ht)-F(x)}{t}
\\&=
\lim_{t\to0}\left(2\langle\alpha x-A^*Ax+A^*y,h\rangle_X+t\|Ah\|_Y^2+t\alpha\|h\|_X^2\right)
\\&=
2\langle\alpha x-A^*Ax+A^*y,h\rangle_X.
\end{split}
$$
The Gâteaux derivative $D_GF(x)$ is an element of $X^*$ which can be identified with $X$ via the inner product.
After this identification
$$
D_GF(x)
=
2(\alpha x-A^*Ax+A^*y).
$$
In fact, since $A$ is continuous (follows from compactness), the function $F$ is actually Fréchet differentiable and the two derivatives agree.
If $A$ were not linear, the derivative would include a derivative of $A$.
I assume nonlinearity was not intended in your exercise.
