Eigenvectors of this matrix - what's the relation to rotation operator? 
I have found the eigenvalues to be 0, 1 and 2.
The corresponding eigenvectors are: $\frac{1}{\sqrt 2} (1 , -1, 0)$ and $(0, 0, 1)$ and $\frac{1}{\sqrt 2}(1, 1, 0)$.
I found that when $x^2 + 2xy + y^2 + z^2 = 1$ was expressed in $x^T A x$.
So, to diagonalize A, the diagonalizing matrix is made up of columns of eigenvectors which turns out to be the rotation matrix, rotating about z-axis.
Then I found the rotation angle $\theta = 45^o$.
But when i find diagonalized form $x^TA'x = 2y'^2 + z'^2 = 1$. This is strange.
 A: Generally, when you diagonalize $Q: \mathbb{R}^3 \rightarrow \mathbb{R}$ you find
$Q(x',y',z') =  \lambda_1(x')^2+\lambda_2(y')^2+\lambda_3(z')^2$. This is what you found. However, $\lambda_1=0$ so that term appears to be absent.
Geometrically, you have an elliptical cylinder where $x'$ is the axis-coordinate.
Added in response to comment: Consider,
$$ x^2 + 2xy + y^2 + z^2 = 1$$
simplifies to:
$$ (x + y)^2 + z^2 = 1$$
But, we must divide and multiply by two to pick up an orthonormal coordinate change:
$$ 2\left(\frac{x + y}{\sqrt{2}}\right)^2 + z^2 = 1$$
Then, as $y' = \frac{x + y}{\sqrt{2}}$ and $z=z'$ we obtain your formula. On the other hand, $x' =\frac{x - y}{\sqrt{2}}$ does not appear nontrivially in the quadratic form.
I can get into the matrix of the coordinate change if need be, I'm not sure what more to say here, the $\lambda=0,2$ e-vectors are in the $xy$-plane whereas $\lambda=1$ corresponds to the $z$-axis which serves as the rotation axis for the coordinate change in question. Furthermore, if you look at your first and third e-vectors it can be seen those are a rotation by $45^o$ in the $xy$-plane.
