Rotation matrix from given axis Let $T\colon\mathbb{R^3}\to\mathbb{R^3}$ denotes linear transformation which rotates by $\frac{\pi}{3}$ counter-clockwise along the vector $u=(1,1,1)$.
If $T(0,1,0)=(a,b,c)$, Find $3a^2+b^2+c^2$.

My Attempt
Consider a plane which contains a point $(0,1,0)$ and uses $u$ as normal vector.
Then I got : $x+y+z=1$.
Since $T$ is rotation, $T(0,1,0)=(a,b,c)$ must be on the same plane $x+y+z=1$.
This means $a+b+c=1$.
Again, $T$ preserves norm of vector : $1=|(0,1,0)|=|T(a,b,c)|=a^2+b^2+c^2$.
So, I got two equations :
$$a+b+c=1 \\ a^2+b^2+c^2=1$$
I want to Find $a, b, c$ without finding exact form of $T$ but I need one more equation about $a,b,c$ to solve this system.
Is there any property of rotation which can give me one more equation about $a,b,c$?
 A: The Rodrigues' Rotation Matrix Formula states that
$R = {a a}^T + (I - {a a}^T) \cos \theta + S_a \sin \theta $
where $a$ is the unit vector of the axis, and
$ S_a = \begin{bmatrix} 0 && - a_z && a_y \\ a_z && 0 && -a_x \\ -a_y && a_x && 0 \end{bmatrix} $
Multiply this matrix by $e_2 = [0, 1, 0]^T $ to get
$[a, b, c]^T = {a a}^T e_2 + (I - {a a}^T) e_2 \cos \theta + S_a e_2 \sin \theta$
The rest is just computation, we have
$ {a a}^T = \dfrac{1}{3} \begin{bmatrix} 1 && 1 && 1 \\ 1 && 1 && 1 \\ 1 && 1 && 1 \end{bmatrix}$
$ I - {a a}^T = \dfrac{1}{3} \begin{bmatrix} 2 && -1 && -1 \\ -1 && 2 && -1 \\ -1 && -1 && 2 \end{bmatrix} $
$S_a = \dfrac{1}{\sqrt{3}} \begin{bmatrix} 0 && -1 && 1 \\ 1 && 0 && -1 \\ -1 && 1 && 0 \end{bmatrix}    $
Therefore,
$ \begin{bmatrix} a \\ b \\ c \end{bmatrix} = \dfrac{1}{6} \begin{bmatrix} 1 \\ 4 \\ 1 \end{bmatrix} + \dfrac{1}{2} \begin{bmatrix} -1 \\ 0 \\ 1 \end{bmatrix} = \dfrac{1}{6} \begin{bmatrix} -2 \\ 4 \\ 4 \end{bmatrix} $
Therefore, $ 3 a^2 + b^2 + c^2 = \dfrac{1}{36} (12 + 16 + 16) = \dfrac{44}{36} = \dfrac{11}{9} $
A: The additional equation comes from the angle of rotation.
Set $A=(0,1,0)$ and $B=(a,b,c)$, then consider the point where the rotation axis intersects the given plane
$$
P_0 = \left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)
$$
and consider the two vectors
\begin{align}
u_A &= A-P_0 = \left(-\frac{1}{3},\frac{2}{3},-\frac{1}{3}\right) \\
u_B &= B-P_0 = \left(a-\frac{1}{3},b-\frac{1}{3},c-\frac{1}{3}\right) \\
\end{align}
Then
$$
\frac{u_A \cdot u_B}{\|u_A\|\|u_B\|} = \cos(\pi/3) = \frac{1}{2}
$$
so
$$
-\frac{a-2 b+c}{\sqrt{6 \left(a^2+b^2+c^2\right)-4 (a+b+c)+2}}=-\frac{1}{2}(a-2b+c) = \frac{1}{2}
$$
The system becomes
\begin{align}
& a+b+c=1 \\
& a^2+b^2+c^2 =1 \\
& a-2b+c= -1
\end{align}
and has two solutions. The correct one is determined by the counter-clockwise information, that could be formalized by the condition
$$
u_A \times u_B \cdot u > 0
$$
A: Another ad-hoc approach. Passing to the plane $x+y+z=1$ where the action takes place, we can see that $(0,1,0)$ rotates around $\left(\frac13, \frac13, \frac13\right)$ within the plane by the angle $\frac{\pi}3$ counter-clockwise.
By definition of $T$ we have

*

*The angle $\angle (0,1,0),\left(\frac13, \frac13, \frac13\right),(a,b,c)$ is equal to $\frac{\pi}3$,

*The distances from $\left(\frac13, \frac13, \frac13\right)$ to the points $(a,b,c)$ and $(0,1,0)$ are equal.

$\hspace{2.5cm}$
It follows that the triangle with vertices $(0,1,0),\left(\frac13, \frac13, \frac13\right),(a,b,c)$ is equilateral so in particular
\begin{align}
\left(a-\frac13\right)^2 + \left(b-\frac13\right)^2 + \left(c-\frac13\right)^2 &= d\left(\left(\frac13, \frac13, \frac13\right), (a,b,c)\right)^2 \\
&= d\left((0,1,0),(a,b,c)\right) \\
&= a^2+(b-1)^2+c^2
\end{align}
which is equivalent to $a-2b+c=1$, so this is your third equation as stated by @enzotib.
You can further simplify the system of equations if you show that the triangle with vertices $(0,0,1),\left(\frac13, \frac13, \frac13\right),(a,b,c)$ is also equilateral (and hence congruent to the aforementioned one) so in particular we have
$$(a-1)^2+b^2 + c^2 = d((0,0,1),(a,b,c)) = d((0,1,0),(a,b,c)) = a^2+(b-1)^2+c^2$$
which immediately implies $a=b$, eliminating one of the solutions.
