Can this symmetric matrix be an orthogonal matrix? So the question is to prove whether this matrix M is orthogonal?
$$
        M = \begin{bmatrix}
        a & k & k \\
        k & a & k \\
        k & k & a \\
        \end{bmatrix}
$$
My attempt to find the inverse of M :
$$det(M)=(a+2k)(a-k)(a-k)$$
The inverse of $M$ is :
$$
        \begin{matrix}
        (a+k)/((a-k)(a+2k)) & (-k)/((a-k)(a+2k)) & (-k)/((a-k)(a+2k))\\
        (-k)/((a-k)(a+2k)) & (a+k)/((a-k)(a+2k)) & (-k)/((a-k)(a+2k))\\
        (-k)/((a-k)(a+2k)) & (-k)/((a-k)(a+2k)) & (a+k)/((a-k)(a+2k))\\
        \end{matrix}
$$
$M$ is orthogonal if $M^T = M^{-1}$
So I end up with a system of 2 equations to solve :
$a=\dfrac{a+k}{(a-k)(a+2k)}$ and $k=\dfrac{-k}{(a-k)(a+2k)}$
and now I am stucked.
I would really appreciate feedbacks about whether my steps are all correct and if yes, what to do next ?
 A: 
So the question is to prove whether this matrix M is orthogonal?

It's not clear what exactly what you mean by this.  Perhaps your question is:

Is $M$ orthogonal for all $a,k \in \Bbb R$?

The answer to this is clearly no.  For instance: if $a = k$, then $M$ fails to be invertible, let alone orthogonal. It is also notable that orthogonal matrices have a determinant of $\pm 1$.
Perhaps your question is:

For which $a,k \in \Bbb R$ is $M$ orthogonal?

Note that $M$ is orthogonal if and only if $MM^T = I$.  However, $M$ is symmetric, so $M = M^T$, and the above can be rewritten as $M^2 = I$. To that end:
$$
M^2 = \pmatrix{a^2 + 2k^2 & 2ak + k^2 & 2ak + k^2 \\  \vdots & \ddots}
$$
So, our matrix will be orthogonal if and only if $M$ is the identity matrix, which is to say that
$$
a^2 + 2k^2 = 1\\
2ak + k^2 = 0
$$
The last equation can be written as $k(2a + k) = 0$. If $k = 0$, then the first equation becomes $a^2 = 1$ so that $a = \pm 1$ are the values for which $M$ is orthogonal. If $k = -2a$, then $9a^2 = 1 \implies a = \pm 1/3$.  So, we have the additional solutions $a=1/3,k=-2/3$ and $a = -1/3, k = 2/3$.

This is all made much easier using eigenvalues, and noting that $M$ has the form
$$
M = k xx^T + (a - k)I
$$
where $x$ is the column vector of $1$s and $I$ is the identity matrix.
A: Since $M$ is symmetric, $M=M^T$, so to check orthogonality, we compute $MM^T=M^2$ and get
$$
\begin{bmatrix}
a^2+2k^2&2ak+k^2&2ak+k^2\\
2ak+k^2&a^2+2k^2&2ak+k^2\\
2ak+k^2&2ak+k^2&a^2+2k^2
\end{bmatrix}
$$
Therefore, you need $2ak+k^2=0$ and $a^2+2k^2=1$.  Factoring the first equation, we have $k(2a+k)=0$.  Therefore, either $k=0$ or $k=-2a$.


*

*When $k=0$, the second equation simplifies to $a^2=1$, so $a=\pm 1$ gives orthogonality.

*When $k=-2a$, the second equation simplifies to $9a^2=1$, so $a=\pm\frac{1}{3}$ with $k=\mp \frac{2}{3}$ gives orthogonality.
A: It would be easier to solve $$A^\top A = I $$.
$$
\begin{eqnarray*}
\left[\begin{array}{ccc}
a & k & k\\
k & a & k\\
k & k & a
\end{array}\right]\left[\begin{array}{ccc}
a & k & k\\
k & a & k\\
k & k & a
\end{array}\right] & = & \left[\begin{array}{ccc}
1 & 0 & 0\\
0 & 1 & 0\\
0 & 0 & 1
\end{array}\right]\\
\left[\begin{array}{ccc}
a^{2}+2k^{2} & 2ak+k^{2} & 2ak+k^{2}\\
2ak+k^{2} & a^{2}+2k^{2} & 2ak+k^{2}\\
2ak+k^{2} & 2ak+k^{2} & a^{2}+2k^{2}
\end{array}\right] & = & \left[\begin{array}{ccc}
1 & 0 & 0\\
0 & 1 & 0\\
0 & 0 & 1
\end{array}\right]
\end{eqnarray*}
$$
Now you can solve the following equations. 
$$
\begin{eqnarray*}
a^{2}+2k^{2} & = & 1\\
2ak+k^{2} & = & 0
\end{eqnarray*}
$$
