Orthogonal vectors such that $X^T S X + Y^T S Y = 0$ (in $\mathbb{R}^3$) Say I have a 3x3 symmetric matrix $S$, having two positive and one negative eigenvalue.
I'm looking for two orthogonal unit vectors $X$ and $Y$ such that
$$X^T S X + Y^T S Y = 0$$
I have tried to solve it using spherical or cartesian parametrizations of $X$ and $Y$ but it quickly becomes painful. Do you know a way to solve this ?
Edit : I should insist on the unit vector requirement, otherwise the problem is indeed easier.
 A: Call the three eigenvalues $a,b$ and $-c$, where $a\ge b\ge0\ge-c$. Complete $\{x,y\}$ to an orthonormal basis $\{x,y,z\}$ and let $Q$ be the augmented matrix $[x,y,z]$. Then $Q$ is orthogonal and
$$
a+b-c=\operatorname{tr}(S)=\operatorname{tr}(Q^TSQ)=x^TSx+y^TSy+z^TSz=z^TSz.
$$
Since $-c\le z^TSz\le a$, we see that a solution $\{x,y\}$ exists if and only if $-c\le a+b-c\le a$, i.e. iff $c\ge b$.
In case $c\ge b$ is indeed true, take any unit vector $z$ such that $z^TSz=a+b-c$ and any two unit vectors $x,y$ that form an orthonormal basis with $z$ would give a solution to your problem. Edit: for instance, let $u,v,w$ be three orthonormal eigenvectors for the eigenvalues $a,b,-c$ respectively. Then $z=\sqrt{\frac{a+b}{a+c}}\,u+\sqrt{\frac{c-b}{a+c}}\,w$ will satisfy the equation $z^TSz=a+b-c$. Now you may take $x=\sqrt{\frac{a+b}{a+c}}\,w-\sqrt{\frac{c-b}{a+c}}\,u$ and $y=v$.
A: Here is another track, not along the same way as @user1551, that can be considered as complementary to his.
Firstly, we (very slightly!) modify the issue by replacing unit vectors by vectors with $\dfrac{1}{\sqrt{2}}$ norm.
I keep notations $a \geq b \geq 0 \geq -c$ for the eigenvalues of $S$. 
Let us define $Q(X,Y):=\begin{pmatrix}X^T&Y^T\end{pmatrix}\underbrace{\begin{pmatrix}S&I_3\\I_3&S\end{pmatrix}}_{\Sigma}\begin{pmatrix}X\\Y\end{pmatrix}=\underbrace{X^TSX+Y^TSY}_{=0}+2X^TI_3Y=2X^TY$
that ultimately should be $0$ itself. 
The eigenvalues of $\Sigma$ are the eigenvalues of $S$ to which $\pm1$ is added, i.e., are
$$\tag{1}\{a+1,a-1,b+1,b-1,-c+1,-c-1\}$$
(not necessarily in that order). See proof of (1) at the end of this answer.
Thus, the extreme eigenvalues of $\Sigma$ are $a+1$ and $-c-1$.
Let $U=\binom{X}{Y}$. It is clear that $\|U\|^2=\|X\|^2+\|Y\|^2 \in \mathbb{R}^6.$
The Rayleigh quotient associated with $\Sigma$ is (for $U$ such that  $\|U\|=1$) nothing else than $Q(U)$ ; it is well known that it takes every value between extreme eigenvalues $a+1$ and $-c-1$, i.e., for every $t, 0 \leq t \leq 1$, there exist $U$ such that :
$Q(U)=X^TY=t(-c-1)+(1-t)(a+1)$
In particular, the LHS is zero for the following positive value of $t$:
$$\tag{3}t=\dfrac{a+1}{a+c+2}$$
which is less than $1$, because $c \geq 0$.  BUT such orthogonal $X$ and $Y$ have no reason to be such that $\|X\|=\|Y\|=\dfrac{1}{\sqrt{2}};$
(One can have for example $U^T=(1,0,0,0,0,0)$ meaning $X^T=(1,0,0)$ and $Y^T=(0,0,0)$.)
The interest of formula (3) is when condition $c\geq b$ established by user1551 is not fulfilled. In this case, (3) provides a start for finding two vectors $X,Y$ that, can be at the same time not far from being orthogonal and not far to have a common norm $\dfrac{1}{\sqrt{2}}$. 

Proof of (1): (1) can be established 


*

*either directly by expressing  


$\tag{2}\Sigma=I_2\otimes S+\binom{0 \ 1}{1 \ 0}\otimes I_3$
where $\otimes$ is the Kronecker product. 
(2) is called a ''Kronecker sum'' (notation $M \oplus N=I_n\otimes M+N\otimes I_p$); a classical theorem says that the eigenvalues of $M \oplus N$ are obtained by summing, in all possible ways, an eigenvalue of $M$ with an eigenvalue of $N$.


*

*or in a computational way by applying  Schur's complement identity:


$$det\begin{pmatrix}A&C\\B&D\end{pmatrix}=det(D)det(D-BA^{-1}C)$$
to a diagonalized form of $\Sigma-\lambda I_6$.
