How can I get x such that $x^T Ax<0$ for this given A?

If $A=\begin{bmatrix}5 & 6 \\ 6 & 7 \end{bmatrix}$ then how can I find an X such that $X^TAX<0$ ? I found the eigen values $6\pm\sqrt{37}$ and A is not positive definite. I can't think what to do?help me... regards

You can try $x = (1, a)^\top$ for suitably chosen $a \in \mathbb{R}$.

Note that if you know how to calculate an eigenvector assosciated with a given eigenvalues, then you're done, since

$$x^TAx = \lambda x^Tx = \lambda ||x||^2$$

which clearly has the same sign of $\lambda$

The upshot of the procedure below is this: given real numbers $r,s,t > 0$ and symmetric matrix $$A = \left( \begin{array}{rr} r & s \\ s & t \end{array} \right)$$ with negative determinant $rt-s^2,$ we can just take $$X = \left( \begin{array}{r} -s \\ r \end{array} \right)$$ to get $$X^T A X = r(rt-s^2) < 0.$$ We could also take $$Y = \left( \begin{array}{r} -t \\ s \end{array} \right)$$ to get $$Y^T A Y = t(rt-s^2) < 0.$$

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

ORIGINAL: It is not strictly necessary to deal with the eigenvalues when dealing with a symmetric matrix: one may find a congruent diagonal matrix, that is some nonsingular $P$ such that $P^T A P = D$ is diagonal. By Sylvester's law of Inertia, one of the diagonal elements will be negative, although it need not be an eigenvalue. It will then be obvious how to choose a column vector $y$ such that $y^T D y < 0.$ However, this means $$0 > y^T D y = y^T P^T A P y = (Py)^T A (Py).$$ Taking $x=Py$ then gives $x^T A x < 0.$

$$P = \left( \begin{array}{rr} 1 & -\frac{6}{5} \\ 0 & 1 \end{array} \right)$$ works. The outcome is $P^T A P = D$ with $$D = \left( \begin{array}{rr} 5 & 0 \\ 0 & -\frac{1}{5} \end{array} \right)$$

People do not seem to be familiar with this sort of thing. One place to read about it is TREIL, Linear Algebra Done Wrong. In Chapter 7, Section 2.2 is called Non-orthogonal diagonalization. Section 2.2.1 is called Diagonalization by completion of squares. The way I have shown is section 2.2.2, called Diagonalization using row/column operations. This part has page numbers 202-205, but the pdf thinks they are pages 212-215, which is what the printer needed to know.

There is also a description (different book) of how to do it as an algorithm at reference for linear algebra books that teach reverse Hermite method for symmetric matrices