Vector that maximizes $(x^TAx)(x^TBx)$ subject to $x^Tx=1$ ? (A,B symmetric, positive definite) I've been playing with eigenvector-type optimisation problems where a vector maximally projects onto two matrices. The sum version of this problem (maximize $x^TAx + x^TBx$) is fairly straightforward (leading eigenvector of $A+B$), but the product version in the title has completely stumped me.
To restate the problem, I aim to find a vector $x$ such that $x^Tx=1$ that maximizes the quantity $(x^TAx)(x^TBx)$. Here A and B can be taken to be symmetric and positive definite matrices.
My attempted solution involved a substitution $p=B^{-1/2}x$, such that the quantity to be maximized is $(p^TB^{-1/2}AB^{-1/2}p)(p^Tp)$, but $p^Tp=x^TB^{-1}x$ is unknown, so we are back to square one. I also considered that, as A and B are positive definite, perhaps the vector that maximizes the sum also maximizes the product. This of course doesn't hold though, as either of the terms may be <1.
Any pointers would be received gratefully.
 A: We are attempting to maximize a real valued function subject to a constraint, and the standard tool is Lagrange multipliers.
Let $f(x)=(x^TAx)(x^TBx)$, and $g(x)=x^Tx-1$.  Since $\nabla x^TMx=(M+M^T)x$, the product rule yields $\nabla f(x)=2[(x^TAx)Bx + (x^TBx)Ax]$. We note that $x^T\nabla f(x)=4f(x)$ (which actually follows from the fact that $f(x)$ is homogenous of degree 4), and $.
The equation dividing the equation $\nabla f(x)=\lambda \nabla g(x)$ by 2 yields
$$[(x^TAx)Bx + (x^TBx)Ax]=\lambda x$$
and multiplying by $x^T$ and using the fact that $x^Tx=1$ yields $\lambda=2f(x)$ at a critical point.  Plugging in for $\lambda$ and then dividing both sides by $f(x)$ (since this cannot be $0$ at a maximum unless $f$ is identically $0$) yields
$$\frac{Ax}{x^TAx}+\frac{Bx}{x^TBx}=2x.$$
Thus, $x$ is an eigenvector of a linear combinaton of $A$ and $B$ with eigenvalue $2$.  Thus, maximizing $f(x)$ involves finding linear combinations $\alpha A + \beta B$ that have an eigenvalue of $2$ which can be done by finding roots of $P(\alpha,\beta)=\det(2I-\alpha A - \beta B)$, verifying that if $x$ is an eigenvector with length $1$ then $x^TAx=1/\alpha, x^TBx=1\beta$, and then the potential maximum will be $1/(\alpha\beta)$.
For every ratio $\alpha/\beta$, this gives us at least $n$ possible choices for $x$ (one for each eigenvector, a family's worth if there is a repeated eigenvalue, as renormalizing the matrix can make the eigenvalue into 2), but without a symbolic way to describe them, I don't see a way to do all the verification steps smoothly.
