generalized Rayleigh Quotient If $M$ is positive definite, $H$ is self-adjoint. Now consider the minimization problem:$$\min_{x\neq 0}\frac{(x,Hx)}{(x,Mx)}.$$
Note that this functional is homogeneous of degree 0. So we can just search the minimum on the unite sphere. And because of the continuity of the functional and that the unit sphere of a finite dimensional space is compact. We can find a solution $f$, and by further calculation(variational method), we can show that $f$ satisfies:$$Hf=bMf,\qquad \hbox{where $b=\frac{(f,Hf)}{(f,Mf)}$}.$$ Now here's my question. Continue to search the minimum in the subspace $\{y:(y,Mf)=0\}$. By the same continuity and compactness argument, the solution exists. Denote it $g$, i.e.$$\min_{y\neq 0, \ (y,Mf)=0}\frac{(y,Hy)}{(y,My)}=\frac{(g,Hg)}{(g,Mg)}$$
How can I show that $g$ satisfies $$Hg=cMg\qquad \hbox{where $c=\frac{(g,Hg)}{(g,Mg)}$}?$$
It's probably quite simple... I might think it to be too complicated. Anyone who can give me a hint?
I've just realized that one can always change this kind of generalized Rayleigh Quotient back to the standard Rayleigh Quotient. By changing back and using what has already been established about the relation between minimizing Rayleigh Quotient and the eigenvector of a Hermitian (if in real case, symmetric), I can derive the required equality. But it seems I'm taking a detour. Is there any way to make it more straightforward?
 A: Find a Cholesky type decomposition of $M$ as 
$$ M = L L^T   $$ 
For our purpose, it does not matter that $L$ is lower triangular,  any such decomposition will do.
Take
$$ E = \left( L^T \right)^{-1},  $$ so that
$$ E^T M E = I,  $$
and
$$   M =   \left( E^T \right)^{-1} E^{-1}.     $$
Write your Rayleigh quotient as
$$ \frac{x^T H x}{x^T M x}  $$
for column vectors $x.$ Now, make the (invertible) substitution
$$ x = E z,  $$ the quotient becomes
$$ \frac{z^T E^T H E z}{z^T E^T M E z} =  \frac{z^T (E^T H E) z}{z^T z}  $$
So, define
$$ G = E^T H E $$ and we have
$$ \frac{x^T H x}{x^T M x}  = \frac{z^T G z}{z^T z}  $$
so we are just asking about eigenvectors of $G,$ which are pairwise orthogonal, call them $z_j$. If we take the first optimum, call it $z_1,$ your 
$$  f = E z_1.  $$
You wanted to next have $$ g^T M f = 0.   $$ However, if we simply take $$  g =  E z_2,   $$ then
$$  g^T M f  = z_2^T E^T  \left( E^T \right)^{-1} E^{-1} E z_1 = z_2^T z_1 = 0. $$
A: The argument goes using the same variational method modified with respect to the set of admissible variations. Denote
$$A = \left\{y:\left(y,Mf\right)=0 \right\}$$
Now for an admissible variation $y\in A$ the statement that $g$ gives the ratio a minimum is expressed as follows:
$$\frac{\left(g+\epsilon y,H(g+\epsilon y)\right)}{\left(g+\epsilon y,M(g+\epsilon y)\right)}>c$$
$$\left(g+\epsilon y,H(g+\epsilon y)\right)-с\left(g+\epsilon y,M(g+\epsilon y)\right)>0$$
Now we have to make sure we exhaust the entire $A$ in the following argument. A simple construction is suggested by the Gram-Smidt orthogonalization process, namely, let
$$y=h-(h,Mf)f$$
with normalization condition $(f,Mf)=1$. Indeed
$$(y,Mf)=(h,Mf)-(h,Mf)(f,Mf)=0$$
Expanding, for example the expression of the numerator, bearing in mind that $H$ is self-adjoint, we obtain:
$$\left(g+\epsilon y,H(g+\epsilon y)\right)=(g,Hg)+2\epsilon (y,Hg)+\epsilon^2(h,h)=c(g,Mg)+2\epsilon(y,Hg)+\epsilon^2(y,Hy)$$
Now
$$(y,Hg)=(h,Hg)-(f,Hg)(y,Mf)=(h,Hg)$$
Performing the same for the denominator and subtracting:
$$\left(g+\epsilon y,H(g+\epsilon y)\right)-с\left(g+\epsilon y,M(g+\epsilon y)\right)=2\epsilon\left( (h,Hg-cMg)\right)+\epsilon^2\left((y,Hy)-c(y,My)\right)$$
Since $h$ is now ranges over $X$, the only way for the RHS to remain positive is for
$$Hg=cMg$$
to hold.
Note: The above argument can be made slightly more concise if scalar product on $X$ is redefined in terms if $M$, that is $\left<x,y\right>=(x,My)$
