Existence of the conditional expectation $\mathbb E[X\mid \mathcal G]$ Let $(\Omega ,\mathcal F,\mathbb P)$ a probability space, $X$ a $L^2(\Omega )$ r.v. and $\mathcal G\subset \mathcal F$ a $\sigma -$algebra. The existence of the conditional expectation $\mathbb E[X\mid \mathcal G]$ is proved in my lecture using Radon-Nikodym. Could we prove it as follow ?
Let $\mathcal S=\{Y : \Omega \to \mathbb R\mid Y\text{ is $\mathcal G-$measurable and } \mathbb E[Y^2]<\infty \}.$ This is a subspace of $L^2(\Omega )$. Moreover, $\left<X,Y\right>:= \mathbb E[XY]$ is a scalar product on $L^2(\Omega )$. By linear algebra, there exist a unique $Z\in \mathcal S$ s.t. for all $S\in \mathcal S$, $$\left<X,S\right>=\left<Z,S\right>.$$
We denote $Z:=\mathbb E[X\mid \mathcal G]$. 
Q1) Does this work ?
Q2) If yes, why do we normally use Radon-Nikodym that looks to be a more difficult theorem ?
 A: Yes, this works. However, some remarks are needed here. First of all, I wouldn't call this result linear algebra, but instead functional analysis, since you need a result for infinite-dimensional spaces. Also, notice that this only works for variables $Y\in L^2(\Omega)$. However, the Radon-Nikodym result is for $Y \in L^1(\Omega)$. It is possible to use $Y \in L^2(\Omega)$ first and then make an approximation in $L^1$ to get the full result, but there are details to fill in this proof.
A: As people pointed out, use of Radon-Nikodym Theorem gives us existance in $L^1$.
Your work proved it (well, maybe Orthogonal projection in Hilbert Space instead of that "Linear Algebra", but the scheme is okay, it isn't hard to fulfill it) only in $L^2$.
However, you can then prove linearity of Cond. Exp in $L^2$, show that when $X \ge 0$ then $\mathbb E[X|\mathcal G] \ge 0$ a.s, and if $X_1 \ge X_2$ then $\mathbb E[X_1 | \mathcal G] \ge \mathbb E[X_2 | \mathcal G]$ a.s
Then you can try to prove the result for $X\ge 0$, $X \in L^1$, by taking sequence of $X_n \in L^2$, such that for every $\omega \in \Omega$ you have $X_n(\omega)$ inscrease to $X(\omega)$ ( for example $X_n = \min\{X,n\}$). Then by what we proved earlier, sequence $\mathbb E[X_n | \mathcal G]$ is non-descreasing, and by monotone convergence theorem (Lebesgue) you can prove that the limit is equal to $\mathbb E[X|\mathcal G]$.
And for any $X \in L^1$ use the decompositio $X = X^+ - X^-$, where $X^+ = \max\{X,0\}$ and $X^- = \max\{-X,0\}$, and use above with some algebraic  work
A: If you're just doing conditional expectaion on $L^2$, then the most natural way is saying, as you do, that the orthogonal projection of $X$ onto the $\mathcal{G}$-measurable $L^2$-variables defines a conditional expectation (you can check that your construction really yields $Z$ as the orthogonal projection of $X$).
Where you want the Radon-Nikodym theorem (or, really, the Jordan-Hahn decomposition) is when you're proving that conditional expectations exist in $L^1$! That is, if $E|X|<\infty,$ then $E[X|\mathcal{G}]$ is well-defined.
