Conditional Expectation with transformation Let $Y_1, Y_2,\ldots, Y_n$ be independent $N(0,1)$ random variables. Define $ X_i =\sum_{j=1}^n c_{i,j}Y_j$, $i=1,2,\ldots,n$, where $c_{i,j}$ are real constants.
Show that $E(X_i\mid X_k)=\dfrac{\sum_{j=1}^n c_{i,j}c_{k,j}}{\sum_{j=1}^nc_{k,j}^2}X_k$.
What is $\operatorname{Var}(X_i\mid X_k)$?
I can only think of $X$ as a transformation of $Y$, which gives me the result that $X$ is Gaussian with $N(0, CC')$. But I got no idea how to get the conditional expectation and variance.
 A: You are right that $X\sim N(0,CC')$.  Conditional distributions are somewhat subtler.  One can look at
$$
\begin{bmatrix} X_i \\ X_j \end{bmatrix} \sim N\left(\begin{bmatrix} 0 \\ 0 \end{bmatrix}, \begin{bmatrix} \sum_{\ell=1}^n c_{i,\ell}^2, & \sum_{\ell=1}^n c_{i,\ell}c_{j,\ell} \\  \sum_{\ell=1}^n c_{i,\ell}c_{j,\ell}, & \sum_{\ell=1}^n c_{j,\ell}^2 \end{bmatrix} \right) = N\left( \begin{bmatrix} 0 \\ 0 \end{bmatrix}, \begin{bmatrix} a & \rho ab \\  \rho ab & b \end{bmatrix} \right),
$$
so that $\rho$ is the correlation.
Maybe I'll come back later and write proofs of some of the assertions below, but for now I'll say
$$
\begin{align}
& E(X_i\mid X_j) = cX_j+d \quad\text{i.e. one needs to fit a straight line,} \\[8pt]
& X_i - E(X_i \mid X_j) \text{ is independent of $X_j$, i.e. the $``$errors" are independent of the predictors,} \\[8pt]
& \operatorname{var}(X_i \mid X_j) \text{ does not depend on $X_j$ (homoscedasticity)}, \\[8pt]
& \mathbb E (\operatorname{var}(X_i\mid X_j)) + \operatorname{var}(\mathbb E(X_i\mid X_j)) = \operatorname{var}(X_i), \\[8pt]
& \text{when $X_j$ is $z$ S.D.s above the mean of $X_j$, then $\mathbb E(X_i\mid X_j)$ is $\rho z$ S.D.s above the mean of $X_i$.}
\end{align}
$$
You can work it out with some algebra based on that.  I'll be back later and maybe add something then.
A: Conditional distributions in normal families are always linear (which explains partly the ubiquity of the Gaussian distribution in modeling), for example the fact that
$$X_i=\alpha_{i,k}X_k+\beta_{i,k}Z_{i,k}+\gamma_{i,k},
$$ 
for some deterministic coefficients $\alpha_{i,k}$, $\beta_{i,k}$ and $\gamma_{i,k}$, and for a standard normal random variable $Z_{i,k}$ independent of $X_{i,k}$ implies that
$$
E(X_i\mid X_k)=\alpha_{i,k}X_k+\gamma_{i,k},
$$ 
and
$$
\mathrm{var}(X_i\mid X_k)=\beta_{i,k}^2.
$$ 
To identify these coefficients, note that, for every $i\ne k$,
$$
E(X_i)=E(X_k)=0,\qquad\mathrm{var}(X_i)=\sum_jc_{i,j}^2,
$$
and
$$
\mathrm{Cov}(X_i,X_k)=\sum_jc_{i,j}c_{k,j},
$$
on the one hand, and that
$$
E(X_i)=\gamma_{i,k},\qquad\mathrm{var}(X_i)=\alpha^2_{i,k}\mathrm{var}(X_k)+\beta^2_{i,k},
$$
and
$$
\mathrm{Cov}(X_i,X_k)=\alpha_{i,k}\mathrm{var}(X_k),
$$
on the other hand. Hence
$$
\gamma_{i,k}=0,\qquad\alpha_{i,k}=\frac{\sum\limits_jc_{i,j}c_{k,j}}{\sum\limits_jc_{k,j}^2},
$$
and
$$
\beta^2_{i,k}=\sum_jc_{i,j}^2-\frac{\left(\sum\limits_jc_{i,j}c_{k,j}\right)^2}{\sum\limits_jc_{k,j}^2}.
$$
Considering the canonical scalar product $\langle\ ,\ \rangle$ and the tuples $c_i=(c_{ik})_k$, this can be rewritten as
$$
\alpha_{i,k}=\frac{\langle c_i,c_k\rangle}{\langle c_k,c_k\rangle},
\qquad
\beta^2_{i,k}=\langle c_i,c_i\rangle-\frac{\langle c_i,c_k\rangle^2}{\langle c_k,c_k\rangle}.
$$
