Soft Question - Conditioning and Derivatives I've noticed a few "similarities" or "correspondences" between certain quantities in probability and calculus.  I'm hoping that somebody can point me to references where I can solidify my intuition, or confirm that my intuition is misguided.
Is there a "basic" relationship between the covariance matrix for a random vector and "some" Jacobian matrix?  Intuitively, since the covariance measures the "dependence" of one variable with another, it seems (to me!) that the Jacobian and covariance matrices are measuring the same thing, at least on "different scales".  But I'm not sure what a covariance matrix is the "Jacobian of".
Similarly (this correspondence lead to my hypothesis), the conditional expectation formula for a bivariate normal random variable looks a lot like the "a best affine approximation":
$$
E(Y|X = x) = E(Y) + \frac{C(X,Y)}{V(X)} (x - E(X))
$$
versus
$$
F(x) = F(p) + J_f(p)(x - p)
$$
For the purposes of this comparative analysis, we set $F(x) = E(Y | X = x)$ and $p = E(X)$ to yield:
$$
\begin{align}
E(Y|X=x) &= E(Y|X=E(X)) + J_f(E(X))(x - E(X)) \\
         &= E(Y) + J_f(E(X))(x - E(X))
\end{align}
$$
So that
$$\frac{C(X,Y)}{V(X)} = J_f(E(X))$$
So it looks as though the covariance between $X$ and $Y$, scaled by the variance of $X$, is the derivative of the density $f$ evaluated at $E(X)$.  (Of course, I started working "backwards", so I know this might not hold in general)
So, what's going on, and where can I find out more?
 A: To correct a mistake, your conjecture is actually that "the covariance between X and Y, scaled by the variance of X, is the derivative" of the conditional expectation function - not "the density".  
As for the actual question, I don't know of a source that discusses "best affine approximation" and "conditional expectation function" side-by-side - most probably because the connection is weak and does not extend in general. Meaning:
"Best affine approximation" is an alternative name for a 1st-order Taylor expansion of a function around a point of its domain (in your notation, around point $p$). But it is "best" by necessity, since all subsequent terms are non-linear.  
Now, the conditional expectation function (CEF) between two variables, does not always have the form you stated in your question. It holds for the magical Normal distribution, but not in general.
Moreover, even if we restrict ourselves to the normal world, then one could point out that the CEF is the "best linear predictor" of the random variable $Y$ which is also a function. Hmm, this sounds a lot like "best affine approximation"... the problem is that the CEF is "best linear predictor" under a specific optimality criterion : it minimizes $E\left(Y-h(X)\right)^2$, i.e. expected squared deviation. This is the most widely used one, but there are others. 
So while "best affine approximation" depends on a much more "wide" optimality criterion (that the remainder approaches zero at a sufficiently fast rate), the CEF is "best linear predictor" given a much more specific and hence, restrictive optimality criterion , which weakens the conceptual analogy.
