2
$\begingroup$

Let $(\Omega,\Sigma,P)$ be some probability space and let $Y: \Omega \to \mathbb{R} $ be a random variable. Let $x, \beta \in \mathbb{R}^n$. In a linear model we assume something like this: $E(Y|x)=\beta^Tx$. I know conditional expectation, where you condition on either an event $E \in \Sigma$ or on a sub-$\sigma$-algebra $\Sigma' \subseteq \Sigma$. As a special case of the latter, we can condition on another random variable. But the variable $x$ is not random, so how would you rigorously define $E(Y|x)$?

$\endgroup$
2
  • $\begingroup$ You can condition on $X = x.$ $\endgroup$
    – William M.
    Feb 29 at 19:31
  • $\begingroup$ But what is $X$? $\endgroup$ Feb 29 at 19:51

2 Answers 2

4
$\begingroup$

The typical way linear models work is to assume that $Y = \beta^T X + \varepsilon$ where $X$ is some random variable that we believe is related to $Y$ and $\varepsilon$ is another random variable independent of $X$, and $\beta$ is constant (i.e. deterministic), but possibly unknown. For example, $Y$ might be a person's weight and $X$ might be a person's height.

There is a theorem that states that (under certain conditions) there exists a measurable, deterministic function $f$ such that $\mathbb{E}[Y|X] = f(X)$. One can then define $\mathbb{E}[Y|x] = f(x)$ to avoid conditioning on null events like $X = x$ when $X$ is a continuous random variable.

$\endgroup$
4
  • $\begingroup$ Thx! Two questions here: 1. People often say that the variable X is not random in the linear model. Are they wrong? 2. Can you maybe give a reference for the theorem? $\endgroup$ Feb 29 at 20:30
  • $\begingroup$ Ah you probably mean the Doob-Dynkin Lemma, right? $\endgroup$ Feb 29 at 20:39
  • $\begingroup$ 1. I'm not sure I've seen linear models where $X$ is not modeled as a random variable. One often tries to estimate the unknown parameter $\beta$ from some observations of $X$, which are not random. After estimating $\beta$, one can try to estimate an realization of $Y$ from a realization of $X$, and that realization of $X$ can be thought of as deterministic, but generally the underlying model includes a random variable $X$. 2. It's a consequence of the Doob-Dynkin lemma, sorry, I forgot the name. $\endgroup$ Feb 29 at 20:40
  • $\begingroup$ Thx, I think I am happy now. I still have some problems with the words estimation / estimator, which seem to have different meanings in different contexts but that's another story ;) $\endgroup$ Feb 29 at 21:11
2
$\begingroup$

There are several models that can be called "linear regression."

The most straightforward model is to assume we have $p$ $n$-vectors $X = [x_{(1)}, \ldots, x_{(p)}]$ and a "response" $y$ in $\mathbf{R}^n$ both of which are known, and we want to obtain the projection of $y$ onto the span of $X.$ There are no probability assumptions here.

A second model is to assume $(Y,X)$ is multinormal and then $E(Y \mid X)$ is linear. Here we explicitly assume $(Y,X)$ follow a particular probability distribution.

The third is to observe that for $L^2$ random variables, $Y = E(Y \mid \mathscr{H}) + (Y - E(Y \mid \mathscr{H}))$ and we see that $E(Y \mid \mathscr{H})$ is the orthogonal projection of $Y$ onto $L^2(\mathscr{H})$; when $\mathscr{H} = \sigma(X),$ then $E(Y \mid \mathscr{H}) = E(Y \mid X) = f(X)$ for some (deterministic but unknown) measurable function $f$; often we approximate linearly $f(x) \approx \beta^\intercal x$ and then we recover Ordinary Linear Regression. Here, we only need that $Y$ has second moment.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .