1
$\begingroup$

If we have the linear model $$ y = \beta^Tx + \epsilon $$ and assuming $\epsilon \sim N(0, \sigma^2)$, we can write \begin{align} p(\epsilon) = \frac{1}{\sqrt{2\pi}\sigma}\exp\left({-\frac{(\epsilon)^2}{2\sigma^2}}\right) \end{align}

Since we know $\epsilon = y - \theta^Tx$, we can write

\begin{align} p(y - \theta^Tx) = \frac{1}{\sqrt{2\pi}\sigma}\exp\left({-\frac{(y - \theta^Tx)^2}{2\sigma^2}}\right) \end{align}

and apparently the above is equivalent to writing \begin{align} p(y|x; \theta) = \frac{1}{\sqrt{2\pi}\sigma}\exp\left({-\frac{(y - \theta^Tx)^2}{2\sigma^2}}\right) \end{align}

I am confused with going from the 2nd to last to last equation. Why is it that we can turn the marginal probability distribution of $\epsilon$ into a conditional distribution of $y$ given $x$?

Why can't the left hand side of the question be a joint probability, e.g., $p(y,x; \theta)$ instead or even $p(x | y; \theta)$?

$\endgroup$

1 Answer 1

1
$\begingroup$

$\epsilon \sim N(0,\sigma^2) \implies y \sim N(\beta^Tx,\sigma^2)$ since we generally assume $x$ is fixed in linear regression: $E[y|x] = \beta^Tx$.

This means that $y$ is simply a translated version of $x$ which makes it trivial to get the conditional distribution of $y$. Note that $x$ is generally considered fixed so a joint distribution doesn't make sense.

In general, it's tempting to do "normal algebra" with random variables, but that doesn't always work out like you think. You need to be very clear about what is random and what is not and not treat them all on the same footing.

$\endgroup$
2
  • $\begingroup$ So the conditioning on $x$ is a derivative of the linear regression model and not that of any underlying MLE characteristic? $\endgroup$
    – David
    Jul 19, 2020 at 4:51
  • $\begingroup$ @David not sure I understood your comment -- you're not doing anything with MLE here. You are proposing a probability model for $y$ given $\beta,x$. No where do you specify the distribution for $x$ and in general it is considered a fixed input. This makes it very easy to derive the conditional distribution of $y$. Imagine if x were also normally distributed with zero mean and some variance -- you'd not get the same symmetry since you'd have to account for the variability of $x$. $\endgroup$
    – Annika
    Jul 19, 2020 at 4:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .