1
$\begingroup$

in Andrew Ng's Machine learning course http://cs229.stanford.edu/notes/cs229-notes1.pdf

Page 12,it said: $x$ and $y$ has linear relationship $$y(i) = \theta^Tx^{(i)}+\epsilon^{(i)}$$ where $$ \epsilon \sim N(0,\sigma^2)$$ is assumed to be IID random Gaussian variable.Therefore $$ p(\epsilon^{(i)}) = \frac{1}{\sqrt{2\pi}\sigma}exp(-\frac{(\epsilon^{(i)})^2}{2\sigma^2})$$ and if we write $$\epsilon^{(i)} = y(i) - \theta^Tx^{(i)}$$ we can obtain a conditional probability: $$ p(y^{(i)} | x^{(i)};\theta) = \frac{1}{\sqrt{2\pi}\sigma}exp(-\frac{(y(i) - \theta^Tx^{(i)})^2}{2\sigma^2})$$

My question is : why it's an conditional probability? rather than a 2D-Joint probability? That is, why can't I write it as this: $$ p(y^{(i)} , x^{(i)};\theta) = \frac{1}{\sqrt{2\pi}\sigma}exp(-\frac{(y(i) - \theta^Tx^{(i)})^2}{2\sigma^2})$$

one more question, in the notes, $y^{(i)}$ and $x^{(i)}$ are samples, not random variants, was it OK to write $ p(y^{(i)} | x^{(i)} )$ instead of $ p(Y | X)$

Thank you Brilliant guys!

$\endgroup$
1
$\begingroup$

$p(\epsilon^{(i)})$ is the probability density function of the error term $\epsilon^{(i)}$.   It expresses how the unmoddled effects influence the value of $y^{(i)}$; that is the density function of the influence on $y^{(i)}$ that is not produced by random variable $x^{(i)}$ (or the parameter $\theta$).

In other words, for a fixed value of $x^{(i)}$, then $p(\epsilon^{(i)})$ is the conditional density function of $y^{(i)}$ given that contraint.

$$p(y^{(i)}\mid x^{(i)};\theta)~=~p(\epsilon^{(i)})$$


one more question, in the notes, $y^{(i)}$ and $x^{(i)}$ are samples, not random variants, was it OK to write $ p(y^{(i)} | x^{(i)} )$ instead of $ p(Y | X)$

Yes..ish.   It is actually shorthand for $p_{\lower{0.5ex}{X\mid Y}}(y^{(i)}\mid x^{(i)})$, just as the $p(\epsilon^{(i)})$ is actually $p_{\lower{0.5ex}\mathcal E}(\epsilon^{(i)})$ .   It is just that the subscripts are sometimes omitted when it is considered unambiguous as to what variable the value refers.

$\endgroup$
  • $\begingroup$ Thanks Graham! May I say it like this: "$\epsilon^{(i)}$ depends on only y, not x, therefore to describe the distribution of $\epsilon^{(i)}$, we can use the form of $p(Y|X)$ instead of $p(X,Y)$"? Was it proper to say this? $\endgroup$ – Long Jun 1 '16 at 9:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.