2
$\begingroup$

In his machine learning notes (Chapter 3) http://cs229.stanford.edu/notes/cs229-notes1.pdf, Andrew Ng glosses over some math. In quantifying the random variable y: $$y^{(i)}=\theta ^{T}x^{(i)}+e^{(i)}$$

e is conjured to represent random noise/factors not accounted by the hypothesis model. If this is the case why must x also be counted a random variable? Also, why is e assumed to have mean 0? Would not a more general case assume a mean of μ? Furthermore, I understand the mathematical form of the gaussian distribution, $$\frac{1}{\sqrt{2\pi \sigma }}e^{-\frac{(q-\mu)^{2}}{2\sigma ^{2}}}$$ but I do not understand how one can conclude that the hypothesis function, $$\theta ^{T}x^{(i)}$$ constiutes the mean of the random variable y, therefore implying: $$p(y^{(i)}|x^{(i)};\theta )=\frac{1}{\sqrt{2\pi \sigma }}e^{-\frac{[y^{(i)}-\theta ^{T}x^{(i)}]^{2}}{2\sigma ^{2}}}$$ Thank you for your help in understanding these subtleties.

$\endgroup$

2 Answers 2

2
$\begingroup$

I will use Andrew Ng's notation (which is a little unusual). In the section Probabilistic Interpretation he makes several assumptions:

  1. $y^{(i)} = \theta^T x^{(i)} + \epsilon^{(i)}$ (there is a linear relationship between $y^{(i)}$ and $x^{(i)}$),
  2. The $\epsilon^{(i)}$ terms are random noise that are modeled as independent identically distributed (iid) Gaussian random variables with mean zero and some standard deviation $\sigma$.

You could model $\epsilon^{(i)}$ has having a more general mean but it is unnecessary because he assumes a bias term $\theta_0$ and $x_0=1$, that is, $$ y^{(i)} = \theta_0\cdot 1 + \theta_1 x_1^{(i)} + \ldots + \theta_n x_n^{(i)} + \epsilon^{(i)} $$ and the regression problem is generally understood as estimating $\theta$ so as to arrive at the average value of $y$ for a given value of $x$. Remember, for a fixed value of $x$ there can be multiple values of $y$ (noisy $y$) and to have a function between $x$ and $y$ you need to pick one (representative) value of $y$. Traditionally this choice has been the mean of $y$.

This presentation of linear regression assumes a linear relationship between $x$ and $y$ where the variation observed in $y$ arises from the noise term $\epsilon$. Since the $\epsilon^{(i)}$ terms are assumed to be iid they all have the same mean (they are independent and have the same distribution which means they have the same mean). If the mean of the $\epsilon^{(i)}=\mu$ then you could transfer the mean from the noise terms to the bias term $\theta_0$ and end up with the same answers for the $y^{(i)}$ values. So for simplicity (or convention) most people assume that the noise term has a mean of zero.

Statements about $y$ are generally made conditional on $x$. So, for example, the mean of $y^{(i)}$ conditional on $x^{(i)}$ is $$ \mathbb{E}\left [y^{(i)}\mid x^{(i)}; \theta\right ]= \mathbb{E}\left [\theta^{T} x^{(i)} + \epsilon^{(i)} \mid x^{(i)}; \theta\right ] = \theta^{T} x^{(i)} + \mathbb{E}[\epsilon^{(i)} \mid x^{(i)}; \theta] = \theta^{T}x^{(i)} $$ since $\theta^{T} x^{(i)}$ is constant given $x^{(i)}$ and $\epsilon^{(i)}$ has a mean of zero. Similarly, $y^{(i)}$ does not necessarily have a normal distribution in this type of linear regression model, but the assumptions above imply that the conditional distribution of $y^{(i)}$ given $x^{(i)}$ is normally distributed with mean $\theta^{T}x^{(i)}$ and standard deviation $\sigma$.

$\endgroup$
0
$\begingroup$

A fitted value is supposed to be an estimate of the average $y$-value for a given $x$-value. The average amount by which a random variable deviates from its average is $0.$

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .