0
$\begingroup$

Suppose $(x^{i},y^{i})$ are observed data for the $i$th data set, let us assume for simplicity that there is only one single $x$ variable in each data set. We want to estimate using the linear model $y^i = \theta x + \epsilon^{i}$, where $\epsilon^{i} \sim N(0, \sigma^2)$. Maximum likelihood says that we want to maximize

$P(y^{i}|x^{i}, \theta) = \frac{1}{\sqrt{2 \pi }\sigma}e^-{\frac{(y^{i} - \theta x^{i})^2}{2\sigma^2}}$.

I am confused as to:

1). Is $P(y^{i}|x^{i}, \theta)$ referring to $P(Y = y^{i}|X = x^{i}, \theta)$ the conditional probability? Isn't $P(Y = y^{i}|X = x^{i}, \theta)$ zero for continuous random variables?

2).How does one deduce that $P(y^{i}|x^{i}, \theta) = \frac{1}{\sqrt{2 \pi }\sigma}e^-{\frac{(y^{i} - \theta x^{i})^2}{2\sigma^2}}$ if it refers to the conditional probability.

3).Even though we assume it is discrete shouldn't we be maximizing $P(Y = \theta x^i|X = x^{i}, \theta)$ because we want our model to be as close as possible.

$\endgroup$
1
  • 2
    $\begingroup$ For continuous random variables, you maximize the conditional probability density evaluated at the data as a function of the unknown parameter. It is not technically a probability. $\endgroup$
    – Ian
    Jun 1, 2020 at 3:43

1 Answer 1

1
$\begingroup$

1) No, it does not refer to $P(Y = y^i \mid X = x^i, \theta)$. Reasons like this are why I do not like to use $P$ to represent the density function; maximum likelihood aims to maximize $$f_{Y_1, \dots, Y_n}(y_1, \dots, y_n) = \prod_{i=1}^{n}f_{Y_i}(y_i)$$ where $f_{Y_i}(y_i)$ is the probability density function of $Y_i$ (and the product occurring due to independence of $Y_1, \dots, Y_n$).

2) If $y^i = \theta x^i + \epsilon^i$, where $\epsilon^i \sim \mathcal{N}(0, \sigma^2)$, what you have is a normal random variable plus a constant ($\theta x^i$). Thus $y^i$ has mean $\theta x^i + 0 = \theta x^i$ by linearity of expectation, and variance $\sigma^2$, hence $y^i \sim \mathcal{N}(\theta x^i, \sigma^2)$, thus leading to the provided density function as you stated. Given $\theta$ and $x^i$, the distribution of $y^i \mid (\theta, x^i)$ would be the same as $y^i$ since $\theta$ and $x^i$ are constants.

3) I am confused by what you are trying to ask here. By maximum likelihood estimation, you would maximize $$f_{Y_1, \dots, Y_n}(y_1, \dots, y_n) = \prod_{i=1}^{n}f_{Y_i}(y_i) = \prod_{i=1}^{n}\dfrac{1}{\sqrt{2\pi}\sigma}e^{-[1/(2\sigma^2)](y_i - \theta x^i)^2}$$

$\endgroup$
4
  • $\begingroup$ Thank you very much for your detailed answer. Why are we treating $x$ variables as constants? Is it conditional density? $\endgroup$
    – z.z
    Jun 1, 2020 at 4:17
  • $\begingroup$ @z.z When you condition on some random variables, that is the exact same thing as assuming that those same random variables are constants (i.e., they are equal to a particular value with probability 1). I will leave it to you to prove this. As to why one might do this, it's related to the regression problem in statistics: in this case, you have a "response variable" $y^i$ you suspect is proportional (by $\theta$) to $x^i$, with an added error term, and you could estimate $\theta$ through a number of ways. $\endgroup$ Jun 1, 2020 at 4:21
  • $\begingroup$ I do not quite see how that can follow from the definition which asserts that $f_{Y_i}(y_i) = \frac{f_{Y_i, X_i}(x_i, y_i)}{f_{X_i}(x_i)}$. Do you maybe have a reference to this fact? $\endgroup$
    – z.z
    Jun 1, 2020 at 4:51
  • $\begingroup$ @z.z Observe that $f_{Y_i \mid X_i}(y_i \mid x_i) = f_{Y_i, X_i}(y_i, x_i)$ if and only if $f_{X_i}(x_i) = 1$ at a particular value of $x_i$. $\endgroup$ Jun 1, 2020 at 5:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .