Rigorous interpretation of the maximum likelihood

Suppose $$(x^{i},y^{i})$$ are observed data for the $$i$$th data set, let us assume for simplicity that there is only one single $$x$$ variable in each data set. We want to estimate using the linear model $$y^i = \theta x + \epsilon^{i}$$, where $$\epsilon^{i} \sim N(0, \sigma^2)$$. Maximum likelihood says that we want to maximize

$$P(y^{i}|x^{i}, \theta) = \frac{1}{\sqrt{2 \pi }\sigma}e^-{\frac{(y^{i} - \theta x^{i})^2}{2\sigma^2}}$$.

I am confused as to:

1). Is $$P(y^{i}|x^{i}, \theta)$$ referring to $$P(Y = y^{i}|X = x^{i}, \theta)$$ the conditional probability? Isn't $$P(Y = y^{i}|X = x^{i}, \theta)$$ zero for continuous random variables?

2).How does one deduce that $$P(y^{i}|x^{i}, \theta) = \frac{1}{\sqrt{2 \pi }\sigma}e^-{\frac{(y^{i} - \theta x^{i})^2}{2\sigma^2}}$$ if it refers to the conditional probability.

3).Even though we assume it is discrete shouldn't we be maximizing $$P(Y = \theta x^i|X = x^{i}, \theta)$$ because we want our model to be as close as possible.

• For continuous random variables, you maximize the conditional probability density evaluated at the data as a function of the unknown parameter. It is not technically a probability.
– Ian
Jun 1, 2020 at 3:43

1 Answer

1) No, it does not refer to $$P(Y = y^i \mid X = x^i, \theta)$$. Reasons like this are why I do not like to use $$P$$ to represent the density function; maximum likelihood aims to maximize $$f_{Y_1, \dots, Y_n}(y_1, \dots, y_n) = \prod_{i=1}^{n}f_{Y_i}(y_i)$$ where $$f_{Y_i}(y_i)$$ is the probability density function of $$Y_i$$ (and the product occurring due to independence of $$Y_1, \dots, Y_n$$).

2) If $$y^i = \theta x^i + \epsilon^i$$, where $$\epsilon^i \sim \mathcal{N}(0, \sigma^2)$$, what you have is a normal random variable plus a constant ($$\theta x^i$$). Thus $$y^i$$ has mean $$\theta x^i + 0 = \theta x^i$$ by linearity of expectation, and variance $$\sigma^2$$, hence $$y^i \sim \mathcal{N}(\theta x^i, \sigma^2)$$, thus leading to the provided density function as you stated. Given $$\theta$$ and $$x^i$$, the distribution of $$y^i \mid (\theta, x^i)$$ would be the same as $$y^i$$ since $$\theta$$ and $$x^i$$ are constants.

3) I am confused by what you are trying to ask here. By maximum likelihood estimation, you would maximize $$f_{Y_1, \dots, Y_n}(y_1, \dots, y_n) = \prod_{i=1}^{n}f_{Y_i}(y_i) = \prod_{i=1}^{n}\dfrac{1}{\sqrt{2\pi}\sigma}e^{-[1/(2\sigma^2)](y_i - \theta x^i)^2}$$

• Thank you very much for your detailed answer. Why are we treating $x$ variables as constants? Is it conditional density?
– z.z
Jun 1, 2020 at 4:17
• @z.z When you condition on some random variables, that is the exact same thing as assuming that those same random variables are constants (i.e., they are equal to a particular value with probability 1). I will leave it to you to prove this. As to why one might do this, it's related to the regression problem in statistics: in this case, you have a "response variable" $y^i$ you suspect is proportional (by $\theta$) to $x^i$, with an added error term, and you could estimate $\theta$ through a number of ways. Jun 1, 2020 at 4:21
• I do not quite see how that can follow from the definition which asserts that $f_{Y_i}(y_i) = \frac{f_{Y_i, X_i}(x_i, y_i)}{f_{X_i}(x_i)}$. Do you maybe have a reference to this fact?
– z.z
Jun 1, 2020 at 4:51
• @z.z Observe that $f_{Y_i \mid X_i}(y_i \mid x_i) = f_{Y_i, X_i}(y_i, x_i)$ if and only if $f_{X_i}(x_i) = 1$ at a particular value of $x_i$. Jun 1, 2020 at 5:03