I am reading "A Primer in Econometric Theory" by John Stachurski and reading the part on Conditional Maximum Likelihood. There I have seen the same kind of maximization I have seen before in other sources too: In order to estimate the parameter of a distribution, author uses conditional maximum likelihood and he does not take into account the marginal density of inputs when maximizing the objective function. I have seen it being done before, but he openly says marginal density is independent of the parameter we are estimating. Let me explain what I am asking:

Suppose we have inputs and outputs in our model, $x$ being the input and $y$ being the output: $x_1, x_2$...$x_N$ and $y_1, y_2$...$y_N$. They come in pairs. Each observation $x_i$ is probably a vector but $y_i$ is just a scalar. Our pair is $(x_i, y_i)$. Our aim is to estimate $\theta$ in $p(y|x;\theta)$ in order to pin down the conditional density of $y$ given $x$.

So we maximize the following condition:

$l(\theta) = \sum_{n=1}^{N}\text{ln}\,p(x_n,y_n;\theta)$ where $p$ is the joint density of $(x_n,y_n)$.

Letting $\pi$ be the marginal density of $x$, we can decompose the joint density as:

$p(x,y;\theta) = p(y|x;\theta)\pi(x)$.

Here author says the following:

"The density $\pi(x)$ is unknown but we have not parameterized it because we aren't trying to estimate it. We can now rewrite the log likelihood as

$l(\theta) = \sum_{n=1}^{N}\text{ln}\,p(y_n|x_n;\theta) + \sum_{n=1}^{N}\text{ln}\,\pi(x_n)$

The second term on the right-hand side is independent of $\theta$ and as such it does not affect the maximizer"

And he goes on just maximizing the first part, the conditional probability.

Here is my question: Is not $\pi(x)$ dependent on $\theta$ somehow? My thinking is convoluted but both $p(y|x;\theta)$ and $\pi(x)$ are derived from the same underlying joint density: $p(x, y;\theta)$. $\pi(x)$ is just a short hand for

$\pi(x) = \int p(x,y;\theta)dy$ which is clearly dependent on $\theta$. Once you plug this into your maximization problem, it becomes:

$l(\theta) = \sum_{n=1}^{N}\text{ln}\,p(y_n|x_n;\theta) + \sum_{n=1}^{N}\text{ln}\,\int p(x,y;\theta)dy$

Now the second item is also dependent on $\theta$ and need to be taken into account when maximizing wrt $\theta$. Am I missing something here?



It really depends on the way you set up your model. For example, this is a perfectly valid model:

$$ X \sim N(0, \sigma^2) \\ Y|X \sim N( \mu, \gamma^2) $$

Then, we may write $\theta = (\sigma^2, \mu, \gamma^2)$ and so:

$$ p(X,Y;\theta) = p(Y|X;\theta)p(X;\theta) = p(Y|X; \mu, \gamma^2) p(X|\sigma^2) $$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.