I'm having a hard time intuitively understanding what this means in a machine learning context. When using the variables $A$ or $B$ or some trivial example, it all makes sense, but when looking at machine learning formulas where there are real variables its harder to see exactly what is meant. For example, if $t$ is what I am trying to predict and $x$ is the training example or input...

$$ p(t|x) = \frac{p(x|t)p(t)}{p(x)} $$

What is meant by $p(x)$? if $x$ is a training example, does it mean the probability of seeing $x$ out of all possible training examples (kind of like the probability of drawing $x$ from a hat)? the probability of seeing $x$ out of the previously known distribution of examples? or something else?

Sometimes I see this with model parameters such as $\theta$ as well which raises the same sort of questions.

  • $\begingroup$ You have a prior distribution for $t$ as well as a conditional probability function for $x$ given $t$ so you could say $p(x)=\int p(x \mid t) \, p(t) \, dt$ changing the integration to a summation if $t$ has a discrete distribution. You are integrating or summing over the parameter $t$, not over the possible observations $x$ $\endgroup$ – Henry May 16 '19 at 13:16
  • $\begingroup$ @Henry, in your comment you said "probability function for 𝑥 given 𝑡," my question is asking how to intuitively understand what the probability of $x$ is? How can I have a probability of a training example or a model parameter? what does that mean? what am I measuring? $\endgroup$ – deltaskelta May 16 '19 at 13:26
  • $\begingroup$ deltaskelta - I have tried to give an example in an answer. Here $t$ is a value your uncertain parameter can take, while $x$ is a value your observation can take $\endgroup$ – Henry May 16 '19 at 14:16

Let's take your dice example to try to illustrate the issue. Here $T$ is your uncertain parameter and $t$ a value it can take, while $X$ is your observation and $x$ a particular value it can take.

  • Suppose you have a $t$-sided fair die, but you do not know what value $t$ has. You do have a prior distribution for $t$ of $P(T=t) = \frac{t}{2^{t+1}}$ for $t \in \{1,2,\ldots\}$.

  • You roll the die and observe a value of $X=x$. Since this is a fair die, you know $P(X=x \mid T=t) = \frac{1}{t}$ for $x \in \{1,2,\ldots\,t\}$

  • You can at this stage ask what is the unconditional $P(X=x)$? In other words, at the start what do you think the probability is of rolling a particular value $x$ even though you do not know how many sides the dice has? As a simple application of conditional probability $$P(X=x) = \sum P(X=x \mid T=t) P(T=t) = \sum\limits_{t=x}^\infty \frac{1}{2^{t+1}} = \frac{1}{2^{x}}$$

As examples, from the first bullet $P(T=6)=\frac{6}{128}$ and $P(T=7)=\frac{7}{256}$ etc. So the unconditional or marginal probability of rolling $X=6$ is $$P(X=6) = \frac{1}{6} \times \frac{6}{128} + \frac{1}{7} \times \frac{2}{256}+ \cdots = \frac{1}{64}= \frac{1}{2^6}$$

If you do roll a $6$ then you then know the number of sides $T \ge 6$, and you get a posterior probability mass function $$P(T=t \mid X=6) = \frac{\frac{1}{2^{t+1}}}{\frac{1}{2^{6}}} = \frac{1}{2^{t-5}}$$ for $t \ge 6$, so $P(T=6 \mid X=6)= \frac12$, $P(T=7 \mid X=6)= \frac14$, etc.

  • $\begingroup$ You reversed my variables so now the question is involving $t$. So I can say that $p(t)$ is the probability that $t$ is the correct number of sides for the die? In the case of $theta$ parameters, it would the the probability that they are the true $theta$ parameters¿ $\endgroup$ – deltaskelta May 16 '19 at 23:04
  • $\begingroup$ @deltaskelta In my statement, I treated $X$ as the observation/evidence/die throw, which is what I thought you did in your question $\endgroup$ – Henry May 17 '19 at 0:10
  • $\begingroup$ I may have used the wrong word somewhere, but my intention was that $x$ was something like the parameters of the dice or $t$ in your example, in which case $p(t)$ means the probability that $t$ is the correct parameters that produced the outcome witnessed (kind of like a confidence in $t$). Is that correct? $\endgroup$ – deltaskelta May 17 '19 at 0:52
  • $\begingroup$ @deltaskelta The Bayesian formula you wrote $p(t|x) = \frac{p(x|t)p(t)}{p(x)}$ is for finding the posterior distribution of the parameter $T$ by updating your prior distribution $p(t)$ using the evidence of your observation $X=x$. This is close to what you wrote in your original question and what I tried to give as an example in my answer $\endgroup$ – Henry May 17 '19 at 7:29
  • $\begingroup$ Yes. I was able to get the answer I wanted from your explanation. I was asking a much more general question about the implications of what $p(x)$ means when x is the parameters (rules of rhe game, or sides of the die). Anyway thanks for the answer $\endgroup$ – deltaskelta May 17 '19 at 12:26

Well, if you have the joint probability $p_{X,Y}(x,y)$, then $p_X(x)=\sum_y p_{X,Y}(x,y)$ is a marginal probability.

$p_{X|Y} = ( p_{Y|X} * p_X ) / p_Y$ has the form

Posterior = ( Likelihood $*$ Prior ) / Evidence.

  • $\begingroup$ I know that, the formula makes sense, but intuitively I don't know what to think about it when it is something like a training example or a model parameter. If you tell me the probability of rolling a 6 is 1/6 because only 1 out of the 6 outcomes is a 6 then I know what it means. Like my question says $p(x)$ means what intuitively? $p(x)$ out of all possible examples? $\endgroup$ – deltaskelta May 16 '19 at 12:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.