# What is p(Evidence) exactly in a bayesian model?

I'm having a hard time intuitively understanding what this means in a machine learning context. When using the variables $$A$$ or $$B$$ or some trivial example, it all makes sense, but when looking at machine learning formulas where there are real variables its harder to see exactly what is meant. For example, if $$t$$ is what I am trying to predict and $$x$$ is the training example or input...

$$p(t|x) = \frac{p(x|t)p(t)}{p(x)}$$

What is meant by $$p(x)$$? if $$x$$ is a training example, does it mean the probability of seeing $$x$$ out of all possible training examples (kind of like the probability of drawing $$x$$ from a hat)? the probability of seeing $$x$$ out of the previously known distribution of examples? or something else?

Sometimes I see this with model parameters such as $$\theta$$ as well which raises the same sort of questions.

• You have a prior distribution for $t$ as well as a conditional probability function for $x$ given $t$ so you could say $p(x)=\int p(x \mid t) \, p(t) \, dt$ changing the integration to a summation if $t$ has a discrete distribution. You are integrating or summing over the parameter $t$, not over the possible observations $x$ – Henry May 16 '19 at 13:16
• @Henry, in your comment you said "probability function for 𝑥 given 𝑡," my question is asking how to intuitively understand what the probability of $x$ is? How can I have a probability of a training example or a model parameter? what does that mean? what am I measuring? – deltaskelta May 16 '19 at 13:26
• deltaskelta - I have tried to give an example in an answer. Here $t$ is a value your uncertain parameter can take, while $x$ is a value your observation can take – Henry May 16 '19 at 14:16

Let's take your dice example to try to illustrate the issue. Here $$T$$ is your uncertain parameter and $$t$$ a value it can take, while $$X$$ is your observation and $$x$$ a particular value it can take.

• Suppose you have a $$t$$-sided fair die, but you do not know what value $$t$$ has. You do have a prior distribution for $$t$$ of $$P(T=t) = \frac{t}{2^{t+1}}$$ for $$t \in \{1,2,\ldots\}$$.

• You roll the die and observe a value of $$X=x$$. Since this is a fair die, you know $$P(X=x \mid T=t) = \frac{1}{t}$$ for $$x \in \{1,2,\ldots\,t\}$$

• You can at this stage ask what is the unconditional $$P(X=x)$$? In other words, at the start what do you think the probability is of rolling a particular value $$x$$ even though you do not know how many sides the dice has? As a simple application of conditional probability $$P(X=x) = \sum P(X=x \mid T=t) P(T=t) = \sum\limits_{t=x}^\infty \frac{1}{2^{t+1}} = \frac{1}{2^{x}}$$

As examples, from the first bullet $$P(T=6)=\frac{6}{128}$$ and $$P(T=7)=\frac{7}{256}$$ etc. So the unconditional or marginal probability of rolling $$X=6$$ is $$P(X=6) = \frac{1}{6} \times \frac{6}{128} + \frac{1}{7} \times \frac{2}{256}+ \cdots = \frac{1}{64}= \frac{1}{2^6}$$

If you do roll a $$6$$ then you then know the number of sides $$T \ge 6$$, and you get a posterior probability mass function $$P(T=t \mid X=6) = \frac{\frac{1}{2^{t+1}}}{\frac{1}{2^{6}}} = \frac{1}{2^{t-5}}$$ for $$t \ge 6$$, so $$P(T=6 \mid X=6)= \frac12$$, $$P(T=7 \mid X=6)= \frac14$$, etc.

• You reversed my variables so now the question is involving $t$. So I can say that $p(t)$ is the probability that $t$ is the correct number of sides for the die? In the case of $theta$ parameters, it would the the probability that they are the true $theta$ parameters¿ – deltaskelta May 16 '19 at 23:04
• @deltaskelta In my statement, I treated $X$ as the observation/evidence/die throw, which is what I thought you did in your question – Henry May 17 '19 at 0:10
• I may have used the wrong word somewhere, but my intention was that $x$ was something like the parameters of the dice or $t$ in your example, in which case $p(t)$ means the probability that $t$ is the correct parameters that produced the outcome witnessed (kind of like a confidence in $t$). Is that correct? – deltaskelta May 17 '19 at 0:52
• @deltaskelta The Bayesian formula you wrote $p(t|x) = \frac{p(x|t)p(t)}{p(x)}$ is for finding the posterior distribution of the parameter $T$ by updating your prior distribution $p(t)$ using the evidence of your observation $X=x$. This is close to what you wrote in your original question and what I tried to give as an example in my answer – Henry May 17 '19 at 7:29
• Yes. I was able to get the answer I wanted from your explanation. I was asking a much more general question about the implications of what $p(x)$ means when x is the parameters (rules of rhe game, or sides of the die). Anyway thanks for the answer – deltaskelta May 17 '19 at 12:26

Well, if you have the joint probability $$p_{X,Y}(x,y)$$, then $$p_X(x)=\sum_y p_{X,Y}(x,y)$$ is a marginal probability.

$$p_{X|Y} = ( p_{Y|X} * p_X ) / p_Y$$ has the form

Posterior = ( Likelihood $$*$$ Prior ) / Evidence.

• I know that, the formula makes sense, but intuitively I don't know what to think about it when it is something like a training example or a model parameter. If you tell me the probability of rolling a 6 is 1/6 because only 1 out of the 6 outcomes is a 6 then I know what it means. Like my question says $p(x)$ means what intuitively? $p(x)$ out of all possible examples? – deltaskelta May 16 '19 at 12:40