Difficult Coin Toss Probability Problem (Probability, Calculus) I posted a question earlier, but I couldn't speak English, so the explanation was lacking. sorry. (so I'm asking a question again using a translator.)
Anyway, I would like to ask a question about the problem of tossing a coin, which is classified as calculus.
.
.
Problem : If a coin with "probability of coming up heads" p is tossed n1 times, it comes up heads m1 times.
Also, when a coin with "probability of coming up heads" q is tossed n2 times, it comes up heads m2 times.
In this case, find the probability that p < q.
ex ) n1 = 2, m1 = 1, n2 = 4, m2 = 3 : Ans = 5/7
ex2 ) n1 = 8, m1 = 4, n2 = 16, m2 = 8 : Ans = 1/2
ex3 ) n1 = 2, m1 = 0, n2 = 6, m2 = 1 : Ans = 8/15
ex4 ) n1 = 2, m1 = 0, n2 = 2, m2 = 1 : Ans = 4/5
.
.
The equation I established
The equation I established was as follows, but when I substituted n1 = 2, m1 = 0, n2 = 2, m2 = 1, I could see that the equation was wrong because 11/70 came out. (Since the answer needs to be 4/5...) (+ Now that I look at it, I wrote the relationship between p and q in reverse.)
Please help me with how to set up the equation, or solve this problem.
Thank you.
 A: As clarified in the comments, the question can’t be answered without assuming a prior for $p$ and $q$, and the given solutions assume that $p$ and $q$ are uniformly and independently drawn from $[0,1]$.
This sort of problem is often easier to solve with symmetry arguments than with integrals. If you’re not yet familiar with this technique, it might make sense to look at it applied in a somewhat simpler context, e.g.  at How to prove the rule of succession without calculus?, before applying it to this problem.
The key observation is that tossing a coin with success probability $p$ can be replaced by uniformly and independently drawing a number from $[0,1]$, with success if the number drawn is less than $p$.
This allows us to reformulate the problem like this: Uniformly and independently draw $n_1+n_2+2$ numbers from $[0,1]$, the last two being $p$ and $q$. Given that $m_1$ of the $n_1$ numbers are less than $p$ and $m_2$ of the $n_2$ numbers are less than $q$, what is the probability that $p\lt q$?
In this formulation, the exact values of the numbers are irrelevant; all that matters is their order, and all orders are equiprobable by symmetry. So we can reformulate the problem further: Label one ball with $p$, one ball with $q$, $n_1$ balls with $1$ and $n_2$ balls with $2$. What is the probability that in a uniformly random permutation of these $n_1+n_2+2$ balls the ball labelled $p$ comes before the ball labelled $q$, given that $m_1$ balls labelled $1$ come before the ball labelled $p$ and $m_2$ balls labelled $2$ come before the ball labelled $q$?
This is not too difficult to calculate: If $q$ comes after $p$ and there are $k$ balls before $p$, and thus $n_1+n_2+1-k$ balls after $p$, then there are $\binom k{m_1}$ ways to choose the $m_1$ balls labelled $1$ that come before $p$ and $\binom{n_1+n_2+1-k}{n_1-m-1}$ ways to choose the $n-1-m_1$ balls labelled $1$ that come after $p$. (The position of $q$ among the remaining balls is then fixed by $m_2$.) Without the condition $p\lt q$, there are $\binom{n_1+n_2+2}{n_1+1}$ ways to choose positions for the $n+1$ balls labelled $p$ or $1$. (The positions of $p$ and $q$ are then fixed by $m_1$ and $m_2$, respectively.) Thus, the desired probability is
$$
\binom{n_1+n_2+2}{n_1+1}^{-1}\sum_{k=m_1}^{m_1+m_2}\binom k{m_1}\binom{n_1+n_2+1-k}{n_1-m_1}\;.
$$
I don’t think this sum can be simplified. For example, in your first case, with $n_1=2$, $m_1=1$, $n_2=4$, $m_2=3$, we have
$$
\binom{2+4+2}{2+1}^{-1}\sum_{k=1}^{1+3}\binom k1\binom{2+4+1-k}{2-1}=\frac{1\cdot6+2\cdot5+3\cdot4+4\cdot3}{8\cdot7}=\frac57\;,
$$
in agreement with your result.
A: Your computation is close, but the $a$-integral you have in the numerator is extraneous. You should have
$$
\frac{\displaystyle \int_0^1 {}_{n_1}C_{m_1} p^{m_1}(1-p)^{n_1-m_1} \int_p^1 {}_{n_2}C_{m_2} q^{m_2}(1-q)^{n_2-m_2}\,dq\,dp   }{\displaystyle \int_0^1 {}_{n_1}C_{m_1} p^{m_1}(1-p)^{n_1-m_1} \,dp \int_0^1 {}_{n_2}C_{m_2} q^{m_2}(1-q)^{n_2-m_2}\,dq   }.
$$
The question is asking for the conditional probability that $p<q$ given the observed data. The numerator represents the probability that $p<q$ and that we observed what we did, and the denominator represents the probability of observing what we did.
