Extracting two balls with the same color vs distinct color I am trying to find which has higher probability when extracting from a box without replacement: two balls of the same color, or two balls with different colors.
Let's assume that in the box are $a$ white balls and $b$ black balls.
$$P(\text{distinct color})=P(\text{first white})P(\text{second black/first white})+P(\text{first black})P(\text{second white/first black})$$
$$=\frac{a}{a+b}\frac{b}{a+b-1}+\frac{b}{a+b}\frac{a}{a+b-1}=\frac{2ab}{(a+b)(a+b-1)}$$
$$P(\text{same color})=P(\text{first white})P(\text{second white/first white})+P(\text{first black})P(\text{second black/first black})$$
$$=\frac{a}{a+b}\frac{a-1}{a+b-1}+\frac{b}{a+b}\frac{b-1}{a+b-1}=\frac{a^2+b^2-(a+b)}{(a+b)(a+b-1)}$$
But there is still needed to find which  is more probably  to happen, the case which I don't know how to proceed. Do I have to solve (I don't know how) for which domain of $a$ and $b$ does the following inequality hold true? 
$$a^2+b^2-(a+b)< 2ab \equiv (a-b)^2<a+b $$
This is like in the below answer, so I did it correctly, but still that doesn't help me to conclude an answer.. 
 A: Total number of possible extractions is $\binom{a+b}{2}$
Number of ways to extract:


*

*two white balls: $\binom{a}{2}$

*two black balls: $\binom{b}{2}$

*one of each color: $ab$
Probabilities can be calculated as $$\frac{\binom{a}{2}+\binom{b}{2}}{\binom{a+b}{2}}\text{ for same color and }\frac{ab}{\binom{a+b}{2}}\text{ for different colors}$$
Since you only want to know which has the higher probability, you can simply compare $\binom{a}{2}+\binom{b}{2}$ with $ab$ to see which is greater. Dependent on the values of $a$ and $b$, this could go either way.
Here's a chart detailing which outcome is more likely for $a,b\le30$. A plus (+) indicates that one ball of each color is more likely, a minus (-) indicates that two balls of one color is more likely, and the equal (=) indicates that the probability is split 50-50. Notice that they are equal when $a$ and $b$ are consectutive triangular numbers (1,3,6,10,15...).
Generally, when there are approximately equal numbers of each color, then drawing one ball of each color is more likely.
   1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0
 1 + + = - - - - - - - - - - - - - - - - - - - - - - - - - - -
 2 + + + + - - - - - - - - - - - - - - - - - - - - - - - - - -
 3 = + + + + = - - - - - - - - - - - - - - - - - - - - - - - -
 4 - + + + + + + - - - - - - - - - - - - - - - - - - - - - - -
 5 - - + + + + + + - - - - - - - - - - - - - - - - - - - - - -
 6 - - = + + + + + + = - - - - - - - - - - - - - - - - - - - -
 7 - - - + + + + + + + + - - - - - - - - - - - - - - - - - - -
 8 - - - - + + + + + + + + - - - - - - - - - - - - - - - - - -
 9 - - - - - + + + + + + + + - - - - - - - - - - - - - - - - -
10 - - - - - = + + + + + + + + = - - - - - - - - - - - - - - -
11 - - - - - - + + + + + + + + + + - - - - - - - - - - - - - -
12 - - - - - - - + + + + + + + + + + - - - - - - - - - - - - -
13 - - - - - - - - + + + + + + + + + + - - - - - - - - - - - -
14 - - - - - - - - - + + + + + + + + + + - - - - - - - - - - -
15 - - - - - - - - - = + + + + + + + + + + = - - - - - - - - -
16 - - - - - - - - - - + + + + + + + + + + + + - - - - - - - -
17 - - - - - - - - - - - + + + + + + + + + + + + - - - - - - -
18 - - - - - - - - - - - - + + + + + + + + + + + + - - - - - -
19 - - - - - - - - - - - - - + + + + + + + + + + + + - - - - -
20 - - - - - - - - - - - - - - + + + + + + + + + + + + - - - -
21 - - - - - - - - - - - - - - = + + + + + + + + + + + + = - -
22 - - - - - - - - - - - - - - - + + + + + + + + + + + + + + -
23 - - - - - - - - - - - - - - - - + + + + + + + + + + + + + +
24 - - - - - - - - - - - - - - - - - + + + + + + + + + + + + +
25 - - - - - - - - - - - - - - - - - - + + + + + + + + + + + +
26 - - - - - - - - - - - - - - - - - - - + + + + + + + + + + +
27 - - - - - - - - - - - - - - - - - - - - + + + + + + + + + +
28 - - - - - - - - - - - - - - - - - - - - = + + + + + + + + +
29 - - - - - - - - - - - - - - - - - - - - - + + + + + + + + +
30 - - - - - - - - - - - - - - - - - - - - - - + + + + + + + +

A: Your inequality
\begin{align*}
(a-b)^2< a + b
\end{align*}
can be expressed as
\begin{align*}
a - \sqrt{2a + \frac{1}{4}}+\frac{1}{2} < b < a + \sqrt{2a + \frac{1}{4}}+\frac{1}{2}
\end{align*}
or
\begin{align*}
\left|b - a - \frac{1}{2}\right| < \sqrt{2a + \frac{1}{4}}
\end{align*}
This domain isn't pretty in any way, but we can make statements about its behavior as $a,b \rightarrow \infty$. For example, dividing both sides by $a$,
\begin{align*}
\left|\frac{b}{a} - 1 - \frac{1}{2a}\right| < \sqrt{\frac{2}{a} + \frac{1}{4a^2}}
\end{align*}
Therefore, in order for the probability of distinct colors to be greater than probability of the same, we need $b = a$ for sufficiently large values of $a$.
