Probability with balls and a box - complementary event of “exactly”

There are $12$ balls in a box: $b$ blue, $y$ yellow and $3$ red balls. Three balls are randomly chosen. If the probability of choosing one blue, one yellow and one red is $3/11$, find the number of yellow balls in a box.

Attempt:

If $t$ is the total number of balls in a box, then: $t=3+b+y$

$A$: "We choose exactly one blue, yellow and red ball."

$$P(A)=\frac{b}{t}\cdot\frac{y}{t-1}\cdot\frac{3}{t-2}=\frac{3}{11}$$

Substituting $t$ gives

$$P(A)=\frac{b}{3+b+y}\cdot\frac{y}{2+b+y}\cdot\frac{3}{1+b+y}=\frac{3}{11}$$

Now we have one equation with two unknowns, so we need to define another event.

Because we already know $P(A)$, that new event should be the complementary event of $P(A)$.

How to define that event and how to evaluate it?

We have,

b + y + 3 = 12

b + y = 9

b = 9 - y

Probability = $\frac{\binom{9 - y}1 \times \binom{y}1 \times \binom{3}1}{\binom{12}3}$

$\frac 3{11} = \frac{\frac{(9 - y)!}{(8 - y)!.1!} \times \frac{(y)!}{(y - 1)!.1!} \times \frac{3!}{2!.1!}}{\frac{12!}{3!.9!}}$

$\frac 3{11} = \frac{\frac{(9 - y)(8 - y)!}{(8 - y)!} \times \frac{(y)(y - 1)!}{(y - 1)!} \times 3}{2.11.10}$

$\frac 3{11} = \frac{(9 - y) \times (y) \times 3}{2.11.10}$

$20 = (9 - y)(y)$

$y^2 - 9y + 20 = 0$

On solving yellow y = 4 or y = 5.

Then blue b = 5 or b = 4.

Remember that you are given $t=12$.

Also,

$$\frac{b}{12}*\frac{y}{11}*\frac{3}{10}=\frac{3}{11}$$ is not correct, as the order in which you draw the balls doesn't matter. There are 6 total ways to draw three distinctly colored balls, so your equation should be:

$$6*\frac{b}{12}*\frac{y}{11}*\frac{3}{10}=\frac{3}{11}$$ This leads to: $$by=20$$

Now, since we also know that $b+y=9$, we can conclude there are either 4 or 5 yellow balls.

Forget all of that. You don't need to write $t$ in terms of $b$ and $y$, notice that you can always write $P(A)$ as \begin{align*} P(A) = 3! \cdot \frac{b}{12} \cdot \frac{y}{11} \cdot \frac{3}{10} = \frac{3}{11} \end{align*} (since the order in which you select the balls matters, and there are 3! ways to pick 1 of each color). Simplifying, we now have \begin{align*} by = 20 \end{align*} So you just need to find two integers $b$ and $y$ such that their product is equal to 20 and their sum is equal to 9. Thus the way the question is phrased, $y$ could be either 5 or 4.