What does the “n choose multiple numbers” symbol stands for?

The question is:

How many ways can you align 3 red balls, 2 blue balls and 2 yellow balls in a row given that at the beginning and the end of the row there is a red ball? (balls of the same color aren't unique).

My solution was - since we have to put two red balls at the beginning and end then we need to order only 1 red ball, 2 blue and 2 yellow. Let's say the order matters, then there are 5! = 120 ways to order the balls. Since we counted as though 2 blue balls differ we need to correct that and divide by 2. Same goes for the yellow balls.

Then the answer is 30.

My professor came up with a different way when he published the answer. Simply: (5 choose 2,2,1) = 30. He didn't explain what this is and how to compute it.

I'm assuming it's an abbreviation for saying - from 5 balls choose 2 blue, 2 yellow and 1 red but can't find any reference to it. Can someone help?

It is denoted, $$\displaystyle \binom{n}{k_1, k_2..., k_m} = \dfrac{n!}{k_1! \times k_2! \times \cdots \times k_m!}$$ and in your case, $(5\;\text{choose}\, 2, 2, 1)$ is denoted and computed as: $$\binom{5}{2, 2, 1} = \frac{5!}{2!\,2!\,1!} = \dfrac{120}{2\cdot 2 \cdot 1} = \dfrac{120}{4} = 30$$
Recall, for any number $n,\,$ $n\,! = n(n-1)(n-2)\cdots(2)(1)$.
After placing a red ball in front and a red ball behind, one can place the remaining two blue, two yellow and one red ball in $\frac{5!}{2!2!1!}$ ways. This is an example of a multinomial coefficient.
$${n\choose{k_1, k_2, ..., k_m}}=\frac{n!}{k_1!\times k_2!\times...\times k_m!}$$