Combination with repetitions when there is an upper bound on the number of repetitions? Say you have finite number of $3$ different items. You have $3$ As, $7$ Bs, $5$ Cs.  And you want to count the number of combinations (order doesn't matter) you can make by picking $6$ from these items.
For example $BCCABA$.
I want to model this problem as finding the number of solutions to the following equation
$$x_1 + x_2 + x_ 3 = 6$$
where $x_1$ is the number of A's, $x_2$ is the number of B's and $x_3$ is the number of C's.
Normally, I would solve this using stars and bars. However, now there are upper bounds to these variables as the number of items you are picking could exceed the number of As and Cs.
So $0 \leq x_1 \leq 3$, $0 \leq x_ 2$, $0 \leq x_3 \leq 5$
Hence the following arrangement, would be invalid ...
$$****|*|*$$
since you only have $3$ As and you're using $4$ in this case.
How do you solve this problem under these conditions?
 A: We wish to find the number of nonnegative integer solutions of the equation 
$$x_1 + x_2 + x_3 = 6 \tag{1}$$
subject to the constraints $x_1 \leq 3$, $x_2 \leq 7$, and $x_3 \leq 5$.  
In the absence of those constraints, a solution corresponds to the insertion of two addition signs in a row of six ones.  For instance, 
$$1 1 1 1 + + 1 1$$
corresponds to the solution $x_1 = 4$, $x_2 = 0$, and $x_3 = 2$, while 
$$1 + 1 1 + 1 1 1$$
corresponds to the solution $x_1 = 1$, $x_2 = 2$, and $x_3 = 3$.  The number of such solutions is 
$$\binom{6 + 2}{2} = \binom{8}{2}$$
since we must choose which two of the eight positions required for six ones and two addition signs will be filled with addition signs.
The equation
$$x_1 + x_2 + x_3 + \ldots + x_k = n$$
has 
$$\binom{n + k - 1}{k - 1}$$
solutions in the nonnegative integers since we must choose which $k - 1$ of the $n + k - 1$ positions required for $n$ ones and $k - 1$ addition signs will be filled with addition signs.
From these solutions, we must exclude those in which $x_1 > 3$, $x_2 > 7$, or $x_3 > 5$.  Clearly, $x_2$ cannot exceed $7$.  Moreover, the constraints $x_1 \leq 3$ and $x_3 \leq 5$ cannot be violated simultaneously since $4 + 6 = 10 > 6$.
Suppose $x_1 > 3$.  Since $x_1$ is an integer, $x_1 \geq 4$.  Let $x_1' = x_1 - 4$. Then $x_1'$ is a nonnegative integer.  Substituting $x_1' + 4$ for $x_1$ in equation 1 yields
\begin{align*}
x_1' + 4 + x_2 + x_3 & = 6\\
x_1' + x_2 + x_3 & = 2 \tag{2}
\end{align*}
Equation 2 is an equation in the nonnegative integers with 

 $$\binom{2 + 2}{2} = \binom{4}{2}$$

solutions.
By inspection, the only solution with $x_3 > 5$ is $x_1 = x_2 = 0$ and $x_3 = 6$.  Hence, the number of solutions of equation 1 that do not violate the constraints is 

 $$\binom{8}{2} - \binom{4}{2} - \binom{2}{2}$$ 

A: Here is a generating functions approach: The number $N$ you are looking for is the coefficient of the $x^6$ term in  the following product:
$$p(x)=(1+x+x^2+x^3)(1+x+\ldots+x^7)(1+x+\ldots+x^5)\ .$$
Now you can write
$$p(x)={1-x^4\over 1-x}\cdot{1-x^8\over 1-x}\cdot{1-x^6\over 1-x}=(1-x^4)(1-x^8)(1-x^6)\sum_{k=0}{-3\choose k}(-x)^k\ .$$
The first three factors on the RHS amount to $1-x^4-x^6$ plus terms of degree $>6$. We therefore obtain
$$N={-3\choose6}(-1)^6-{-3\choose2}(-1)^2-{-3\choose0}(1)^0=21\ .$$
