Complex combinatorics with colors 
In the picture above, I've created an example. We've got eight squares being chosen from, and five squares choosing, $8 \choose 5$. It may look like only four squares, but that rightmost square is actually two squares, sharing a spot.
So, in this method of combination, two squares can occupy the same spot. However, no more than two squares can occupy the same spot.
However, there is more: The different colors don't have the same reach. So, for example, the color blue can exist anywhere along the 8 squares. The color green can exist anywhere along the inner 6 squares. The color red can exist anywhere along the inner 4 squares. The subtractor is always $2n$, because whatever
$n$ is, it is subtracted from both ends.
So, to clarify my illustration: Above here, the number being chosen from, or the element set, is 8. The number of choosing is 5. Those 5 squares have 3 different colors. Those colors come with restrictions. It is also possible for two different squares, whether of the same color or a different one, to occupy the same spot.
So, is there an equation or formula that can do this: there are a lot of variables here, so I understand if it is too complex for any general formula. The variables are: the number of squares in the main set (which is 8 here), the number of squares in the sub-set (which is 5 here), the number of colors (which is 3 here) and the individual subtractors for the different colors.
However, the number of subtractors is always the same the number of colors. The colors are different precisely because they have different restrictions. Blue has the subtractor $0$, green has the subtractor $2$ and red has the subtractor $4$.
EDIT: So what I'm looking for is a general formula that gives me the number of combinations given these restrictions. The input values are the length of the row, the number of colors and the number of squares. The colors carry with them a value, which is the value subtracted from both sides of the row. So, is there a formula for this?
 A: There are 18,948 possible colorings for your example.
This is based on the assumption that

*

*There are three colors.

*There are eight spaces.

*We must choose and color five spaces from these eight.

*Each space can be chosen up to two times.

*When a space is chosen twice, order does not matter. In this case, [red, blue] is equivalent to [blue, red]. (I wasn't sure whether you wanted this rule, so I guessed. Feel free to clarify.)

*Colors follow the specific palette rule you specify (the first color can be used everywhere; the next color can be used in all but the outer two squares; the next color can be used in all but the outer four; and so on.)

I calculated this by obtaining a recursive formula for counting two quantities:

*

*$f(k,m)$ counts how many ways there are to color exactly $2k$ blocks according to the above rules, when you can make $m$ selections. By coloring the middle two squares and working your way out, you can write this as a recursive sum in terms of $f(k-1, m)$, $f(k-1, m-1)$, $f(k-1, m-2)$, $f(k-1, m-3)$.

*$q(k, m, N)$ counts how many ways there are to color $N$ blocks when you can make $m$ selections according to the above rules— but we drop the palette requirement and instead allow all $k$ colors throughout. By coloring the middle square (if $N$ is odd) or middle two squares (if $N$ is even) and working your way outward in pairs, you can similarly get a recursive formula for $q$.

Then if $S$ is the total number of blocks, $k$ is the total number of colors, and $m$ is the total number of selections to be made, your total counting formula is:
$$\sum_{j=0}^{2N} q(k, j, S-2(k-1)) \cdot f(k-1, m-j) $$
which in your specific case $(k=3; m=5; N=8)$ is:
$$18948 = 78 + 864 + 3588 + 6804 + 5670 + 1944 + 0 + 0$$
Details follow.

You can build up a full solution by relaxing some of the constraints.
We can start from a simpler problem: suppose you have $k$ colors and $2k$ blocks. The middle 2 blocks can be colored by all $k$ colors. The two beyond that can be colored with $k-1$ colors. The two beyond that can be colored with $k-2$ colors, and so on until the outer two can only be colored with one remaining color.
We can find the function $g(k,m)$ which counts how many ways there are to color such a system if you have $m$ spaces left to color, and $k$ colors left in your palette. (Unlike in the original problem, each square can be picked at most once.)
We color from the inside outward, which gives us a recursive coloring procedure. First we consider the innermost 2 spaces. We decide whether we will pick 0, 1, or 2 of them to color. Then we decide what color they will be. We deduct the number of spaces we've chosen from our number $m$ of remaining spaces, then proceed to the next outermost spaces.  This gives us the recursive step of our recursive relationship where at each step we're considering a pair of squares: to pick $m$ out of $2k$ spaces to color, first decide whether to pick zero, one, or two of the middle spaces; color each one with one of the $k$ colors; then color the remaining outer spaces.
If, during our coloring process, we run out of spaces to color ($m>2k$), we have failed. If we color too many $(m<0)$, we have failed. If we run out of spaces just as we run out of colors, we successfully finish $(m=k=0)$. These give us the base case of our recursive relationship.
Altogether, we have:
$$g(k,m) = \begin{cases}0 & \text{if }m < 0\\ 1 & \text{if }m=0\text{ and }k=0\\0 & \text{if }m>0\text{ and }k=0\\ \sum_{i=0}^2 k^i {2 \choose i} g(k-1,m-i)&\text{otherwise}\end{cases}$$
Here, $k$ is the number of colors left in the palette; $m$ is the number of spaces we can choose; $i$ is the number of spaces you've chosen to color from the pair of spaces you're considering this round. The recursive step formula is more simply written
$$g(k-1, m) + 2k\,g(k-1, m-1) + k^2\, g(k-1,m-2).$$
Starting from smaller values and working your way up, you can build a table of values of $g$ for each choice of $k$ and $m$; these will count the possible ways of choosing and coloring $m$ of the $2k$ spaces according to the palette constraints of the game.

Now how to modify this to achieve the formula for your more complicated game? If the innermost color can appear in more than two spaces—e.g. the red color in your example can appear in four spaces—then we must adjust the first term in our recurrence relation. Let $h(k,m,N)$ denote the number of ways to pick $m$ of the available spaces and color them with a palette of $k$ colors according to the rules, when the rarest color has $N$ available spaces.
Then $h(k,m,N)$ is just $$\sum_{j=0}^N {N \choose j} k^j \cdot g(k-1,m-j)$$
which recurses into the formula for $g$ that we've already described.
What if we want to allow each space to be selectable up to two times? Well, naively we could add $g(k,m-i)$ and $g(k,i)$ together ("choose $i$ spaces the first time, then choose $m-i$ potentially overlapping spaces) but this would result in double-counting; more finesse is required.
Assuming that when two colors are in the same space, the order doesn't matter (so, [red blue] in the same space is the same as [blue, red]), we have the following counting results:
When you have two spaces under consideration and each space can be colored up two twice, and your palette has $k$ colors:

*

*You can pick just one space and color it. There are $2k$ possibilities.

*You can pick two spaces to color. In particular, you could pick two different squares, with $k^2$ possible colorings. Or you could pick two of the same square— either both in the left square or both in the right— with $k(k+1)/2$ colorings each. In total, there are $k^2+2k(k+1)/2 = k^2+k^2+k = k(2k+1)$ colorings when you select two spaces.

*You can pick three spaces to color. Two will necessarily be in a single square [$k(k+1)/2$ options there] and one will be alone [$k$ options]. Exchanging whether the double-occupancy is on the left or right doubles the number of possibilities. In total, there are $k^2(k+1)$ coloring possibilities.

*You can pick four spaces to color. There are $k(k+1)/2$ for each double-occupied space, which can be picked independently of the other double-occupied space. Hence there are $k^2(k+1)^2/4$ possibilities.

*You can pick zero spaces to color. It's allowed; there's just one way to do that.

Hence the number of ways to pick and color $m$ positions out of $2k$ squares, where each square has two indistinguishable spaces, and you have a palette of $k$ colors, is given by the recurrence:
$$\begin{align*}f(k-1, m)&=f(k-1, m) \\ &\;+\; 2k\,f(k-1, m-1)\\ &\;+\; k(2k+1)\,f(k-1, m-2)\\ & \;+\; k^2(k+1)\,f(k-1, m-3) \\&\;+\; k^2(k+1)^2/4\, f(k-1,m-4).\end{align*}$$
