The number of ways of choosing $k$ objects from $n$ choices is given by
which is known as a binomial coefficient (see also here).
Also, if you have $x$ options for decision #1, and $y$ options for decision #2, and which choice you make in either decision does not affect the other (i.e., they are independent decisions), then there are a total of $xy$ options all together.
So, if you want to pick 3 math courses out of 7, the number of options you have for the math class decision is
The number of ways of picking 2 science courses out of 4 is
The number of ways of picking 1 language or history course out of a total of $7=4+3$ language and history courses is
Now the total number of options you have is simply
$$35\cdot 6\cdot 7=1470.$$
The question gets a bit tougher when your choices are not independent, like in your variation (if you choose history courses in your second decision, you might be forced to choose language courses in your third decision). However, it's not exactly clear to me what your variation means; when you say you want "2 science or history courses", does that mean you either want 2 science courses or 2 history courses, or does it mean that you want to choose any 2 courses from the total of 7 science and history courses?