How many 7-note musical scales are possible within the 12-note system? This combinatorial question has a musical motivation, which I provide below using as little musical jargon as I can. But first, I'll present a purely mathematical formulation for those not interested in the motivation:
Define a signature as a 7-tuple over the set $\{1,2,3\}$ such that the sum of the elements in the tuple is $12$. Two signatures are said to be equivalent if they are either identical or there is a circular shift relation between them (i.e. one can be circularly shifted between 1 and 6 times to obtain the other).
How many unique signatures are there?

Most modern, Western music is based on the equal temperament system, which divides the octave logarithmically equally into 12 notes.
Let's refer to 2 adjacent notes from these 12 as being "1 step" apart, two notes with one skipped note in between them as being "2 steps" apart, and so on.
The (arguably) most natural scale is the major scale, which uses 7 notes from these 12, and has the following signature:
$$\text{major signature}=(2,2,1,2,2,2,1)$$
This means that we can construct a major scale as follows: given any note to start from, the second note is 2 steps away from first note, the third note is 1 step away from the 2nd note, and so on according to the above signature.
Now this signature really has not just one, but seven scales embedded in it. This is because we can circularly shift the signature, effectively meaning that we are picking a different degree of the major scale to serve as our home note (these seven 'permutations' are called the modes of the signature). Thus, for example, $(1,2,2,2,1,2,2)$ is not a new signature, but just the major signature circularly shifted left twice (and is called the Phrygian mode).
My question is: how many unique 7-note signatures are there under the restriction that any signature must not contain an interval greater than 3 steps (this is to respect the fact that any 7-note scale in common use uses only 1, 2 and 3 step intervals, to the best of my knowledge).
Some signatures in common use are:
$$\text{harmonic minor signature} = (2,1,2,2,1,3,1)$$
$$\text{melodic minor signature} = (2,1,2,2,2,2,1)$$
$$\text{harmonic major signature} = (2,2,1,2,1,3,1)$$
The number of possible 7-note scales within the 12-note system is simply given by multiplying the number of unique signatures by 7.

 A: Since $7$ is prime and there is no $7$-note signature that sums to $12$ with all $7$ steps identical, we don't have to worry about periodicity; we can just divide by $7$ in the end. Thus, we just have to count the number of ways of distributing $12-7=5$ balls into $7$ bins with capacity $3-1=2$. There are $\binom75=21$ ways to have $5$ steps of $2$, $\binom7{1,3,3}=140$ ways to have $3$ steps of $2$ and $1$ step of $3$, and $\binom7{2,1,4}=105$ ways to have $1$ step of $2$ and $2$ steps of $3$, for a total of $21+140+105=266$ scales in $266/7=38$ cyclically inequivalent types.
In the present case, inclusion-exclusion would be a bit of an overkill, but since you said you'd like a method that generalises to any number of notes with any number of maximum steps, let's generalise: For $k$ notes with a maximum of $m$ steps that sum to $12$, we want to distribute $12-k$ balls into $k$ bins with capacity $m-1$. As explained at Balls In Bins With Limited Capacity, inclusion-exclusion yields a count of
$$
\sum_{t=0}^{12-k}(-1)^t\binom{12-k}t\binom{12-k+k-tm-1}{12-k-1}=\sum_{t=0}^{12-k}(-1)^t\binom{12-k}t\binom{11-tm}{11-k}\;,
$$
where, contrary to convention, binomial coefficients with negative upper index are taken to be zero. For the present case of $k=7$, $m=3$, this again yields
$$
\sum_{t=0}^7(-1)^t\binom7t\binom{11-3t}6=\binom{11}6-\binom71\binom86=266
$$
signatures. If $k$ isn't prime, or if it divides $12$, then you have to do a bit more to deal with periodicity; otherwise, you can just divide the above result by $k$.
A: We begin by writing a list of partitions of $12$ (with the numbers in descending order).  The restricted partitions of $12$ into specifically seven parts with maximum part size $3$ are the following:
$3,3,2,1,1,1,1$
$3,2,2,2,1,1,1$
$2,2,2,2,2,1,1$
The list was constructed with the number of threes used in mind.  It is clear that no other partitions match the desired conditions.
As we are counting the number of arrangements where cyclic shifts are irrelevant, we notice that those that fall into the first two cases can be described with respect to the location of the number that occurs exactly once.  Without loss of generality, let us count then the number of these where the unique number occurs at the beginning.
There are then, $\binom{6}{2}=15$ and $\binom{6}{3}=20$ of these respectively
The final case can be counted by considering the smallest distance between the ones.  The smallest distance is always either $1,2$ or $3$, corresponding to $(1,1,2,2,2,2,2),(1,2,1,2,2,2,2),(1,2,2,1,2,2,2)$ respectively.
There are then a total of $38$ possible heptatonic scales.
A: Here is another method. A scale contains the tonic, by definition, and 6 of the other 11 pitch classes. There are $\binom{11}{6}=462$ ways to choose these. As joriki notes, none of these are modes of limited transposition, so the number of signatures is $1/7$ of this, namely $66$. But from these, we must remove:


*

*$(6,1,1,1,1,1,1)$

*the $6$ ways to substitute $2$ for a $1$ in $(5,1,1,1,1,1,1)$

*the $6$ ways to substitute $3$ for a $1$ in $(4,1,1,1,1,1,1)$

*the $\binom{6}{2}=15$ ways to substitute $2$ for two of the $1$s in $(4,1,1,1,1,1,1)$


leaving $66-1-6-6-15=38$ signatures with no interval greater than $3$ semitones.
A: Here is another approach, similar to how JMoravitz did but focusing on half-steps (minor seconds), not minor thirds
Consider a note a collection of frequencies, a mode a collection of notes, and a scale a collection of modes.  Hence, CDEFGAB is a collection of notes, the Ionian mode, and its seven cyclic permutations reflect the Major scale.  These definitions are unambiguous and in line with musical usage.
Now, consider family of scale as any scale (collection of 7 modes) that has the same number of half-step in their spelling.  You get the following families:


*

*Step-Scale: 2 half-steps, 3 member. (spelled with  m2 and M2)

*Skip-Scale: 3 half-steps, 15 members (spelled with m2, M2, and m3)

*Double-Skip-Scale: 4 half-step, 20 members (spelled with m2, M2, m3, m3)


So these are the 38 scales and 266 modes of the initial question.   I only mention this method because it allows you to easily extend this other families:


*

*Major-Skip-Scale: 4 half-steps, 15 members  (spelled with m2, M2, M3)

*Major-Minor-Skip-Scale: 5 half-steps, 6 members  (spelled with m2, m3, M3) 

*Tritone-Scale: 6 half-steps, 1 member (spelled with m2 and one TT)


and shows that the total number of possible heptatonic scales is 60 encompassing a total of 420 modes.
A: I have been looking at scales with only half-steps and whole-steps, so a subset of the ones mentioned in the above discussion. I wondered if there were any other such scales besides the Major and Melodic Minor scales and their associated modes. What I came up with is that there is only one other possibility, and I'm curious if I'm correct. That other scale is one I can't find a name for. It's intervallic signature is 1222221, so that the two half-steps are on either end of the scale, or adjacent when you play multiple octaves. It can be described as Phrygian with a raised 6th AND 7th, or melodic minor with a flat 2nd, but I can find no mention of the scale anywhere.
Some observations:
- There must be exactly 2 semitones in a scale with only half and whole steps. I like the mathematical way this is shown in previous posts, which I was not previously aware of. The addition of a third half-step, or having only a single half-step makes the total less than or greater than 12, respectively.
- The distance between the two semitones is a convenient way to characterize the different possibilities of parent scale type. In Major/Ionian, the distance is 4 (or its complement, 3). In Melodic Minor, the distance is 5 (or its complement, 2). Looking at it this way was how I came to the conclusion that there is only one other possibility, the aforementioned scale, which has a distance between its two semitone intervals of 6 (or its complement, 1). It is clear that there are no other ways to "slice" a scale with only half and whole steps.
Is this analysis correct?
If so, I wonder why the third type (Melodic Minor b9??) hasn't, as far as I know, been used much in music.
By the way, it has a rather interesting harmonic aspect. The 2nd and 3rd degrees are both augmented triads, the 5th degree is diminished, and the 6th degree is half-diminished.
A: Keep in mind that there are other musical scales that will not be as outrageous as those generated by combinatorics.  For example, in African American blues, we have gaps of 3,2,2,3,2 halftones.   Somwetimes a tritone (half octave) is added.  Some Spanish music is based on the Locrian mode.   
A good source of alternate scales is the "DeTwelveulate" album by Ivor Darreg.   Some is nice, some is really strange.
A: Were you so minded (and it is optional!), you may further reduce the 38 scale 'shapes' to 20 by considering that 32 of them turn up as 16 inverse pairs where each pair's 'signature' (or interval string as it's otherwise known) is the reverse of the other. The remaining 6 signatures, when reversed, are identical (the diatonic scale/modes is one of these 6 symmetric ones). It all depends on whether you're willing to consider that - for example - the Locrian Ultra (1212213) might be considered the same 'thing' as the Indian (1213122) as both are within 'modal reversal' range of each other.
But I suppose most listeners, at least, would regard them as completely different 'scales' despite one's 'shape' being just a mirror image of the other's.
A: Some of the math in the other answers is truly beautiful and worth appreciating for its own sake. But if all one wishes is the answer to the original question and others similar to it, they are all in the Scale-Chord Synopticon (see, e.g., Amazon.com or StrunzAndFarah.com; I should mention that I am a co-author but no longer affiliated with the company, and I do not receive royalties, but I retain my original purpose for participating, namely to expand the tools available to musicians).
This book introduced a new musical concept, the scale family, a description of the number of intervals of various sizes in a given scale. For example, the major scale is in family 52, meaning that it has two minor seconds and five major seconds. Harmonic minor is in family 133. The least significant digit is the number of minor seconds, next to it is the number of major seconds, one position higher is the number of minor thirds, etc.
The Synopticon shows 66 heptatonic scales, denoted 7-1 to 7-66, forming families 52, 133, 214, 1024, 1105, 10015, and 100006. Since the original question was clarified to involve no intervals higher than a minor third, the book shows that these are 7-1 through 7-38, i.e., 38 scales as found in other answers herein. The other 28 scales are clearly quirkier than these 38, but one purpose of the book is to be mathematically complete, leaving no question in the reader’s mind regarding the possibility that something else exists but was not mentioned. The context of this completeness is 12-tone scales with five to nine notes per octave and repeating over the octave (i.e., not Slonimsky scales; for that one might take a look at the “Albonimsky” program).
Another advantage of the book is that it is also mathematically complete regarding chords, although in a slightly different sense, since tradition has given us “equivalent” chords like the Maj6 and min7 (e.g., C Maj6, CEGA, and A min7, ACEG), and “chords” with more than five notes are more unruly than most, having in some cases to be referred to their equivalent scales.
All these scales are cross-referenced to the chords they contain, and vice versa. I have found exploring the scale families useful for obtaining some familiar scale “feeling” but with some refreshing differences. For example, I am partial to Harmonic Minor, but there are 19 other scales in the 133 family, all with sibling-like relationships.
A: I know this thread is a bit old, but for completeness, "somebody" (unfortunately I've lost the OP), wrote a parametric google sheet that builds every possible 12-tone scale: All Possible Scales
The section for 7-note scales does have 462 rows, because the the 7 scales that contain CDEFGAB, would be unique if your "starting point" (aka tonal center) was any of the individual note names (e.g. CDEFGAB (C-Major) is unique from ABCDEFG (A-Minor)).
