When a spade is not a spade Typically, when one sticks an adjective in front of a noun, the resulting noun phrase refers to a subclass of the things that the bare noun refers to.  For example, a red truck is a special type of truck.
There are certain standard exceptions, notably when the adjective serves a negating or broadening function.  For example, a near success is not a success, and a would-be intellectual is not an intellectual.
Some of the most confusing terms in mathematics are those which violate the above principles.  Three (admittedly rather arcane) examples that come to mind are:

*

* A quantum group is not a group

* A perverse sheaf is not a sheaf

* A Boolean-valued model is not a model

What are some other examples?  I feel that there are probably many examples that I've gotten so used to that I no longer notice the "illogicality."  I think it would be useful to compile a list of these so that people who teach math can be aware of them, and point out the possible confusion to students.
Note that examples involving adjectives such as "pseudo," "quasi," "almost," etc., don't really count in my book because these adjectives are widely understood to negate or partially negate the noun in question.
EDIT: Here is another example that occurred to me:
A fractional ideal
is not necessarily an ideal.
 A: A Hilbert-basis is not a basis.
A: A rational function is typically not a function.
A: A right inverse is not necessarily an inverse! Likewise, if a functor is left exact there is no guarantee it is exact. Watch out for left/right!
A: The signed measures, the outer measures, and the inner measures are supersets, not subsets of the measures.
Not sure if this counts, but ergodic theory concerns a great deal more than ergodicity or ergodic theorems.
I don't think there's anything unusual about this.  It's common for adjectives or noun pairs to create meanings other than by restriction.  I don't recall the linguistic terms for the different roles such pairings created, but I've come across some of them.
Further, in math, you can pick a phrase that kind-of sort-of feels right, and then just give it a definition, in virtue of which it thereby becomes precisely correct.
(fwiw, I've always suspected that a several things that aren't entirely intimately connected are called "ergodic" because people thought that "ergodic" sounded cool.)
A: This phenomenon is sometimes referred to as the "red herring principle".  This phrase is used, for example, on a page on nlab (a wiki for category theory).  Knowing the phrase might help you find more examples.
Examples are common in category theory, and I'm surprised that the nlab page doesn't list very many of these.  Here are some I know of:

*

*There are various concepts called "[something] category" that are not (in general) categories, e.g. enriched category, internal category, double category.  If you're including prefixes, you can add bicategory, tricategory, n-category and multicategory to that list.

*Various concepts in category theory can be weakened or made lax (laxened?), and a "weak [something]" or "lax [something]" is not normally a "[something]", e.g. a weak n-category is not an n-category, a weak limit is not a limit, a weakly initial object is not an initial object, a lax monoidal category is not a monoidal category, a lax functor is not a functor, etc.  These kinds of examples can get pretty esoteric, e.g. lax trimodification.

*A simplicial set is not a set; likewise a globular set is not a set.  There might be similar examples with other types of presheaf.

Additionally, I think there's an extra level of implicit subtlety to the question.  In maths, the answer to the question "is a red herring a herring?" is not a simple yes or no – as I see it, there are three possible cases:

*

*A red herring is a herring with the property of being red.

*A red herring is a herring equipped with some extra structure that makes it red.

*A red herring is not a herring.

I think case 2 is a grey area, neither a definite yes nor no.  To give a mathematical example, a monoidal category is a category equipped with a monoidal product (a piece of extra structure).  Thus a monoidal category has an underlying category, but strictly speaking perhaps one shouldn't say that it is a category, in the same way that one wouldn't say a group is a set.  Depending on whether you think case 2 is a yes or a no, this might help you find more examples.
A: Almost Upper Bound is an example that is a broadening function. Even more confusing that the word "almost" is typically not broadening. Caused me quite a headache a few days ago (see here: Why is an almost upper bound named so?)
A: I wrote a blog post about this a while back. Examples include:

*

*A partial function is not (in general) a function

*A contravariant functor is not a functor

*A hom-set is not a set

*A skew field is not a field

*A snub cube is not a cube

*A quantum group is not a group

*A Gaussian integer need not be an integer

*At one time, affine spaces were known as "affine vector spaces", despite not usually being vector spaces.

As Mars says in another answer, this is common outside of mathematics also.  For example, a toy ball is a ball, but a toy fire engine is not a fire engine.
A: Manifold-with-boundary is not (unless the boundary is empty) a manifold, a persistent source of confusion.
Also: "delta function." Sigh.
Others please feel free to add your contributions.
A: An ill-defined function is not a function.
