I think several answers have touched on this, but haven't stated it directly --
One of the features that defines human intelligence (and which has allowed humans to become the pervasive force on the face of the earth) is the ability to form inferences -- to extrapolate from observation to hypothesis. This is innate, and we would not be humans without it, and it's nonsensical to expect students to not employ the technique.
However, in many scenarios (social, political, economic, etc -- not just mathematical) it's possible to jump too quickly from inference to conclusion, bypassing experiment/analysis, and end up being just plain wrong. This in fact happens all the time. (Heck, it probably happened between you and your wife yesterday.) But this is not a reason to stop using such a powerful tool. Rather, students (and non-students) need to be taught (or learn from sometimes bitter experience) that not all inferences are correct, and that while an inference can "inform" a subsequent experiment or analysis, one needs to tread lightly before jumping from inference to (presumed valid) assumption without the intervening experiment/analysis step.
For the OP's situation I would first say, "Lighten up!" This is human nature, don't take it quite so personally! After that it might actually be worthwhile to discuss this aspect of human intelligence with the class in general terms, without direct linkage to mathematics, giving some examples of good and bad inferences from other aspects of life. This might help the students understand that the issue is not about some rigid oddity of mathematics but is about "life skills" that can be applied everywhere.