I'm over here from the philosophy page since a very similar question that I asked there a couple times wasn't ever properly answered and I think statisticians here might be able to provide a helpful answer. It has to do with the use of statistical (primarily Bayesian) inferences as applied to scientific inquiry as a whole.
There is an argument in philosophy known as the "bad lot" objection made by a guy called Bas van Fraassen. His argument goes like this. You have some set of data or evidence that you want to explain so you (naturally) generate some set of hypotheses (or models, what you want to call them) and see how these hypotheses hold up to the data you have and test their predictions. Eventually one hypothesis may come out clearly on top and generally in science we may consider this hypotheses true. Since this hypothesis has beaten its rivals and been well confirmed by the evidence (according to Bayes' theorem), we will want to consider this hypothesis an accurate representation of reality. Often, this will mean that we have inferred the truth of processes that we have not directly observed. Van Fraassen's objection to this form of reasoning is that we may just have the best of a bad lot. That is, due to limitations on human creativity or even bad luck, there may be possible hypotheses that we have not considered which would be just as well if not better confirmed by the evidence than the one we currently hold to be the best. Since we have no reason to suppose that absolute best hypothesis is among those generated by scientists, we have no reason to believe that the best hypothesis of our set is an accurate representation of what's going on.
This seems like a hard hit to most forms of inquiry that involve hypothesis generation to account for data in order to gain knowledge about a system.
I have seen in multiple papers a suggestion which is supposed to 'exhaust' the theoretical space of possibilities. Namely, by using a "catch-all" negation hypothesis. These are primarily philosophy papers but they make use of statistical tools. Namely, again, Bayesian statistical inference. If you can get access to this paper, the response to van Fraassen's argument begins on page 14. This paper also treats the argument very quickly. You can find it if you just do a search for the term "bad lot" since there is only one mention of it. The solution provided is presented as trivial and obvious Bayesian statistical solution. There are other examples of this solution being presented in academic philosophy too.
So suppose we have some set of hypothesis:
H1, H2, H3...
We would generate a "catch-all hypothesis" Hc which simply states "all other hypotheses in this set are false" or something along those lines. It is the negation of the rest of the hypotheses. The most simple example is when you have one hypothesis and its negation ~H. So your set of hypotheses looks like this:
Since the total prior probability of these hypotheses sums to 1 (this is obvious), we have successfully exhausted the theoretical space and we need only consider how these match up to the data. If P(H) ends up considerably higher than P(~H) according to Bayesian updating with the evidence, we have good reason to believe that H is true.
All of this makes very intuitive sense, of course. But here is what I don't understand**:**
If you only have H and ~H, are there not other possible hypotheses (in a way)? Say for example that you have H and ~H and after conditionalizing on the evidence according to Bayes theorem for a while, you find H comes out far ahed. So we consider H true. Can I not run the same argument as before still, though?
Say, after doing this and concluding H to be successful, someone proposes some new hypothesis H2. H2 and our original hypothesis H are competing hypotheses meaning they are mutually exclusive. Perhaps H2 is also very well confirmed by the evidence. But since H2 entails ~H (due to H and H2 being mutually exclusive) doesn't that mean that we wrongly thought that ~H was disconfirmed by the evidence? Meaning that we collected evidence in favour of H but this evidence shouldn't actually have disconfirmed ~H. This seems very dubious to me.
I'm sure I don't need to but I'll elaborate with an example. A very mundane and non-quantitative example. One that might (does, I would argue) take place quite often.
I come home from work to find my couch ripped to shreds and couch stuffing is everywhere. I want to explain this. I want to know why it happened so I generated a hypothesis H.
H: The dog ripped up the couch while I was at work.
My set of hypotheses, then is H and ~H. (~H: The dog did not rip up the couch while I was at work).
Lets say that I know that my dog looks obviously guilty (as dogs sometimes do) when they know they've done something wrong. So that means H predicts fairly strongly that the dog will look guilty. When I find my dog in the other room, he does look extremely guilty. This confirms and increases the probability of H and disconfirms, decreasing the probability of ~H. Since P(H) > P(~H) after this consideration, I conclude (fairly quickly) that H is true.
However, the next day my wife offers an alternative hypothesis which I did not consider H2.
H2: The cat ripped up the couch while you were out and the dog didn't rip up the couch but did something else wrong which you haven't noticed.
This hypothesis, it would seem, predicts just as well that the dog would look guilty. Therefore H2 is confirmed by the evidence. Since H2 entails ~H, however, does that not mean that ~H was wrongly disconfirmed previously? (Of course this hypothesis is terrible. It assumes so much more than the previous one, perhaps giving us good reason to assign a lower prior probability but this isn't important as far as I can tell).
Sorry for the massive post. This has been a problem I've been wracking my brain over for a while and can't come round to. I suspect it has something to do with a failure of understanding rather than a fault with the actual calculations. The idea that we can collect evidence in favour of H and ~H is not disconfirmed seems absurd. I also think it may be my fault because the papers that I have seen this argument in treat this form of reasoning as an obvious way of using Bayesian inference and I've seen little criticism of it (but then again I'm using this inference myself here so perhaps I'm wrong after all). Thanks to anyone that can help me out.
Quick note: I'm no stats expert. I study mathematics at A-level which may give you some idea of what kind of level I'm at. I understand basic probability theory but I'm no whizz. So I'd be super happy if answers were tailored to this. Like I said, I have philosophical motivations for this question.
Big thanks to any answers!!!