down vote favorite
I would try to clarify the problem and then ask the questions.
The problem (variable names are masked due to confidentiality):
I ran a binary logistic regression, in which there were 5 independent variables (IVs): A, B, C, D, and E. A and B are my concern. C is also my concern and I would talk about it later. They have two factors each (A1 and A2, B1 and B2). When I ran the estimation, some significant findings appeared. Nevertheless, the direction of the coefficients (beta values) for A and B was the opposite of what I expected! According to the most of the literature (not all of it), I expected A and B to have a positive beta, while they both had a negative coefficient.
First I checked the models for about three full days. There was no mistake in them, and the directions did not change whatever changes I made to the models (except that in none of those changes, I attempted to drop the interactions). The log likelihoods as well showed that I am in a good direction.
Then I decided to put my subjective view against the strange results aside and trust the results of the regression analysis. Then I passed to discuss those strange results and tried to justify the controversial findings. While discussing, I came to this point that "those two variables are heavily interconnected. Firstly, they had a significant interaction. Secondly, the distribution of the predictor A was heavily affected on the B, and according to the literature, A and B could have opposite effects; it could be important in my sample which was not balanced in terms of B. This imbalance could confound the effect of A as well.
So I though maybe these are causing some problems. Then since C was as well strange, I thought maybe the whole model is being affected in a bad way by problems such as multicollinearity. I asked myself "what will happen if I isolate only A and B in the model?" If the interactions between A, B, and C are some sources of bias, can reducing the number of IVs lead to different results? The answer was yes: when I excluded all the other IVs from the model and left only A, B, and A*B, one of the coefficients became favorable and more in line with the literature and common sense. Thus I might tell that some errors do exist in my model which disrupt the main model (such as multicollinearity maybe).
The I decided to examine every strange predictor, in isolation. When I excluded the interactions and left only the five IVs, the results seemed Much more consistent. Apparently, the problem begins when some specific interactions (but not all of them) are added to the model. After adding them, the directions of betas for A and C get reversed. It is a little annoying since by adding those specific interactions, the log likelihood reduces considerably (from about -75 to -48), so I cannot easily ignore those interactions.
Questions (the main ones are 3 and 4, but an answer to the rest is as well much appreciated):
When the model acts strangely, but LRT and log likelihood tell that it is fine, which one should we choose? The subjective common sense, or the objective statistical measures?
Do you think is there a "problem" in my case, in the first place? Maybe everything is fine. If you wished, I can provide the raw data too.
What would you do in my case? At least three choices can be made: A. Dropping the interactions. B. Not dropping them and reporting the strange model. C. Reporting both the models with and without interactions, and also models of limited numbers of IVs (for example only A and B), and then try to subjectively discuss that "it is the interactions that cause the main large model strange".
I am going to do the latter (3.C). But that would be so messy and not so good looking. I wonder if there is an elegant, objective way of finding the source of error in the main model (well if there is any errors, of course), so instead of subjective discussions, I can substantiate my claims on some objective statistical measures. For example, is there a way to highlight the problematic interactions according to some statistics?
Do you have any other valuable suggestions or ideas?