I have conducted an experiment which manipulated three factors (Factor 1: 3 levels, Factor 2: 2 levels, Factor 3: 2 levels). The response variable is binomially distributed (1 = correct or 0 = not correct).
I have fitted various logistic regression models with combinations from small models only containing one of the factors (e.g. RESPONSE ~ FACTOR1) to a model containing all three factors and their interactions (RESPONSE ~ FACTOR1 + FACTOR2 + FACTOR3 + FACTOR1xFACTOR2.. etc.).
Model comparison based on AIC, BIC, and DIC (using the JAGS Gibbs sampler, prior is uninformative) always points to the same model which looks like (RESPONSE ~ FACTOR2 + FACTOR3 + FACTOR1xFACTOR2). However, when I estimate the regression coefficients for this best fit model, confidence intervals (as well as Bayesian credibility intervals) overlap with 0 for one factor. The ML estimate is around -0.01 and the confidence/credibility interval around [-0.04; 0.02].
If I remove this factor, model fit decreases. What is wrong here? Shouldn't the best fit model include all "reliable" factors?