# Model selection in regression: Estimated parameters seem to be “non-significant”

I have conducted an experiment which manipulated three factors (Factor 1: 3 levels, Factor 2: 2 levels, Factor 3: 2 levels). The response variable is binomially distributed (1 = correct or 0 = not correct).

I have fitted various logistic regression models with combinations from small models only containing one of the factors (e.g. RESPONSE ~ FACTOR1) to a model containing all three factors and their interactions (RESPONSE ~ FACTOR1 + FACTOR2 + FACTOR3 + FACTOR1xFACTOR2.. etc.).

Model comparison based on AIC, BIC, and DIC (using the JAGS Gibbs sampler, prior is uninformative) always points to the same model which looks like (RESPONSE ~ FACTOR2 + FACTOR3 + FACTOR1xFACTOR2). However, when I estimate the regression coefficients for this best fit model, confidence intervals (as well as Bayesian credibility intervals) overlap with 0 for one factor. The ML estimate is around -0.01 and the confidence/credibility interval around [-0.04; 0.02].

If I remove this factor, model fit decreases. What is wrong here? Shouldn't the best fit model include all "reliable" factors?

Alex

• Hi, was all ok with your analysis? – Anatoly Jul 29 '14 at 21:05

It is rather common that AIC-based models provide nonsignificant values for estimated parameters. AIC and its variants can be interpreted as penalized versions of the log-likelihood, which are not necessarily related to p-values. In particular, AIC quantifies goodness-of-fit by favouring lower residual error and penalises for including further predictors, thus minimizing the risk of overfitting. In this regard, looking at individual p-values in a AIC-based model may be misleading, as a nonsignificant $p$ does not necessarily imply that a variable is useless. If the use of AIC for the selection of the "best" model was predefined, you can keep the model as it is, without looking at p-values, confidence intervals, or Bayesian credibility intervals.