1
$\begingroup$

I have conducted an experiment which manipulated three factors (Factor 1: 3 levels, Factor 2: 2 levels, Factor 3: 2 levels). The response variable is binomially distributed (1 = correct or 0 = not correct).

I have fitted various logistic regression models with combinations from small models only containing one of the factors (e.g. RESPONSE ~ FACTOR1) to a model containing all three factors and their interactions (RESPONSE ~ FACTOR1 + FACTOR2 + FACTOR3 + FACTOR1xFACTOR2.. etc.).

Model comparison based on AIC, BIC, and DIC (using the JAGS Gibbs sampler, prior is uninformative) always points to the same model which looks like (RESPONSE ~ FACTOR2 + FACTOR3 + FACTOR1xFACTOR2). However, when I estimate the regression coefficients for this best fit model, confidence intervals (as well as Bayesian credibility intervals) overlap with 0 for one factor. The ML estimate is around -0.01 and the confidence/credibility interval around [-0.04; 0.02].

If I remove this factor, model fit decreases. What is wrong here? Shouldn't the best fit model include all "reliable" factors?

Alex

$\endgroup$
1
  • $\begingroup$ Hi, was all ok with your analysis? $\endgroup$ – Anatoly Jul 29 '14 at 21:05
0
$\begingroup$

It is rather common that AIC-based models provide nonsignificant values for estimated parameters. AIC and its variants can be interpreted as penalized versions of the log-likelihood, which are not necessarily related to p-values. In particular, AIC quantifies goodness-of-fit by favouring lower residual error and penalises for including further predictors, thus minimizing the risk of overfitting. In this regard, looking at individual p-values in a AIC-based model may be misleading, as a nonsignificant $p$ does not necessarily imply that a variable is useless. If the use of AIC for the selection of the "best" model was predefined, you can keep the model as it is, without looking at p-values, confidence intervals, or Bayesian credibility intervals.

It should also be pointed out that neither AIC nor p-values have been designed for stepwise model selection. Therefore, you could consider to use other strategies such as the LASSO or LAR, which are valid alternatives if you need an automated model selection. These methods are less biased than the stepwise procedure. You can find an interesting paper on the use of LASSO for logistic regression here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.