Information Criteria Confidence Intervals

Description

Calculate confidence intervals of AIC and BIC for non-nested models.

Usage

icci(object1, object2, conf.level = 0.95, ll1 = llcont, ll2 = llcont)

Arguments

Details

Functionality is currently available for models of classes lm, glm, glm.nb, clm, hurdle, zeroinfl, mlogit, nls, polr, rlm, and lavaan.

Users should take care to ensure that the two models have the same dependent variable (or, for lavaan objects, identical modeled variables), with observations ordered identically within each model object. Assuming the same data matrix is used to fit each model, observation ordering should generally be identical. There are currently no checks for this, however.

Note: if models are nested or if the "variance test" from vuongtest() indicates models are indistinguishable, then the intervals returned from icci() will be incorrect.

Value

an object of class icci containing test results.

Author(s)

Ed Merkle and Dongjun You

References

Vuong, Q. H. (1989). Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica, 57, 307-333. <DOI:10.2307/1912557>

Merkle, E. C., You, D., & Preacher, K. (2016). Testing non-nested structural equation models. Psychological Methods, 21, 151-163. <DOI:10.1037/met0000038>

Examples

## Not run: 
## Count regression comparisons
require(MASS)
house1 <- glm(Freq ~ Infl + Type + Cont, family=poisson, data=housing)
house2 <- glm(Freq ~ Infl + Sat, family=poisson, data=housing)

## CI for BIC
icci(house2, house1)

## Further comparisons to hurdle, zero-inflated models
require(pscl)
bio1 <- glm(art ~ fem + mar + phd + ment, family=poisson, data=bioChemists)
bio2 <- hurdle(art ~ fem + mar + phd + ment, data=bioChemists)
bio3 <- zeroinfl(art ~ fem + mar + phd + ment, data=bioChemists)
icci(bio2, bio1)
icci(bio3, bio1)
icci(bio3, bio2)

## Latent variable model comparisons
require(lavaan)
HS.model <- 'visual  =~ x1 + x2 + x3
              textual =~ x4 + x5 + x6
              speed   =~ x7 + x8 + x9 '
fit1 <- cfa(HS.model, data=HolzingerSwineford1939, meanstructure=TRUE)
fit2 <- cfa(HS.model, data=HolzingerSwineford1939, group="school")
icci(fit1, fit2)

## End(Not run)

Individual Log-Likelihoods

Description

Obtain log-likelihood values associated with individual observations, evaluated at the ML estimates.

Usage

llcont(x, ...)

Arguments

Details

This is a S3 generic function. Currently, the method is defined for lm, glm, glm.nb, clm, hurdle, zeroinfl, mlogit, nls, polr, rlm, lavaan, vglm, mirt, and OpenMx objects.

Value

An object of class numeric containing individuals' contributions to the log-likelihood. The sum of these contributions equals the model log-likelihood.

Author(s)

Ed Merkle, Dongjun You, Lennart Schneider, Mauricio Garnier-Villarreal, and Phil Chalmers

Examples

## Fit gamma glm, check that sum of llcont() equals
## the model loglikelihood:
clotting <- data.frame(u = c(5,10,15,20,30,40,60,80,100),
                       lot1 = c(118,58,42,35,27,25,21,19,18),
                       lot2 = c(69,35,26,21,18,16,13,12,12))
gam1 <- glm(lot1 ~ log(u), data = clotting, family = Gamma)
sum(llcont(gam1))
logLik(gam1)

Vuong Tests for Model Comparison

Description

Test pairs of models using Vuong's (1989) <DOI:10.2307/1912557> theory. This includes a test of model distinguishability and a test of model fit.

Usage

vuongtest(
  object1,
  object2,
  nested = FALSE,
  adj = "none",
  ll1 = llcont,
  ll2 = llcont,
  score1 = NULL,
  score2 = NULL,
  vc1 = vcov,
  vc2 = vcov
)

Arguments

Details

For non-nested models, the test of distinguishability indicates whether or not the models can possibly be distinguished on the basis of the observed data. The LRT then indicates whether or not one model fits better than another.

For nested models (nested=TRUE), both tests serve as robust alternatives to the classical likelihood ratio tests. In this case, the adj argument is ignored.

Users should take care to ensure that the two models have the same dependent variable (or, for lavaan objects, identical modeled variables), with observations ordered identically within each model object. Assuming the same data matrix is used to fit each model, observation ordering should generally be identical. There are currently no checks for this, however.

Value

an object of class vuongtest containing test results.

Author(s)

Ed Merkle and Dongjun You

References

Vuong, Q. H. (1989). Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica, 57, 307-333. <DOI:10.2307/1912557>

Merkle, E. C., You, D., & Preacher, K. (2016). Testing non-nested structural equation models. Psychological Methods, 21, 151-163. <DOI:10.1037/met0000038>

Examples

## Not run: 
## Count regression comparisons
require(MASS)
house1 <- glm(Freq ~ Infl + Type + Cont, family=poisson, data=housing)
house2 <- glm(Freq ~ Infl + Sat, family=poisson, data=housing)
house3 <- glm(Freq ~ Infl, family=poisson, data=housing)
## house3 is nested within house1 and house2
anova(house3, house1, test="Chisq")
anova(house3, house2, test="Chisq")

## house 2 is not nested in house1, so this test is invalid
anova(house2, house1, test="Chisq")

## Use vuongtest() instead
vuongtest(house2, house1)

## Application to models with different distributional assumptions
require(pscl)
bio1 <- glm(art ~ fem + mar + phd + ment, family=poisson, data=bioChemists)
bio2 <- hurdle(art ~ fem + mar + phd + ment, data=bioChemists)
bio3 <- zeroinfl(art ~ fem + mar + phd + ment, data=bioChemists)
vuongtest(bio2, bio1)
vuongtest(bio3, bio1)
vuongtest(bio1, bio2)
vuongtest(bio1, bio3)
vuongtest(bio3, bio2)

## Application to latent variable models
require(lavaan)
HS.model <- 'visual  =~ x1 + x2 + x3
              textual =~ x4 + x5 + x6
              speed   =~ x7 + x8 + x9 '
fit1 <- cfa(HS.model, data=HolzingerSwineford1939)
fit2 <- cfa(HS.model, data=HolzingerSwineford1939, group="school")
vuongtest(fit1, fit2)

## Supplying custom vcov function
require(lme4)
require(merDeriv)

fm1 <- lmer(Reaction ~ Days + (Days | Subject), sleepstudy, REML=FALSE)
fm2 <- lmer(Reaction ~ Days + (Days || Subject), sleepstudy, REML=FALSE)

vcl <- function(obj) vcov(obj, full=TRUE)
vuongtest(fm1, fm2, vc1=vcl, vc2=vcl, nested=TRUE)


## End(Not run)