Properties of Variance Estimators in Finite Sample Sizes

Abstract

This study examines the stability properties of several variance estimators associated with generalized linear models (GLMs). In particular, six variance estimators were studied: Poisson, quasi-Poisson, sandwich-based, unconditional bootstrap, conditional bootstrap, and negative binomial. Performance was assessed via simulation under three data generating mechanisms (DGMs): Poisson (i.e., no over- or under-dispersion), quasi- Poisson, and negative binomial. We considered sample sizes ranging from n=40 to n=1,280. Validity was assessed by comparing average estimated standard errors to the empirical standard errors. Stability was assessed by comparing the standard deviation of the estimated standard errors between different estimators. The Poisson-based standard error demonstrated superior stability across all DGMs but was biased under non-Poisson DGMs. Robust methods (i.e., those based on the sandwich and bootstrap) exhibited greater instability and bias compared to parametric estimators presuming a known mean-variance model; this was especially apparent under smaller sample sizes. These methods were applied to text message response data from the recent Family/friends Activation to Motivate Self-care (FAMS) clinical trial on diabetes self management. The various estimation approaches offer differential degrees of precision; the results of our simulation suggest that it is challenging to attribute this variation specifically to finite-sample bias or instability. This research highlights potential trade-offs between validity and stability for estimators of variance. It suggests that parametric methods may by favorable over robust ones when we are confident in the mean model and mean-variance relationship. Otherwise, robust estimators provide valid estimates, potentially at the cost of stability.