A Comprehensive Simulation Study on Coverage Probabilities of Double Machine Learning: Analytical versus Bootstrapped Confidence Intervals

Abstract

Double Machine Learning (DML) is a principled approach used for causal effect estimation in various settings, which allows a wide range of flexible machine learning methods to be used for nuisance parameter estimation while preserving valid inference. In practice, however, applied researchers must choose among many machine learning algorithms for nuisance models, and the impact of this choice on the variance estimation of DML is not well characterized. We conduct a comprehensive simulation study to compare the coverage probability of DML confidence intervals across different machine learning algorithms. In this study, we compare (1) analytical confidence intervals derived by DML theory versus (2) bootstrapped confidence interval. To achieve this, we use a set of learners including linear models, regularized regression, tree-based methods, and neural networks, to evaluate coverage probability, bias, and interval width under both methods. In simulation studies, the data-generating processes for outcome and treatment models include linear and/or nonlinear relationships, interaction terms, and high-dimensional covariates. Our results show substantial variability in coverage performance across analytical and bootstrapped fits, highlighting that learner choice plays a critical role in reliable DML inference, with no single learner performing well across the board. We further investigate coverage probabilities using a real dataset on rural–urban differences among U.S. counties, showing its potential in practical inference.