Evaluation of Supervised Learning Approaches for Assessing Heterogeneity of Treatment Effect in Clinical Trials

Abstract

Inferring heterogeneity of treatment effect is a popular secondary aim of clinical trials. While there are several methods available to estimate conditional average treatment effects (CATEs) in clinical trials, they are often applied in settings with lower sample sizes than were included in corresponding seminal methodological work, making the validity of inference in these settings unclear. To provide practical guidance, we conducted a simulation study to evaluate the performance of different estimators for the CATE, including ordinary least squares (OLS) and causal forests, in a variety of settings. We evaluated 95% confidence interval (CI) coverage, bias, and variance under linear and non-linear data generating mechanisms (DGM) in the presence of 0-40 nuisance covariates and 0-16 effect modifying covariates. We found that while tree-based ensembles like causal forests can be quite flexible to linear or nonlinear settings, they can have meaningfully impaired coverage in many settings at sample sizes which constitute most trial applications. As expected, OLS has superior performance under linear DGMs but poor performance under nonlinear DGMs. We conclude with recommendations.