On False Discovery Rates for Second-Generation p-Values

Abstract

The False Discovery Rate (FDR) was introduced as an alternative multiple comparisons adjustment to controlling the family wise error rate. While many variations of the FDR have been proposed, the weakness of the p-value – namely that it does not consider the scientific relevance of the finding – remains a challenge for these methods. The second-generation p-value (SGPV, Blume et al. 2018) is an alternative that uses the scientific relevance of the findings to screen out false discoveries. It does this by employing an interval null hypothesis to denote null and practically null findings. This dissertation explores and defines false discovery concepts and quantities for SGPVs. First, a review of fundamental FDR concepts for classical p-values, including empirical estimation, is provided. Second, the definition and estimation of the positive false discovery rate (pFDR) is extended to SGPVs. In particular, an approach which addresses the interval null hypothesis is needed. The pFDR is the appropriate measurement of the reliability of a set of findings in a large-scale inference procedure. Finally, FDR control and power using SGPV methods, along with comparison of SGPV false discovery quantities with other multiple testing methods, is examined. The handling and identification of null effects is critical, as FDR methods routinely misclassify tiny departures from the null as findings, especially in large samples. The SGPV framework directly addresses this, but at the cost of some global performance properties. Overall, SGPV methods can be beneficial for impactful scientific inference, and further work on empirical estimation schemes is needed for wide adoption in practice.