On second-generation p-values for equivalence testing and study planning, and flexible false discovery rate computation for classical p-values

Abstract

Second-generation p-values (SGPVs) have been proposed and discussed in the literature as an alternative inferential statistic to indicate when the data support the null or alternative hypothesis, or when the data are inconclusive. As they are inferentially non-denominational, SGPVs can be used with frequentists, likelihood or Bayesian methods. This dissertation compares the behavior of SGPVs to classical equivalence tests, explores the operational characteristics of SGPV for study planning, and presents a tool for flexible false discovery rate computations for classical p-values.

First, we derive the mathematical relationship between SGPVs and traditional equivalence testing. We provide a conceptual framework for comparing the two approaches and conclude that the flexibility of the second-generation p-value framework offers notable advantages including ease of use, clear interpretation, and improved statistical properties. Second, we investigate different ways to specify the "interval null hypothesis" or indifference zone that is critical to the second-generation p-value (SGPV). We propose allowing the indifference zone to shrink as the sample size grows as a way of mitigating collaborator uncertainty about the indifference zone. Shrinking the indifference zone can balance the power and errors in a classical sense, but it is only practically useful in certain settings. Third, we introduce a new user-friendly R package for estimating FDRs and computing adjusted p-values for FDR control. A key contribution of this package is that it distinguishes between these two quantities while also offering several refined algorithms for estimating them. In conclusion, this work identifies the most flexible and easy-to-use method for establishing equivalence, proposes a new concept to adjust for collaborator uncertainty in SGPV methods, and creates a new user-friendly package for computing FDRs and adjusted p-values. The motivation behind these contributions is that the reporting of second-generation p-values and false discovery rates greatly improves the dissemination, transparency, and accessibility of statistical analyses.