• About
    • Login
    View Item 
    •   Institutional Repository Home
    • Electronic Theses and Dissertations
    • Electronic Theses and Dissertations
    • View Item
    •   Institutional Repository Home
    • Electronic Theses and Dissertations
    • Electronic Theses and Dissertations
    • View Item
    JavaScript is disabled for your browser. Some features of this site may not work without it.

    Browse

    All of Institutional RepositoryCommunities & CollectionsBy Issue DateAuthorsTitlesSubjectsDepartmentThis CollectionBy Issue DateAuthorsTitlesSubjectsDepartment

    My Account

    LoginRegister

    Stabilizing Calibration of Clinical Prediction Models in Non-Stationary Environments: Methods Supporting Data-Driven Model Updating

    Davis, Sharon Elizabeth
    : https://etd.library.vanderbilt.edu/etd-10162019-152157
    http://hdl.handle.net/1803/14327
    : 2019-10-17

    Abstract

    Risk prediction models are increasingly employed in clinical environments to support population health management, quality assessment, and clinical decision support tools directly integrated with electronic health records. However, patient populations, clinical practice, workflows, and information systems continuously evolve over time. Calibration of prediction models often deteriorates as training and application populations become increasingly disparate, and mis-identification of appropriate patients for a given use case can result in sub-optimal care and potential patient safety issues. As a result, updating strategies to sustain performance over time are becoming critical components of model implementations. Variations in the response of models to changes in clinical environments may impact the timing, extent, and form of drift in accuracy, yet common pre-defined updating plans fail to account for such differences. As an alternative, we develop a suite of methods supporting data-driven model updating strategies in order to consistently retain model calibration over time for both regression and machine learning models. We first develop the notation of dynamic calibration curves to maintain an evolving assessment of model performance. These curves bring together methods of stringent model validation and online learning from data streams to continuously evaluate performance as new patient data is observed. Leveraging these dynamic calibration curves, we construct a calibration drift detection system to trigger model updating as performance declines and inform the updating process with insight into selecting updating datasets. Finally, we define a non-parametric testing procedure to select between available updating methods, including recalibration and model refitting. This test aims to recommend simple updating methods without compromising performance achievable with more complex adjustments. All methods are designed to be widely applicable to dichotomous outcome models regardless of the underlying learning algorithm and customizable to meet the needs of clinical use cases. This work promotes a shift away from insufficient “one-size fits all” strategies. Individually and in concert, these methods tailor updating to the requirements of specific prediction models and use cases. They lay the ground work for the design of automated, electronic health record-embedded prediction model surveillance tools to promote long-term performance and utility of prediction models underlying a variety of informatics applications.
    Show full item record

    Files in this item

    Icon
    Name:
    Davis_Dissertation_Oct2019.pdf
    Size:
    18.87Mb
    Format:
    PDF
    View/Open

    This item appears in the following collection(s):

    • Electronic Theses and Dissertations

    Connect with Vanderbilt Libraries

    Your Vanderbilt

    • Alumni
    • Current Students
    • Faculty & Staff
    • International Students
    • Media
    • Parents & Family
    • Prospective Students
    • Researchers
    • Sports Fans
    • Visitors & Neighbors

    Support the Jean and Alexander Heard Libraries

    Support the Library...Give Now

    Gifts to the Libraries support the learning and research needs of the entire Vanderbilt community. Learn more about giving to the Libraries.

    Become a Friend of the Libraries

    Quick Links

    • Hours
    • About
    • Employment
    • Staff Directory
    • Accessibility Services
    • Contact
    • Vanderbilt Home
    • Privacy Policy