Integrating Multimodal Data for Predictive Modeling in Neurology and Psychiatry

Abstract

Background: Healthcare predictive modeling often uses structured electronic health record (EHR) data, but integrating multimodal sources—such as unstructured clinical notes, imaging, and patient-generated communications—may improve clinical decision support. The effectiveness of such integration varies by domain and data type, requiring systematic evaluation. Objective: This dissertation examined multimodal machine learning for predictive modeling in neurology and psychiatry, focusing on: (1) combining computed tomography (CT)-derived brain imaging with clinical data to predict deep brain stimulation (DBS) responsiveness in Parkinson’s disease, (2) incorporating temporal clinical note features for suicide attempt risk prediction, and (3) assessing patient portal messages (PPMs) as an additional suicide risk predictor. Methods: Using Vanderbilt University Medical Center EHR data, Study 1 analyzed 105 Parkinson’s patients undergoing DBS, integrating CT-derived brain regional volumes with clinical and neurocognitive assessments via support vector machines, logistic regression, random forests, and k-nearest neighbors. Study 2 examined 2,364,183 visit clusters (2010–2022), extracting temporal concept identifiers from notes using natural language processing and hybrid Long Short-Term Memory neural networks for 30-, 90-, and 365-day suicide attempt prediction. Study 3 extended this model by adding PPM features using bag-of-words and sentiment analysis. Results: Multimodal integration showed domain-specific benefits. In DBS prediction, a multimodal support vector classifier achieved AUROC 0.90, outperforming expert neurologists (0.56); larger left putamen volume predicted better response. For suicide risk, temporal note features substantially improved performance (AUPRC 0.056 vs. 0.015 for structured-data-only). Key predictors included suicide-related terms, hopelessness, and self-injury. PPMs added minimal value (AUPRC 0.061 vs. 0.054). Conclusions: Effective multimodal modeling depends on complementary data, clinical relevance, sufficient density, temporal alignment, and processing methods suited to data characteristics. Neuroimaging and notes offered substantial gains; PPMs did not, likely due to overlap with existing documentation. These findings guide precision medicine approaches and practical implementation of multimodal predictive models in clinical care.